Hierarchical deep reinforcement learning routing protocol method based on multi-link ad hoc network

Abstract

The application provides a layered deep reinforcement learning routing protocol method based on a multi-link ad hoc network, relates to the technical field of communication routing, and solves the problem of computing efficiency degradation in the current multi-link ad hoc network. The method first constructs and dynamically updates a global network view for all nodes in the network, and then constructs a routing selection model, which is divided into a high-level controller and a low-level controller. The high-level controller plans the routing from a global perspective and balances the global load. The low-level controller selects a target node according to the planning result of the high-level controller, and allocates the optimal communication link for the target node in combination with the link state information. The training and optimization of the network of each controller are independently carried out, while an external reward is introduced for the high-level controller and an internal reward is introduced for the low-level controller. The external reward is used to optimize the global performance of the high-level controller. The internal reward is used as an index value of the real-time feedback of the link state of the low-level controller, so that the high-level controller and the low-level controller are cooperatively optimized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of communication routing, in particular to a layered deep reinforcement learning routing protocol method based on multi-link ad hoc networks. BACKGROUND

[0002] Currently, the node cluster coordination system has become an important development direction of distributed network architecture, and its core is to realize efficient interconnection and collaborative work between nodes. Especially in the multi-link ad hoc network scenario, the system needs to meet the high accuracy of control instructions, low delay transmission and high bandwidth carrying demand of data services at the same time. This puts forward strict requirements for the underlying routing strategy: on the one hand, it needs to dynamically adapt to network topology changes and link state fluctuations, and on the other hand, it needs to consider the differentiated quality of service of control flow and data flow. Therefore, designing a routing mechanism that can adaptively optimize path selection and resource allocation has become a key technical challenge to improve the efficiency of cluster coordination.

[0003] Existing routing strategies mainly optimize around a single performance dimension. Common routing strategies include link quality or path cost-based schemes, which calculate path cost based on metrics such as hop count, transmission delay, available bandwidth, or link stability, and prefer to select the path with the lowest overall cost for data transmission. Another type of strategy focuses on load awareness, dynamically shunts traffic to avoid high-load nodes by monitoring link congestion and node buffer status in real time, thereby achieving balanced distribution of traffic in the network. In addition, stability-based schemes prefer to select nodes with high stability as relays by introducing reliability indicators such as link survival time and node reachability, in order to enhance the anti-disturbance ability of routing. In recent years, intelligent routing strategies have gradually emerged, such as solutions combining reinforcement learning frameworks, which use reward functions to integrate multi-dimensional parameters such as hop count, link quality, and node load to guide nodes to make autonomous decisions to approximate the global optimal path.

[0004] Although the above strategies have improved path selection efficiency or load balancing capability in specific scenarios, their design is mainly aimed at single-link or static network environments, and it is difficult to effectively cope with the complexity of multi-link dynamic ad hoc networks. The main limitations are as follows: First, traditional strategies usually treat node selection and link allocation as independent problems, lacking a collaborative mechanism for joint optimization of the two. In the multi-link parallel transmission scenario, the node processing capacity, link capacity, and service type matching degree are not considered comprehensively, resulting in low resource utilization or substandard service quality. Second, although intelligent solutions based on reinforcement learning have adaptive potential, they rely on maintaining a single and large Q-value table to cover all possible states. As the network node size increases, the number of links grows, and the state parameter dimension increases, the Q-value table faces the problem of explosive expansion of the state space, resulting in exponential increase in computational complexity, and dramatic increase in storage overhead, which seriously restricts its practical deployment feasibility in large dynamic networks.

[0005] The above deficiencies highlight the bottleneck of the prior art in terms of multi-link collaborative optimization and scalability. There is an urgent need for a new routing mechanism that can achieve collaborative scheduling of nodes and links in a dynamic topology environment while avoiding the degradation of computational efficiency caused by state space expansion, thereby providing efficient and reliable control instruction transmission and data sharing services for node clusters. SUMMARY

[0006] The present application aims to solve the problems of node-link collaborative scheduling deficiency in multi-link ad hoc networks and computational efficiency degradation caused by state space expansion, and therefore proposes a hierarchical deep reinforcement learning routing protocol method based on multi-link ad hoc networks. The present application effectively reduces the complexity of the state and action space through a hierarchical architecture, improving decision-making efficiency; introduces a dual feedback mechanism of rewards to optimize global path and local link performance; combines a priority-based experience pool replay mechanism and a target network to improve model training stability and data utilization efficiency, achieving collaborative optimization of path selection and link allocation, and significantly improving routing performance and adaptability in multi-link dynamic networks.

[0007] The present application adopts the following technical solutions to achieve the purpose:

[0008] A hierarchical deep reinforcement learning routing protocol method based on multi-link ad hoc networks, comprising the following steps:

[0009] S1. In the link perception phase, each node in the network establishes and maintains neighbor link state information through periodic hello message interaction, and dynamically selects a multi-point relay forwarding node; the multi-point relay forwarding node diffuses network topology changes and experience data through multi-point TC message forwarding, enabling all nodes in the network to build and dynamically update a global network view;

[0010] S2. A routing selection model for routing decisions in the global network view is constructed, and the routing selection model is divided into a high-level controller and a low-level controller; the high-level controller plans routes from a global perspective and balances global load, and the low-level controller selects target nodes according to the planning results of the high-level controller and allocates optimal communication links for the target nodes in combination with link state information;

[0011] S3. Hierarchical deep reinforcement learning is used to achieve collaborative optimization of the high-level controller and the low-level controller, so that the high-level controller and the low-level controller independently learn to complete the training and optimization of their own networks for different sub-tasks, and implicitly maintain the Q-value table of nodes and links in the routing strategy collaboration process;

[0012] S4, when training and optimizing the high-level controller and the low-level controller, an external reward is introduced for the high-level controller, and an internal reward is introduced for the low-level controller; the external reward is used for optimizing the global performance of the high-level controller; the internal reward is used as an index value of the real-time feedback link state of the low-level controller, so that the high-level controller and the low-level controller are cooperatively optimized.

[0013] Further, in step S2, the modeling parameters of the high-level controller include a high-level state , a high-level action , and an external reward ; the modeling parameters of the low-level controller include a low-level state , a low-level action , and an internal reward .

[0014] Further, in step S3, the high-level controller learns the mapping relationship between the high-level state and the action through a high-level Q network; when training the high-level Q network, an adaptive anti-redundancy search strategy is adopted; the low-level Q network of the low-level controller optimizes the selection of link allocation through the guidance of the local link information and the target node specified by the high-level controller; when training the low-level Q network, the same adaptive anti-redundancy search strategy as training the high-level Q network is adopted.

[0015] Further, in step S4, the external reward is composed of a function formula including three types of judgment factors of delay, accuracy and hop count; the internal reward is composed of a function formula including two types of judgment factors of link signal-to-noise ratio and bandwidth utilization rate.

[0016] As described above, since the technical solution is adopted, the application has the following beneficial effects:

[0017] The application effectively solves the problem in multi-link ad hoc network routing decision by introducing a hierarchical reinforcement learning framework and a cooperative optimization mechanism. First, a hierarchical task decomposition architecture is adopted to divide the routing decision into two cooperative stages of high-level global target node selection and low-level link resource allocation. The high-level dynamically plans the transmission path based on the global network state, and the low-level optimizes the resource scheduling combined with the real-time link state, thereby significantly reducing the dimension of the state and action space through hierarchical decoupling, improving the decision efficiency, and realizing the joint optimization of path planning and link allocation.

[0018] The application then designs a double feedback reward mechanism, the external reward of the high-level controller optimizes the end-to-end performance indicators such as transmission delay, data accuracy and hop count, and the internal reward of the low-level controller real-time feedbacks the link-level state parameters including signal-to-noise ratio and bandwidth utilization rate. The mechanism cooperatively drives the synchronous improvement of the global path performance and the local resource utilization efficiency, and enhances the adaptability to the dynamic network environment.

[0019] In addition, the application also proposes an adaptive heavy-avoidance exploration strategy, which dynamically adjusts the action selection probability in the early training stage, guides the network to fully explore the global topology structure, and avoids falling into a suboptimal solution due to repeated selection of local Q value path; gradually converges to the global optimal path in the later training stage, significantly improves the exploration efficiency and convergence stability of the corresponding model. At the same time, combined with the experience replay mechanism based on priority, the key experience is given a higher learning weight according to the time difference error, and the target network is introduced to stabilize the training process, which improves the learning efficiency and training robustness of the model together, and effectively overcomes the problem of insufficient data utilization.

[0020] Therefore, the application has obvious advantages in reducing computational complexity, avoiding local optimization, improving training stability and strengthening global-local cooperation, and provides a high adaptability and low overhead routing decision solution for multi-link dynamic networks. BRIEF DESCRIPTION OF DRAWINGS

[0021] The embodiments and technical solutions of the application are further described in detail by the following drawings, specifically including three drawings, as follows:

[0022] Figure 1 The overall flow chart of the hierarchical deep reinforcement learning routing protocol method of the application is briefly described;

[0023] Figure 2 The overall collaborative architecture of the hierarchical reinforcement learning framework in the method of the application is schematically shown;

[0024] Figure 3 The detailed schematic diagram of the application process and the application process of the routing selection model is shown. DETAILED DESCRIPTION

[0025] In order to make the purpose, technical scheme and advantages of the embodiments of the application clearer, the technical scheme in the embodiments of the application will be described clearly and completely in conjunction with the drawings in the embodiments of the application. Obviously, the described embodiments are part of the embodiments of the application, not all the embodiments. The components of the embodiments of the application described and shown in the drawings can be arranged and designed in various different configurations.

[0026] Therefore, the following detailed description of the embodiments of the application provided in the drawings is not intended to limit the scope of the claimed application, but only represents selected embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the application.

[0027] Embodiment 1

[0028] A hierarchical deep reinforcement learning routing protocol method based on multi-link ad hoc network, the overall flow chart of the method can be referred toFigure 1 The key steps of the method can be summarized as follows:

[0029] S1, in the link sensing phase, each node in the network establishes and maintains the neighbor link state information through periodic hello message interaction, and dynamically selects the multi-point relay forwarding node; the multi-point relay forwarding node diffuses the network topology changes and experience data through the multi-point relay TC message, so that all nodes in the network can build and dynamically update the global network view;

[0030] S2, a routing selection model for routing decision in the global network view is constructed, and the routing selection model is divided into a high-level controller and a low-level controller; the high-level controller plans the route from the global perspective and balances the global load, and the low-level controller selects the target node according to the planning result of the high-level controller, and allocates the optimal communication link for the target node in combination with the link state information;

[0031] S3, the hierarchical deep reinforcement learning is used to realize the cooperative optimization of the high-level controller and the low-level controller, so that the high-level controller and the low-level controller learn to complete the training and optimization of their own networks independently according to different subtasks, and implicitly maintain the Q value table of nodes and links in the routing strategy cooperation process;

[0032] S4, when training and optimizing the high-level controller and the low-level controller, an external reward is introduced for the high-level controller, and an internal reward is introduced for the low-level controller; the external reward is used to optimize the global performance of the high-level controller; the internal reward is used as an index value of the real-time feedback link state of the low-level controller, to realize the cooperative optimization of the high-level controller and the low-level controller.

[0033] The above method of the embodiment introduces a hierarchical reinforcement learning framework in view of the deficiencies of the traditional routing strategy and the routing strategy based on reinforcement learning in dealing with multi-link ad hoc networks. The hierarchical design divides the routing task into two stages of high-level target selection and low-level link allocation. The high-level dynamically selects the target node through the global network state, and realizes path planning; the low-level optimizes link allocation in combination with link state information, and ensures that the resource utilization efficiency is maximized. Through this hierarchical architecture, the complexity of the state and action space is effectively reduced, and the decision efficiency is improved. In addition, the embodiment also introduces a double feedback mechanism of external reward and internal reward to optimize the global path and local link performance.

[0034] The embodiment can also adopt an adaptive heavy search strategy during optimization training, fully explore the global network topology in the early training stage, avoid repeated selection of a link with a high Q value due to insufficient training data, model structure limitations, and other reasons, and thus fall into a local suboptimal solution. In combination with the priority-based experience pool playback mechanism and the target network, the embodiment further improves the training stability and data utilization efficiency of the corresponding model, realizes the cooperative optimization of path selection and link allocation, and significantly improves the routing performance and adaptability in the multi-link dynamic network.

[0035] Embodiment 2

[0036] Based on embodiment 1, the details and preferred contents of the method are described in detail. Referring to the schematic of Figure 2 The hierarchical reinforcement learning framework solves the global planning and local optimization problem in complex decision-making tasks by introducing a cooperative mechanism of high-level controllers and low-level controllers. In view of the current situation that node selection and link allocation lack joint optimization mechanism, the high-level controller plans the path from a global perspective and balances the global load to ensure the maximum path efficiency; the low-level controller performs optimal link allocation locally around the path target specified by the high-level controller to achieve effective use of local resources.

[0037] To cope with the problem of state space expansion caused by the increase of network size and parameters, the framework of the embodiment introduces a neural network in deep reinforcement learning as a function approximator to replace the explicit storage mode of the traditional Q value table for policy optimization. At the same time, the search range of the low-level controller is constrained by the high-level controller, effectively reducing the complexity of the state-action space; the embodiment makes targeted improvements to the network topology establishment and maintenance of the optimized link state routing (OLSR) protocol, and the detailed process of the entire method during implementation can be referred to the schematic of Figure 3 The embodiment focuses on optimizing path selection in multi-link ad hoc networks.

[0038] The embodiment next corresponds to the four key steps of the method in embodiment 1, and introduces the details and preferred contents from four parts of network topology establishment, routing selection model modeling, network training and optimization, and reward function design.

[0039] I. Network topology establishment

[0040] Each node detects the link state between itself and the one-hop neighbor node by periodically flooding the HELLO greeting message carrying neighbor information, and divides the link state into "symmetric" or "asymmetric"; the symmetric link state represents that it has bidirectional effective transmission, and the asymmetric link state is considered invalid due to the uncertainty of wireless propagation.

[0041] Based on the detection process, a one-hop neighbor table is constructed to maintain the link and node state information of the one-hop neighbor, and a two-hop neighbor table is generated through dynamic interaction update; the neighbor table content can refer to Table 1 below. The lowest number of bidirectional neighbor nodes is selected from the two-hop neighbor table as the multi-point relay forwarding node (Muti-Point Replay, MPR) to ensure that all two-hop neighbors can reach at least one MPR node. In the network initialization phase, all bidirectional neighbors are temporarily selected as MPR nodes to quickly establish initial topology information. Subsequently, the MPR set is updated and a new HELLO greeting message is embedded and broadcasted for interaction, and the interaction result is recorded as the MPR corresponding selector table (MPR Selector, MS), that is, the MS table content shown in Table 2 below.

[0042] Table 1 Example of Neighbor Table

[0043]

[0044] Table 2 Example of MS Table

[0045]

[0046] Subsequently, the MPR periodically sends the TC message to multicast the own MS table and the high and low layer experience data to realize the latest and consistent network topology and experience data; after any node receives the TC message, the validity is judged according to the serial number in it to avoid repeated or outdated processing. After the valid TC message is parsed, the node relationship information in it is recorded into the network topology table, the content of which is defined as shown in Table 3 below, and the global network view is gradually constructed by associating the previous hop node and the target node to support efficient routing decision and avoid loop.

[0047] Table 3 Example of Network Topology Table

[0048]

[0049] As a preferred embodiment of the present embodiment, for experience data, it is stored in the experience pool queue according to its type and priority weight; the type of experience data is uniquely represented by its node information, and the priority weight is represented based on the temporal difference error (Temporal Difference, TD) between the predicted value and the actual reward value of the current routing strategy in hierarchical deep reinforcement learning; after the experience data stored in the experience pool queue is classified according to different types and assigned with priority weights, it is stored in the experience table for experience management by the corresponding node.

[0050] When experience management is performed, the node cleans up expired experience through a time window, and detects and avoids redundant storage by using the sequence number of the TC message; when the MPR set or the preset key experience data changes, the corresponding node synchronizes the network information by temporarily increasing the frequency of sending the TC message under the constraint of the minimum interval limit, to realize dynamic updating of the global network view. Through these mechanisms, the network topology is continuously and dynamically updated, consistency is maintained, and training samples are provided for subsequent route optimization.

[0051] II. Modeling of the routing selection model

[0052] In this embodiment, the routing selection model is divided into a high-level controller and a low-level controller, and the decision strategies of both comply with the Markov decision process. Both the high-level controller and the low-level controller execute corresponding actions by perceiving the environment state, and optimize the routing decision strategy according to the reward feedback of themselves.

[0053] The modeling parameters of the high-level controller include:

[0054] High-level state : used to reflect the current running environment of the network, including network topology information, node state information and link quality information;

[0055] High-level action : used to determine the globally optimal direction of routing selection, that is, to select the next hop node from the target neighbor set ; n represents the th neighbor node, and N represents the total number of neighbor nodes;

[0056] External reward : used to convert the first performance indicators including the delay, accuracy and hop count into the corresponding reward value according to the real-time state of the network, as the guidance basis for dynamic adjustment of the strategy, so as to better adapt to the changes of the network environment.

[0057] The low-level controller allocates the optimal communication link for the target node based on the node selected by the high-level controller and the link state information; the modeling parameters of the low-level controller include:

[0058] Low-level state : used to reflect the state information of different links in the network;

[0059] Low-level action : used to determine the optimal allocation of the link, that is, to select the best link from the links between the two nodes ; l represents the th link, and L represents the total number of links;

[0060] ​​Internal rewards It is used to provide real-time feedback on link utilization efficiency and stability, converting secondary performance indicators, including link signal-to-noise ratio and bandwidth utilization, into corresponding reward values, which serve as a guide for its dynamic adjustment strategy.

[0061] Unlike traditional Q-learning-based routing protocols that rely on a single Q-value table to directly maintain routing information, this embodiment achieves collaborative optimization through hierarchical reinforcement learning. The higher and lower layer controllers learn independently for different subtasks, implicitly maintaining the Q-value tables of nodes and links during policy coordination. This hierarchical architecture can more efficiently handle dynamic multi-link network environments and adapt to topology changes in real time. The optimization training process will be detailed in Part Three.

[0062] III. Network Training and Optimization

[0063] In this embodiment, the high-level controller learns the mapping relationship between high-level states and actions through a high-level Q-network to approximate the traditional Q-value update function as shown below:

[0064]

[0065] In the formula, The learning rate is used to control... The step size for value updates; For the current value function of the high-level Q-network, The target value function is used to update the current value function of the higher-level Q-network; This is a discount factor used to weigh the importance of current rewards against future rewards; To perform the action The new state afterwards; Indicates the state Take action below The expected total return obtained afterward is calculated by multiplying itself by... Then, add the learning rate. With TD target items The product is obtained; the TD objective term is the immediate reward. Add discount factor The product of the maximum expected total return in the next state.

[0066] The Q-value function posits that actions should maximize the expected reward at each time step, thereby maximizing the overall reward. To this end, the following loss function is used to optimize high-level Q-networks, making their traditional Q-value update function approximate the optimal Q-value function:

[0067]

[0068] In the formula, The value is , represents the high-level target Q value; is a desired value operator; is a high-level main network parameter; is used to measure the prediction Q value and the target Q value error; is an experience pool, used to store training samples corresponding experience data, by randomly extracting samples to break the time correlation, improve data utilization efficiency and reduce training variance. At the same time, the experience pool allows the agent to replay key experience multiple times, strengthens the learning of rare states, and optimizes the strategy. In order to stabilize training, is an introduced high-level independent target network parameter, which periodically synchronizes the high-level main network parameter to avoid frequent fluctuations.

[0069] The high-level strategy based on the current highest Q value may select a fixed neighbor node, resulting in insufficient exploration of the global topology structure by the agent, and forming a local optimal route. To this end, when training the high-level Q network, the adaptive duplicate avoidance search strategy in the following form is adopted:

[0070]

[0071] In the formula, is an adaptive adjustment factor, which gradually decreases as the training progresses; is the number of times of repeatedly selecting actions ; is a set composed of all actions is the th action in the set. When is larger, the selection probability of each action is closer, and the strategy is more biased towards random exploration, that is, insensitive to the difference in Q value of all actions; when is smaller, the probability of actions with large Q values increases significantly, and the strategy is more biased towards using the current optimal action. This encourages the model to explore in the early stage, and due to incomplete training data, model structure limitations, and other reasons, the network tends to use known actions and ignore the exploration of new actions. This will lead to overestimation of the Q value of part of the path by the network, and then fall into a local optimal solution. As the training progresses, the value of

[0072] is gradually reduced, so that the strategy gradually shifts to using the existing optimal strategy. This way ensures that the agent can fully explore the global topology in the early stage, and focus on the optimized path in the later stage.

[0073] ​​In this embodiment, the low-layer Q-network of the low-layer controller optimizes the selection of link allocation through local link information and the guidance of the target node designated by the high-layer controller. The approximate Q-value update formula of the low-layer Q-network is as follows:

[0074]

[0075] In the formula, the overall represents the low-layer Q-value function at the designated high-layer action ; the new state is the state after the execution of the action ; the current value function of the low-layer Q-network ; and the target value function for updating the current value function of the low-layer Q-network .

[0076] The loss function of the low-layer Q-network is as follows:

[0077]

[0078] In the formula, the value of the low-layer target Q-value is represented by ; the low-layer main network parameter ; and the introduced low-layer independent target network parameter that periodically synchronizes the low-layer main network parameter . When training the low-layer Q-network, the same adaptive avoidance search strategy as that for training the high-layer Q-network is adopted.

[0079] IV. Reward function design

[0080] This embodiment introduces external reward and internal reward design to realize double optimization of global network performance and local resource allocation.

[0081] The definition of the external reward function covers information related to nodes, links and global network topology to support the evaluation and decision of routing selection performance. The external reward function is centered around three judgment factors, i.e., delay, accuracy and hop count, and is specifically as follows:

[0082] The delay is the transmission delay of a data packet from a sending node to a receiving node, and the expression is as follows:

[0083]

[0084] In the formula, the average delay obtained by averaging the current delays of all packets in the network is represented by ; the preset maximum tolerable delay is represented by ; and the load balancing penalty factor for preventing the routing decision strategy from excessively relying on a path with short delay is represented by .

[0085] The latency can specifically include propagation latency, processing latency, and queuing latency, etc., and the average latency may be expressed as follows:

[0086]

[0087] In the formula, Here, the number of successfully transmitted data packets per unit time is represented; The total time that the th data packet stays at the source node due to queuing and processing is represented; The total time that the th data packet arrives at the target node via different links is represented.

[0088] In addition, the present embodiment introduces a load balancing penalty factor in the latency factor to prevent the route from relying too much on the path with short latency, and to avoid congestion, increased latency, or even packet loss due to long-term high load of some nodes. At the same time, by dynamically adjusting the load distribution between nodes, the route selection is guided to more evenly utilize network resources, improving overall efficiency and reducing congestion risk. The factor is designed as follows:

[0089]

[0090] In the formula, and are the load degrees of the source node and the target node, respectively, and the load degree is measured by the occupancy rate of the node buffer. When the load difference between the source node and the target node is large, the load balancing penalty factor is close to 0, thereby reducing the priority of the path. By dynamically adjusting the node load distribution in the route selection process, the factor guides the route selection to tend to network balance, improves overall efficiency, and reduces congestion risk.

[0091] Accuracy rate is the proportion of data packets successfully arriving at the target node, which is used to evaluate the transmission reliability of the network, and is expressed by the delivery rate, as follows:

[0092]

[0093] In the formula, represents the packet loss rate of the th link; N is the total number of links; is the weight of the th link. The expression of the packet loss rate is as follows:

[0094]

[0095] In the formula, Represents a node The number of data packets sent. Represents a node The number of successfully received data packets. For multiple links connecting two nodes, the link with higher bandwidth carries more traffic, so its delivery rate has a more significant impact on overall network performance; the link with lower bandwidth has a smaller impact even if its delivery rate is low. Therefore, this embodiment achieves the aforementioned accuracy rate. The expression is implemented by using a weighted aggregation strategy to calculate the packet loss rate of multiple links.

[0096] Number of jumps The number of routing nodes that a data packet passes through from the source node to the destination node is expressed as follows:

[0097]

[0098] In the formula, Represents the hop count of the current route; This represents the maximum number of hops allowed by default. This represents the minimum number of hops from each node to other nodes.

[0099] As the minimum hop count, a two-dimensional array is initialized based on the topology table maintained in the OLSR protocol. , among which Represents a node To the node The minimum number of hops. Next, the node. Select the node with the smallest hop count from the unvisited node set, and determine if this node can optimize the minimum hop count from the current node to other nodes; if the condition is met, update... Repeat this process until all nodes have been traversed and finally... The minimum number of hops to all other nodes is recorded in the data.

[0100] Number of jumps The closer the path is to 0, the closer it is to the optimal path, meaning the hop count is close to 0. ; number of jumps The closer the number of hops is to 1, the closer the current path's hop count is to the maximum allowed hop count. .

[0101] Based on the aforementioned functions of latency, accuracy, and hop count, the external reward function... It is expressed as follows:

[0102]

[0103] In the formula, , and These are the corresponding weight values.

[0104] In this embodiment, the internal reward function is designed by incorporating the unique attributes of different links to reflect their quality and impact on communication performance. It includes two types of judgment factors: link signal-to-noise ratio and bandwidth utilization.

[0105] Link signal-to-noise ratio The signal-to-noise ratio (SNR) is the ratio of signal power to noise power in a data link, used to measure signal clarity; a higher SNR indicates better link quality and more reliable data transmission. Its expression is as follows:

[0106]

[0107] In the formula, Signal power, representing the strength of the useful signal; The noise power represents the intensity of the background noise.

[0108] Bandwidth utilization Bandwidth is the ratio of the actual data rate transmitted on the current link to the link's maximum carrying capacity. Bandwidth represents the maximum data rate a link can handle and directly affects throughput; larger bandwidth supports higher transmission rates and improves data transmission efficiency. Its expression is as follows:

[0109]

[0110] In the formula, This represents the current available bandwidth of the link. This represents the basic data bandwidth of the link.

[0111] Example 3

[0112] Based on the above embodiments, this embodiment introduces the relevant situation of the method in practical application with more specific exemplary operation steps.

[0113] Assuming a network consisting of 5 nodes, its topology is as follows:

[0114] The neighbor node set is ; The neighbor node set is ; The neighbor node set is ; The neighbor node set is ; The neighbor node set is .

[0115] and The inter-node link set is ; With the inter-node link set being ; With the inter-node link set being ; With the inter-node link set being ; With the inter-node link set being .

[0116] Now a message needs to be delivered from node to node , and the node selection and link allocation process is as follows:

[0117] Step 1: Each node periodically sends a HELLO message, and by recording the link allocation, link state, etc. of the neighboring nodes, a one-hop neighbor table, a two-hop neighbor table and a link table are constructed to realize dynamic perception of link state and neighbor discovery.

[0118] Step 2: Based on the one-hop neighbor set and the two-hop neighbor set, the node preferentially selects a one-hop neighbor node that can cover the isolated two-hop neighbors as an MPR node. In the case of incomplete coverage, from the remaining one-hop neighbors, the number of covered two-hop neighbors is selected in descending order until all two-hop neighbors are covered.

[0119] Step 3: The selected MPR set is included in the updated HELLO message and broadcast to the network. The selected MPR nodes and their related information are stored in the MS table to support dynamic maintenance of the topology.

[0120] Step 4: The relay node periodically sends a TC message, and the multi-point forwarding MS table and playback experience data. The receiving node judges the validity of the message according to the sequence number, updates the topology table after analyzing the content, and gradually constructs a global network view. In addition, the playback experience data is stored in the experience table according to type and TD error weight, providing training samples for subsequent routing optimization and link allocation.

[0121] Step 5: Node maps the global network state information into an input tensor with dimensions , where the number of rows 2 represents the number of neighbor node sets in the network , and the number of columns 3 represents the state feature dimensions of the data transmission delay, data delivery rate and hop count of each node in .

[0122] In this embodiment, a sliding window mechanism is introduced in step five to uniformly manage the network state input. This mechanism updates the maximum and minimum values ​​of the state information, as well as statistical characteristics (such as mean and standard deviation), in real time. This adaptively normalizes the dynamically changing state input, preventing large differences in feature values ​​from affecting the model's training stability and convergence speed. The high-level controller consists of two fully connected neural networks: the first layer is a hidden layer with 256 neurons, using the ReLU activation function; the second layer is the output layer, outputting a target number of 2. value The high-level controller employs an adaptive deduplication search strategy. In the form of one-hot encoding, such as Select next jump target .

[0123] Step 6: Jointly encode the local link information between nodes with the target selected by the higher-level controller to form a dimension. The input tensor, where row number 3 represents a node. target node Between The number of links, with 2 columns representing the state feature dimension including bandwidth and signal-to-noise ratio (same as in step 5, using adaptive normalization on the state input), and 1 representing the joint feature dimension of higher-level target information. The lower-level controller network is the same as the higher-level controller, but the output layer outputs Q values ​​with the number of targets C. and in the form of one-hot encoding, such as Select Link .

[0124] Step 7: After selecting a link each time, the lower-level controller calculates the internal reward function. The higher-level controller generates an external reward function by integrating reward factors such as latency, accuracy, and hop count. Internal and external rewards, actions performed, current state, and next state are represented as experience tuples. The data are stored in the corresponding experience tables of the high-level controller and the low-level controller, respectively.

[0125] Step 8: The high- and low-level controllers, based on the empirical weights of the time difference error, extract a batch size from the empirical table. Based on empirical samples, the current Q-value and target Q-value are calculated using the target network and Bellman equation. Error is calculated using Huber loss. and Then, gradient backpropagation is performed to update the parameters. After the update is complete, the parameters of the current Q-network are synchronized to the target network.

[0126] Step nine: continuously perform steps five to eight, dynamically adjust the routing and link allocation strategies of each node to adapt to the real-time changes of the network environment until the routing process of the nodes to the nodes is completed.

Claims

1. A hierarchical deep reinforcement learning routing protocol method based on multi-link ad hoc networks, characterized in that, Includes the following steps: S1. During the link awareness phase, each node in the network establishes and maintains neighbor link status information through periodic greeting messages and dynamically selects multi-point relay forwarding nodes. Multi-point relay forwarding nodes spread network topology changes and experience data by multi-point relaying TC messages, enabling all nodes in the network to build and dynamically update the global network view. S2. Construct a routing model for routing decisions in the global network view, and divide the routing model into a high-level controller and a low-level controller. The higher-level controller plans routes and balances the global load from a global perspective. The lower-level controller selects the target node based on the planning results of the higher-level controller and allocates the optimal communication link to the target node in combination with the link status information. S3. Layered deep reinforcement learning is used to achieve collaborative optimization between high-level controllers and low-level controllers. This allows high-level controllers and low-level controllers to independently learn and complete the training and optimization of their own networks for their different sub-tasks, and implicitly maintain the Q-value table of nodes and links during the routing policy collaboration process. S4. When training and optimizing the high-level controller and the low-level controller, introduce external rewards for the high-level controller and internal rewards for the low-level controller. External rewards are used to optimize the global performance of the high-level controller. Internal rewards are used as indicators for the real-time feedback of link status by the low-level controller, so as to achieve collaborative optimization between the high-level controller and the low-level controller.

2. The hierarchical deep reinforcement learning routing protocol method according to claim 1, characterized in that: In step S1, each node periodically floods HELLO greeting messages carrying neighbor information to detect the link status between itself and its one-hop neighbor nodes, and classifies the link status as symmetric or asymmetric; a symmetric link status indicates that it has bidirectional effective transmission, while an asymmetric link status is considered invalid. Based on this detection process, a one-hop neighbor table is constructed, and then a two-hop neighbor table is generated through dynamic interaction updates; the minimum number of bidirectional neighbor nodes are selected from the two-hop neighbor table as multipoint relay forwarding nodes (MPRs). Embed a new HELLO greeting message into the MPR and broadcast the interaction. Record the interaction result as the selector table corresponding to the MPR, i.e., the MS table.

3. The hierarchical deep reinforcement learning routing protocol method according to claim 2, characterized in that: MPR periodically sends TC messages to multiple points to relay its own MS table and high- and low-layer experience data, so as to ensure that the network topology and experience data are up-to-date and consistent. After receiving a TC message, any node determines its validity based on the sequence number. Once a valid TC message is parsed, the node relationship information is recorded in the network topology table. By associating the previous hop node with the target node, a global network view is gradually constructed.

4. The hierarchical deep reinforcement learning routing protocol method according to claim 3, characterized in that: For experience data, it is stored in the experience pool queue according to its type and priority weight; the type of experience data is uniquely represented by its node information, and the priority weight is represented by the temporal difference error TD between the predicted value and the actual reward value of the current routing policy in the hierarchical deep reinforcement learning. The experience data stored in the experience pool queue is classified according to different types and assigned priority weights, and then stored in the experience table for the corresponding nodes to manage experience. When performing experience management, nodes clean up expired experiences through time windows, and at the same time use the sequence number of TC messages to detect and avoid redundant storage; when the MPR set or preset key experience data changes, the corresponding nodes synchronize the information of the entire network by temporarily increasing the frequency of TC message sending under the constraint of minimum interval limit, so as to realize the dynamic update of the global network view.

5. The hierarchical deep reinforcement learning routing protocol method according to claim 1, characterized in that: In step S2, the routing decision strategies of both the high-level controller and the low-level controller conform to the Markov decision process; both the high-level controller and the low-level controller execute corresponding actions by sensing the environmental state and optimize the routing decision strategy based on their corresponding reward feedback.

6. The hierarchical deep reinforcement learning routing protocol method according to claim 1, characterized in that, In step S2, the modeling parameters of the high-level controller include: High-level status : Used to reflect the current operating environment of the network, including network topology information, node status information, and link quality information; High-level actions Used to determine the globally optimal direction for route selection, i.e., from the target neighbor set. Select the next hop node; Representing the One neighboring node, This represents the total number of neighboring nodes. External rewards It is used to convert the first performance indicators, including latency, accuracy and hop count, into corresponding reward values ​​based on the real-time network status, as a guide for its dynamic adjustment strategy; The modeling parameters for the low-level controller include: Low-level state Used to reflect the status information of different links in the network; Low-level actions Used to determine the optimal link allocation, i.e., the link between two nodes. Select the best link; Representing the Link, This represents the total number of links. Internal rewards It is used to provide real-time feedback on link utilization efficiency and stability, converting secondary performance indicators, including link signal-to-noise ratio and bandwidth utilization, into corresponding reward values, which serve as a guide for its dynamic adjustment strategy.

7. The hierarchical deep reinforcement learning routing protocol method according to claim 6, characterized in that: In step S3, the higher-level controller learns the mapping relationship between higher-level states and actions through the higher-level Q-network to approximate the traditional Q-value update function as shown below: In the formula, The learning rate is used to control... The step size for value updates; For the current value function of the high-level Q-network, The target value function is used to update the current value function of the higher-level Q-network; This is a discount factor used to weigh the importance of current rewards against future rewards; To perform the action The new state afterwards; Indicates the state Take action below The expected total return obtained afterward is calculated by multiplying itself by... Then, add the learning rate. With TD target items The product is obtained; the TD objective term is the immediate reward. Add discount factor The product of the maximum expected total return in the next state; The following loss function is used to optimize high-level Q-networks so that their traditional Q-value update function approximates the optimal Q-value function: In the formula, The value is , representing the Q value of the high-level objective; For expected value operators; For high-level main network parameters; Used to measure the predicted Q value and target Q value The error; This is the experience pool, used to store training samples. The corresponding empirical data was used for training by randomly sampling samples; This is an introduced high-level independent target network parameter, which is periodically synchronized with the high-level master network parameter. ; When training high-level Q-networks, the following adaptive deduplication search strategy is adopted: In the formula, This is an adaptive adjustment factor, whose value gradually decreases as training progresses; To repeat the action selection The number of times; For all actions The set constituted For the set of One action .

8. The hierarchical deep reinforcement learning routing protocol method according to claim 7, characterized in that, The lower-level Q-network of the lower-level controller optimizes link allocation by using local link information and guidance from the higher-level controller regarding the target node. The approximate Q-value update formula for the lower-level Q-network is as follows: In the formula, Overall, it indicates actions at a specified high level. The low-level Q-value function at that time; To perform the action The new state afterwards; This is the current value function of the lower-level Q-network. This is the target value function used to update the current value function of the lower-level Q-network; The loss function of the low-level Q-network is as follows: In the formula, The value is , representing the Q value of the low-level target; These are the parameters for the lower-level main network. This is a low-level independent target network parameter that is periodically synchronized with the low-level master network parameter. When training the low-level Q-network, the same adaptive deduplication search strategy as when training the high-level Q-network is adopted.

9. The hierarchical deep reinforcement learning routing protocol method according to claim 1, characterized in that, In step S4, the external reward consists of a function that includes three types of judgment factors: latency, accuracy, and hop count, where: Delay The transmission delay of a data packet from the sending node to the receiving node is expressed as follows: In the formula, This represents the average latency obtained by averaging the latency of all current packets in the network. This represents the preset maximum tolerable delay; This represents the load balancing penalty factor, used to prevent routing decision strategies from relying too heavily on paths with short latency. accuracy The percentage of data packets that successfully reach the target node is expressed as follows: In the formula, Representing the Packet loss rate of each link; N is the total number of links; For the first The weight of each link; Number of jumps The number of routing nodes that a data packet passes through from the source node to the destination node is expressed as follows: In the formula, Represents the hop count of the current route; This represents the maximum number of hops allowed by default. This represents the minimum number of hops from each node to other nodes; Based on the aforementioned functions of latency, accuracy, and hop count, the external reward function... It is expressed as follows: In the formula, , and These are the corresponding weight values.

10. The hierarchical deep reinforcement learning routing protocol method according to claim 1, characterized in that, In step S4, the internal reward consists of a function that includes two types of judgment factors: link signal-to-noise ratio and bandwidth utilization, where: Link signal-to-noise ratio The ratio of signal power to noise power in the link is expressed as follows: In the formula, Signal power; Noise power; Bandwidth utilization This is the ratio of the actual data rate transmitted on the current link to the maximum carrying capacity of the link, expressed as follows: In the formula, This represents the current available bandwidth of the link. This is the basic data bandwidth of the link; Based on the above link signal-to-noise ratio Bandwidth utilization The functional, internal reward function It is expressed as follows: In the formula, and These are the corresponding weight values.

Images