Hierarchical reinforcement learning routing decision method, device, storage medium and computer program product

Abstract

This application discloses a hierarchical reinforcement learning routing decision-making method, device, storage medium, and computer program product, relating to the field of telecommunications technology. The method includes: when an access point selects a next-hop node for a data packet to be forwarded, constructing a network state vector based on collected network state information; inputting the network state vector into an actor network to obtain an action probability distribution for all feasible neighbor nodes; and determining the next-hop node according to the action probability distribution, wherein the next-hop node includes terminal sites within the same cluster or other access points outside the current access point. This application ensures low-latency communication for routing decisions and improves the reliability of routing decisions.

Description

Technical Field

[0001] This application relates to the field of telecommunications technology, and in particular to a hierarchical reinforcement learning routing decision-making method, device, storage medium, and computer program product. Background Technology

[0002] Wireless Mesh Networks (WMNs) are a core networking technology supporting large-scale Internet of Things (IoT) and Industrial Internet. The efficiency of their routing protocols directly determines the network's throughput, latency, and reliability. With the continuous expansion of network scale and the increasing mobility of nodes, traditional routing decisions face severe challenges in dynamic environments.

[0003] While schemes based on centralized path calculation using a network manager facilitate global optimization, all routing decisions rely on a central node. This leads to an exponential increase in control signaling overhead with network size, making the network manager a performance bottleneck and a single point of failure. Such schemes struggle to detect and respond in real-time to transient changes such as link quality fluctuations, node movement, or localized interference, failing to meet the deterministic, low-latency communication requirements of scenarios like the Industrial Internet of Things (IIoT). Summary of the Invention

[0004] The main purpose of this application is to provide a hierarchical reinforcement learning routing decision method, device, storage medium and computer program product, which aims to solve the technical problem that routing decision cannot guarantee low-latency communication.

[0005] To achieve the above objectives, this application proposes a hierarchical reinforcement learning routing decision method applied to access points of a hierarchical network. The hierarchical network includes a global coordination layer composed of network managers and a local autonomous layer composed of access points. The access points manage a cluster of terminal sites. The hierarchical reinforcement learning routing decision method includes: When the access point selects the next-hop node for the data packet to be forwarded, a network state vector is constructed based on the collected network state information; The network state vector is input into the actor network to obtain the action probability distribution facing all feasible neighbor nodes; The next hop node is determined based on the action probability distribution. The next hop node includes terminal sites within the same cluster or other access points outside the current access point.

[0006] In one embodiment, after the step of determining the next hop node based on the action probability distribution, the method further includes: Forward the data packet to the next-hop node; After the data packet is forwarded, an immediate reward is calculated, and the parameters of the actor network and the critic network are updated based on the immediate reward.

[0007] In one embodiment, the step of updating the parameters of the actor network and the critic network based on the instant reward includes: Based on the instantaneous reward, calculate the time-series difference error; The value function parameters of the critic network are updated using the temporal difference error. The policy parameters of the actor network are updated using the temporal difference error through the policy gradient method.

[0008] In one embodiment, the step of calculating the immediate reward includes: The instant reward is calculated based on the packet delivery rate, link performance metrics, and cluster utilization corresponding to the access point. The link performance metric includes end-to-end latency and / or energy consumption information. The instant reward is positively correlated with the packet delivery rate and the cluster utilization rate, while the instant reward is negatively correlated with the end-to-end latency and the energy consumption information.

[0009] In one embodiment, the step of calculating the instant reward based on packet delivery rate, link performance metrics, and cluster utilization corresponding to the access point includes: The instant reward is calculated based on the first formula, the packet delivery rate, the link performance metric, and the cluster utilization rate corresponding to the access point. The first formula includes: ; in, Indicates the success rate of packet delivery. Indicates end-to-end delay. Indicates energy consumption information. Indicates cluster utilization. to These represent the corresponding weight parameters.

[0010] In one embodiment, before the step of calculating the instant reward based on packet delivery rate, link performance metrics, and cluster utilization corresponding to the access point, the method further includes: The link success probability under the retry mechanism is determined based on the single-attempt success probability of the wireless communication link and the configured retransmission budget. The data packet delivery rate is determined based on the actual single-hop transmission result of the data packet and / or the link success probability of the current link.

[0011] In one embodiment, the step of determining the link success probability under the retry mechanism based on the single-attempt success probability of the wireless communication link and the configured retransmission budget includes: Based on the second formula, the single-attempt success probability of the wireless communication link, and the configured retransmission budget, determine the link success probability under the retry mechanism. The second formula is shown below: ; in, Indicates the probability of link success. Indicates that the budget will be retransmitted. This represents the probability of success for a single attempt within the allocated retry budget.

[0012] In one embodiment, the probability of success in a single attempt is obtained through the following steps: The link loss weight is calculated based on the path loss index, and the radio reliability is determined based on the link loss weight. Determine the competition avoidance coefficient based on the probability of competition; Link availability is determined based on a random waypoint movement model, whereby link availability characterizes whether a node is within communication range. The success probability of a single attempt for a wireless communication link is determined based on the radio reliability, the contention avoidance coefficient, and the link availability.

[0013] In one embodiment, the step of determining the single-attempt success probability of a wireless communication link based on the radio reliability, the contention avoidance coefficient, and the link availability includes: The success probability of a single attempt of a wireless communication link is determined based on the third formula, the radio reliability, the contention avoidance coefficient, and the link availability. The third formula is as follows: ; in, This represents the probability of success in a single attempt. Indicates radio reliability, Indicates the link loss weight. Represents the competition avoidance coefficient, where Represents the probability of competition; and Indicates link availability.

[0014] In one embodiment, before the step of calculating the instant reward based on packet delivery rate, link performance metrics, and cluster utilization corresponding to the access point, the method further includes: Determine the routing type corresponding to the data transmission request, where the routing type includes intra-cluster routing or inter-cluster routing; The corresponding delay scaling factor is determined based on the route type, and the end-to-end delay is determined based on the delay scaling factor and the baseline delay. The corresponding energy consumption scaling factor is determined based on the routing type, and the energy consumption information is determined based on the energy consumption scaling factor and the baseline energy consumption.

[0015] In one embodiment, the step of determining the corresponding delay scaling factor based on the routing type includes: Determine the delay scaling factor based on average transmission distance, traffic load, and routing complexity; Specifically, when the routing type is intra-cluster communication, the shorter the average transmission distance, the smaller the traffic load, the smaller the routing complexity, and the smaller the delay scaling factor; when the routing type is inter-cluster communication, the longer the average transmission distance, the larger the traffic load, the larger the routing complexity, and the larger the delay scaling factor.

[0016] In one embodiment, the step of determining the corresponding energy consumption scaling factor based on the routing type includes: Determine the energy scaling factor based on the routing distance, retransmission budget, and traffic load. Specifically, when the routing type is intra-cluster routing, the shorter the routing transmission distance, the smaller the retransmission budget, the smaller the traffic load, and the smaller the energy consumption scaling factor; when the routing type is inter-cluster routing, the longer the routing transmission distance, the larger the retransmission budget, the larger the traffic load, and the larger the energy consumption scaling factor.

[0017] In one embodiment, before the step of calculating the instant reward based on packet delivery rate, link performance metrics, and cluster utilization corresponding to the access point, the method further includes: Cluster utilization is determined based on the cluster information managed by the access point; The cluster information includes at least the number of nodes in the cluster, the total traffic load in the cluster, the average queue length of nodes or access points, and the node density in the cluster.

[0018] In one embodiment, the step of determining cluster utilization based on cluster information managed by the access point includes: The cluster utilization rate is calculated based on the number of nodes in the cluster, the total traffic load in the cluster, the average queue length of the nodes or access points, the node density in the cluster, and the corresponding weight parameters.

[0019] In one embodiment, the method further includes: Within each preset telemetry period, link metric information and neighbor node update information within the communication range are reported to the network manager; The link metric information includes link success probability, average latency, and link status interruption caused by node offline or movement; the neighbor node update information includes new node additions, old node departures, and neighbor node changes.

[0020] In one embodiment, the network state vector includes at least: hop distance to the target node, hop count metric, link quality metric of neighboring nodes, statistical summary of single-attempt success probability of neighboring nodes, normalized degree of the current node, packet delivery rate, latency metric, cluster utilization, network size estimate, and link stability metric.

[0021] In one embodiment, the statistical summary of the single-attempt success probability of the neighboring node includes the average attempt success probability, the maximum attempt success probability, and the minimum attempt success probability.

[0022] Furthermore, to achieve the above objectives, this application also proposes a hierarchical reinforcement learning routing decision device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the hierarchical reinforcement learning routing decision method as described above.

[0023] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the hierarchical reinforcement learning routing decision method described above.

[0024] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the hierarchical reinforcement learning routing decision method described above.

[0025] One or more technical solutions proposed in this application have at least the following technical effects: By constructing a layered network architecture that coordinates a global coordination layer and a local autonomous layer, the complexity of large-scale network routing decisions is reduced. Based on this, access points utilize actor networks to adaptively generate action probability distributions for all feasible neighbor nodes based on instantaneous network state information. This supports both efficient local forwarding of terminal sites within the same cluster and cross-domain routing between clusters via other access points, improving the real-time performance and environmental adaptability of routing decisions, ensuring low-latency communication for routing decisions, and enhancing the reliability of routing decisions. Attached Figure Description

[0026] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0027] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a schematic diagram of the structure of the hierarchical network provided in the embodiment of the hierarchical reinforcement learning routing decision method of this application; Figure 2 This is a flowchart illustrating intra-cluster routing in a hierarchical network in an embodiment of the hierarchical reinforcement learning routing decision method of this application. Figure 3 This is a flowchart illustrating the inter-cluster routing of a hierarchical network in an embodiment of the hierarchical reinforcement learning routing decision method of this application. Figure 4 This is a flowchart illustrating an embodiment of the hierarchical reinforcement learning routing decision method of this application. Figure 5 A flowchart illustrating the link metric information and neighbor node update information of the access point in the hierarchical reinforcement learning routing decision method of this application; Figure 6 This is a flowchart illustrating Embodiment 2 of the hierarchical reinforcement learning routing decision method of this application. Figure 7 A flowchart illustrating the actor network and critic network of the hierarchical reinforcement learning routing decision method in this application; Figure 8 This is a flowchart illustrating Embodiment 3 of the hierarchical reinforcement learning routing decision method of this application; Figure 9 This is a schematic diagram of the device structure of the hardware operating environment involved in the hierarchical reinforcement learning routing decision method in the embodiments of this application.

[0029] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0030] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0031] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0032] As a core networking technology supporting large-scale Internet of Things (IoT) and Industrial Internet, the efficiency of wireless mesh networks directly determines the network's throughput, latency, and reliability. With the continuous expansion of network scale and the increasing mobility of nodes, traditional routing decisions face severe challenges in dynamic environments.

[0033] While schemes based on centralized path calculation using a network manager facilitate global optimization, all routing decisions rely on a central node. This leads to an exponential increase in control signaling overhead with network size, making the network manager a performance bottleneck and a single point of failure. Such schemes struggle to detect and respond in real-time to transient changes such as link quality fluctuations, node movement, or localized interference, failing to meet the deterministic, low-latency communication requirements of scenarios like the Industrial Internet of Things (IIoT).

[0034] The main solution of this application embodiment is as follows: when the access point selects the next-hop node for the data packet to be forwarded, a network state vector is constructed based on the collected network state information; the network state vector is input into the actor network to obtain the action probability distribution for all feasible neighbor nodes; the next-hop node is determined according to the action probability distribution, and the next-hop node includes terminal sites in the same cluster or other access points outside the current access point.

[0035] In this embodiment, for ease of description, the hierarchical reinforcement learning routing decision device will be used as the execution subject for the following description.

[0036] This application provides a solution that reduces the complexity of large-scale network routing decisions by constructing a hierarchical network architecture that coordinates a global coordination layer and a local autonomous layer. Based on this, access points utilize actor networks to adaptively generate action probability distributions for all feasible neighbor nodes based on instantaneous network state information. This supports both efficient local forwarding of terminal sites within the same cluster and cross-domain routing between clusters via other access points, improving the real-time performance and environmental adaptability of routing decisions, ensuring low-latency communication for routing decisions, and enhancing the reliability of routing decisions.

[0037] It should be noted that the hierarchical reinforcement learning routing decision-making device, the execution subject in this embodiment, can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device, access point, or terminal site capable of performing the above functions. The following description uses an access point as an example to illustrate this embodiment and the subsequent embodiments.

[0038] This application applies to hierarchical networks, which include a global coordination layer consisting of network managers (NM) and a local autonomous layer consisting of access points (AP). Access points are used to manage clusters of terminal stations (STA). Access points or network managers are configured as routing decision nodes in hierarchical networks.

[0039] The network manager manages at least one access point, each access point manages one cluster, and each cluster manages at least one end site. (See reference...) Figure 1 The network manager manages two access points, each managing its corresponding terminal sites. Optionally, the access points can be industrial routers, wireless mesh access points, IoT gateways, etc. Optionally, the terminal sites include, but are not limited to, mobile terminals such as mobile phones, drones, laptops, routers, digital broadcast receivers, personal digital assistants (PDAs), tablets (PADs), portable media players (PMPs), in-vehicle terminals, etc., as well as fixed terminals such as digital TVs, desktop computers, etc.

[0040] A layered, two-tier architecture is adopted to distribute control intelligence and alleviate the bottleneck of the network manager. The complete network is formally represented as the union of interconnected graphs. Viewing the deterministic wireless mesh network as a graph structure, where the upper layer serves as the global graph, can be represented as: ; in, This represents the global coordination layer consisting of the network manager and access points. This indicates the wireless connection between the access point and the network manager. In the local autonomy tier, each... Manage the local cluster graph, where And 1≤i≤N, specifically: ; in, This represents a combination consisting of an access point (APi) and its associated terminal sites (STAs). This represents the set of intra-cluster wireless links between access points and terminal sites. The complete network topology is a hierarchical combination of a global graph and all local cluster graphs, as shown below: ; in, Representing a global graph With each local cluster graph Hierarchical relationships between them, union operator All access points are aggregated into clusters. This layered network architecture enables distributed control, with intra-cluster routing handled locally at the access points, while inter-cluster routing is coordinated globally through the network manager.

[0041] In a hierarchical network, intra-cluster routing is performed by the access point to which the source terminal site belongs. The access point possesses a corresponding intra-cluster topology map, including wireless communication links between all terminal sites within the cluster. Based on this topology map, the access point can locally generate candidate paths within the cluster. Intra-cluster routing is intelligently performed locally by the access point, and the intra-cluster routing path sequentially includes: terminal site, its associated access point, and another terminal site within the same cluster.

[0042] In a hierarchical network, inter-cluster routing is performed by the network manager or coordinated by relevant access points under the guidance of the network manager. The network manager maintains a global topology map, which is aggregated from information reported by all access points. The network manager can generate cross-cluster paths based on the global topology map. Inter-cluster routing requires the network manager's coordination, and the inter-cluster routing path includes, in sequence: source terminal site, source access point, destination access point, and destination terminal site. The destination access point is determined by the network manager in coordinating the path.

[0043] Therefore, data forwarding can be divided into two cases: one is that the source node and the target node are located in the same cluster, and the other is that the source node and the target node belong to different clusters, where the source node is the source terminal site and the target node is the target terminal site.

[0044] In one embodiment, reference is made to Figure 2 When data is forwarded within the cluster, i.e., during intra-cluster routing, the source node and the destination node are located in the same cluster. Therefore, the access point does not need to query the network manager for destination node information. When the access point receives a message from the source node, it can directly forward it to the destination node.

[0045] In one embodiment, reference is made to Figure 3 When data is forwarded between clusters, i.e., when routing between clusters, and the source and destination nodes are located in different clusters, data is forwarded through coordination among the cluster nodes. Initially, the source node forwards the message to the source access point. The network manager retrieves the destination node address from the global routing table and forwards the routing request to the cluster head of the destination node, i.e., the destination access point. After establishing a connection with the destination access point, data is forwarded from the source node to the destination node through the cluster head coordination mechanism.

[0046] Based on this, embodiments of this application provide a hierarchical reinforcement learning routing decision method, referring to... Figure 4 , Figure 4This is a flowchart illustrating the first embodiment of the hierarchical reinforcement learning routing decision method of this application.

[0047] In this embodiment, the hierarchical reinforcement learning routing decision method includes steps S10 to S30: Step S10: When the access point selects the next-hop node for the data packet to be forwarded, a network state vector is constructed based on the collected network state information.

[0048] In this application, a hierarchical network and an Actor-Critic Network (AC) are used to endow access points with localized and intelligent routing decision-making capabilities. Locally, an Actor-Critic reinforcement learning model is deployed at the access point. When an access point needs to forward a data packet, it does not simply query a fixed routing table or request instructions from the network manager, but instead initiates a local intelligent routing decision-making process.

[0049] Network state vectors integrate information from multiple aspects such as topology, link quality, historical performance, real-time load, and mobility, providing rich environmental representations for actor and critic reinforcement learning models, enabling them to make adaptive and efficient routing decisions.

[0050] In an optional embodiment, the network state vector includes at least one of the following: hop distance to the target node, hop count metric, link quality metric of neighboring nodes, statistical summary of single-attempt success probability of neighboring nodes, normalized degree of the current node, packet delivery rate, latency metric, cluster utilization, network size estimate, and link stability metric. Optionally, the link stability metric includes steady-state on-network probability and / or expected link duration. Optionally, the statistical summary of single-attempt success probability of neighboring nodes includes average attempt success probability, maximum attempt success probability, and minimum attempt success probability.

[0051] node i The network state vector aggregates comprehensive network condition metrics. For example, the network state vector is shown in the following formula: ; in, This represents the estimated distance to the destination; This indicates the number of jumps. This represents the link quality metrics of neighboring nodes; , , ) represents a statistical summary of the single-attempt success probability of feasible neighbor nodes; deg represents the normalized degree of the current node; pDR represents the packet delivery rate estimate; Indicates a time delay metric; Indicates cluster utilization; Indicates network size estimation; This represents the steady-state network presence probability based on a stochastic routing model. Indicates the expected link duration.

[0052] It should be noted that the hop count distance to the destination node refers to the estimated minimum number of forwarding hops from the current node to the destination node of the data packet. The hop count distance to the destination node is calculated based on the local routing table or topology information, reflecting the length complexity of the path and serving as an important basis for selecting the shortest path. The hop count metric represents the cumulative number of hops a data packet has traversed from the source node to the current node. This metric is used to avoid routing loops and control path length, preventing excessive detours in the network. The link quality metric for neighboring nodes is a comprehensive evaluation value reflecting the average transmission quality of the link between the current node and all its directly communicating neighbors. It can be calculated from physical layer parameters such as received signal strength, signal-to-noise ratio, or bit error rate, and is used to comprehensively assess the channel conditions of the neighboring area. The single-attempt success probability of a neighboring node includes the average, standard deviation, and minimum single-transmission success probabilities to each neighbor. These values ​​are estimated in real-time based on historical data and link models, used to quantify the instantaneous reliability of each hop. The normalized degree of the current node refers to the ratio of the number of the current node's neighbors to the maximum possible degree of the network, normalized to the interval [0, 1]. This metric characterizes the local connectivity density of a node. High connectivity may mean more routing options, but it may also bring higher interference and contention. Packet delivery rate is an estimate of the successful forwarding rate of packets by the current node in its history, reflecting the node's reliability as a forwarding node and is an important indicator of its long-term performance. Latency metric represents the estimated end-to-end latency from the current node to the destination node, or the average queuing and processing latency at the current node. Latency metric integrates link transmission time, queuing time, and propagation delay, and is a key parameter for ensuring real-time performance. Access cluster information is an identifier indicating the cluster to which the current node belongs. If it is an access point, it identifies the cluster it manages; if it is a terminal site, it identifies the cluster it accesses. This information is used to distinguish between intra-cluster and inter-cluster routes and is the basis for implementing hierarchical routing. Network size estimation is the current node's approximate judgment of the total number of active nodes in the network. This information may be obtained through periodic updates from the network manager or local listening, helping the agent understand the overall scalability of the network. Link stability metric includes two key indicators: one is the steady-state probability of the link being in a connected state, calculated based on the mobility model; the other is the expected average duration of the link. Both characteristics together characterize the persistence and reliability of the link in node mobility scenarios, which is crucial for selecting a robust path.

[0053] In an optional embodiment, the steady-state on-network probability is determined based on the communication distance, the geometry of the mobile area, the node speed distribution, and the pause time distribution; the average expected link duration is calculated based on the communication distance and the expected relative speed between the two nodes.

[0054] Given the communication distance R and the expected relative speed between the two nodes Under the given conditions, calculate the statistical average length of the time from the moment the two nodes enter each other's communication range until either node moves out of the communication range, which is the average expected link duration.

[0055] Based on the set network parameters, such as region size, node speed distribution, pause time distribution, and communication distance R, different communication distances R and... can be pre-calculated by offline simulation of a random waypoint movement model or by solving more accurate integral equations. Combination The value is used to generate a two-dimensional lookup table. At runtime, a node only needs to measure or estimate the current value. You can quickly obtain the information by looking up the table. This is an efficient and accurate method in engineering.

[0056] In an optional embodiment, the steady-state on-network probability reflects the long-term statistical characteristics of two nodes within the communication range, while the average expected link duration reflects the link's sustainability. To characterize the long-term statistical characteristics of the link, its stationary properties are examined. The stationary on-network probability π represents the steady-state on-network probability of two nodes being within the communication range, determined based on communication distance, mobile area geometry, node speed distribution, and pause time distribution. Its expression is: ; Here, It is a function where R represents the communication distance and domain represents the geometry of the mobile domain, such as a rectangular area; Represents the probability distribution of node velocities; This indicates the distribution of pause times at waypoints.

[0057] The communication distance R directly affects the steady-state network presence probability. A larger communication distance R results in a higher steady-state network presence probability. The geometry of the mobile area determines the node position and distance distribution. It is typically assumed to be rectangular or circular, with different calculation formulas for different shapes. The node velocity probability distribution affects the speed at which a node reaches steady state. The pause time distribution is one of the most important influencing factors. If a node's pause time is not zero, its position is fixed when it pauses at a waypoint. A longer average pause time means that nodes stay at waypoints longer, making their position distribution closer to a uniform distribution. This significantly increases the chance of being in the edge region, potentially reducing the probability of being in the center region, directly impacting the calculation of the steady-state network presence probability.

[0058] Based on defined network parameters, such as region shape and size, communication distance, node speed probability distribution, and pause time distribution, the steady-state on-network probability is calculated offline in advance. The results of the steady-state on-network probability are compiled into a lookup table and stored in the access point's software or configuration file. Nodes only need to know their current communication distance R to instantly obtain the steady-state on-network probability by looking up the table.

[0059] Step S20: Input the network state vector into the actor network to obtain the action probability distribution for all feasible neighbor nodes.

[0060] An adaptive routing framework is implemented in the actor and critic network, which integrates all predefined physical layer, media access control layer and mobility metrics to make intelligent forwarding decisions.

[0061] Actor network as a policy function The approximator receives the current network state vector s, outputs a probability distribution or deterministic decision for all possible actions, and directly executes routing and forwarding actions; the commentator network acts as the value function. or action value function The estimator is responsible for evaluating the long-term cumulative reward of the current state or state-action pair, providing advantage estimates or temporal difference error guidance for the policy update of the actor network.

[0062] The actor network is a neural network whose parameters encode routing strategies learned through long-term learning. After analyzing the current state, the actor network does not directly assign a next hop, but instead calculates a probability value for each possible action—that is, choosing a feasible neighbor node as the next hop. The action probability distribution reflects the intelligence of the strategy, assigning higher probabilities to neighbor nodes that are more likely to bring high long-term rewards in the current state. At the same time, the probability distribution itself also retains a certain degree of randomness or exploratory nature to cope with changes in the network environment.

[0063] Optionally, the network state vector is sent to the actor network local to the access point, where action selection follows a constraint policy, and the action probability distribution is shown in the following formula: ; in, A parameterized strategy for the actor network, which is constrained by node roles and grid operation requirements in a feasible neighbor set. The next hop neighbor node is randomly selected.

[0064] The decisions made by the actor network are not static; a corresponding critic network is responsible for evaluating the value of these decisions. After each data packet forwarding, a reward signal is calculated. The critic network uses this reward and the changes in state before and after to calculate the time difference error, and then updates its own value estimation function. Then, the actor network updates its policy parameters based on the value guidance and policy gradient provided by the critics, making it more inclined to make decisions that yield higher rewards in similar future states. This dual update mechanism—actors learning to improve their policies and critics learning to evaluate values—allows each access point to continuously optimize its local routing strategy during operation.

[0065] Step S30: Determine the next-hop node based on the action probability distribution. The next-hop node includes terminal sites within the same cluster or other access points outside the current access point.

[0066] Based on the generated action probability distribution, the access point selects the next-hop node. The selection process can have a degree of randomness to continuously explore better paths. Alternatively, it can tend to select the node with the highest probability after a certain period to utilize known optimal strategies. Regardless of which neighboring node is chosen as the next hop, this decision is based on an intelligent assessment of the current complex network state, rather than static rules. If the target node of the data packet is located within the cluster managed by this access point, the next hop is usually a terminal site within the cluster, achieving efficient local switching. If the target node is a node in another cluster, the next hop is usually an access point connected to the backbone network or other clusters, thus achieving cross-cluster routing.

[0067] Optionally, after step S30, the method further includes: within each preset telemetry period, the access point reports the link metric information and neighbor node update information within the communication range to the network manager, referring to... Figure 5 The link metrics include link success probability, average latency, and link status interruption caused by node offline or movement; the neighbor node update information within the communication range includes new node additions, old node departures, and neighbor node changes.

[0068] Telemetry intervals are a periodic process, for example, every few seconds or tens of seconds. Real-time reporting of every link change incurs huge control overhead, overwhelming the data channel. Periodic reporting significantly reduces signaling load while ensuring relatively fresh information. Short-term, instantaneous network fluctuations are not reported immediately; instead, statistically significant changes over a period of time are reflected in periodic reports, making the global network topology more stable.

[0069] It should be noted that link metric information reflects changes in wireless communication quality. Link update information refers to link metrics that have changed significantly, learned and perceived by the access point through its local actor and critic framework. Examples include a sharp drop in the estimated success rate of a link, an increase in average latency, or a link interruption caused by a neighbor node going offline / moving. Here, the link success probability is the statistical probability of a data packet being successfully transmitted on the link, and the statistical change in average latency represents the recent fluctuations in the end-to-end latency of the link. Link state interruption is a connection interruption event caused by a neighbor node going offline, moving, or channel fading.

[0070] Neighbor node changes reflect dynamic changes in the local topology. A new node joining means a new terminal site or access point enters the communication range. An old node leaving means an existing neighbor node moves out of the communication range or goes offline. A neighbor node role change refers to a change in the functional role of a neighbor node, such as a regular node being upgraded to a cluster head or an access point switching its operating mode.

[0071] The network manager updates the global network topology and reallocates scheduling information based on information reported by all access points. This includes marking failed links, updating link weights and costs, adding new nodes and connections, and removing offline nodes. This ensures that the global network topology maintained by the network manager is always synchronized with the physical and logical state of the network.

[0072] Based on an updated and more accurate global topology map, the network manager performs global optimization calculations by reallocating scheduling information. This scheduling information includes TDMA (Time Division Multiple Access) time slot allocation, fine-tuning of global routing policies, and resource and role assignment. TDMA time slot allocation, in networks using TDMA access, involves the network manager recalculating and allocating time slices for data transmission across access points and critical links to avoid conflicts and optimize channel utilization. Fine-tuning of global routing policies involves the network manager calculating new, better cross-cluster backbone paths and distributing this path information as policy hints or initial policies to relevant access points, guiding their local learning. Resource and role assignment, for example, involves the network manager instructing some terminal sites to switch associated access points to achieve load balancing if it finds that access points in one area are overloaded while APs in another area are less busy.

[0073] By establishing a periodic status reporting mechanism within a preset telemetry period, access points can promptly synchronize link metrics such as link success probability, average latency, and link status interruption, as well as topology change information such as new node additions, old node departures, and neighbor node changes, to the network manager. This achieves efficient information interaction between the local autonomy layer and the global coordination layer in a hierarchical network. This mechanism ensures that the network manager can build and maintain a global network topology view in real time, dynamically grasp the link quality distribution and resource availability among clusters, and support macro-level cluster reconstruction, resource reallocation, and global optimization decisions. It also provides global state context compensation for the local actor and critic reinforcement learning of access points, avoiding the local optima and global information silos that may be caused by purely distributed decision-making. At the same time, by reporting only statistical changes rather than continuous data streams, control signaling overhead is reduced, achieving a good balance between information real-time performance and communication overhead. This realizes the organic combination of global coordination and local autonomy in a hierarchical network architecture, improving the overall manageability and routing optimization efficiency in a large-scale dynamic network environment.

[0074] In this embodiment, when an access point selects a next-hop node for a data packet to be forwarded, it constructs a network state vector based on collected network state information. The network state vector is then input into an actor network to obtain an action probability distribution for all feasible neighbor nodes. The next-hop node is determined based on this action probability distribution, and the next-hop node includes terminal sites within the same cluster or other access points outside the current access point. By constructing a hierarchical network architecture that coordinates the global coordination layer and the local autonomous layer, the complexity of large-scale network routing decisions is reduced. Furthermore, by utilizing the actor network, the access point can adaptively generate an action probability distribution for all feasible neighbor nodes based on instantaneous network state information. This supports efficient local forwarding for terminal sites within the same cluster and cross-domain routing between clusters via other access points, improving the real-time performance and environmental adaptability of routing decisions, ensuring low-latency communication for routing decisions, and enhancing the reliability of routing decisions.

[0075] Based on the first embodiment of this application, in the second embodiment of this application, the same or similar content as the above embodiment can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 6 After step S30, the hierarchical reinforcement learning routing decision method further includes: Step S40: Forward the data packet to the next-hop node; Step S50: After the data packet forwarding is completed, calculate the instant reward and update the parameters of the actor network and the critic network based on the instant reward.

[0076] After the data packet is forwarded to the selected next-hop node, a learning and optimization loop begins. The immediate reward is a comprehensive signal designed to reflect the multiple objectives of the routing decision.

[0077] In one embodiment, an immediate reward is calculated based on the actual transmission result of the data packet, which includes whether the transmission was successful or failed, end-to-end latency, throughput, packet loss rate, or link load status, etc. Optionally, in the immediate reward, successful delivery to the target node will receive a positive reward; end-to-end latency incurred during transmission will be converted into a negative penalty, with higher latency resulting in a larger penalty; packet loss or failure will generate a significant negative reward; in addition, the energy consumed in this forwarding may also be considered to encourage energy-saving paths. The reward function typically balances these competing objectives using weighted coefficients.

[0078] Upon receiving an immediate reward, this serves as the core feedback, driving parameter updates for both the actor and critic networks. The critic network is responsible for estimating the long-term expected value obtained by following the current policy in a given state. Temporal-Difference (TD) error is used to update the critic network. The TD error is the difference between the actual immediate reward plus the estimated value for the next state and the original estimated value for the current state. By minimizing this error, the critic network continuously refines its value estimation function, making it more accurate in predicting future returns.

[0079] The actor network is responsible for generating the policy, i.e., the probability distribution of action choices. It optimizes its policy based on more accurate value guidance provided by the critic network. Using the policy gradient method, the actor network's parameters are updated along directions that increase the probability of high-value actions and decrease the probability of low-value actions. Simply put, if choosing a certain next-hop node brings a high reward, the actor network fine-tunes its internal parameters to be more inclined to make that high-reward choice in similar situations in the future.

[0080] In an optional embodiment, the temporal difference error is calculated based on the immediate reward; the value function parameters of the critic network are updated using the temporal difference error; and the policy parameters of the actor network are updated using the policy gradient method with the temporal difference error.

[0081] Optionally, the update strategy for the actor and critic network is shown in the following formula: ; ; in, This represents the temporal difference error used to update the critic value function Vw. The policy gradient update for the actor parameter θ is defined. This dual update mechanism enables the system to simultaneously learn value estimation and improve decision-making policies based on empirical rewards.

[0082] Reference Figure 7By introducing temporal difference error as a shared learning signal for both the actor and critic networks, a single-step online update mechanism is implemented after packet forwarding is completed. On the one hand, the temporal difference error is used to update the value function parameters of the critic network, making its estimate of the future cumulative reward of the current state closer to the true value, thus providing an accurate policy evaluation benchmark. On the other hand, the same error term is used to update the policy parameters of the actor network through the policy gradient method, adjusting the action selection probability in the direction where the temporal difference error is positive. At the same time, the value baseline is used to effectively reduce the gradient variance, accelerating the convergence speed of the routing policy. This enables access points to quickly learn an adaptive forwarding policy that balances immediate transmission quality and long-term path optimality in dynamic network environments, improving the real-time performance, sample efficiency, and decision stability of hierarchical reinforcement learning routing.

[0083] In the technical solution of this embodiment, after the data packet is actually forwarded, the immediate reward is calculated based on the transmission result, and the policy parameters of the actor network and the value estimation parameters of the critic network are updated synchronously. This realizes real-time adaptive optimization and continuous policy evolution of routing decisions. This closed-loop process is completed entirely within the local autonomous layer, without frequent interaction with the global coordination layer, reducing control signaling overhead and decision latency. At the same time, the ability of the critic network to evaluate long-term cumulative benefits guides the routing policy gradient of the actor network, effectively balancing immediate rewards and long-term routing performance, accelerating policy convergence, and enabling access points to quickly learn and evolve the optimal forwarding strategy with low latency, high reliability, and load balancing in complex scenarios such as dynamic changes in network topology and fluctuations in traffic load. This improves the overall transmission efficiency and robustness of the hierarchical network.

[0084] Based on any embodiment of this application, in the third embodiment of this application, the content that is the same as or similar to the above embodiments can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 8 Step S50 includes: Step S51: Calculate the instant reward based on the packet delivery rate, link performance metric, and cluster utilization rate corresponding to the access point.

[0085] Link performance metrics include end-to-end latency and / or energy consumption information. Immediate rewards are positively correlated with packet delivery rate and cluster utilization, while rewards are negatively correlated with end-to-end latency and energy consumption. Packet delivery rate measures transmission reliability, and cluster utilization indicates resource load balancing.

[0086] Optionally, packet delivery rate refers to the ratio of the number of packets successfully forwarded to the total number of packets attempted to be forwarded by a network node, such as an access point or terminal node, within a statistical period. It is used to measure the long-term reliability and stability of the node as a data relay station.

[0087] Link performance metrics refer to statistical models or estimates of the historical performance and inherent properties of a specific communication link, pre-stored and maintained within the system. Optionally, link performance metrics include end-to-end latency and / or energy consumption information. Optionally, link performance metrics include single-hop success rate estimates, such as the probability of a successful transmission attempt calculated based on historical transmission statistics and current signal-to-noise ratio, contention probability, etc. Link performance metrics include single-hop estimated latency, such as the transmission, queuing, and processing time typically required for a data packet to pass through the link. Link performance metrics include single-hop estimated energy consumption, such as the energy typically consumed to send a data packet through the link.

[0088] Cluster utilization is a key, dynamically changing network status parameter. It characterizes the local network load, reflecting not the quality of a single link, but the macroscopic operational status of the entire cluster managed by the access point—the local area. Cluster utilization directly reflects the network's busyness, resource strain, and potential congestion risk in that area. It is correlated with the ratio of active connections, channel occupancy time, buffer utilization, and data throughput ratio, all of which can be measured and calculated in real-time or near real-time from network interfaces, drivers, or system counters.

[0089] In one embodiment, the instant reward provides a positive incentive for successful packet delivery and imposes a negative penalty on transmission delay and energy consumption. Simultaneously, a cluster utilization metric reflecting network load is introduced as an adjustment term. The instant reward is calculated based on a first formula, packet delivery rate, link performance metrics, and the cluster utilization corresponding to the access point. The first formula includes: ; in, This indicates successful data packet delivery. Indicates end-to-end delay and Indicates energy consumption, and This indicates an additional reward based on the reasonable utilization rate of the access cluster. Weighting coefficient. to This represents the weighting parameters, which balance these competing objectives.

[0090] In an optional embodiment, before step S51, the method further includes: determining the link success probability under the retry mechanism based on the single-attempt success probability of the wireless communication link and the configured retransmission budget; and determining the data packet delivery rate based on the single-hop actual transmission result of the data packet and / or the link success probability of the current link.

[0091] By combining the success probability of a single attempt with the configured retransmission budget, the link success probability under the retry mechanism is quantified, explicitly characterizing the role of the retransmission strategy in improving transmission reliability. Based on this, or based on the actual single-hop transmission results, the packet delivery rate can be accurately determined, thus providing a reliability evaluation benchmark that closely reflects the real transmission characteristics of the physical layer for subsequent multi-dimensional real-time reward calculation.

[0092] Optionally, the packet delivery rate (PDR) is determined based on the actual single-hop transmission results. Each time a node completes a packet forwarding, the actual single-hop transmission result is recorded, such as success or failure. Then, an algorithm such as a sliding window or exponentially weighted moving average is used to update the historical success rate statistics. For example, if an access point recently forwarded 100 packets, of which 98 received the next-hop packet, then the PDR is updated to 98%.

[0093] Optionally, the packet delivery rate can be determined based on the link success probability of the current link. When there is insufficient historical data or a need to respond quickly to drastic changes, a node can use the newly calculated link success probability to a specific neighbor as an estimate or initial value to represent the expected packet delivery rate for its current forwarding in that direction. If the link model to neighbor A shows a success probability as high as 99%, the current packet delivery rate to A is considered high.

[0094] Alternatively, in a stable network, packet delivery rate is continuously and smoothly updated primarily based on actual results, providing a solid historical foundation. When link states change abruptly or metrics are established for new links, the current theoretical probabilities are used to quickly initialize or correct the packet delivery rate estimate. This combination ensures that the packet delivery rate metric possesses both historical robustness and the agility to respond to real-time changes.

[0095] To account for the MAC layer retransmission mechanism, the success probability of each hop in a retry follows a geometric distribution. Based on the second formula, the single-attempt success probability of the wireless communication link, and the configured retransmission budget, the link success probability under the retry mechanism is determined. The second formula is shown below: ; in, Indicates the probability of link success. The retransmission budget configured for the MAC layer. It describes the probability that at least one transmission attempt will succeed within the allocated retry limit.

[0096] In an optional embodiment, the single-attempt success probability is obtained through the following steps: calculating the link loss weight based on the path loss index, and determining the radio reliability based on the link loss weight; determining the contention avoidance coefficient based on the contention probability; determining the link availability based on the random waypoint movement model, where link availability characterizes whether a node is within communication range; and determining the single-attempt success probability of the wireless communication link based on the radio reliability, contention avoidance coefficient, and link availability.

[0097] In wireless mesh network scenarios, some terminal devices or nodes are mobile, thus requiring a mobility model. The dynamic connectivity between nodes i and j is determined by their mobility. Under the random waypoint mobility model, the instantaneous state of the wireless communication link is determined by its link availability. The definition is shown in the following formula: ; in, Let represent the instantaneous Euclidean distance between nodes i and j at time t, and This is for fixed wireless communication range. The indicator function {} outputs a value of 1 when the condition is true, indicating that the link is active; otherwise, it outputs 0.

[0098] Optionally, the success probability of a single attempt for transmission on link (i, j) is determined based on the third formula, radio reliability, contention avoidance coefficient, and link availability; the third formula is shown below: ; in, This represents the probability of success in a single attempt. Indicates radio reliability, ∈[0,1] represents the normalized link packet loss weight of FSPL, i.e., the link loss weight; This represents the competition avoidance coefficient. Indicates the probability of competition; Indicates link availability, taken from the binary indicator function of the RWP mobility model, and is only 1 when the node is within communication range.

[0099] Based on a preset mapping relationship, the time slot load factor of the time division multiple access mechanism is mapped to the contention probability of the wireless communication link. Optionally, the mapping function is a monotonic function, and step S21 includes: based on the monotonic function, mapping the time slot load factor of the time division multiple access mechanism to the contention probability of the wireless communication link, where the contention probability characterizes the probability of collision caused by simultaneous transmission attempts. Its expression is: ; Here, It is a monotonic function that maps time slot load sub-slots to contention probabilities, thereby capturing the collision probability caused by simultaneous transmission attempts.

[0100] By integrating three dimensions—radio reliability, contention avoidance coefficient, and link availability—to construct the success probability of a single attempt, a full-stack accurate modeling and dynamic prediction of wireless link quality is achieved. Specifically, radio reliability calculated based on path loss accurately reflects physical layer signal quality; the avoidance coefficient derived using contention probability effectively quantifies the inhibitory effect of MAC layer channel contention on transmission success rate; and the introduction of a random waypoint movement model to evaluate link availability proactively addresses the risk of communication interruption caused by node movement. This multi-dimensional evaluation mechanism enables routing decisions to comprehensively perceive multiple challenges such as signal attenuation, channel contention, and dynamic topology changes, significantly improving the accuracy and robustness of link state prediction. It avoids the drawbacks of traditional methods that overestimate link performance due to ignoring mobility or contention effects, thus enabling the selection of paths with truly continuous and stable transmission capabilities, effectively reducing packet loss rate and retransmission overhead, and ensuring reliable end-to-end service delivery. It is particularly suitable for wireless mesh network scenarios with highly mobile terminals and frequent topology changes.

[0101] In an optional embodiment, the end-to-end latency is obtained through the following steps: determining the routing type corresponding to the data transmission request, where the routing type includes intra-cluster routing or inter-cluster routing; determining the corresponding latency scaling factor based on the routing type, and determining the end-to-end latency based on the latency scaling factor and the baseline latency; determining the corresponding energy consumption scaling factor based on the routing type, and determining energy consumption information based on the energy consumption scaling factor and the baseline energy consumption.

[0102] Path performance is characterized by latency and energy consumption metrics, and intra-cluster and inter-cluster communication are distinguished: ; ; in, and These represent the baseline latency and energy model, respectively, depending on the hop count H and bandwidth B, while the scaling factor... and This reflects the unique characteristics of intra-cluster and inter-cluster communication. These fundamental metrics ultimately form a comprehensive routing objective function, which can optimize paths based on multiple criteria.

[0103] Path performance is characterized by latency and energy consumption metrics, and intra-cluster and inter-cluster communication are distinguished: ; ; in, and These represent the baseline latency and energy model, respectively, depending on the hop count H and bandwidth B, while the scaling factor... and This reflects the unique characteristics of intra-cluster and inter-cluster communication. These fundamental metrics ultimately form a comprehensive routing objective function, which can optimize paths based on multiple criteria.

[0104] Further, in one embodiment, the step of determining the corresponding delay scaling factor based on the routing type includes: determining the delay scaling factor based on the average transmission distance, traffic load, and routing complexity, as shown in the following formula: ; in, d Indicates the average transmission distance; L Indicates the traffic load or contention level; M Indicates routing complexity; , , This represents the normalization coefficient.

[0105] Specifically, when the routing type is intra-cluster communication, a shorter average transmission distance results in lower traffic load, lower routing complexity, and a smaller delay scaling factor; conversely, when the routing type is inter-cluster communication, a longer average transmission distance results in higher traffic load, higher routing complexity, and a larger delay scaling factor. Optionally, .

[0106] In another embodiment, the step of determining the corresponding energy consumption scaling factor based on the route type includes: determining the energy consumption scaling factor based on the route transmission distance, retransmission budget, and traffic load, as shown in the following formula: ; in, R represents the route transmission distance and path loss effect; R represents the average number of retransmissions, which is related to the retry budget; L represents the traffic load or contention level. , , This represents the weighting coefficient.

[0107] Specifically, when the routing type is intra-cluster routing, the shorter the route transmission distance, the smaller the retransmission budget, the lower the traffic load, and the smaller the energy scaling factor; when the routing type is inter-cluster routing, the longer the route transmission distance, the larger the retransmission budget, the greater the traffic load, and the larger the energy scaling factor. Optionally, .

[0108] and All parameters are dynamically calculated based on real-time network measurement data, such as distance, load, mobility, and retries. For intra-cluster routing, these parameters are updated periodically at the access point layer; for inter-cluster routing, they are updated periodically at the network manager layer. These parameters ensure that routing decisions accurately reflect the higher latency and energy consumption associated with inter-cluster communication.

[0109] By prioritizing the intra-cluster or inter-cluster routing types of data transmission requests and dynamically configuring differentiated latency and energy scaling factors based on these types, refined modeling of heterogeneous link costs is achieved. This hierarchical metric mechanism enables actor and critic networks to accurately perceive the essential differences between short-range, low-energy forwarding within their local autonomous layers and high-overhead cross-cluster backbone links. It explicitly guides the actor network to prioritize low-cost intra-cluster paths and prudently evaluate inter-cluster forwarding decisions during policy learning, improving the state discriminability and environmental adaptability of reward signals, accelerating the convergence speed of policy gradients in complex hierarchical topologies, and effectively avoiding the local optimum trap caused by a unified metric standard. Ultimately, it achieves a smart routing strategy that is load-balanced across the entire network, energy-efficient, and consistent with the design intent of a hierarchical network architecture.

[0110] In an optional embodiment, cluster utilization is determined based on cluster information managed by the access point; wherein the cluster information includes at least the number of nodes in the cluster, the total traffic load in the cluster, the average queue length of the nodes or access point, and the node density in the cluster.

[0111] Furthermore, based on the number of nodes in the cluster, the total traffic load in the cluster, the average queue length of nodes or access points, the node density in the cluster, and the corresponding weight parameters, a weighted value is calculated as the cluster utilization rate, as exemplified by the following formula: ; in, This indicates the number of nodes in the cluster, reflecting the cluster size. This indicates the total traffic load in the cluster, reflecting the degree of congestion. The average queue length of a node or access point represents the buffer latency. Indicates the node density or connectivity within the cluster; , , , Let be the weight coefficient, and satisfy... .

[0112] parameter This represents a comprehensive metric that reflects the overall condition of the cluster and its applicability in routing decisions. This parameter is incorporated into the reward function to guide the learning process by including cluster-level network status. The value reflects how advantageous the cluster is for routing: lower load and shorter queues result in higher routing efficiency; moderate node density leads to better connectivity and redundancy; and a balanced cluster size results in higher resource utilization. Therefore, As a compact representation of cluster-level quality, it enables routing policies to prioritize clusters with better communication conditions.

[0113] In an optional embodiment, cluster utilization is determined based on cluster information managed by the access point; wherein the cluster information includes at least the ratio of active connections, the ratio of channel occupancy time, the buffer occupancy rate, and the data throughput ratio. Cluster utilization is an indicator used to measure the current load or busy level of a local cluster managed by a single access point. Cluster utilization refers to the utilization of communication resources within the cluster, the density of data traffic load, or the proportion of active communication tasks.

[0114] Incorporating cluster utilization into the calculation of immediate rewards aims to reinforce the awareness of global load balancing among access points when making routing decisions. If an access point manages a cluster with high utilization, continuing to route new data streams through that access point may lead to congestion within that cluster, increasing latency and packet loss risk for all data packets passing through it. The reward mechanism incentivizes the algorithm to find lighter cluster paths by imposing slight negative penalties or reducing positive rewards in such cases. This guided learning algorithm intelligently distributes network traffic across different clusters and access points, thereby improving the overall network throughput and stability.

[0115] By integrating multi-dimensional cluster status indicators such as the proportion of active connections, the proportion of channel occupancy time, the buffer occupancy rate, and the data throughput ratio, a refined cluster utilization evaluation system was constructed, enabling comprehensive and real-time perception of the local autonomous layer resource congestion status. This multi-dimensional resource profile accurately reflects the overall load level of the cluster in terms of connection density, channel contention, queue backlog, and traffic intensity. It provides a high-information state representation and reward calculation benchmark for the actor and critic reinforcement learning framework, enabling access points to effectively identify and avoid high-load nodes in routing decisions, dynamically balance traffic distribution within or between clusters, and prevent buffer overflow, channel conflict, or rapid energy depletion caused by local resource overload. At the same time, this fine-grained utilization metric significantly enhances the sensitivity and discriminative power of reward signals to network status, guiding the actor network to spontaneously form load balancing preferences in policy gradient updates, accelerating policy convergence in complex heterogeneous network environments, and ultimately achieving globally optimized routing decisions that balance resource efficiency and service quality.

[0116] The active connection ratio is the number of terminal devices currently communicating divided by the total number of terminal devices in the cluster. The channel occupancy time ratio is the proportion of wireless channel time occupied by the cluster within the statistical period. The queue length or buffer occupancy rate is the average occupancy rate of the access point's buffer for forwarding data packets. The data throughput ratio is the ratio of the current actual throughput to the cluster's theoretical maximum throughput.

[0117] In the technical solution of this embodiment, by constructing a multi-dimensional reward function that integrates packet delivery rate, link performance measurement, and cluster utilization, a refined quantitative evaluation and multi-objective collaborative optimization of routing decision quality are achieved. This layered and decoupled reward design enables the local autonomous layer to self-awarely balance high-reliability transmission, low-latency guarantee, high-energy-efficiency routing, and cluster load balancing without relying on global coordination. This avoids network congestion, energy depletion, or QoS degradation caused by optimizing a single indicator. At the same time, it provides dense reward signals with rich state discrimination for the actor and critic reinforcement learning framework, effectively guiding the policy gradient to update in the direction of Pareto optimality, accelerating the convergence speed and exploration efficiency of the agent in complex dynamic environments, and ultimately achieving adaptive and robust routing decisions that maximize the utilization of network resources while meeting the service quality requirements of differentiated services.

[0118] In one specific embodiment, once the optimal path is determined, the data forwarding process is executed through an Adaptive Actor and Critics (AC) decision layer, which continuously optimizes the routing strategy based on real-time network feedback. Each access point and end node maintains a state vector encapsulating network parameters such as hop count, neighbor quality, link reliability, latency, and packet delivery rate. The network selects the optimal next-hop node through a policy function. Simultaneously, the critic is rewarded based on transmission success rate, latency, and energy consumption. Optionally, the AC model uses time difference (TD) learning for iterative updates.

[0119] SmartMesh dynamically adjusts policy parameters and improves decision accuracy over time. The local learning process at access points ensures the routing mechanism remains adaptive under changes in mobility, interference, or load, while the network manager periodically aggregates feedback to guarantee global stability. To maintain network consistency, access points and the network manager periodically exchange telemetry updates. These updates refresh link metrics and ensure routing table synchronization, thus enabling self-learning and self-healing capabilities throughout the SmartMesh topology.

[0120] Specifically, the online reposting, actor and critic decisions, and maintenance are shown in the table below:

[0121] By constructing a layered network architecture that coordinates a global coordination layer and a local autonomous layer, the complexity of large-scale network routing decisions is reduced. Based on this, access points utilize actor networks to adaptively generate action probability distributions for all feasible neighbor nodes based on instantaneous network state information. This supports both efficient local forwarding of terminal sites within the same cluster and cross-domain routing between clusters via other access points, improving the real-time performance and environmental adaptability of routing decisions, ensuring low-latency communication for routing decisions, and enhancing the reliability of routing decisions.

[0122] After the data packet is actually forwarded, the instant reward is immediately calculated based on the transmission result, and the policy parameters of the actor network and the value estimation parameters of the critic network are updated synchronously. This realizes real-time adaptive optimization and continuous policy evolution of routing decisions. This process is completed entirely within the local autonomous layer, without frequent interaction with the global coordination layer, reducing control signaling overhead and decision latency. At the same time, the ability of the critic network to evaluate long-term cumulative benefits guides the routing policy gradient of the actor network, balancing instant rewards and long-term routing performance, accelerating policy convergence, and enabling access points to quickly learn and evolve the optimal forwarding strategy with low latency, high reliability, and load balancing in complex scenarios such as dynamic changes in network topology and fluctuations in traffic load. This improves the overall transmission efficiency and robustness of the hierarchical network.

[0123] It should be noted that the above examples are only for understanding this application and do not constitute a limitation on the hierarchical reinforcement learning routing decision method of this application. Any simple modifications based on this technical concept are within the protection scope of this application.

[0124] This application provides a hierarchical reinforcement learning routing decision device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the hierarchical reinforcement learning routing decision method in the above embodiment 1.

[0125] The following is for reference. Figure 9 It shows a schematic diagram of a structure suitable for implementing a hierarchical reinforcement learning routing decision device in the embodiments of this application. Figure 9 The hierarchical reinforcement learning routing decision device shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0126] like Figure 9As shown, the hierarchical reinforcement learning routing decision device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the hierarchical reinforcement learning routing decision device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. The communication device 1009 allows the hierarchical reinforcement learning routing decision device to communicate wirelessly or wiredly with other devices to exchange data. While the figure shows hierarchical reinforcement learning routing decision devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0127] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0128] The hierarchical reinforcement learning routing decision device provided in this application, employing the hierarchical reinforcement learning routing decision method in the above embodiments, can solve the technical problem that routing decision cannot guarantee low-latency communication. Compared with the prior art, the beneficial effects of the hierarchical reinforcement learning routing decision device provided in this application are the same as those of the hierarchical reinforcement learning routing decision method provided in the above embodiments, and other technical features in this hierarchical reinforcement learning routing decision device are the same as those disclosed in the previous embodiment method, and will not be repeated here.

[0129] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0130] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0131] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the hierarchical reinforcement learning routing decision method in the above embodiments.

[0132] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0133] The aforementioned computer-readable storage medium may be included in the hierarchical reinforcement learning routing decision device; or it may exist independently and not be assembled into the hierarchical reinforcement learning routing decision device.

[0134] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the hierarchical reinforcement learning routing decision-making device, the device performs the following actions: when an access point selects a next-hop node for a data packet to be forwarded, it constructs a network state vector based on collected network state information; it inputs the network state vector into an actor network to obtain an action probability distribution for all feasible neighbor nodes; and it determines the next-hop node based on the action probability distribution. The next-hop node includes terminal sites within the same cluster or other access points outside the current access point. By constructing a hierarchical network architecture that coordinates the global coordination layer and the local autonomous layer, the complexity of large-scale network routing decisions is reduced. Furthermore, by utilizing the actor network, the access point can adaptively generate an action probability distribution for all feasible neighbor nodes based on instantaneous network state information. This supports efficient local forwarding of terminal sites within the same cluster and cross-domain routing between clusters via other access points, improving the real-time performance and environmental adaptability of routing decisions, ensuring low-latency communication for routing decisions, and enhancing the reliability of routing decisions.

[0135] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0137] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0138] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described hierarchical reinforcement learning routing decision method, thereby solving the technical problem that routing decisions cannot guarantee low-latency communication. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the hierarchical reinforcement learning routing decision method provided in the above embodiments, and will not be repeated here.

[0139] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the hierarchical reinforcement learning routing decision method as described above.

[0140] The computer program product provided in this application can solve the technical problem that routing decisions cannot guarantee low-latency communication. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the hierarchical reinforcement learning routing decision method provided in the above embodiments, and will not be repeated here.

[0141] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A hierarchical reinforcement learning routing decision method, characterized in that, An access point is applied to a hierarchical network, which includes a global coordination layer composed of network managers and a local autonomous layer composed of access points. The access point is used to manage a cluster of terminal sites. The hierarchical reinforcement learning routing decision method includes: When the access point selects the next-hop node for the data packet to be forwarded, a network state vector is constructed based on the collected network state information; The network state vector is input into the actor network to obtain the action probability distribution facing all feasible neighbor nodes; The next hop node is determined based on the action probability distribution. The next hop node includes terminal sites within the same cluster or other access points outside the current access point.

2. The hierarchical reinforcement learning routing decision method as described in claim 1, characterized in that, After the step of determining the next hop node based on the action probability distribution, the method further includes: Forward the data packet to the next-hop node; After the data packet is forwarded, an immediate reward is calculated, and the parameters of the actor network and the critic network are updated based on the immediate reward.

3. The hierarchical reinforcement learning routing decision method as described in claim 2, characterized in that, The step of updating the parameters of the actor network and the critic network based on the instant reward includes: Based on the instantaneous reward, calculate the time-series difference error; The value function parameters of the critic network are updated using the temporal difference error. The policy parameters of the actor network are updated using the temporal difference error through the policy gradient method.

4. The hierarchical reinforcement learning routing decision method as described in claim 2, characterized in that, The steps for calculating the instant reward include: The instant reward is calculated based on the packet delivery rate, link performance metrics, and cluster utilization corresponding to the access point. The link performance metric includes end-to-end latency and / or energy consumption information. The instant reward is positively correlated with the packet delivery rate and the cluster utilization rate, while the instant reward is negatively correlated with the end-to-end latency and the energy consumption information.

5. The hierarchical reinforcement learning routing decision method as described in claim 4, characterized in that, The step of calculating the instant reward based on packet delivery rate, link performance metrics, and cluster utilization corresponding to the access point includes: The instant reward is calculated based on the first formula, the packet delivery rate, the link performance metric, and the cluster utilization rate corresponding to the access point. The first formula includes: ； in, Indicates the success rate of packet delivery. Indicates end-to-end delay. Indicates energy consumption information. Indicates cluster utilization. to These represent the corresponding weight parameters.

6. The hierarchical reinforcement learning routing decision method as described in claim 4, characterized in that, Before the step of calculating the instant reward based on packet delivery rate, link performance metrics, and cluster utilization corresponding to the access point, the method further includes: The link success probability under the retry mechanism is determined based on the single-attempt success probability of the wireless communication link and the configured retransmission budget. The data packet delivery rate is determined based on the actual single-hop transmission result of the data packet and / or the link success probability of the current link.

7. The hierarchical reinforcement learning routing decision method as described in claim 6, characterized in that, The step of determining the link success probability under the retry mechanism based on the single-attempt success probability of the wireless communication link and the configured retransmission budget includes: Based on the second formula, the single-attempt success probability of the wireless communication link, and the configured retransmission budget, determine the link success probability under the retry mechanism. The second formula is shown below: ； in, Indicates the probability of link success. Indicates that the budget will be retransmitted. This represents the probability of success for a single attempt within the allocated retry budget.

8. The hierarchical reinforcement learning routing decision method as described in claim 6, characterized in that, The probability of success in a single attempt is obtained through the following steps: The link loss weight is calculated based on the path loss index, and the radio reliability is determined based on the link loss weight. Determine the competition avoidance coefficient based on the probability of competition; Link availability is determined based on a random waypoint movement model, whereby link availability characterizes whether a node is within communication range. The success probability of a single attempt for a wireless communication link is determined based on the radio reliability, the contention avoidance coefficient, and the link availability.

9. The hierarchical reinforcement learning routing decision method as described in claim 8, characterized in that, The step of determining the single-attempt success probability of a wireless communication link based on the radio reliability, the contention avoidance coefficient, and the link availability includes: The success probability of a single attempt of a wireless communication link is determined based on the third formula, the radio reliability, the contention avoidance coefficient, and the link availability. The third formula is as follows: ； in, This represents the probability of success in a single attempt. Indicates radio reliability, Indicates the link loss weight. Represents the competition avoidance coefficient, where Represents the probability of competition; and Indicates link availability.

10. The hierarchical reinforcement learning routing decision method as described in claim 4, characterized in that, Before the step of calculating the instant reward based on packet delivery rate, link performance metrics, and cluster utilization corresponding to the access point, the method further includes: Determine the routing type corresponding to the data transmission request, where the routing type includes intra-cluster routing or inter-cluster routing; The corresponding delay scaling factor is determined based on the route type, and the end-to-end delay is determined based on the delay scaling factor and the baseline delay. The corresponding energy consumption scaling factor is determined based on the routing type, and the energy consumption information is determined based on the energy consumption scaling factor and the baseline energy consumption.

11. The hierarchical reinforcement learning routing decision method as described in claim 10, characterized in that, The step of determining the corresponding delay scaling factor based on the routing type includes: Determine the delay scaling factor based on average transmission distance, traffic load, and routing complexity; Specifically, when the routing type is intra-cluster communication, the shorter the average transmission distance, the smaller the traffic load, the smaller the routing complexity, and the smaller the delay scaling factor; when the routing type is inter-cluster communication, the longer the average transmission distance, the larger the traffic load, the larger the routing complexity, and the larger the delay scaling factor.

12. The hierarchical reinforcement learning routing decision method as described in claim 10, characterized in that, The step of determining the corresponding energy consumption scaling factor based on the routing type includes: Determine the energy scaling factor based on the routing distance, retransmission budget, and traffic load. Specifically, when the routing type is intra-cluster routing, the shorter the routing transmission distance, the smaller the retransmission budget, the smaller the traffic load, and the smaller the energy consumption scaling factor; when the routing type is inter-cluster routing, the longer the routing transmission distance, the larger the retransmission budget, the larger the traffic load, and the larger the energy consumption scaling factor.

13. The hierarchical reinforcement learning routing decision method as described in claim 4, characterized in that, Before the step of calculating the instant reward based on packet delivery rate, link performance metrics, and cluster utilization corresponding to the access point, the method further includes: Cluster utilization is determined based on the cluster information managed by the access point; The cluster information includes at least the number of nodes in the cluster, the total traffic load in the cluster, the average queue length of nodes or access points, and the node density in the cluster.

14. The hierarchical reinforcement learning routing decision method as described in claim 13, characterized in that, The step of determining cluster utilization based on the cluster information managed by the access point includes: The cluster utilization rate is calculated based on the number of nodes in the cluster, the total traffic load in the cluster, the average queue length of the nodes or access points, the node density in the cluster, and the corresponding weight parameters.

15. The hierarchical reinforcement learning routing decision method as described in any one of claims 1 to 14, characterized in that, The method further includes: Within each preset telemetry period, link metric information and neighbor node update information within the communication range are reported to the network manager; The link metric information includes link success probability, average latency, and link status interruption caused by node offline or movement; the neighbor node update information includes new node additions, old node departures, and neighbor node changes.

16. The hierarchical reinforcement learning routing decision method as described in any one of claims 1 to 14, characterized in that, The network state vector includes at least: hop distance to the target node, hop count metric, link quality metric of neighboring nodes, statistical summary of single-attempt success probability of neighboring nodes, normalized degree of the current node, packet delivery rate, latency metric, cluster utilization, network size estimate, and link stability metric.

17. The hierarchical reinforcement learning routing decision method as described in claim 16, characterized in that, The statistical summary of the single-attempt success probability of the neighboring nodes includes the average success probability, the maximum success probability, and the minimum success probability.

18. A hierarchical reinforcement learning routing decision-making device, characterized in that, The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the hierarchical reinforcement learning routing decision method as described in any one of claims 1 to 17.

19. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the hierarchical reinforcement learning routing decision method as described in any one of claims 1 to 17.

20. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the network control method as described in any one of claims 1 to 17.

Images