A wireless self-organizing network service-aware cross-layer optimization method and system based on hierarchical reinforcement learning

Abstract

This application discloses a service-aware cross-layer optimization method and system for wireless ad hoc networks based on hierarchical reinforcement learning, relating to the field of wireless ad hoc networks. The method includes defining each service flow to be controlled in the wireless ad hoc network as an agent, and constructing the joint optimization problem of routing and resource allocation for multiple service flows as a Markov decision process model. For any agent, a local observation graph is constructed with the agent's current node as the central node, and features are extracted based on the local observation graph to obtain the agent's current state representation. The agent's current state representation is used to make decisions based on a pre-trained hierarchical reinforcement learning network, selecting and outputting joint actions from the joint action space. The joint actions selected by each agent are distributed to the corresponding physical nodes for execution, realizing service-aware cross-layer optimization in wireless ad hoc networks. This application can effectively solve the problems of action space explosion and credit allocation.

Description

Technical Field

[0001] This application relates to the field of wireless ad hoc networks, and in particular to a service-aware cross-layer optimization method and system for wireless ad hoc networks based on hierarchical reinforcement learning. Background Technology

[0002] Wireless ad hoc networks (WANETs, ​​such as drone relays, IoT, and disaster relief scenarios) geared towards future 6G are self-organized by nodes without fixed infrastructure. Nodes act as both terminals and relays, enabling multi-hop communication. In such networks, routing, channel access, and transmit power control are strongly coupled: for example, choosing the shortest geographical path may exacerbate co-channel interference; while increasing transmit power may improve link quality, it will accelerate node energy depletion and shorten network lifespan.

[0003] Related deep reinforcement learning methods (such as DQN and MADRL) have the following drawbacks: 1. Curse of Dimension: The action space of joint decision-making (simultaneously selecting the next hop, channel, and power) is huge and grows exponentially with the number of candidate neighbors, channels, and power levels, resulting in extremely low exploration efficiency and difficulty in convergence.

[0004] 2. Poor generalization ability: Traditional neural networks require fixed-dimensional input and output, making it difficult to adapt to the dynamically changing number of nodes, neighbor sets, and topology in WANETs.

[0005] 3. Credit allocation problem (sparse rewards): In multi-hop routing, the contribution of early route choices to the final successful arrival at the destination node is difficult to effectively evaluate, resulting in sparse rewards, delayed feedback, and low policy learning efficiency.

[0006] Therefore, how to implement generalizable and real-time executable cross-layer collaborative optimization strategies in complex environments, and solve the problems of action space explosion and credit allocation, are key technical issues that urgently need to be addressed. Summary of the Invention

[0007] The purpose of this application is to provide a service-aware cross-layer optimization method and system for wireless ad hoc networks based on hierarchical reinforcement learning, which can effectively guarantee the end-to-end service quality of multiple service flows.

[0008] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a service-aware cross-layer optimization method for wireless ad hoc networks based on hierarchical reinforcement learning, including: Each service flow to be controlled in a wireless ad hoc network is defined as an agent, and the joint optimization problem of routing and resource allocation of multiple service flows is constructed as a Markov decision process model. The Markov decision process model includes a global state space, a local observation space, and a joint action space. The joint action space includes the next hop, channel, and power level.

[0009] For any agent, a local observation graph is constructed with the agent's current node as the center node, and features are extracted based on the local observation graph to obtain the agent's current state representation.

[0010] The agent's current state is represented by a pre-trained hierarchical reinforcement learning network that makes decisions, selects and outputs joint actions from the joint action space; the hierarchical reinforcement learning network decomposes the joint action value into a sum of routing value component and resource allocation value component through a hierarchical action value function, wherein the routing value component only scores the next-hop selection action, and the resource allocation value component scores the combination action of channel and power level under the condition that the next-hop action has been selected.

[0011] The joint actions selected by each intelligent agent are sent to the corresponding physical nodes for execution, thereby achieving cross-layer optimization of wireless ad hoc network service awareness.

[0012] Secondly, this application provides a service-aware cross-layer optimization system for wireless ad hoc networks based on hierarchical reinforcement learning, including: a model building unit, a local observation graph construction and feature extraction unit, a joint action unit, and an execution unit.

[0013] The model building unit is used to define each service flow to be controlled in the wireless ad hoc network as an agent, and to construct the joint optimization problem of routing and resource allocation of multiple service flows into a Markov decision process model. The Markov decision process model includes a global state space, a local observation space, and a joint action space. The joint action space includes the next hop, channel, and power level.

[0014] The local observation graph construction and feature extraction unit is used to construct a local observation graph for any agent with the agent's current node as the center node, and to extract features based on the local observation graph to obtain the agent's current state representation.

[0015] The joint action unit is used to represent the agent's current state based on a pre-trained hierarchical reinforcement learning network to make decisions, select and output joint actions from the joint action space; the hierarchical reinforcement learning network decomposes the joint action value into a sum of routing value component and resource allocation value component through a hierarchical action value function, wherein the routing value component only scores the next-hop selection action, and the resource allocation value component scores the combination action of channel and power level under the condition that the next-hop action has been selected.

[0016] The execution unit is used to distribute the joint actions selected by each agent to the corresponding physical nodes for execution, thereby realizing cross-layer optimization of wireless ad hoc network service awareness.

[0017] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a service-aware cross-layer optimization method and system for wireless ad hoc networks based on hierarchical reinforcement learning. Each service flow to be controlled in the wireless ad hoc network is defined as an agent, and the joint optimization problem of routing and resource allocation for multiple service flows is constructed as a Markov decision process model. For any agent, a local observation graph is constructed with the agent's current node as the center node, and features are extracted from the local observation graph to obtain the agent's current state representation, transforming the dynamic topology into graph-structured data and improving generalization. The agent's current state representation is then used to make decisions based on a pre-trained hierarchical reinforcement learning network, selecting and outputting joint actions from the joint action space. The learning network decomposes the joint action value into a sum of routing value and resource allocation value components through a hierarchical action value function. The routing value component only scores the next-hop selection action, while the resource allocation value component scores the combination of channel and power level actions given that the next-hop action has been selected. This decouples the complex joint action space (next hop, channel, power level) into two layers: routing selection and channel-power allocation, significantly reducing the complexity of single-step decision-making and effectively solving the problems of action space explosion and credit allocation. The joint actions selected by each agent are distributed to the corresponding physical nodes for execution, realizing cross-layer optimization of wireless ad hoc network service awareness and ensuring end-to-end service quality for multiple service flows. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1This is a flowchart illustrating a service-aware cross-layer optimization method for wireless ad hoc networks based on hierarchical reinforcement learning, provided as an embodiment of this application.

[0020] Figure 2 This is a schematic diagram of the functional modules of a service-aware cross-layer optimization system for wireless ad hoc networks based on hierarchical reinforcement learning, provided as another embodiment of this application.

[0021] Figure 3 A comparison diagram of ablation experiment results for 2-hop neighbor aggregation, hierarchical decision-making, and service value-assisted learning, provided for another embodiment of this application, is shown. Figure 3 (a) in the figure represents the average reward comparison. Figure 3 (b) in the figure represents a comparison of connection probabilities. Figure 3 (c) in the figure represents a comparison of connection bottleneck rates. Figure 3 (d) in the figure represents the average transmit power comparison. Figure 3 (e) in the figure represents a comparison of service demand. Figure 3 (f) in the figure represents the comparison of demand satisfaction rates.

[0022] Figure 4 This is a comparison chart of the robustness of various algorithms under different interference intensities, provided in another embodiment of this application. Figure 4 (a) in the figure represents the average reward comparison. Figure 4 (b) in the figure represents a comparison of connection probabilities. Figure 4 (c) in the figure represents a comparison of connection bottleneck rates. Figure 4 (d) in the figure represents the average transmit power comparison. Figure 4 (e) in the figure represents a comparison of service demand. Figure 4 (f) in the figure represents the comparison of demand satisfaction rates.

[0023] Figure 5 This is a robustness comparison chart of various algorithms under different numbers of channels, provided in another embodiment of this application. Figure 5 (a) in the figure represents the average reward comparison. Figure 5 (b) in the figure represents a comparison of connection probabilities. Figure 5 (c) in the figure represents a comparison of connection bottleneck rates. Figure 5 (d) in the figure represents the average transmit power comparison. Figure 5 (e) in the figure represents a comparison of service demand. Figure 5 (f) in the figure represents the comparison of demand satisfaction rates.

[0024] Figure 6 This is a robustness comparison chart of various algorithms with different numbers of agents provided in another embodiment of this application, wherein... Figure 6 (a) in the figure represents a comparison of average rewards. Figure 6 (b) in the figure represents a comparison of connection probabilities. Figure 6(c) in the figure represents a comparison of connection bottleneck rates. Figure 6 (d) in the figure represents the average transmit power comparison. Figure 6 (e) in the figure represents a comparison of service demand. Figure 6 (f) in the figure represents the comparison of demand satisfaction rates.

[0025] Figure 7 A robustness comparison diagram of various algorithms under different energy budgets is provided for another embodiment of this application, wherein, Figure 7 (a) in the figure represents a comparison of average rewards. Figure 7 (b) in the figure represents a comparison of connection probabilities. Figure 7 (c) in the figure represents a comparison of connection bottleneck rates. Figure 7 (d) in the figure represents the average transmit power comparison. Figure 7 (e) in the figure represents a comparison of service demand. Figure 7 (f) in the figure represents the comparison of demand satisfaction rates. Detailed Implementation

[0026] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0027] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0028] In addition, the relevant solutions also have the following problems: Traditional heuristic methods (such as routing based on hop count, link quality, and remaining energy) rely only on local or short-term states, making it difficult to characterize the long-term impact of the current decision on subsequent interference distribution, link bottlenecks, and node energy consumption. They are prone to getting trapped in local optima and perform poorly in dynamic and multi-service flow scenarios.

[0029] Model-driven optimization methods: Modeling is done through mathematical programming or game theory. However, in dynamic networks, node movement and disturbance changes make it difficult to obtain model parameters in real time, resulting in excessive computational complexity that cannot meet the needs of real-time decision-making.

[0030] Related hierarchical reinforcement learning methods: Although they can reduce the complexity of action space by decomposing tasks, most of them do not fully consider the end-to-end service benefits in multi-hop scenarios, and high-level routing strategies are still easily affected by local metrics and fall into suboptimal solutions.

[0031] Therefore, how to implement a generalizable and real-time executable cross-layer (network layer routing, MAC layer channel access, physical layer power control) collaborative optimization strategy in complex environments that rely solely on local observation information, dynamic topology changes, energy constraints, and interference from multiple service flows, and how to solve the problems of action space explosion and credit allocation, are key technical issues that urgently need to be addressed.

[0032] like Figure 1 As shown, a service-aware cross-layer optimization method for wireless ad hoc networks based on hierarchical reinforcement learning is provided, including the following steps 201 to 204.

[0033] Step 201: Define each service flow to be controlled in the wireless ad hoc network as an agent, and construct the joint optimization problem of routing and resource allocation of multiple service flows as a Markov decision process model. The Markov decision process model includes a global state space, a local observation space, and a joint action space. The joint action space includes the next hop, channel, and power level.

[0034] Step 202: For any agent, construct a local observation graph with the current node of the agent as the center node, and extract features based on the local observation graph to obtain the current state representation of the agent.

[0035] Step 203: The agent's current state is represented by a pre-trained hierarchical reinforcement learning network for decision-making, and a joint action is selected and output from the joint action space. The hierarchical reinforcement learning network decomposes the joint action value into a sum of a routing value component and a resource allocation value component through a hierarchical action value function. The routing value component only scores the next-hop selection action, and the resource allocation value component scores the combination of channel and power level actions under the condition that the next-hop action has been selected.

[0036] In another exemplary embodiment of this application, the hierarchical reinforcement learning network is trained using a centralized training and distributed execution framework. The hierarchical reinforcement learning network includes an agent Q-network and a QMIX hybrid network. The agent Q-network outputs the local action values ​​of each agent based on local observations. The QMIX hybrid network mixes the local action values ​​of each agent during the centralized training phase to obtain a joint action value. During the centralized training phase, the QMIX hybrid network performs centralized value decomposition of the local action values ​​of each agent and introduces a service value-assisted learning mechanism to construct an auxiliary loss function. The auxiliary loss function and the main loss function are weighted and jointly updated to update the network parameters. The auxiliary loss function is only used to update the parameters in the agent Q-network (specifically, the parameters of the routing scoring part in the agent network), and the main loss function is used to update the parameters of both the agent Q-network and the QMIX hybrid network. During the distributed execution phase, each agent independently performs the action selection process.

[0037] In another exemplary embodiment of this application, a service value-assisted learning mechanism is introduced to construct an auxiliary loss function, specifically including: using the cumulative demand satisfaction rate of the discount as the service value target to construct an auxiliary loss function.

[0038] In another exemplary embodiment of this application, the auxiliary loss function is a service value loss, and the expression for the service value loss is: .

[0039] in, For the loss of service value, This represents the sample set of effective routing decisions obtained from experience replay. For time, For business flow, For intelligent agents In the current local observation Next hop node selection The route score, The service value target is to achieve the cumulative demand satisfaction rate for discounts.

[0040] In another exemplary embodiment of this application, the main loss function is a temporal difference loss, and the expression for the temporal difference loss is: .

[0041] in, For time-series difference loss, For experience replay pool, To replay the experience pool Calculate the mathematical expectation of the sampled data. The target value for time-series difference is... The joint action value output by the QMIX hybrid network, This is the global state. For the joint action of all intelligent agents, These are online network parameters.

[0042] In another exemplary embodiment of this application, the hierarchical reinforcement learning network employs... Greedy strategies select joint actions based on probability. Select the action with the highest action value from the available actions, based on probability. Uniformly randomize from all available actions.

[0043] Step 204: The joint action selected by each agent is sent to the corresponding physical node for execution, thereby realizing cross-layer optimization of wireless ad hoc network service awareness.

[0044] By implementing steps 201 to 204 above, this application defines each service flow to be controlled in the wireless ad hoc network as an agent, and constructs the joint optimization problem of routing and resource allocation of multiple service flows as a Markov decision process model. For any agent, a local observation graph is constructed with the current node of the agent as the central node, and feature extraction is performed based on the local observation graph to obtain the current state representation of the agent, transforming the dynamic topology structure into graph structure data and improving generalization. The current state representation of the agent is used to make decisions based on a pre-trained hierarchical reinforcement learning network, selecting and outputting joint actions from the joint action space. The hierarchical reinforcement learning network, through hierarchical... The action value function decomposes the joint action value into the sum of a routing value component and a resource allocation value component. The routing value component only scores the next-hop selection action, while the resource allocation value component scores the combined action of channel and power level under the condition that the next-hop action has been selected. This decouples the complex joint action space (next hop, channel, power level) into two levels: routing selection and channel-power allocation, significantly reducing the complexity of single-step decision-making and effectively solving the problems of action space explosion and credit allocation. The joint actions selected by each agent are distributed to the corresponding physical nodes for execution, realizing cross-layer optimization of wireless ad hoc network service awareness and ensuring end-to-end service quality for multiple service flows.

[0045] In another exemplary embodiment of this application, a local observation graph is constructed with the node where the agent is currently located as the central node, including the following steps 301 to 302.

[0046] Step 301: Using the current node of the agent as the central node, determine the set of locally observable nodes and the set of locally observable links; the set of locally observable nodes includes the current node, the set of candidate next-hop nodes, and the set of two-hop neighbor nodes (i.e., subsequent candidate nodes) of the candidate next-hop nodes; the set of locally observable links includes the set of one-hop edges and the set of two-hop edges.

[0047] Step 302: Construct a local observation graph based on the set of locally observable nodes and the set of local links.

[0048] In another exemplary embodiment of this application, feature extraction based on the local observation map is used to obtain the current state representation of the agent, which is replaced by the following steps 401 to 402.

[0049] Step 401: Extract spatial features from the local observation map using a graph aggregation encoder to obtain a spatial feature embedding vector.

[0050] Step 402: The spatial feature embedding vector is temporally encoded based on a gated recurrent unit to obtain the current state representation of the agent.

[0051] In another exemplary embodiment of this application, the local observation map is subjected to two-stage spatial feature extraction based on a graph aggregation encoder, including the following steps 501 to 503.

[0052] Step 501: Based on the local observation graph, obtain the initial features of the current node, the initial features of each one-hop candidate neighbor node, the initial features of the one-hop edge, the initial features of the two-hop neighbor nodes of each one-hop candidate neighbor node, and the initial features of the two-hop edge.

[0053] Step 502: In the graph aggregation encoder, phase one executes one-hop message passing and two-hop message passing.

[0054] The one-hop message passing is as follows: taking the initial features of the current node, the initial features of each one-hop candidate neighbor node, and the initial features of the one-hop edge as input, a message vector is generated through the edge message function and then aggregated and node updated to obtain a one-hop enhanced representation.

[0055] The two-hop message passing is as follows: taking the initial features of each one-hop candidate neighbor node, the initial features of each one-hop candidate neighbor node's two-hop neighbor node, and the initial features of the two-hop edges as input, a message vector is generated through the edge message function and then aggregated and node updated to obtain the enhanced representation of the candidate node.

[0056] Step 503: In the graph aggregation encoder, stage two executes a second one-hop message passing (where "second" refers to running the one-hop message passing mechanism a second time to realize the back transmission of two-hop information to the current node): The second one-hop message passing is as follows: the enhanced representation of the candidate node obtained in stage one is used to replace the initial feature of the one-hop candidate neighbor node as the updated feature of the candidate node, and the one-hop enhanced representation and the initial feature of the one-hop edge in stage one remain unchanged; the one-hop enhanced representation, the updated feature of the candidate node and the initial feature of the one-hop edge are used as inputs to re-execute the one-hop message passing to obtain the updated current node representation, which is used as the spatial feature embedding vector.

[0057] This application addresses the technical problem in dynamic wireless ad hoc networks where the decision-making coupling between routing, channel allocation, and power control, as well as the difficulty in time-series credit allocation due to multi-hop decision-making, makes it impossible for related methods to guarantee end-to-end service quality for multiple service flows. It provides a service-aware hierarchical reinforcement learning control method. This method constructs an agent that integrates graph neural networks and gated recurrent units to encode spatiotemporal features of local topology and historical states. Combined with hierarchical action value decomposition and end-to-end service value-assisted learning, it achieves joint optimization of routing and resources within a multi-agent centralized training and distributed execution framework. This results in improved network service satisfaction and system robustness in dynamic environments.

[0058] The service-aware hierarchical reinforcement learning method proposed in this application is used to jointly optimize the next-hop routing, transmission channel and transmit power of multiple service flows in dynamic wireless ad hoc networks, and specifically includes the following steps.

[0059] Step S1: Construct a multi-agent partially observable Markov decision process model.

[0060] Each service flow to be controlled in the network is defined as an agent, and the joint optimization problem of routing and resource allocation of the multiple service flows is constructed as a multi-agent partially observable Markov decision process model, expressed as: .

[0061] in, To construct a multi-agent partially observable Markov decision process model, For the number of controlled business flows, For the global state space, For local observation space, For joint action space, Let be the state transition probability. For instant rewards, This is the discount factor.

[0062] Any intelligent agent At its current node Joint actions at the location Defined as: .

[0063] in, Indicates the next hop selection. Indicates the channel number. Indicates the power level.

[0064] The reward function Constructed based on the incremental demand satisfaction (or demand satisfaction rate) of business flows between adjacent decision time slots, defined as: .

[0065] in, For business flow priority weights, For the first Each business flow The degree of demand satisfaction (or demand satisfaction rate) at any given moment.

[0066] Step S2: Construct a local observation map and extract spatiotemporal features.

[0067] For any agent, in its current decision time slot, perform the following sub-steps: S2.1: Based on the node where the agent is currently located Using the central node, obtain the following features and construct a local observation map. .in, It is a set of locally observable nodes. For business flow At any moment The set of its candidate next-hop nodes, Represents business flow The set of currently visited routing nodes. Represents a node The remaining energy, The energy threshold required to perform minimum reliable transmission. Its node characteristics include node location, remaining energy, mobility speed, service demand rate, and priority.

[0068] It is a set of local links and a set of one-hop edges. Indicates the current node to each one-hop candidate neighbor node The link. Its link characteristics include link distance, signal-to-interference-plus-noise ratio (SINR), and channel availability. Specifically: Business Flow Current node eigenvectors Defined as: .

[0069] in, This represents the position coordinates of the current node in the two-dimensional plane. This represents the relative position vector from the current node to the destination node. The normalized representation of the remaining energy of a node. A normalized representation of business flow priority. A normalized representation of business requirements. This represents the normalized representation of the current node's velocity.

[0070] Candidate Neighbor Nodes Features: For candidate next-hop nodes Its eigenvectors Includes the relative position with respect to the current node, normalized remaining energy, and relative velocity: .

[0071] 1-hop edge characteristics: A 1-hop edge represents the current node. With candidate next hop nodes The forwardable relationship between them, and its edge characteristics Defined as: .

[0072] in, From the current node to the candidate node In the channel SINR, this value is adopted Compression is performed to avoid excessive differences in numerical scale; Let be the channel availability vector, if the th If a channel is available for the link, the corresponding element is 1; otherwise, it is 0. This feature characterizes the immediate feasibility and link quality of the current node forwarding to a candidate neighbor.

[0073] 2-hop neighbor edge features: To avoid policy shortsightedness, this application further constructs 2-hop edges to describe the subsequent reachability after the candidate next hop. For candidate next hop nodes... Let its subsequent candidate nodes (i.e., two-hop neighbor nodes) be... 2-hop edge features Defined as: .

[0074] S2.2: Business flow At any moment Local observation map Input a graph aggregation encoder to extract hidden node and edge representations of the fused topological context. Let... For nodes initial characteristics, Representing edge features. Specifically, this includes the following two stages: Phase 1: One-hop message passing and two-hop message passing.

[0075] One-hop message passing: based on the initial characteristics of the current node Initial features of each one-hop candidate neighbor node and the initial features of a jump edge As input, the slave node is generated through the edge message function. To the node Message vectors passed : .

[0076] Among them, the edge message function This is a multilayer perceptron (MLP) containing activation functions. Agent node. Aggregate messages from all its 1-hop neighbors: .

[0077] The 1-hop augmented representation (i.e., one-hop augmented representation) is obtained through the node update function. : .

[0078] in, This is the node message function.

[0079] Two-hop message passing: For each one-hop candidate neighbor node With its initial characteristics The second-hop candidate nodes (i.e., the second-hop neighbor nodes) initial features and initial features of two-jump edges As input, perform the same edge message aggregation and node update operations as the one-hop message passing to obtain an enhanced candidate node representation that incorporates subsequent forwarding potential information. : .

[0080] in, These are the node message function and edge message function used to aggregate 2-hop features, respectively.

[0081] Phase Two: Secondary One-Hop Message Passing (Message Backhaul).

[0082] Enhance the representation of each candidate node obtained in stage one Replace the initial features of the original one-hop candidate neighbor nodes The updated features of candidate nodes are used; the central node retains the one-hop augmented representation obtained in phase one. Initial features of a jump edge Remain unchanged. As stated... , and For input, re-execute one-hop message passing: .

[0083] .

[0084] .

[0085] Ultimately, business flow Spatial embedding It is taken directly from the updated representation of the current node, that is This embedding integrates spatial features such as the node where the service flow is located, candidate neighbors, 2-hop reachability, and channel state.

[0086] S2.3: Embed the spatial features into the vector The input is fed into a gated recurrent unit and combined with the temporal hidden state of the previous decision slot. Update: .

[0087] Received As a business flow In the time slot The complete state representation includes not only the current spatial structure but also the memory of path establishment process, energy consumption, and service progress, enabling the strategy to maintain continuity and robustness in partially observable dynamic environments.

[0088] Step S3: Decouple decision-making and select actions based on the hierarchical action value function.

[0089] The hidden state output by the GRU in step S2.3 As an intelligent agent During decision-making time slots The complete state representation is input into a hierarchical reinforcement learning network. This network constructs a hierarchical action value function, decomposing the value of joint actions (i.e., joint action value) into the sum of routing value and resource allocation value components: .

[0090] in, For intelligent agents In the current local observation Next joint action value, Select only the next hop To score, This means that the next hop has already been selected. Under these conditions, different channel-power combinations are scored. This decomposition explicitly follows the causal order of "first determining the forwarding direction, then fine-tuning the link resources".

[0091] For including One candidate neighbor One channel and For scenarios with different power levels, the joint action space size is: .

[0092] Action selection adopts Greedy strategy, based on probability Select available actions The action with the largest value is selected; otherwise, random sampling is performed uniformly from all available actions to ensure that every legal action has a chance to be tried, thus preventing the strategy from getting trapped in local optima too early.

[0093] .

[0094] Among them, action mask can be used. It is dynamically generated by the environment based on constraints such as communication distance, channel idle state, and remaining energy, ensuring that all selected actions are physically feasible.

[0095] Step S4: Centralized training and distributed execution of multiple agents.

[0096] During the training phase, this application introduces the QMIX hybrid network to perform centralized value decomposition of the local action values ​​of each agent. Specifically, the... An agent based on local observations Output individual action value Subsequently, the QMIX hybrid network uses the local action values ​​of all agents. and global state Given input, output the value of the combined action: .

[0097] in, For the number of agents, The network parameters are mixed. During the execution phase, each agent independently selects the next hop, channel, and transmit power action based solely on its local observations (including current node state, candidate neighbor features, link state, remaining energy, and local interference information) through a shared agent Q-network. The execution process does not depend on the global network state, satisfying the requirements of distributed routing decision-making. QMIX generates non-negative network weights through a state-dependent supernetwork, thereby ensuring that the joint action value satisfies the monotonicity constraint on the action values ​​of each agent. .

[0098] This constraint ensures that maximizing the value of joint actions aligns with maximizing the value of individual agents' local actions, enabling policies trained centrally to be implemented independently during the distributed execution phase. Specifically, when any business flow... Local action value When increased, global value It will not decrease.

[0099] The objective of Temporal Difference (TD) is defined as follows: .

[0100] in, Indicates the end of the process. Let QMIX be the target network. The corresponding TD loss (instantaneous differential loss) is: .

[0101] in, An online parameter set including agent Q-networks and QMIX hybrid networks. This is an experience replay pool. Indicates online network parameters, This represents the learning rate. The target network is used only to construct a stable TD target and synchronizes parameters from the online network through periodic soft updates. : .

[0102] in, This is the soft update coefficient.

[0103] Step S5: Introduce a service value-assisted learning mechanism to optimize credit allocation.

[0104] During training, in addition to state, action, reward, and termination flag, the experience replay cache also records the demand satisfaction rate signal for each business flow at the corresponding decision step. For business flow Let its effective decision steps set be... This indicates the last valid decision step of the service flow in the current sampling sequence or round. The service value target is defined as the cumulative demand satisfaction rate with discounts. .

[0105] This objective reflects the time frame... The cumulative service revenue from the start to the end of the flow. For the actually selected next hop. Service value loss is defined as: .

[0106] in, This represents the sample set of effective routing decisions obtained from experience replay. This auxiliary learning mechanism encourages the model to attribute the final high demand satisfaction rate to truly effective relay selection in the path, thereby improving the interpretability of routing scores and significantly accelerating the convergence of routing strategies.

[0107] The total loss function is formed by weighted summation of the service value auxiliary loss function (i.e., service value loss) and the time-series difference loss function (i.e., time-series difference loss): .

[0108] in, Weighting of service value to assist loss. Used to update agent Q networks and QMIX hybrid networks; gradient updates only apply to online agent networks and online QMIX hybrid networks: .

[0109] in, Indicates online network parameters, This represents the learning rate.

[0110] The key features of this application are: 1. A hierarchical decision-making architecture: The complex joint action space (next hop, channel, power) is decoupled into two layers: routing selection and channel-power allocation, significantly reducing the complexity of single-step decision-making. 2. A perception mechanism combining graph neural networks and recurrent neural networks: Graph neural networks are used to extract spatial features from dynamic local topology, and gated recurrent units are combined to encode temporal information, enabling the agent to perceive key information such as topology, link status, and node energy within a two-hop range. 3. A service value-assisted learning mechanism: To address the sparse reward and delayed feedback problems in multi-hop routing, a service value-assisted learning mechanism is designed. By predicting the cumulative end-to-end service demand satisfaction rate, it provides dense auxiliary training signals directly related to the final service revenue for intermediate step decisions, thereby guiding the routing strategy to focus on global service quality rather than local indicators.

[0111] Furthermore, based on the core inventive ideas of this application (hierarchical decision-making, graph-time sequence perception, and service value-assisted learning), the following alternatives or variations exist: 1. Alternatives to graph neural network architectures: This application uses a message-passing-based GNN and 2-hop aggregation. An alternative approach would be to use a graph attention network to dynamically compute the importance weights of neighboring nodes, rather than a simple averaging.

[0112] Graph isomorphic networks can be used as encoders, which theoretically have a stronger ability to distinguish different graph structures.

[0113] 2. Replacement of timing encoders: This application uses GRU for temporal encoding. An alternative approach is to use a Long Short-Term Memory (LSTM) network, which, although having more parameters, may perform better on certain tasks requiring longer memory.

[0114] The self-attention mechanism in Transformer can also be used to process historical state sequences and capture long-distance dependencies.

[0115] 3. Alternatives to the multi-agent value decomposition method: This application employs QMIX for centralized training. Alternatives could use VDN (a simpler linear summation) or QTRAN (suitable for more complex non-monotonic scenarios). QMIX was chosen because it can handle a wider range of tasks and guarantees monotonicity, but other methods are also feasible under specific constraints.

[0116] 4. Service value-assisted learning objective calculation alternatives: This application uses the cumulative demand satisfaction rate of the discount as the regression objective. Alternatives could be to directly use Monte Carlo returns or to use a dominance function to assess the relative merit of the selected action.

[0117] The way auxiliary loss is added can also vary; for example, it can be added directly to the loss function of the policy network as a regularization term.

[0118] 5. Substitution of the implementing entity: This application treats each "service flow" as an intelligent agent. An alternative approach could treat each "network node" as an intelligent agent, with the node making decisions for all flows it forwards. The former is more conducive to end-to-end performance optimization, while the latter is more suitable for a fully distributed network architecture. Both approaches have their advantages and disadvantages and are interchangeable.

[0119] 6. Alternatives to reward function design: This application uses the demand satisfaction rate increment as the immediate reward. An alternative approach could be to design a multi-objective reward function that includes multiple components such as connectivity reward, rate reward, and energy penalty, and assign them different weights.

[0120] Based on the same inventive concept, this application also provides a system for implementing the hierarchical reinforcement learning-based service-aware cross-layer optimization method for wireless ad hoc networks, as described above. The solution provided by this system is similar to the implementation described in the above method. Therefore, the specific limitations of one or more embodiments of the hierarchical reinforcement learning-based cross-layer optimization system for wireless ad hoc networks provided below can be found in the limitations of the hierarchical reinforcement learning-based cross-layer optimization method for wireless ad hoc networks described above, and will not be repeated here.

[0121] like Figure 2 As shown, a service-aware cross-layer optimization system for wireless ad hoc networks based on hierarchical reinforcement learning is provided, including: a model building unit, a local observation graph construction and feature extraction unit, a joint action unit, and an execution unit.

[0122] The model building unit is used to define each service flow to be controlled in the wireless ad hoc network as an agent, and to construct the joint optimization problem of routing and resource allocation of multiple service flows into a Markov decision process model. The Markov decision process model includes a global state space, a local observation space, and a joint action space. The joint action space includes the next hop, channel, and power level.

[0123] The local observation graph construction and feature extraction unit is used to construct a local observation graph for any agent with the agent's current node as the center node, and to extract features based on the local observation graph to obtain the agent's current state representation.

[0124] The joint action unit is used to represent the agent's current state based on a pre-trained hierarchical reinforcement learning network to make decisions, select and output joint actions from the joint action space; the hierarchical reinforcement learning network decomposes the value of the joint action (i.e., the joint action value) into the sum of a routing value component and a resource allocation value component through a hierarchical action value function, wherein the routing value component only scores the next-hop selection action, and the resource allocation value component scores the combination action of channel and power level under the condition that the next-hop action has been selected.

[0125] The execution unit is used to distribute the joint actions selected by each agent to the corresponding physical nodes for execution, thereby realizing cross-layer optimization of wireless ad hoc network service awareness.

[0126] Furthermore, this application also provides a routing and resource joint optimization system based on service-aware hierarchical reinforcement learning for executing the above-described method. The system includes: multiple wireless nodes and controlled service flow agents. Each wireless node has a communication module, memory, and processor, used to form a self-organizing network. Each controlled service flow corresponds to an agent, which is logically deployed on a node or a centralized server (training phase), or runs distributed across various nodes (execution phase).

[0127] The intelligent agent includes: a perception module, a graph neural network encoder, a gated recurrent unit, a hierarchical action value network (or a hierarchical reinforcement learning network), and a service value-assisted learning module (training phase only).

[0128] Sensing module: Used to acquire and process local observation information and construct a local observation map.

[0129] Graph Neural Network Encoder: Used to extract spatial features from the local observation map.

[0130] Gated loop unit: used to encode timing information and generate a comprehensive state representation.

[0131] Hierarchical Action Value Network (or Hierarchical Reinforcement Learning Network): Used to output routing scores and channel-power allocation scores, and select joint actions based on the hierarchical action value function.

[0132] Service Value-Assisted Learning Module (Training Phase Only): Used to calculate the cumulative demand satisfaction rate target and update the routing score network.

[0133] The technical solution (SA_HRL) proposed in this application has the following significant technical advantages compared with related technologies (such as the c2Dst_Full heuristic algorithm, standard QMIX, MAPPO multi-agent reinforcement learning algorithm, and the HRL algorithm for serverless assisted learning): 1. Improved business satisfaction and service quality: Performance data: Simulation results show that SA_HRL improves the business requirement fulfillment rate by approximately 28.7% compared to HRL and by approximately 99.6% compared to QMIX (estimated based on Table 1). Its average return is 30.36% higher than HRL and 53.17% higher than c2Dst_Full (see Table 1).

[0134] Table 1. Comparison of quantitative performance of each algorithm in the last 100 rounds.

[0135] Reason: The effectiveness of this technology stems from the service value-assisted learning mechanism and the reward function design centered on demand satisfaction rate. This allows the model to not only pursue connectivity but also optimize end-to-end bottleneck links, ensuring that business flows receive effective service rates and prioritizing network resources for high-priority and high-return services.

[0136] 2. Improved connection reliability and bottleneck link speed: Performance data: SA_HRL achieved the highest connection probability (0.87) and connection bottleneck rate (0.561Mbps), which are significant improvements over HRL (0.808, 0.442Mbps) (see Table 1).

[0137] Reason: The effectiveness of this technology stems from the GNN-GRU encoder's ability to perceive 2-hop neighbor information. By proactively evaluating the subsequent forwarding potential of candidate next hops, the agent can effectively avoid "trap" nodes that may cause routing interruptions or bottlenecks, thereby building a more stable and higher-throughput end-to-end path.

[0138] 3. Improved training efficiency and convergence performance: Performance data: Ablation experiments demonstrate that after incorporating the service value-assisted learning mechanism, the model's early training rewards improve faster, resulting in a higher final convergence value (see [link]). Figure 3 (and Table 1).

[0139] Reason: The effectiveness of this technique stems from the synergistic effect of hierarchical action value functions and service value-assisted learning. The hierarchical structure transforms the joint action space of multiplicative complexity ( Reduced to additive complexity ( This greatly reduces the difficulty of exploration. Meanwhile, service value-assisted learning provides dense, end-reward-related auxiliary signals for high-level routing decisions, effectively alleviating the credit allocation problem in multi-hop scenarios and accelerating the convergence of the policy network.

[0140] 4. Enhanced system robustness: Performance data: Under tests with different interference intensities (number of background flows), different number of channels, different traffic volumes, and different node energy budgets, SA_HRL exhibited the smallest performance degradation, consistently maintaining optimal or near-optimal performance (see [link to relevant documentation]). Figures 4-7 ).

[0141] Reason: The effectiveness of this technique stems from the model's comprehensive encoding of local graph observations (including information such as energy, interference, and topology) and the service value-driven optimization objective. The model learns to dynamically adjust routing and resource allocation strategies based on current resource (energy, channel) and competition (background flow) conditions (e.g., SA_HRL uses differentiated channels and medium power at relay nodes to avoid collisions), rather than using fixed rules (such as maximum power), thus exhibiting stronger adaptability to environmental and resource changes.

[0142] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0143] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A service-aware cross-layer optimization method for wireless ad hoc networks based on hierarchical reinforcement learning, characterized in that, include: Each service flow to be controlled in a wireless ad hoc network is defined as an agent, and the joint optimization problem of routing and resource allocation of multiple service flows is constructed as a Markov decision process model. The Markov decision process model includes a global state space, a local observation space, and a joint action space. The joint action space includes the next hop, channel, and power level. For any agent, a local observation graph is constructed with the agent's current node as the center node, and features are extracted based on the local observation graph to obtain the agent's current state representation; The agent's current state is represented by a pre-trained hierarchical reinforcement learning network for decision-making, selecting and outputting joint actions from the joint action space; the hierarchical reinforcement learning network decomposes the joint action value into a sum of routing value component and resource allocation value component through a hierarchical action value function, wherein the routing value component only scores the next-hop selection action, and the resource allocation value component scores the combination action of channel and power level under the condition that the next-hop action has been selected. The joint actions selected by each intelligent agent are sent to the corresponding physical nodes for execution, thereby achieving cross-layer optimization of wireless ad hoc network service awareness.

2. The service-aware cross-layer optimization method for wireless ad hoc networks based on hierarchical reinforcement learning according to claim 1, characterized in that, A local observation graph is constructed with the current node of the agent as the center node, specifically including: Taking the current node of the agent as the central node, determine the set of locally observable nodes and the set of locally observable links; the set of locally observable nodes includes the current node, the set of candidate next-hop nodes, and the set of two-hop neighbor nodes of the candidate next-hop nodes; the set of locally observable links includes the set of one-hop edges and the set of two-hop edges. A local observation graph is constructed based on the set of locally observable nodes and the set of local links.

3. The service-aware cross-layer optimization method for wireless ad hoc networks based on hierarchical reinforcement learning according to claim 1, characterized in that, Feature extraction is performed based on the local observation map to obtain the agent's current state representation, specifically including: Spatial features are extracted from the local observation map using a graph aggregation encoder to obtain spatial feature embedding vectors; The spatial feature embedding vector is temporally encoded based on a gated recurrent unit to obtain the agent's current state representation.

4. The service-aware cross-layer optimization method for wireless ad hoc networks based on hierarchical reinforcement learning according to claim 3, characterized in that, The local observation map is used for two-stage spatial feature extraction based on a graph aggregation encoder, specifically including: Based on the local observation graph, we obtain the initial features of the current node, the initial features of each one-hop candidate neighbor node, the initial features of the one-hop edge, the initial features of the two-hop neighbor node of each one-hop candidate neighbor node, and the initial features of the two-hop edge. In the graph aggregation encoder, phase one performs one-hop message passing and two-hop message passing; The one-hop message passing is as follows: taking the initial features of the current node, the initial features of each one-hop candidate neighbor node, and the initial features of the one-hop edge as input, a message vector is generated through the edge message function and then aggregated and node updated to obtain a one-hop enhanced representation. The two-hop message passing is as follows: taking the initial features of each one-hop candidate neighbor node, the initial features of each one-hop candidate neighbor node's two-hop neighbor node, and the initial features of the two-hop edges as input, a message vector is generated through the edge message function and then aggregated and node updated to obtain the enhanced representation of the candidate node. In the graph aggregation encoder, phase two performs a second one-hop message passing: The second one-hop message passing is as follows: the enhanced representation of the candidate node obtained in stage one is used to replace the initial feature of the one-hop candidate neighbor node as the updated feature of the candidate node, and the one-hop enhanced representation and the initial feature of the one-hop edge in stage one remain unchanged; the one-hop enhanced representation, the updated feature of the candidate node and the initial feature of the one-hop edge are used as inputs to re-execute the one-hop message passing to obtain the updated current node representation, which is used as the spatial feature embedding vector.

5. The service-aware cross-layer optimization method for wireless ad hoc networks based on hierarchical reinforcement learning according to claim 1, characterized in that, The hierarchical reinforcement learning network is trained using a centralized training and distributed execution framework. The hierarchical reinforcement learning network includes an agent Q network and a QMIX hybrid network. The agent Q network is used to output the local action value of each agent based on local observations. The QMIX hybrid network is used to mix the local action values ​​of each agent during the centralized training phase to obtain a joint action value. During the intensive training phase, a QMIX hybrid network is used to perform centralized value decomposition of the local action values ​​of each agent, and a service value-assisted learning mechanism is introduced to construct an auxiliary loss function. The auxiliary loss function and the main loss function are weighted and jointly updated to update the network parameters. The auxiliary loss function is only used to update the parameters in the agent's Q network, and the main loss function is used to update the parameters of the agent's Q network and the QMIX hybrid network. During the distributed execution phase, each agent independently performs the action selection process.

6. The service-aware cross-layer optimization method for wireless ad hoc networks based on hierarchical reinforcement learning according to claim 5, characterized in that, A service value-assisted learning mechanism is introduced to construct an auxiliary loss function, which specifically includes: using the cumulative demand satisfaction rate of discounts as the service value objective to construct an auxiliary loss function.

7. The service-aware cross-layer optimization method for wireless ad hoc networks based on hierarchical reinforcement learning according to claim 5 or 6, characterized in that, The auxiliary loss function is the service value loss, and the expression for the service value loss is: ； in, For the loss of service value, This represents the sample set of effective routing decisions obtained from experience replay. For time, For business flow, For intelligent agents In the current local observation Next hop node selection The route score, The service value target is to achieve the cumulative demand satisfaction rate for discounts.

8. The service-aware cross-layer optimization method for wireless ad hoc networks based on hierarchical reinforcement learning according to claim 5, characterized in that, The main loss function is the temporal difference loss, and the expression for the temporal difference loss is: ； in, For time-series difference loss, For experience replay pool, To replay the experience pool Calculate the mathematical expectation of the sampled data. The target value for time-series difference is... The joint action value output by the QMIX hybrid network, This is the global state. For the joint action of all intelligent agents, These are online network parameters.

9. The service-aware cross-layer optimization method for wireless ad hoc networks based on hierarchical reinforcement learning according to claim 1, characterized in that, The hierarchical reinforcement learning network adopts Greedy strategies select joint actions based on probability. Select the action with the highest action value from the available actions, based on probability. Uniformly randomize from all available actions.

10. A service-aware cross-layer optimization system for wireless ad hoc networks based on hierarchical reinforcement learning, characterized in that, include: The model building unit is used to define each service flow to be controlled in the wireless ad hoc network as an agent, and to construct the joint optimization problem of routing and resource allocation of multiple service flows into a Markov decision process model. The Markov decision process model includes a global state space, a local observation space, and a joint action space. The joint action space includes the next hop, channel, and power level. The local observation graph construction and feature extraction unit is used to construct a local observation graph for any agent with the agent's current node as the center node, and to extract features based on the local observation graph to obtain the agent's current state representation; The joint action unit is used to make decisions based on a pre-trained hierarchical reinforcement learning network to represent the current state of the agent, select and output joint actions from the joint action space; the hierarchical reinforcement learning network decomposes the joint action value into the sum of routing value component and resource allocation value component through a hierarchical action value function, wherein the routing value component only scores the next hop selection action, and the resource allocation value component scores the combination action of channel and power level under the condition that the next hop action has been selected; The execution unit is used to distribute the joint actions selected by each agent to the corresponding physical nodes for execution, thereby realizing cross-layer optimization of wireless ad hoc network service awareness.

Images