Distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization

Abstract

The invention discloses a distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization, and the method comprises the steps: abstracting a to-be-established end-to-end data stream into a partially observable Markov hop-by-hop decision process when a data stream communication request from a source node to a destination node is received; wherein in each hop, the current node only carries out joint decision-making of routing and transmitting power based on local observation information. In the decision of each hop, irregular topology and interference relation between adjacent nodes are flexibly described by using a graph; designing a GNN-based model architecture as a strategy of an agent node, and processing input / output fluctuation dimensions by using a graph relationship; in the optimization process, the information exchange between the nodes strictly follows the actual limitation of the sensing ability, and accords with the distributed configuration of the data link self-organizing network; the learned strategy can quickly establish a high-quality routing path between a source node and a target node, and shows strong generalization ability for random network topology.

Description

Technical Field

[0001] This invention relates to the field of data flow ad hoc network communication technology, and in particular to a distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization. Background Technology

[0002] Ad hoc datalink networks can adapt to flexible topologies and spontaneously establish communication in emergency situations where cellular infrastructure is unavailable. To establish high-quality routing paths between specified source and destination nodes, a series of optimizations, including optimal routing and radio resource allocation, must be considered. To limit the complexity of system design, the traditional OSI model divides these optimization problems into multiple layers and expects adjustments to parameters from different layers to be made independently. However, due to the openness of the radio medium and the flexibility of the datalink ad hoc network structure, optimal routing at the network layer and radio resource allocation at the physical link layer are interdependent: on the one hand, optimal routing needs to consider the quality of the physical link, which depends on transmit power and spectrum allocation. On the other hand, radio resource allocation must manage interference relationships between nodes related to the optimal routing outcome.

[0003] While cross-layer design in wireless communication has been a classic research area, insufficient attention has been paid to the interaction between the physical link layer and the network layer. Based on classical optimization theory, multivariate cross-layer problems are generally decomposed into multiple subproblems and solved by alternately updating partial variables. This approach faces daunting complexity and slow convergence. In fact, this problem is very common in various scenarios when resource allocation is combined with the optimization of other network parameters. Furthermore, most of these works neglect interference problems, which are prevalent in datalink ad hoc networks and often require non-convexity for optimization. Effective solutions remain lacking for routing and wireless resource allocation in interference channels within datalink ad hoc networks.

[0004] In recent years, deep reinforcement learning (DRL) has demonstrated its capabilities in resource allocation and routing in data link ad hoc networks. Agents learn using approximation strategies from deep neural networks (DNNs) to solve complex optimization problems without requiring further specialized knowledge. With proper training, agents can exhibit superior optimization performance and lower execution latency compared to traditional iterative algorithms. However, the accuracy of agents largely depends on the specific design of the neural network architecture and training process. Neural network architectures in standard DRL algorithms often face scalability issues when applied to network optimization. Graph neural networks (GNNs) have been used in various works to handle diverse relationships in communication networks of different sizes. At the network layer, GNNs can effectively extract features of the network topology represented as a graph and use them to assist routing decisions. Simultaneously, GNNs are used in DRL-based routing algorithms to handle parameterized strategies, and wireless interference relationships at the physical link layer can also be flexibly represented using graphs. Due to the ability of GNNs to handle relationships between the network and physical link layers in communication networks, incorporating GNNs into DRL could effectively solve cross-layer problems in distributed data link ad hoc networks. Summary of the Invention

[0005] The present invention aims to at least partially solve one of the technical problems existing in the related art.

[0006] The purpose of this invention is to provide a distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization. It incorporates GNN into DRL to solve the joint problem of optimal route selection and resource allocation. Through a series of hop-by-hop strategies of local decision-making, it is implemented in a distributed manner and extended to different topologies.

[0007] To achieve the above objectives, this invention provides a distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization, wherein when a route is received from the source node... to the destination node When requesting data stream communication, the end-to-end data stream to be established is abstracted into a partially observable Markov hop-by-hop decision process, where each hop is decided solely by the current node based on local observation information, involving a joint decision on routing and transmit power; including the following steps:

[0008] Set the source node as the initial proxy node. The local observation information is represented by a graph structure;

[0009] In the Each decision step, the current agent node Collect local network information within its sensing range to form graph-structured observations. Current proxy node This observation value The input is fed into a pre-trained, parameterized deep reinforcement learning policy based on a GNN model architecture. This policy makes single-hop optimization decisions and outputs a joint action. This action simultaneously determines the next-hop relay node. and by the current proxy node To the next hop relay node Transmission power According to joint actions Establish a new physical link and connect the current proxy node. Passed to the selected next-hop relay node ;

[0010] Each hop is repeated iteratively to observe the Markov hop-by-hop decision process until the current agent node iterates to the destination node. The problem may occur if: the current proxy node has no valid neighbors to choose from; or the path hop count has reached the maximum limit. Complete the data stream communication request.

[0011] A further preferred embodiment of the present invention is that, in the decision-making process of each decision step, the optimization objective is to maximize the bottleneck throughput of the data flow. The bottleneck throughput It is determined by the bottleneck link with the lowest signal-to-interference-plus-noise ratio (SINR) on the routing path;

[0012] The constraints are:

[0013] The selection of all next-hop relay nodes is within the local awareness range of the current proxy node;

[0014] Avoid loops in the routing path;

[0015] Except for the connection points of continuous links, a single node is not shared by the routing paths of multiple data streams;

[0016] The transmit power of each node is limited to a preset maximum value. Inside.

[0017] Preferably, the method of representing the local observation information with a graph structure specifically includes:

[0018] The vertices of the graph correspond to the current proxy node and its one-hop and two-hop neighbor nodes; vertex attributes include the relative position of the node to the destination node and its cumulative interference level;

[0019] The edges of a graph represent the information exchange relationship between the nodes corresponding to the vertices; the attributes of the edges include the channel gain and the relative position between the two nodes connected by the edge.

[0020] As a preferred approach, the partially observable Markov jump-by-jump decision process is formally defined as:

[0021] Define the observations of the graph structure Including the current proxy node All neighboring nodes within range Relative to the current proxy node position Position relative to the destination node Channel gain to the current proxy node And at the node Cumulative interference measured at [location] ;

[0022] Define the joint action of the output It is from a set of optional neighbor nodes and discrete power level set Choose from the Cartesian product action space formed by the action;

[0023] Define rewards It exhibits sparsity, awarding a positive reward upon successful establishment of a routing path. This reward is a function of the logarithmic SINR value on the bottleneck link of the path, while the reward is 0 for all intermediate steps.

[0024] Preferably, the GNN model architecture includes:

[0025] A graph-based encoder for processing observations represented in graph form. Perform information aggregation and feature extraction;

[0026] The recursive module is used to maintain and pass the hidden state across different decision steps. This is to correlate the local decisions of each hop with the final overall path performance;

[0027] The graph-based decision module is used to process the action space with dynamically changing dimensions and output decisions.

[0028] Preferably, the graph-based encoder employs non-uniform graph convolution operations, using independent update functions for one-hop and two-hop relationships in the observation graph. , and , Perform phased message delivery and aggregation;

[0029] The recursive module is a gated recursive unit (GRU).

[0030] The graph-based decision-making module makes decisions in the following way:

[0031] a. The current agent node and all its optional neighbor nodes. Construct a temporary decision graph between them;

[0032] b. Define an edge update function This function calculates a score vector for each edge in the decision graph, generating a score vector for each possible neighbor node.

[0033] c. The dimension of the score vector is equal to the number of discrete power levels. Each element in the vector corresponds to the estimate when selecting that neighboring node and using the corresponding transmit power. value;

[0034] d. Decompose multivariate action selection to accommodate the number of available neighbor nodes. The dynamic changes.

[0035] Preferably, the training of the deep reinforcement learning strategy is performed offline on a randomly generated network topology, and the training process employs a deep recursive Q-learning algorithm, specifically as follows:

[0036] a. Through the agent - A greedy strategy interacts with the environment to generate a transformation sequence, and then applies a fixed-length sequence. Continuous transformation sequence data blocks Save to Experience Replay middle;

[0037] b. Randomly select a small batch of samples from the experience replay unit and calculate the time difference target according to the Bellman equation. ;

[0038] c. By minimizing The parameters of the policy network are updated using the mean squared error loss function between the network estimate and the TD objective. ;

[0039] d. Use a separate target Network computing TD target Values, and parameters of the target network. Perform soft synchronization updates to enhance training stability.

[0040] In another aspect, the present invention provides a non-transitory computer-readable storage medium having computer instructions stored thereon, which cause a computer to execute the above-described distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization.

[0041] In another aspect, the present invention provides an electronic device, comprising: a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus, and the processor calls logical instructions in the memory to execute the above-described distributed data link self-organizing network routing construction method based on DRL and cross-layer optimization.

[0042] In another aspect, the present invention provides a computer program product, the computer program product comprising a computer program stored on a non-transitory computer-readable storage medium, wherein when the computer program is executed by a processor, the computer executes the above-described distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization.

[0043] Beneficial effects: The distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization of this invention uses a graph to flexibly describe the irregular topology and interference relationships between neighboring nodes in each hop decision. In addition, this invention designs a model architecture based on graph neural network (GNN) for decision-making to utilize these graph relationships to handle the fluctuation dimension of input / output. During the optimization process, the information exchange between nodes strictly follows the actual limitations of their perception capabilities, which is consistent with the distributed configuration of data link ad hoc networks. The learned strategy can quickly establish a high-quality routing path between the source node and the target node, demonstrating a strong generalization ability to random network topologies. Attached Figure Description

[0044] Figure 1 This is the distributed data link self-organizing network multi-hop transmission system model of the present invention.

[0045] Figure 2 This is a flowchart of the jump-by-jump layer optimization process of the present invention.

[0046] Figure 3 This is a 2-hop observation graph when node 1 is considered the current proxy node.

[0047] Figure 4 This is a flowchart of the decision-making process for a single hop in this invention.

[0048] Figure 5 A computer program for training deep reinforcement learning strategies using the deep recursive Q-learning algorithm. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, embodiments of this invention, and should not be construed as limiting the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention. In the description of this invention, it should be understood that the terminology used is for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0050] Example 1: Research has found that traditional DRL strategies based on multi-layer perceptron (MLP) parameterization are difficult to adapt to the variability of distributed data link ad hoc network topologies and the limitations of information exchange. Therefore, a solution incorporating a Generative Neural Network (GNN) into DRL is designed to circumvent these practical problems. The learned strategy can establish high-quality routing paths in a distributed and efficient manner.

[0051] Based on this, this embodiment mainly achieves the following objectives:

[0052] (1) Considering the necessity of joint optimization of network layer routing and physical link layer resource allocation in data link ad hoc networks, this embodiment proposes a hop-by-hop strategy through a series of local decisions, which is implemented in a distributed manner and extended to different topologies.

[0053] (2) Considering the limited perception capabilities and uncertain geographical distribution of nodes in real-world data link ad hoc networks, a graph is used to represent the irregular topological and interference relationships between nodes. The graph structure allows for flexible adjustment of the data flow between nodes during the decision-making process.

[0054] (3) The DRL was used to solve the joint problem of optimal routing and resource allocation, and a novel model structure was designed as the learning strategy for the agent nodes. The model effectively handles the irregular relationships between local nodes using graphs and uses them as input to the agent, while enabling the agent to make decisions based on the time-varying size-related set of neighbor nodes. In addition, a recursive module is used to maintain the storage of hops, which consistently improves the agent's performance.

[0055] The following is combined with Figures 1-5 This invention describes a distributed data link self-organizing network routing construction method based on DRL and cross-layer optimization.

[0056] First, establish a distributed data link self-organizing network multi-hop transmission system and identify optimization issues.

[0057] The distributed data link self-organizing network multi-hop transmission system is a multi-hop transmission model, in one Region, a group Nodes with the same attributes are randomly distributed. The bandwidth is... The spectrum is shared by all nodes, and each node is equipped with a single omnidirectional antenna. Each node is composed of This indicates that the indices of all nodes are in the set. In the middle. Assuming each node It can obtain its own geographical location During message transmission in a data link ad hoc network, from arrive The channel gain of the link is It depends on and The distance between them. When dealing with data streams When making a communication service request, it is expected that a slave node will be created. to its designated destination The path, where Define the set of all data streams in a distributed data link ad hoc network as... When multiple data streams exist simultaneously, that is... The concurrent transmission of these data streams will cause mutual interference. In the data stream During the optimization process, It is considered as the surrounding data stream.

[0058] like Figure 1 As shown, a route needs to be established from the source node to the destination node. (Node) Based on its neighbors ( , and ) can make decisions based on available local information, while there may be other node links (such as arrive This can cause interference. The cross-layer design needs to consider the following design factors: 1) Network layer: finding relay nodes to form a network from... arrive 1) Routing; 2) Physical link layer: Allocate radio resources for the physical link between each pair of consecutive nodes along the routing path.

[0059] vector Represents data stream The complete path, where Indicates the first The node at the jump point This represents the total number of hops. For connected routing paths, , ,and Used as a relay node. Then, create... Multi-hop links, where the rate depends on physical link layer parameters. Resource allocation is achieved by adjusting transmit power, further improving the performance of multi-hop paths. Node definition. The transmission power is From node To the node The signal-to-interference-plus-noise ratio (SINR) of the link is

[0060] ;

[0061] in It is a node The power of the additive white Gaussian noise (AWGN) at that location. Besides the data stream... In addition to cross-link interference within the network, formula (1) also considers when... At that time, potential interference from surrounding data streams. Node To the node The capacity of the link is calculated as .

[0062] The maximum reachable throughput of a multi-hop routing path is limited by the worst-performing link along the path; this is known as the bottleneck link. (Data flow) The bottleneck throughput is defined as:

[0063] ;

[0064] The results show that Equation (2) is closely related to the SINR of all physical links along the routing path. The link quality in Equation (1) depends on the channel gain and transmit power. In a real-world environment, integrating geographical and topological information into the routing decision can substantially improve the channel conditions of links along the routing path.

[0065] path The resource allocation scheme for all nodes is represented as follows: ,in Represents a node The transmission power. yes and The function is represented as a necessary factor in cross-layer design.

[0066] Centralized network management is unsuitable for distributed datalink ad hoc networks with flexible topologies because collecting global information leads to excessively high feedback costs and connectivity reliability issues. Therefore, architecture optimization should be distributed, with relay node selection and radio resource allocation being the result of local decisions at each node. Each node must adhere to several practical constraints when making its own decisions:

[0067] 1) Due to the limited sensing capabilities of individual nodes, only incomplete information about the distributed data link ad hoc network can be provided to a single node, affecting its decision-making. Assume each node can only access information within a limited sensing range. The nearby nodes within. Then, the nodes. The adjacent set is defined as:

[0068] ;

[0069] Only about the node set Information can be obtained from nodes The decision-making process directly utilizes this information. Furthermore, it stipulates that a link between two nodes can only be established when both nodes are within each other's sensing range.

[0070] ;

[0071] 2) Each node can be used by at most two consecutive links across all data flows. Once a node has been added to the routing path of a data flow, it is no longer optional for other data flows. Definition In data stream The identity in the data is represented by binary pointers. ,in

[0072] ;

[0073] Only idle nodes are considered potential candidates for adding to a route, and routes for different data flows should not overlap.

[0074] 3) Due to hardware constraints, the maximum transmit power at the node is limited to [specific value]. ,in .

[0075] In a distributed data link ad hoc network, nodes may randomly move to new locations. Significant movement causes changes in channel conditions and network topology, necessitating adjustments to the transmission scheme. An optimized transmission scheme should be adapted to the random network topology to respond to arbitrary node mobility.

[0076] Based on the proposed system model, the data flow... The cross-layer optimization problem of maximizing bottleneck throughput is formulated as follows:

[0077] ;

[0078] ;

[0079] ;

[0080] ;

[0081] ;

[0082] ;

[0083] in and These represent the data flow routing path from the network layer and the transmit power from the physical link layer, respectively. Condition equation (6a) forces all decisions to be made locally. Condition equation (6b) avoids loops in the routing path. Condition equation (6c) ensures that a node does not appear in the routes of multiple data flows simultaneously. Therefore, a node cannot be shared by two physical links unless they are contiguous in the routes. Condition equation (6d) limits the effective range of transmit power at the node. Note that hop count... It depends on The variables are precisely chosen. Considering the end-to-end delay, condition equation (6e) is... An upper limit has been applied.

[0084] The optimization problem in Equation (6) involves convolutional discrete and continuous variables, and such a mixed-integer non-linear programming (MINLP) problem has been found to be NP-hard. Its complexity is mainly attributed to the coupling of non-convex wireless resource allocation and optimal routing. In order to effectively solve the cross-layer optimization problem in Equation (6), an optimization framework based on DRL and agent learning training of graph models is proposed, adapting to the distributed organizational structure of data link ad hoc networks.

[0085] like Figure 2 The diagram illustrates the hop-by-hop optimization scheme proposed in this invention. The decision for each hop is made locally by the last selected node; therefore, optimization is performed through local cooperation between nodes. Since no centralized controller is involved, this method can adapt to datalink ad hoc networks with arbitrary topologies and distributed structures.

[0086] II. Construction of Distributed Data Link Self-Organizing Network Routing Based on DRL and Cross-Layer Optimization

[0087] To address the characteristics of distributed data link ad hoc networks, such as variable topologies and lack of a central controller, a hop-by-hop strategy is adopted to solve formula (6). The main process is as follows: Figure 2 As shown. Starting from the source node, the initial route path is given as follows: In each iteration, the forward node in the routing process selects the next-hop node and transmit power, adds a physical link to the routing path, and then the selected node acts as an agent to begin optimizing the next hop. Physical links are iteratively inserted into the routing path until a termination condition is met. It's important to note that nodes can only access information about their neighboring nodes. Therefore, effective local coordination among nodes at each decision step can improve overall optimization performance.

[0088] The above process can be naturally formulated as a partially-observable Markov decision process (POMDP), which allows the use of DRL to solve the problem given by equation (6). Currently, standard DRL algorithms (such as DQN) use MLP to handle the input / output of agent policies. However, directly applying these algorithms to the cross-layer design of distributed data link ad hoc networks faces the following problems:

[0089] 1) Simplified representation of input information: To optimize optimal routing and resource allocation across layers, topological and interference interactions between nodes should be considered. This relational data has an irregular shape and cannot be effectively processed using ordinary MLP parameterization strategies.

[0090] 2) A multivariable action space that can be extended to fluctuating dimensions: The combination of possible resource allocation and routing path selection results in a large action space, and the variable number of available nodes per hop further contributes to the fluctuation of the action space dimension. Ordinary DRL algorithms do not support selecting actions from a time-varying space.

[0091] 3) Interdependence between local decision steps: The overall performance of cross-layer optimization is hampered by the incompleteness of information at decision nodes. Furthermore, the accuracy of each decision step remains unknown until the entire multi-hop path is established, and effective DRL largely depends on the timely evaluation of agent decisions. These limitations prevent the agent from effectively learning the optimal strategy for distributed cross-layer optimization.

[0092] To address these issues, this embodiment designs a modified DRL solution. First, a hop-by-hop, layer-wise optimization POMDP formula is proposed, which resolves the aforementioned practical limitations. Then, a graph representation method is proposed for the flexible representation of irregular inputs to the agent in the DRL. Based on these graph relationships, a new model architecture is designed as a parameterized strategy for the agent to solve the POMDP. Specifically, the proposed model architecture can handle the fluctuating action space and utilizes temporal relationships across decision steps. Finally, the training process for the DRL agent is presented.

[0093] Reinforcement learning (RL) solves a sequential decision-making task involving a series of discrete time steps. The key components of RL include an agent and an environment with which it interacts, which is represented as a Markov decision process (MDP). The environment provides the complete features required for the task, and the agent learns a strategy for solving the task from trial and error. A standard MDP is described as a quadruple... All possible environment states and available actions of the agent are represented by sets. and At time step Basic information in the environment is expressed in terms of state. To describe, the agent follows its strategy Choose an action Then, the environment follows the state transition probability function. Transition to the next state, while the agent applies the reward function. Receive reward The agent's long-term goal is to learn a strategy that optimally selects actions that maximize cumulative reward, i.e., the return. .here, It is a discount factor that reflects the importance of future rewards to current decisions. An agent's strategy can be constructed in various ways; here, DRL is chosen and a parameter is used. A neural network is used to approximate the strategy.

[0094] In many cases, the agent cannot access the complete environment state, but can only receive the state. Partial observations By introducing a mapping from state to observation. This extends the standard MDP to POMDP. POMDP can reflect complete information that is not available due to real-world physical limitations, but this introduces additional challenges for the agent.

[0095] To apply DRL to the cross-layer optimization problem defined in Equation (6), the entire process of route path establishment is considered as a task set, and the time step concept in RL corresponds to the optimization of one hop. Therefore, the hop count and time step index can be used interchangeably, i.e. At time step Forward node of the path It plays the role of agent and is responsible for selecting the next node. The index and the first Jumping transmit power Throughout the optimization process of multi-hop paths, the agent's role follows... The nodes in the algorithm. Due to the limitations of node awareness, hop-by-hop optimization of a path can be naturally represented as a POMDP. The key components of POMDP are formally introduced:

[0096] (1) Observation, including information about nodes surrounding the agent's current forward node. The agent can access the coordinates of the destination and itself. It can also receive geographic coordinates, accumulated interference, and channel gain from neighboring nodes. In practical systems, information exchange about neighboring nodes is achieved through the control and non-payload link (CNPC). The above features are used to solve optimal routing and resource allocation, and the scope of information exchange is limited to the sensing range of the nodes. Therefore, the agent does not require more overhead than existing algorithms to obtain local observations. Then, the relative positions of nearby nodes to the agent and the destination can be calculated. Time step The observed value at point is given by the following formula:

[0097] ;

[0098] Among them, set Included in forward nodes The node where information is collected. Typically, Including forward nodes and his neighbors When supporting high-scale collaboration in self-organizing data link networks, It can be greater than .

[0099] (2) Action is a combination of relay node selection and power allocation. At the time step... Overall movement space It is an optional neighboring node. and transmit power point level The Cartesian product. To reduce the complexity of multivariate optimization, discrete power control is employed. Maximum power. Divided into average There are several levels, namely:

[0100] ;

[0101] Based on the constraints in formula (6), the set of optional nodes is given by the following formula:

[0102] ;

[0103] Since the agent role is played by different nodes, the neighbor set It is a dynamic set.

[0104] (3) Reward should reflect the feedback reward for the agent's decisions. The goal of cross-layer optimization is to maximize bottleneck throughput, which is only known after the entire path is successfully established. Therefore, the time step... The reward is defined as follows:

[0105] ;

[0106] Among them, conditions express It was added to the route path, and and The paths between them are connected. Formula (2) for the signal-to-interference-plus-noise ratio (SINR) defines the maximum data flow. The bottleneck throughput replaces the connected link capacity. This is because it is a monotonic function of the latter, and its dynamic fluctuations are more sensitive to changes in link quality. Here, a bias is also added. To ensure The reward is positive at the last time step. The reward for all other time steps except the last one remains 0, making the agent sparse and providing delayed feedback signals.

[0107] (4) The event's termination is triggered under the following circumstances:

[0108] Successfully established from arrive The path.

[0109] The current forward node is orphaned and has no valid neighbors to add to the route, i.e. .

[0110] Reaching the maximum number of jumps .

[0111] The optimization fails when the last two scenarios occur. To explicitly identify whether the task has ended, a binary signal is introduced. . Indicator at time step The process terminates at the point of termination.

[0112] Representing the relationships between nodes using a graph structure, a graph is used to describe entity nodes and the relationships between them. Formally, a graph is defined as a tuple. ,in and It is a set of vertices and edges. Each vertex Represent an entity, edge Indicates from arrive The directed relationship. From the vertex and edge The retained attributes are respectively from and This indicates. Here, and The dimension properties of vertices and edges are specified respectively.

[0113] When using the graph representation of a distributed data link ad hoc network, the vertices in the graph correspond to the nodes in the network, i.e. If and only one node Nodes can be accessed Only when the information is available can an edge be created. During each decision-making step, a process is initiated from each neighboring node. To the forward node currently acting as the agent The edges. Therefore, from the definition in formula (7) Nodes are connected to each other by edges. For example... Figure 3 As shown, the agent is located in Its 1-hop and 2-hop neighbors are respectively... and Provided.

[0114] Furthermore, neighboring nodes of the agent can also gather information from their own neighbors, utilizing multi-hop relationships to assist the agent nodes. Process local decisions. Then, if added... This would require a higher level of collaboration between nodes. Therefore, the hop count is defined as... The scope of coordination between nodes varies It increases with the increase of [something]. Here, the default setting [is used]. This represents a trade-off between collaboration overhead and decision performance. Figure 3 An example of a 2-hop local observation plot is shown.

[0115] In theory, at the time step The agent's 1-hop neighbors and 2-hop neighbors are represented as follows:

[0116] ;

[0117] Then, the nodes within the 2-hop local observation map include A subset of , which is given as:

[0118] ;

[0119] The properties of the observation plot are given below:

[0120] 1) Vertex attributes Including the destination node and relative position and its cumulative interference level.

[0121] 2) Edge attributes Including from arrive Channel gain and relative position .

[0122] This graph representation method flexibly expresses the topological and interference relationships between adjacent nodes, while strictly adhering to the distance limitations for local information exchange.

[0123] To fully utilize the relationships within each observation graph and leverage the correlation of observation graphs across time steps, this embodiment designs a specialized model architecture. This architecture includes a graph-based encoder, a recursive module, and a graph-based decision module. The overall architecture is as follows: Figure 4 As shown. Note that the nodes and relationships in the local message passing process correspond to... Figure 3 The content of the observation map.

[0124] 1) Encoder: Since the observations are represented by a graph, GNNs are used to extract hidden features from the graph. The core component of a GNN is the update function. and aggregate functions The resulting graph network (GN) blocks. Update function. Calculate the features of vertices and edges, while aggregation functions Combine input features from multiple edges. Update function. and aggregate functions The precise design depends on the graph structure and task characteristics. In this work, the following functionalities are used:

[0125] ;

[0126] ;

[0127] ;

[0128] The calculation steps for the GN block are shown below. First, formula (13a) represents the edge update function. A message is constructed on each edge based on the current properties of the edges and their connected vertices. Then, equation (13b) represents the aggregation function. The messages from the incoming edges at each vertex are merged. Finally, the vertex update function in equation (13c) is... The new vertex attributes are calculated based on the old vertex attributes and the message from the aggregation function. Equation (14) represents graph convolution, which uses edges to pass messages. In this embodiment, MLPs are chosen as the update function and the normalized mean is chosen as the aggregation function. The function parameters in the GN block are shared between graphs, and the GN block uses the data structure as a relation inductive bias. Therefore, the GNN encoder can achieve a more efficient learning rate than a regular MLP encoder.

[0129] Furthermore, the observations of the observation map were processed more effectively by modifying the standard map convolution. .because and Each vertex plays a different role in the decision-making process and is considered as a different type of vertex set. Therefore, due to the non-uniformity of the observed graph, separate functions are applied to manage 1-hop and 2-hop node relationships. Jump relations, representing the MLPs of the vertex and edge update functions as follows: and Therefore, the local message passing used for processing observations is given as follows:

[0130] ;

[0131] ;

[0132] ;

[0133] ;

[0134] in, Indicates from node Send to node The news Represents a node The vertex update function properties. So far, local observations... The vertex features in the graph are determined by the vertex located in the forward node. The agent is obtained by aggregation.

[0135] 2) Recursive module: (Based on observations) After extracting features, the agent still cannot make high-performance decisions for two reasons: 1) some observations It cannot reflect the overall state of a distributed data link self-organizing network; 2) The objective function, i.e., the bottleneck throughput of the path, is reflected in the time step. It was not available before.

[0136] Therefore, a recursive module was added after the graph-based encoder. This allows it to maintain hidden states. The model employs memory between decision-making steps, with functions including: 1) making local decisions based on the history of observations to reduce some observation constraints, and 2) associating the local decisions maintained by the agent with the final bottleneck throughput. This model uses a gated recurrent unit (GRU) as an efficient recurrent neural network (RNN). Adding a GRU introduces recursion into the policy model architecture, explicitly establishing connections across time steps (jumps).

[0137] 3) Decision module: The model's output depends on the input from... Extracted features and accumulated historical information Time step The combined action space at the location is represented as , The set dimension fluctuates as the agent moves from one node to another. The main problem is that the output of existing DRL algorithms is generated by an MLP and is restricted to a fixed-dimensional 1-D vector. Furthermore, the performance of MLPs deteriorates as the dimension of the action space increases.

[0138] To address the aforementioned issues, a novel GN block is designed to replace the MLP-based decision-making module in DRL agents. It is assumed that at a time step... The agent and its optional neighboring nodes Form a graph. Create edges from the agent to each of its optional neighbors. Then, define the edge update function. The features generated on each edge are used as the scores of the adjacent nodes connected by that edge. The input includes the latest features from the agent in the recursive module and features from neighboring nodes. The output dimension is equal to the number of power levels. Then, when the associated neighbor is selected as the next hop, the output vector of each edge can be used to estimate the values ​​of different power levels. Finally, the output set of all edges is of dimension [dimensional value missing]. Value The vectors are then used to determine the relay node and power level for the next hop.

[0139] because Both the input and output are of fixed dimensions, so it can be parameterized using an MLP. This strategy can be reused by the agent and passed to each of the available neighbor nodes, indicating that it is invariant to fluctuations in the number of available neighbor nodes. In this way, the multivariate action space is decomposed, and the graph-based decision module can be extended to time-varying cardinality. .at the same time, Output dimension much smaller Remove the dimensional limitations.

[0140] Finally, the deep reinforcement learning strategy is trained using a deep recursive Q-learning algorithm.

[0141] To solve for the optimal policy in a POMDP that characterizes cross-layer optimization, Q-learning is applied. The principle of Q-learning is to establish a Q-function, which is the optimal policy. An approximation of the action value function. For any observation value Both agents can use a parameter as The neural network is used as a parameterization strategy to estimate the value of each action, i.e. . This refers to the set of all parameters in the encoder, recursive module, and decision module. Since the strategy includes a recursive module, deep recursive Q-learning is used to update the parameters. .

[0142] During training, the agent follows Greedy strategies select actions from observations to interact with the environment. Greedy strategies achieve a trade-off between utilizing learning strategies and exploring potentially better options. In terms of probability... In the case of, from the estimated Choose the greedy action from the value; otherwise, select from... Random selection is made from the pool. As training progresses, The value is annealed. Assume the total number of steps is... And in time step The standard transformation at the point is defined as a tuple. In the experience replay The collected transformations serve as the agent's experience. When the experience replayer is full, a first-in, first-out (FIFO) rule is used to save the latest experience.

[0143] Because this strategy includes a recursive module, each training sample should be a continuous sequence of transformations, rather than an independent transformation. The recursive module employed by the agent can fully capture the interrelationships between transformations in the sequence. The continuous sequence of transformations is pruned into data blocks of fixed length. Here, each sample is a data block rather than a single transformation, including... A data block of transformation It is given by the following formula:

[0144] ;

[0145] in yes The initial time step. The loss function is The mean square error between the network's estimated value and its target value is expressed as:

[0146] ;

[0147] here, The target, known as the time difference (TD) target, is defined according to the Bellman equation as follows:

[0148] ;

[0149] It is worth noting that the updated objective also includes bootstrapping estimation from the parameterization policy. Here, to enhance training stability, a parameterized... of A separate copy of the network is used to compute the target. value.

[0150] In actual implementation, network parameters Through small batch samples Stochastic gradient descent (SGD) is used to update the data. The ratio is of Soft synchronization is represented as:

[0151] ;

[0152] Detailed training process steps are as follows Figure 5 As shown.

[0153] After training, policies can be deployed at nodes in the datalink self-organizing network. When a communication request is detected between randomly selected node pairs, optimization begins with the designated source node. In each step, the node acts as an agent and uses the learned policy to determine its transmit power and the next node. The agent's role is then passed to the selected node, and so on, until termination is triggered. The final routing and resource allocation scheme is derived from a series of local decisions made by the nodes equipped with the learned policy.

[0154] Cross-layer optimization is implemented on a fixed topology, while the topology of a real distributed data link ad hoc network can be time-varying. When the network layout changes abruptly, communication quality may deteriorate sharply. Then, if a severe performance degradation is detected, optimization should be restarted to update network parameters rapidly. Therefore, adaptability to topology changes is an essential part of the optimization strategy for distributed data link ad hoc networks. This concern was considered when designing the training program. Since the local decision-making of this DRL agent only requires the relative distance / interference between nodes, the network topology is randomly reset in different training sets, bringing two advantages:

[0155] (1) The learning strategy can be generalized to unseen instances of distributed data link self-organizing networks;

[0156] (2) Even if node mobility significantly changes the layout, optimization can be performed without retraining.

[0157] Training DRL is computationally expensive and requires numerous iterations before convergence. Therefore, training should ideally be performed offline, where the learning speed has no significant impact on execution efficiency. A prerequisite for applying offline training is that the policy accepts arbitrary node locations as input. Thus, even if the layout of the distributed data chain self-organizing network is not visible during training, the agent's optimization performance can be guaranteed.

[0158] In this scenario, when the learned policy is deployed for real-time optimization of a distributed datalink self-organizing network, decisions regarding route selection and transmit power require only forward computation of the neural network policy. The execution latency and computational resources for online optimization then depend solely on the complexity of the forward computation, which in turn depends on the number of hops in the route and the exact number of neighbors at each hop. When the agent uses only information about 1-hop neighbors, the time complexity is O(log n). When the number of jumps increases to 2, the complexity becomes... The additional cost of 2-hop message passing results in performance enhancements. Note that computations on each edge can be performed in parallel, which further reduces the execution latency of this algorithm.

[0159] Example 2: This example provides a non-transitory computer-readable storage medium storing computer instructions that cause a computer to execute a distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization. The method includes the following steps:

[0160] S1, when a message is received from the source node to the destination node When making a data stream communication request, the source node is set as the initial proxy node. ;

[0161] S2, in the Each decision-making step, the current agent node Collect its limited sensing range Internal local network information forms observation values. ;

[0162] S3, Proxy Node Observations The input is fed into a pre-trained deep reinforcement learning policy parameterized by a GNN-based model architecture, which makes single-hop optimization decisions and outputs a joint action. This action simultaneously determines the next-hop relay node. and by arrive Transmission power ;

[0163] S4, based on the action Establish a new physical link and connect the current proxy node. Passed to the selected next-hop relay node ;

[0164] S5. Repeat steps S2 to S4 iteratively until the proxy node becomes the destination node. The problem may occur if: the current proxy node has no valid neighbors to choose from; or the path hop count has reached the maximum limit. .

[0165] Example 3: This example provides an electronic device that may include a processor, a communication interface, memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The processor can call logical instructions from the memory to execute a distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization. This method includes the following steps:

[0166] S1, when a message is received from the source node to the destination node When making a data stream communication request, the source node is set as the initial proxy node. ;

[0167] S2, in the Each decision-making step, the current agent node Collect its limited sensing range Internal local network information forms observation values. ;

[0168] S3, Proxy Node Observations The input is fed into a pre-trained deep reinforcement learning policy parameterized by a GNN-based model architecture, which makes single-hop optimization decisions and outputs a joint action. This action simultaneously determines the next-hop relay node. and by arrive Transmission power ;

[0169] S4, based on the action Establish a new physical link and connect the current proxy node. Passed to the selected next-hop relay node ;

[0170] S5. Repeat steps S2 to S4 iteratively until the proxy node becomes the destination node. The problem may occur if: the current proxy node has no valid neighbors to choose from; or the path hop count has reached the maximum limit. .

[0171] Furthermore, the logical instructions in the aforementioned memory can be implemented as software functional units and sold or used as independent products, and can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] Example 4: This example provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute a distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization. The method includes the following steps:

[0173] S1, when a message is received from the source node to the destination node When making a data stream communication request, the source node is set as the initial proxy node. ;

[0174] S2, in the Each decision-making step, the current agent node Collect its limited sensing range Internal local network information forms observation values. ;

[0175] S3, Proxy Node Observations The input is fed into a pre-trained deep reinforcement learning policy parameterized by a GNN-based model architecture, which makes single-hop optimization decisions and outputs a joint action. This action simultaneously determines the next-hop relay node. and by arrive Transmission power ;

[0176] S4, based on the action Establish a new physical link and connect the current proxy node. Passed to the selected next-hop relay node ;

[0177] S5. Repeat steps S2 to S4 iteratively until the proxy node becomes the destination node. The problem may occur if: the current proxy node has no valid neighbors to choose from; or the path hop count has reached the maximum limit. .

[0178] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0179] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0180] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing routes in a distributed data link ad hoc network based on DRL and cross-layer optimization, characterized in that, When receiving from the source node to the destination node When requesting data stream communication, the end-to-end data stream to be established is abstracted into a partially observable Markov hop-by-hop decision process, where each hop is decided solely by the current node based on local observation information, involving a joint decision on routing and transmit power; including the following steps: Set the source node as the initial proxy node. The local observation information is represented by a graph structure; In the Each decision step, the current agent node Collect local network information within its sensing range to form graph-structured observations. Current proxy node This observation value The input is fed into a pre-trained, parameterized deep reinforcement learning policy based on a GNN model architecture. This policy makes single-hop optimization decisions and outputs a joint action. This action simultaneously determines the next-hop relay node. and by the current proxy node To the next hop relay node Transmission power According to joint actions Establish a new physical link and connect the current proxy node. Passed to the selected next-hop relay node ; Each hop is repeated iteratively to observe the Markov hop-by-hop decision process until the current agent node iterates to the destination node. The problem may occur if: the current proxy node has no valid neighbors to choose from; or the path hop count has reached the maximum limit. Complete the data stream communication request.

2. The distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization according to claim 1, characterized in that, In each decision-making step, the optimization objective is to maximize the bottleneck throughput of the data flow. The bottleneck throughput It is determined by the bottleneck link with the lowest signal-to-interference-plus-noise ratio (SINR) on the routing path; The constraints are: The selection of all next-hop relay nodes is within the local awareness range of the current proxy node; Avoid loops in the routing path; Except for the connection points of continuous links, a single node is not shared by the routing paths of multiple data streams; The transmit power of each node is limited to a preset maximum value. Inside.

3. The distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization according to claim 1, characterized in that, The representation of the local observation information using a graph structure specifically includes: The vertices of the graph correspond to the current proxy node and its one-hop and two-hop neighbor nodes; vertex attributes include the relative position of the node to the destination node and its cumulative interference level; The edges of a graph represent the information exchange relationship between the nodes corresponding to the vertices; the attributes of the edges include the channel gain and the relative position between the two nodes connected by the edge.

4. The distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization according to claim 1, characterized in that, The partially observable Markov jump-by-jump decision process is formally defined as: Define the observations of the graph structure Including the current proxy node All neighboring nodes within range Relative to the current proxy node position Position relative to the destination node Channel gain to the current proxy node And at the node Cumulative interference measured at [location] ; Define the joint action of the output It is from a set of optional neighbor nodes and discrete power level set Choose from the Cartesian product action space formed by the action; Define rewards It exhibits sparsity, awarding a positive reward upon successful establishment of a routing path. This reward is a function of the logarithmic SINR value on the bottleneck link of the path, while the reward is 0 for all intermediate steps.

5. The distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization according to claim 1, characterized in that, The GNN model architecture includes: A graph-based encoder for processing observations represented in graph form. Perform information aggregation and feature extraction; The recursive module is used to maintain and pass the hidden state across different decision steps. This is to correlate the local decisions of each hop with the final overall path performance; The graph-based decision module is used to process the action space with dynamically changing dimensions and output decisions.

6. The distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization according to claim 5, characterized in that, The graph-based encoder employs non-uniform graph convolution operations, using independent update functions for one-hop and two-hop relationships in the observation graph. , and , Perform phased message delivery and aggregation; The recursive module is a gated recursive unit (GRU). The graph-based decision-making module makes decisions in the following way: a. The current agent node and all its optional neighbor nodes. Construct a temporary decision graph between them; b. Define an edge update function This function calculates a score vector for each edge in the decision graph, generating a score vector for each possible neighbor node. c. The dimension of the score vector is equal to the number of discrete power levels. Each element in the vector corresponds to the estimate when selecting that neighboring node and using the corresponding transmit power. value; d. Decompose multivariate action selection to accommodate the number of available neighbor nodes. The dynamic changes.

7. The distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization according to claim 1, characterized in that, The training of the deep reinforcement learning strategy is performed offline on a randomly generated network topology. The training process employs a deep recursive Q-learning algorithm, specifically as follows: a. Through the agent - A greedy strategy interacts with the environment to generate a transformation sequence, and then applies a fixed-length sequence. Continuous transformation sequence data blocks Save to Experience Replay middle; b. Randomly select a small batch of samples from the experience replay unit and calculate the time difference target according to the Bellman equation. ; c. By minimizing The parameters of the policy network are updated using the mean squared error loss function between the network estimate and the TD objective. ; d. Use a separate target Network computing TD target Values, and parameters of the target network. Perform soft synchronization updates to enhance training stability.

8. A non-transitory computer-readable storage medium, characterized in that, It stores computer instructions that cause the computer to execute the distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization as described in any one of claims 1-7.

9. An electronic device, characterized in that, include: The system includes a processor, a communication interface, a memory, and a communication bus. The processor, communication interface, and memory communicate with each other via the communication bus. The processor calls logical instructions from the memory to execute the distributed data link self-organizing network routing construction method based on DRL and cross-layer optimization as described in any one of claims 1-7.

10. A computer program product, characterized in that, The computer program product includes a computer program stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer performs the distributed data link ad hoc network routing construction method based on DRL and cross-layer optimization as described in any one of claims 1-7.

Images