A wireless sensor network routing method based on graph reinforcement learning

Abstract

The application relates to a wireless sensor network routing method based on graph reinforcement learning, comprising the following steps: modeling a wireless sensor network as an undirected graph, wherein a feature vector of each graph node and an adjacency matrix with self-connection constitute graph data; training a wireless sensor network routing model based on graph reinforcement learning based on the graph data, wherein the wireless sensor network routing model based on graph reinforcement learning comprises a graph convolution network and a double deep Q network; and determining a deterministic multi-hop transmission path of each to-be-planned node in a to-be-processed wireless sensor network by using the trained wireless sensor network routing model based on graph reinforcement learning. The application utilizes the graph convolution network to deeply mine network topology features, and utilizes the double deep Q network to construct an end-to-end closed-loop feedback scheme; the application effectively utilizes decision error feedback to enhance the representation ability of state features, realizes dynamic balance of network load, and significantly prolongs the life cycle of the network.

Description

Technical Field

[0001] This application relates to the field of wireless communication, and more specifically, to a routing method for wireless sensor networks based on graph reinforcement learning. Background Technology

[0002] Wireless Sensor Networks (WSNs), with their self-organizing characteristics and scalable deployment advantages, have become a core research direction in the Internet of Things (IoT) field for achieving large-scale environmental sensing and data acquisition. WSNs consist of a large number of miniature sensor nodes deployed within a monitoring area. These nodes self-organize into a multi-hop wireless network through efficient routing protocols, completing hop-by-hop transmission of monitoring data to the base station. In practical applications, these nodes are typically deployed in specific areas with complex terrain, harsh environments, or even areas inaccessible to humans, such as railways and mines. Limited by the deployment environment, sensor nodes usually cannot be externally powered and can only operate using batteries with limited capacity. This extremely limited energy reserve has become a key bottleneck restricting the long-term stable operation and large-scale application of wireless sensor networks. Therefore, designing energy-efficient routing protocols to optimize energy distribution, balance network energy consumption, and thus extend network lifespan is a core research direction for overcoming the application bottlenecks of WSNs.

[0003] Research on multi-hop routing in wireless sensor networks has yielded various technical solutions, mainly categorized into three types: traditional clustered multi-hop routing protocols, clustered routing protocols based on heuristic algorithms, and intelligent routing protocols based on traditional reinforcement learning. Existing multi-hop routing technologies for wireless sensor networks cannot effectively address the shortened network lifetime caused by hotspot effects and limited local visibility in routing decisions under large-scale dynamic scenarios. They also struggle to achieve coordinated optimization among cluster head optimization, multi-hop routing planning, and overall network energy consumption balance, becoming a key bottleneck restricting the engineering application of wireless sensor networks in large-scale, dynamic, and complex scenarios. Summary of the Invention

[0004] To overcome at least one of the shortcomings of the prior art, this application provides a wireless sensor network routing method based on graph reinforcement learning.

[0005] Firstly, a routing method for wireless sensor networks based on graph reinforcement learning is provided, including: Obtain the model training dataset, where the samples are wireless sensor networks; The wireless sensor network is modeled as an undirected graph, whose nodes include sensor nodes and base stations. The adjacency matrix for adding self-connections is determined based on the undirected graph. Based on the physical attributes of each sensor node and base station, the feature vector of each graph node is determined; the feature vector of each graph node and the adjacency matrix with added self-connections constitute the graph data. The graph reinforcement learning-based wireless sensor network routing model is trained using graph data to obtain the trained graph reinforcement learning-based wireless sensor network routing model; the graph reinforcement learning-based wireless sensor network routing model includes graph convolutional networks and dual-depth Q networks; During training, the graph convolutional network performs global feature aggregation and encoding on the graph data to obtain high-dimensional node embedding vectors for routing tasks. The dual-depth Q-network includes an evaluation network and a target network. The evaluation network generates Q-values ​​based on the high-dimensional node embedding vectors. These Q-values ​​are then processed by an action masking mechanism to obtain masked Q-values. The optimal action is determined based on the masked Q-values. The target network calculates the Q-value of the optimal action and generates target labels based on the Q-value of the optimal action, combined with the reward value and discount factor. The target labels are used to calculate the loss. The network weights of the graph convolutional network and the dual-depth Q-network are updated based on the loss. The wireless sensor network to be processed is modeled as an undirected graph, and the graph data is determined based on the undirected graph. Any node to be planned in the wireless sensor network is denoted as the current planning node. The graph data is input into the trained wireless sensor network routing model based on graph reinforcement learning. The graph data passes through the graph convolutional network and the evaluation network in sequence to obtain the next hop node of the current planning node. The next hop node is updated to the current planning node. This process is repeated until the next hop node is the base station, thus obtaining a deterministic multi-hop transmission path.

[0006] In one embodiment, the high-dimensional node embedding vector is:

[0007] in, For high-dimensional node embedding vectors, To add self-connected adjacency matrices, For degree matrix, for Activation function These are the feature vectors of each graph node. This is the weight matrix of the first layer of a graph convolutional network (GCN). This is the weight matrix of the second layer GCN in the graph convolutional network.

[0008] In one embodiment, the state space of a dual-depth Q-network is a high-dimensional node embedding vector.

[0009] In one embodiment, the action space of a dual-depth Q-network is:

[0010] in, In order to be in time step t The space of motion In order to be in time step tSelect node for current node The action that serves as the next hop node. Indicates time step t The action of the current node selecting a base station as the next-hop node.

[0011] In one embodiment, the reward of a dual-deep Q-network is:

[0012] in, In order to be in time step t The reward For combined reward value;

[0013] When k=1, The remaining energy factor; when k=2, The distance progress factor; when k=3, Energy consumption factor;

[0014]

[0015]

[0016] in, Next hop node The remaining energy, Next hop node initial energy, This represents the distance between the current node and the base station. This represents the distance between the next-hop node and the base station. For communication range, The energy consumed from the current node to the next hop node. is a coefficient.

[0017] In one embodiment, the masked Q value is:

[0018] in, The Q value after masking. To evaluate the Q-value generated by the network, In order to be in time step t The state space, In time step t The action, To evaluate the weight of the network, the actions in the effective action set are: the next hop node is a node that is alive and has a communication relationship with the current node within the communication range.

[0019] In one embodiment, the target label is:

[0020] in, For target tags, In order to be in time step t The reward As a discount factor, The Q value represents the optimal action.

[0021] In one embodiment, the loss is:

[0022] in, For loss, As expected, Target tags Evaluate the Q-value generated by the network. In order to be in time step t The state space, During the action at time step t, To evaluate the weights of the network.

[0023] Secondly, a wireless sensor network routing device based on graph reinforcement learning is provided, comprising: The training dataset acquisition module is used to acquire the model training dataset, in which the samples are wireless sensor networks. The undirected graph construction module is used to model wireless sensor networks as undirected graphs. The graph nodes of the undirected graph include sensor nodes and base stations, and the adjacency matrix for adding self-connections is determined based on the undirected graph. The graph data construction module is used to determine the feature vectors of each graph node based on the physical attributes of each sensor node and the base station; the feature vectors of each graph node and the adjacency matrix with self-connection are used to form the graph data. The training module is used to train the graph reinforcement learning-based wireless sensor network routing model based on graph data, resulting in the trained graph reinforcement learning-based wireless sensor network routing model. The graph reinforcement learning-based wireless sensor network routing model includes graph convolutional networks and dual-depth Q-networks. During training, the graph convolutional network performs global feature aggregation and encoding on the graph data to obtain high-dimensional node embedding vectors for routing tasks. The dual-depth Q-network includes an evaluation network and a target network. The evaluation network generates Q-values ​​based on the high-dimensional node embedding vectors. These Q-values ​​are then processed by an action masking mechanism to obtain masked Q-values. The optimal action is determined based on the masked Q-values. The target network calculates the Q-value of the optimal action and generates target labels based on the Q-value of the optimal action, combined with the reward value and discount factor. The target labels are used to calculate the loss. The network weights of the graph convolutional network and the dual-depth Q-network are updated based on the loss. The path generation module is used to model the wireless sensor network to be processed as an undirected graph and determine the graph data based on the undirected graph. Any node to be planned in the wireless sensor network to be processed is marked as the current planning node. The graph data is input into the trained wireless sensor network routing model based on graph reinforcement learning. The graph data passes through the graph convolutional network and the evaluation network in sequence to obtain the next hop node of the current planning node. The next hop node is updated to the current planning node. This process is repeated until the next hop node is the base station, thus obtaining a deterministic multi-hop transmission path.

[0024] Compared with existing technologies, this application has the following advantages: This application utilizes graph convolutional networks to deeply mine network topology features, giving nodes a global forward-looking view of the entire network state; and uses a dual-depth Q network to construct an end-to-end closed-loop feedback scheme, which reverses the interaction error generated by routing decisions to achieve adaptive enhancement of feature representation, resulting in a multi-hop routing path with optimal global energy efficiency and load balance; This application effectively utilizes decision error feedback to enhance the representation capability of state features, achieves dynamic load balancing across the entire network, and significantly extends the network's lifespan. Attached Figure Description

[0025] This application can be better understood by referring to the description given below in conjunction with the accompanying drawings, which, together with the detailed description below, are incorporated in and form part of this specification. In the drawings: Figure 1 A flowchart of a wireless sensor network routing method based on graph reinforcement learning is shown. Figure 2 A schematic diagram of a wireless sensor network routing model based on graph reinforcement learning is shown. Figure 3 The diagram illustrates the actual routing effect of the method described in this application within a network. Figure 4 A comparison chart of the average reward convergence curves of the algorithm under different hyperparameter values ​​is shown; Figure 5 The results of ablation experiments in different scenarios are shown; Figure 6 The performance metrics of various routing algorithms under different network area sizes are shown in the graph. Figure 7 The graph shows a comparison of the performance metrics of various routing algorithms with the same number of nodes. Detailed Implementation

[0026] Exemplary embodiments of the present application will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the development of any such actual embodiment to achieve the developer’s specific objectives, and these decisions may vary as the embodiments differ.

[0027] It should also be noted that, in order to avoid obscuring this application with unnecessary details, only the device structure closely related to the solution of this application is shown in the accompanying drawings, while other details that are not closely related to this application are omitted.

[0028] It should be understood that this application is not limited to the described embodiments by virtue of the following description with reference to the accompanying drawings. In this document, embodiments may be combined with each other, features may be substituted or borrowed between different embodiments, and one or more features may be omitted in one embodiment, where feasible.

[0029] This application provides a wireless sensor network routing method based on graph reinforcement learning, which overcomes the shortcomings of existing multi-hop routing schemes, such as the separation of perception and decision-making modules and the lack of routing task orientation in state representation, which easily leads to routing blindness and energy voids in dynamic and complex environments. The core of its technical concept lies in establishing a joint optimization architecture that deeply couples perception and decision-making. This aims to eliminate information distortion caused by linear state representation (one-dimensional vector) and decision blindness caused by the separation of perception and decision-making modules in traditional intelligent routing through direct learning of graph data (adjacency matrix and feature matrix). This application directly backpropagates the path planning loss generated by reinforcement learning to the graph convolutional feature extraction layer, enabling the network topology representation capability to adaptively enhance based on real-time feedback from the routing task, thereby fundamentally suppressing the generation of energy voids and extending the network lifetime.

[0030] Figure 1 A flowchart of a wireless sensor network routing method based on graph reinforcement learning is shown. See [link / reference]. Figure 1 The method mainly includes the following steps: Step S1: Obtain the model training dataset, where the samples are wireless sensor networks.

[0031] Step S2: Model the wireless sensor network as an undirected graph. Graph nodes of an undirected graph Including sensor nodes and base stations, It is the set of connected edges determined based on the current physical communication environment, and the adjacency matrix for adding self-connections is determined according to the undirected graph.

[0032] The topology of a graph is determined by its adjacency matrix. This indicates that its construction rules strictly follow the communication radius. Physical constraints. For any graph node pair Its connectivity is defined as follows:

[0033] in, Adjacency matrix The elements in for The distance between them.

[0034] Regarding the feature extraction mechanism, in order to preserve the node's own feature information during information transmission and to solve the problem of numerical scale imbalance caused by differences in the degree of different nodes, this application adopts a renormalization technique, which involves adding self-loops to the adjacency matrix in the topological graph. Add an identity matrix To construct an adjacency matrix with self-connections and the corresponding degree matrix Its definition is as follows:

[0035] =

[0036] in, Degree matrix The elements in.

[0037] Step S3: Determine the feature vector of each graph node based on the physical attributes of each sensor node and the base station; the feature vector of each graph node and the adjacency matrix with added self-connections constitute graph data.

[0038] Each graph node's feature vector includes five key attributes, namely:

[0039] in, For graph nodes eigenvectors, and Graph nodes along x direction and y Normalized coordinates of direction, Represents graph nodes energy levels, Represents graph nodes The distance to the base station can help the agent determine the direction of data transmission. This is an identifier used to distinguish the decision-making entity; if the node... Currently, a routing decision is being made. This value is set to 1, while the values ​​of the other nodes are set to 0. This allows the graph convolutional network to focus on the current action executor when aggregating information across the entire network.

[0040] The feature vectors of each graph node are combined to form a vector X, and the graph data is represented as (X, ).

[0041] Step S4: Train the graph reinforcement learning-based wireless sensor network routing model based on graph data to obtain the trained graph reinforcement learning-based wireless sensor network routing model; the graph reinforcement learning-based wireless sensor network routing model includes graph convolutional networks and dual-depth Q-networks. Figure 2 A schematic diagram of a wireless sensor network routing model based on graph reinforcement learning is shown.

[0042] During training, the graph convolutional network performs global feature aggregation and encoding on the graph data to obtain high-dimensional node embedding vectors for routing tasks. The dual-depth Q-network includes an evaluation network and a target network. The evaluation network generates Q-values ​​based on the high-dimensional node embedding vectors. These Q-values ​​are then processed through an action masking mechanism to obtain masked Q-values. The optimal action is determined based on the masked Q-values. The target network calculates the Q-value of the optimal action and generates a target label based on the Q-value of the optimal action, combined with the reward value and discount factor. The target label is used to calculate the loss. The network weights of the graph convolutional network and the dual-depth Q-network are updated based on the loss.

[0043] Here, to overcome the limitations of single-hop information perception, the graph convolutional network includes a two-layer GCN (Graph Convolutional Neural Network) to complete feature encoding and fuse two-hop neighborhood topological information. This ensures global topological perception while avoiding the computational overhead and overfitting risks associated with excessively deep networks. The resulting high-dimensional node embedding vector is:

[0044] in, For high-dimensional node embedding vectors, To add self-connected adjacency matrices, For degree matrix, for Activation function These are the feature vectors of each graph node. This is the weight matrix of the first layer of a graph convolutional network (GCN). This is the weight matrix of the second layer GCN in the graph convolutional network.

[0045] Graph convolutional networks nonlinearly map the original low-dimensional physical properties into high-dimensional node embedding vectors. These vectors not only contain the energy state of individual nodes, but also capture their connectivity role and potential forwarding potential in the overall network topology.

[0046] Traditional routing methods based on reinforcement learning (such as Q-learning and basic DQN) typically flatten and concatenate node states (energy, location, etc.) into a fixed-length one-dimensional vector as input. This approach forcibly maps the non-Euclidean network topology to a low-dimensional linear space, inevitably losing the topological adjacency relationships between nodes and complex spatial evolution information. This embodiment, however, utilizes a message-passing mechanism through GCN to enable each node to aggregate feature information from its two-hop neighborhood. Its output high-dimensional embedding vector not only contains information such as the node's energy and location but also implies the node's relay potential in the global topology.

[0047] The base station needs to use the extracted node embedding vectors for accurate routing decisions. Traditional Deep Q-Networks (DQNs), when dealing with sequential decision-making problems with large state-action spaces, use the same network for both target action selection and value evaluation. This mechanism, using the same set of Q-values ​​for both action selection and value evaluation, is prone to accumulating positive errors in the value of non-optimal relay nodes during complex decision-making processes, leading to severe Q-value overestimation and causing the routing strategy to get stuck in local optima or even fail to converge. To overcome the problem of self-reinforcing errors in a single network during routing decisions, this application introduces a Dual Deep Q-Network (DDQN) to enable the agent to make routing decisions. DDQN constructs an evaluation network and a target network with identical network structures but independent parameters, separating action selection and value evaluation, thereby effectively improving the convergence stability of the algorithm in complex dynamic topologies.

[0048] In the routing decision phase, the high-dimensional embedding vector output by the graph convolutional network, which integrates the node's own attributes and the topological information of its two-hop neighborhood, is first processed. Flatten it out and use it as a time step. status The input is fed into the evaluation network. The evaluation network performs a nonlinear mapping on the state features and outputs a vector consistent with the action space dimension. Each element in this vector corresponds to the original action value of the current decision node in selecting a candidate node as the next-hop forwarding target. However, wireless sensor networks have strict physical communication constraints. For example, some nodes may fail due to energy depletion or exceed the radio frequency communication radius of the current node. Allowing agents to freely explore in an unconstrained full-dimensional action space would not only result in a huge waste of computational power but also easily lead to data packets being routed to unreachable nodes, or even triggering routing loops.

[0049] To ensure the physical feasibility and algorithm convergence of routing decisions, this application introduces an action masking mechanism at the output of the evaluation network. This mechanism constructs an effective action set based on the dynamic adjacency matrix of the current decision node and the liveness status of all nodes in the network. And the Q-value of invalid actions is forcibly penalized to negative infinity. The Q-value after masking is:

[0050] in, The Q value after masking. To evaluate the Q-value generated by the network, In order to be in time step t The state space, In time step t The action, To evaluate the weight of the network, the actions in the effective action set are: the next hop node is a node that is alive and has a communication relationship with the current node within the communication range.

[0051] After acquiring an effective action space, the agent generates experiential data through interaction with the environment. During the training and update phases, the separate dual networks of DDQN play a crucial role. First, the evaluation network evaluates the next state... From the set of valid actions, the optimal action that maximizes the Q-value is selected. Then, the target network utilizes its frozen weights. Calculate the Q-value of the optimal action, and generate the target label based on the Q-value of the optimal action, combined with the reward value and the discount factor:

[0052] in, For target tags, In order to be in time step t The reward As a discount factor, The Q value represents the optimal action.

[0053] In the network optimization process, to effectively overcome the parameter oscillation problem that easily occurs in traditional gradient descent methods under complex dynamic environments, this application employs the RMSPropOptimizer to perform iterative updates of network weights. This optimizer improves upon the gradient oscillation and convergence instability issues common in traditional gradient descent methods. It adaptively adjusts the learning rate by using an exponential moving average of the squared historical gradients, significantly reducing memory consumption and enabling the neural network to converge quickly and stably during training. This is combined with an empirical replay pool. The optimizer minimizes the target label. The mean squared error loss function between the current predictions of the evaluation network is used to backpropagate the gradient and update the weights of the evaluation network. The loss is:

[0054] in, For loss, As expected, Target tags Evaluate the Q-value generated by the network. In order to be in time step t The state space, During the action at time step t, To evaluate the weights of the network.

[0055] Unlike traditional algorithms where the weights of the perception modules are fixed or updated independently, this application applies the gradient signal of the loss function not only to update the network decision weights θ of the DDQN, but also directly backpropagates it to the weight matrix of the front-end Graph Convolutional Network (GCN). This collaborative optimization logic ensures that the feature convolutional kernels of the GCN are aware of the performance of multi-hop routing tasks: if a feature representation leads to high energy consumption or path detours (resulting in low rewards), the gradient signal will drive the GCN to adjust its weights, enabling it to more sensitively capture and highlight key topological information affecting routing quality in subsequent sampling. Through this feedback enhancement, state representation and path decision-making achieve "synchronized" evolution during training, significantly improving the model's accuracy in representing complex dynamic topologies.

[0056] Step S5: Model the wireless sensor network to be processed as an undirected graph and determine the graph data based on the undirected graph; denote any node to be planned in the wireless sensor network as the current planning node; input the graph data into the trained wireless sensor network routing model based on graph reinforcement learning; the graph data passes through the graph convolutional network and the evaluation network in sequence to obtain the next hop node of the current planning node; update the next hop node to the current planning node, and repeat the process until the next hop node is the base station, thus obtaining a deterministic multi-hop transmission path.

[0057] Through the above path generation process, multiple deterministic transmission paths are obtained for each node to be planned.

[0058] After path planning is completed, the base station encapsulates the generated deterministic path into a routing table and broadcasts it to the entire network via the wireless downlink. Upon receiving instructions, each sensor node no longer needs to perform complex local routing calculations; it only needs to execute deterministic data relay actions based on the routing decisions issued by the base station. This mechanism of "centralized computing by the base station and distributed execution by the nodes" not only maximizes the use of the base station's powerful computing capabilities for global optimization but also significantly reduces the control overhead of resource-constrained nodes in complex dynamic topologies. Through periodic network state sampling and model inference, this application ensures that multi-hop backhaul of sensing data is always performed with optimal energy efficiency as the energy distribution of the sensor network continuously evolves, fundamentally achieving efficient energy balance and extended lifespan for the entire network.

[0059] Figure 3The diagram illustrates the actual routing effect of the proposed method in a network. Orange diamonds represent cluster head nodes elected in this round, blue dots represent nodes that communicate directly with the base station (typically hotspot nodes undertaking relay tasks), red pentagrams represent base stations responsible for data collection, and black arrows depict multi-hop forwarding paths from the cluster head to the base station. From the routing effect, the spatial distribution of cluster head nodes is very uniform, effectively dispersing the energy consumption caused by information aggregation. Remote cluster heads select relay nodes with suitable locations and energy levels to transmit data, effectively shortening the single transmission distance. Simultaneously, direct communication is chosen for nodes closer to the base station; these nodes need to undertake a large number of data relay tasks and try to avoid being elected as cluster heads. This strategy effectively alleviates the energy gap problem near the base station.

[0060] This application utilizes graph convolutional networks to deeply mine network topology features, giving nodes a global forward-looking view of the entire network state. Furthermore, it employs a dual-depth Q-network to construct an end-to-end closed-loop feedback scheme, which reverses the interaction errors generated by routing decisions to achieve adaptive enhancement of feature representation, resulting in a globally energy-efficient and load-balanced multi-hop routing path. This application effectively utilizes decision error feedback to enhance the representation capability of state features, achieves dynamic load balancing across the entire network, and significantly extends the network's lifetime.

[0061] In one embodiment, to enable the routing optimization problem to be solved using reinforcement learning, this application formally models the sequential forwarding process of data packets as a quintuple Markov decision process. ,in It is a finite set of states. It is a finite set of actions, representing the state. The following set of executable actions It is the state transition probability matrix, which describes the agent's state transition probability in the current state. Next action Afterwards, the environment changed from state Transition to a new state The probability, This is a reward function used to evaluate the immediate benefit of performing an action in a given state. This is a discount factor used to balance the weighting of immediate rewards and long-term cumulative rewards. The larger the value, the more the agent focuses on long-term cumulative rewards.

[0062] (1) State space The state space describes the set of observable states of a reinforcement learning agent. In this application, routing reasoning decisions are uniformly executed by the base station, and the state space is represented using a global graph structure rather than local node states. (At time step...) At that time, the high-dimensional node embedding vector is flattened and used as the time step. status .

[0063] (2) Action space Action space Defined as the set of all possible next-hop forwarding objects within the communication radius of the current decision node, including all reachable neighbor nodes and base stations. In time steps... The motion space is:

[0064] in, In order to be in time step t The space of motion In order to be in time step t Select node for current node The action that serves as the next hop node. Indicates time step t The action of the current node selecting a base station as the next-hop node.

[0065] (3) Reward function reward function Is the agent in a state? Execute action Then transition to a new state The immediate reward obtained provides real-time feedback on the agent's behavior, evaluating the merits of an action in a given state and influencing future decisions. Through continuous trial and error and adjustments, the agent learns to select high-reward behavioral strategies in different states, thus guiding it to learn efficient and balanced routing strategies. In multi-hop routing scenarios of wireless sensor networks, uneven energy consumption and excessive relay hops are core pain points leading to shortened network lifetimes. Therefore, the reward function needs to guide the agent to avoid low-energy nodes, prioritize paths approaching the base station, and reduce single-hop communication energy consumption. Combining the above optimization objectives, the reward function is defined as:

[0066] in, In order to be in time step t The reward For combined reward value;

[0067] When k=1, The remaining energy factor represents the proportion of remaining energy in the next hop node. This factor encourages agents to choose nodes with sufficient remaining energy as the next hop, avoiding premature failure of energy-deficient nodes due to frequent forwarding, and ensuring a balanced energy load across the entire network. When k=2, The distance progress factor represents the degree of progress towards the base station when hopping from the current node to the next node. It guides data packets to move closer to the base station, and the purpose of this factor is to reduce the number of invalid relay hops and shorten the data packet transmission path. The larger the value, the more significantly the routing decision brings the data packet closer to the base station. The agent will receive a higher reward and will be guided to choose a forwarding scheme with fewer hops and a shorter path. When k=3, The energy consumption factor represents the energy consumption level of single-hop communication. Its core objective is to reduce the energy loss of a single forward and improve energy utilization efficiency.

[0068]

[0069]

[0070] in, Next hop node The remaining energy, Next hop node initial energy, This represents the distance between the current node and the base station. This represents the distance between the next-hop node and the base station. For communication range, The energy consumed from the current node to the next hop node. is a coefficient.

[0071] To prevent the agent from obtaining rewards through meaningless routing behavior, and to guide it to learn a more balanced globally optimal route while constraining the hop count, the reward for the current decision node directly forwarding the data packet to the base station is set to 0, while the joint reward value in a conventional relay forwarding scenario is... Strictly bound to Within this interval, the overall reward is always negative. This design ensures that the agent only receives the highest relative reward when routing to the base station, while any additional invalid relay forwarding will generate negative returns, thus naturally avoiding meaningless multi-hop forwarding. Simultaneously, the joint reward incorporates multiple factors such as energy balance, distance progress, and energy consumption control, enabling the agent to prioritize high-energy nodes, paths that approach the base station, and have lower single-hop energy consumption after completing the routing. This guarantees routing efficiency and guides the agent to learn a globally optimal routing strategy that balances hop count constraints, load balancing, and energy efficiency optimization.

[0072] To further verify the effectiveness of the method in this application, the following experimental analysis was conducted.

[0073] The proposed method (GDRL-MRA) is evaluated through simulation experiments from the perspectives of energy consumption and network lifetime, thereby verifying its effectiveness in extending the lifespan of wireless sensor networks and balancing node energy consumption. First, the simulation experimental environment and basic parameter settings are introduced. Then, ablation experiments are used to analyze the impact of different hyperparameters on network performance. Finally, GDRL-MRA is compared with routing algorithms such as DEEC, MH-LEACH, MRP-GTCO, and EECRAIFA under different simulation environments, comprehensively analyzing the differences in network performance among the algorithms.

[0074] (1) Experimental setup A simulation environment was built using Python 3.11. Efficient matrix operations were implemented using NumPy, and the GCN topology sensing module and DDQN decision network were constructed using the deep learning framework PyTorch 2.1.0. Simultaneously, Pandas was used to store and process the massive log data generated during the experiment, such as node remaining energy and network throughput, in CSV format. To evaluate the performance of the proposed method, the experiment adopted the three simulation scenarios (Scenario I, II, and III) defined in Table 1 for network topology configuration. The initial node energy was uniformly set to 0.5 J, and 4000 bits of sensing data were generated per communication cycle. It was assumed that all sensor nodes had sufficient data transmission time in each round, and the specific multi-hop network parameters are shown in Table 2. The simulation experiment automatically terminated when 80% of the nodes in the network died.

[0075] Table 1 Network topology configurations for experimental scenarios of different scales

[0076] Table 2 Multi-hop network environment parameter settings

[0077] To comprehensively evaluate the overall performance of routing algorithms, the following four key indicators are selected as evaluation criteria.

[0078] Average energy consumption per round: This is the ratio of the total energy of the entire network to the total number of communication rounds. This metric directly reflects the energy utilization efficiency of the routing algorithm; the lower the value, the better the algorithm is in reducing transmission overhead and optimizing energy allocation.

[0079] First Node Dead (FND): Defines the time of death of the first node. FND is an important indicator of network stability and reliability. A later FND value means that the algorithm can effectively prevent some nodes from failing prematurely and ensure the integrity of the initial monitoring coverage.

[0080] Half Node Dead (HND): Defines the time when 50% of the nodes run out of energy. HND can show the trend of node deaths. By recording the half-node dead time, we can observe the trend and rate of network node deaths. A later HND value indicates a more balanced energy consumption in the network.

[0081] Network Failure Time (NFT): Defined as the time when 80% of the nodes in the network die, it is used to define the limit of the network's multi-hop communication capability. At this point, due to the low node density causing the topology connectivity to break, the base station is unable to establish effective multi-hop paths to collect data, marking the complete failure of network function.

[0082] (2) Parameter settings The performance of the method in this application is highly dependent on the hyperparameter configuration of the neural network. In order to enable the agent to maintain a fast learning speed in complex and ever-changing multi-hop routing environments and to converge stably to the global optimum, the training parameters of the DDQN agent were optimized and tested in the basic environment of Scenario II. The average reward obtained during training was used as the core evaluation index, and the convergence performance of the model was compared by analyzing the changes of the following four key hyperparameters: (1) hidden layer dimension, (2) discount factor, (3) target network update frequency, and (4) batch size.

[0083] Figure 4 The graph shows a comparison of the convergence curves of the average reward of the algorithm under different hyperparameter values. (a) shows the effect of the hidden layer dimension on the average reward of the network, (b) shows the effect of the discount factor on the average reward of the network, (c) shows the effect of the target network update frequency on the average reward of the network, and (d) shows the effect of the batch size on the average reward of the network.

[0084] According to (a), when the hidden layer dimension is set to 32, the network's ability to extract local topological features is limited due to insufficient model capacity, resulting in slow reward convergence and a relatively low final reward value. However, when the dimension is increased to 48, 64, or 128, the overall trend stabilizes, and all converge to a higher reward range. Considering the need to minimize the computational complexity and training time of base stations in sensor networks, the algorithm selects 64 dimensions as the optimal hidden layer dimension.

[0085] According to (b), when γ is 0.3, the reward value for final convergence decreases and the convergence speed is slower, while the convergence effect is similar when γ is 0.5 or 0.9. This indicates that a larger discount factor can encourage agents to pay more attention to the impact of long-term paths on the overall network lifetime when performing multi-hop optimization.

[0086] According to (c), when the update frequency is set to a slow 20, the learning efficiency of the model is significantly reduced in the early stages of training due to the severe lag in updating the target Q value, resulting in slow convergence. Increasing the update frequency to 5 allows the network to promptly correct estimation biases in action values, ensuring training stability.

[0087] According to (d), using a large batch size of 48 leads to slower initial convergence and a significantly longer time the algorithm spends in the low-reward region. In contrast, when the batch size is set to 16 or 32, the network can achieve faster initial exploration. A batch size of 32 achieves the best trade-off between gradient estimation accuracy and training speed.

[0088] Based on the comparison and optimization analysis of the above multiple sets of parameters, the parameter settings of the GDRL-MRA routing algorithm were finally determined as shown in Table 3. Table 3

[0089] (3) Ablation test To verify the actual performance improvement of the Dual Deep Q Network (DDQN) and Graph Topology Aware Module (GCN) in this application, and to examine the robustness of these core components under different network scenarios, ablation experiments were conducted on the key modules. The experiments compared and analyzed the following three scenarios: DQN: It only uses a standard deep Q-network model based on the basic reinforcement learning framework, does not introduce graph feature extraction, and has the risk of overestimating action value.

[0090] DDQN: It introduces a dual-network architecture routing decision model on the basic framework, which can effectively alleviate value assessment bias, but lacks global topology awareness.

[0091] The method proposed in this application (GDRL-MRA): The complete routing decision model proposed in this application has both a graph feature aggregation mechanism and a dual-depth network architecture.

[0092] In selecting the testing environment, considering that Scenario I involves a small-scale network with densely distributed nodes and short multi-hop links, the exploration space for routing is limited, making it difficult to fully expose the decision-making differences of different algorithms when dealing with complex topologies. Therefore, Scenario II (100 nodes) and Scenario III (200 nodes) were selected as the benchmark environments.

[0093] The experiment recorded the model's performance in terms of the number of surviving nodes and the total network energy consumption. Figure 5 The results of ablation experiments in different scenarios are shown.

[0094] (a) shows the survival node curves in Scenario II, illustrating the survival node trends of DQN, DDQN, and GDRL-MRA in Scenario II. All three algorithms maintained normal operation of all network nodes in the early stages of simulation. However, as communication rounds progressed, the DQN algorithm with a single network structure was the first to experience node failure. The first node death time (FND) of DQN occurred at round 840, significantly earlier than the other two models. This is because traditional DQN is prone to Q-value overestimation when facing complex routing action spaces, causing the agent's planning trajectory to fall into local optima, resulting in some key relay nodes prematurely exhausting their energy due to excessive forwarding load. The DDQN algorithm, which introduces a dual-network architecture, postpones FND to round 920, demonstrating that the dual-network architecture, which separates action selection and value evaluation, effectively mitigates the bias in value evaluation, extending the network stability period to some extent. The GDRL-MRA model proposed in this application exhibits the best network lifetime, with an FND improvement of 14.3% compared to DQN and 4.3% compared to DDQN. This demonstrates that, in conventional-scale networks, GDRL-MRA can achieve better load balancing by relying on global topology characteristics.

[0095] (c) shows the survival node curves under Scenario III, reflecting the large area and dense node topology environment in Scenario III. The increase in monitoring area leads to a significant increase in the relay hops of nodes, completely differentiating the performance of each module. Under this condition, the possible path combinations for multi-hop routes increase exponentially. DQN and DDQN algorithms, lacking topology awareness, can only make decisions from a local perspective, resulting in a rapid and steep decline in node survival rate. The FND of the DQN model occurs as early as round 54, and the HND occurs at only round 264. In contrast, the DDQN model postpones the FND to round 122, and the HND occurs at round 431. In comparison, the GDRL-MRA model maintains a relatively gentle decline trend. In terms of FND, GDRL-MRA improves by approximately 144.4% and 52.5% compared to DQN and DDQN, respectively, and in terms of HND, it improves by approximately 122.7% and 36.4% compared to DQN and DDQN, respectively, significantly extending the network lifetime. This is mainly attributed to GCN's powerful spatial feature aggregation mechanism, which enables the agent to keenly capture the topological connectivity and energy distribution of the entire network, autonomously avoid low-energy congested areas during the decision-making stage, and thus find the optimal multi-hop transmission path in the environment.

[0096] (b) shows the total network energy consumption curve for scenario II, and (d) shows the total network energy consumption curve for scenario III. (b) and (d) record the consumption of the total remaining network energy. The decreasing slope of the energy consumption curves intuitively reflects the overall energy efficiency level of each algorithm in path planning. In both scenario II and scenario III, the energy consumption rate of GDRL-MRA is consistently lower than that of DQN and DDQN. Furthermore, its leading advantage is particularly significant in large-scale scenarios. Taking the 200th round in (d) as an example, at this point, the total remaining network energy of DQN is approximately 30 J, the remaining energy of DDQN is approximately 55 J, while GDRL-MRA still retains nearly 65 J of sufficient energy. This indicates that the introduction of graph features enables the algorithm to consistently select the optimal path with fewer relay hops and lower communication overhead.

[0097] Ablation experiments fully validate the effectiveness of the multi-module collaborative mechanism, from basic DQN to DDQN that solves the overestimation problem, and then to GDRL-MRA that integrates topology-aware modules. These results demonstrate that the method in this application provides an effective and feasible technical path for solving the multi-hop routing problem in wireless sensor networks.

[0098] (4) Performance comparison under different network regions Performance comparisons of the proposed GDRL-MRA with routing algorithms such as DEEC, MH-LEACH, MRP-GTCO, and EECRAIFA were conducted under different network area scenarios to verify the effectiveness of GDRL-MRA in complex and dynamic network environments. In the experiment, the number of sensor nodes was fixed at 100, with an initial energy of 0.5J, and the base station was located at the center of the monitoring area. Nodes were randomly deployed. , , and Four different network areas of varying sizes were included. These scenarios were set up to simulate wireless sensor networks with different coverage areas, testing the algorithm's performance under different network densities and communication distances.

[0099] Figure 6 The performance metrics of various routing algorithms under different network area sizes are shown in the graphs. Among them, (a) shows the average energy consumption comparison, (b) shows the FND round comparison, (c) shows the HND round comparison, and (d) shows the NFT round comparison.

[0100] In the average energy consumption comparison results in (a), as the network area expands, the communication distance between nodes increases, leading to an exponential increase in the energy consumed for data transmission, and consequently, an increase in the average energy consumption per round across the entire network. However, GDRL-MRA consistently maintains the lowest average energy consumption and exhibits the most gradual growth, with its average energy consumption reduced by approximately 2.5%, 7.7%, 16.1%, and 22.2% compared to the EECRAIFA, MRP-GTCO, DEEC, and MH-LEACH algorithms, respectively. In this scenario, DEEC and MH-LEACH, by considering only energy factors when selecting cluster heads and neglecting important factors such as node location and network topology, experience increased energy consumption due to long-distance node communication. The EECRAIFA and MRP-GTCO algorithms, on the other hand, employ heuristic algorithms to achieve more reasonable cluster head election, improving energy balance compared to traditional protocols. However, both algorithms focus on static optimization objectives, making them less adaptable to large-scale, highly dynamic network scenarios and prone to getting trapped in local optima, resulting in higher energy consumption. In contrast, GDRL-MRA uses a heuristic algorithm to efficiently select cluster heads, taking into account information such as node energy and location, and uses DDQN to continuously learn and determine the optimal multi-hop path.

[0101] In sections (b), (c), and (d), the network lifetime of GDRL-MRA was compared with that of EECRAIFA, MRP-GTCO, DEEC, and MH-LEACH protocols in different network areas. Specific metrics included FND, HND, and NFT. The expansion of network area size led to a more sparse node distribution and a significant increase in the energy demand for inter-node communication, resulting in a shortened network lifetime for all routing protocols. In the FND metric, GDRL-MRA significantly outperformed other algorithms in all areas, maintaining a stable FND round count of over 1000 rounds in smaller environments of 100m and 150m, while algorithms such as EECRAIFA and MRP-GTCO performed below 500 rounds. Even when the area expanded to 250m, GDRL-MRA maintained a stable period of over 600 rounds, while DEEC and MH-LEACH protocols dropped to around 100 rounds. In the HND metric, GDRL-MRA achieved average improvements of approximately 21.3%, 38.8%, 69.1%, and 107.6% compared to EECRAIFA, MRP-GTCO, DEEC, and MH-LEACH algorithms, respectively, significantly extending the survival time of half of the network nodes and demonstrating superior mid-term energy balance and network stability. In the NFT metric, GDRL-MRA maintained its advantage in the 200m and 250m regions, achieving improvements of 3.8% and 5.9% compared to EECRAIFA, respectively. Both MH-LEACH and DEEC protocols employ random cluster head election strategies, failing to consider key factors such as node energy, location, and load. This results in low-energy or remote nodes still being elected as cluster heads, undertaking high-energy-consuming communication, leading to early node death and energy imbalance. Furthermore, both use single-hop or simple multi-hop routing, neglecting real-time node energy and link status, accelerating node failure. The EECRAIFA and MRP-GTCO protocols focus on static optimization goals. While they consider factors such as node location and energy, they are poorly adaptable to dynamic topologies and prone to cluster head overload or routing hotspots in large-scale networks. In contrast, GDRL-MRA utilizes heuristic algorithms to optimize cluster head selection, leverages GCN to enhance the information content of node features, and combines deep reinforcement learning to achieve intelligent decision-making. It can adjust routing strategies in real time according to dynamic network changes, balance node energy consumption, and achieve more efficient energy utilization.

[0102] (5) Performance comparison with different numbers of nodes Furthermore, to verify the scalability and load balancing capabilities of the algorithm under different deployment densities, the performance of GDRL-MRA was compared with that of DEEC, MH-LEACH, MRP-GTCO, and EECRAIFA protocols under different numbers of nodes. The monitoring area size was fixed in the experiment. The base station was located at the center of the region, and the initial energy of all nodes was uniformly set to 0.5J. The number of nodes in the experiment was set to 100, 150, 200, 250 and 300 respectively to simulate a typical scenario from sparse deployment to dense deployment.

[0103] Figure 7 The performance metrics of various routing algorithms with the same number of nodes are shown in the graph. (a) shows the average energy consumption comparison, (b) shows the FND round comparison, (c) shows the HND round comparison, and (d) shows the NFT round comparison.

[0104] (a) shows the average energy consumption results for each protocol. As the number of nodes increases, intra-network communication becomes more frequent, and the total number of data packets for intra-cluster data acquisition and forwarding, and inter-cluster routing relay increases significantly, leading to an increase in the average energy consumption of all protocols. GDRL-MRA, MRP-GTCO, and EECRAIFA have similar energy consumption levels, with their energy consumption curves consistently remaining at a low level. MH-LEACH has the highest energy consumption in all scenarios, attributed to its simple random cluster head election and inefficient multi-hop routing design. DEEC also exhibits high energy consumption due to the limitations of its energy-weighted random election. Although GDRL-MRA does not achieve an absolute lead in this metric, its energy consumption remains consistently low, maintaining a high energy efficiency.

[0105] (b), (c), and (d) illustrate the network lifetime of each protocol under different numbers of nodes. As the number of nodes increases, the HND (Highest Distributed Load) of each algorithm generally shows an upward trend. This is because with increased node density, the network has more nodes available for data forwarding and relay, thus distributing the overall load. Regarding FND (Fulfilled Load), GDRL-MRA outperforms other protocols across all node sizes, demonstrating optimal early energy balancing capabilities. In scenarios with 100 to 250 nodes, the algorithm's load balancing effect is fully realized. However, in high-density scenarios with 300 nodes, FND decreases due to topology complexity and communication overhead, but still maintains a leading advantage. GDRL-MRA's FND is approximately 16.7%, 23.5%, 41.9%, and 61.5% higher than EECRAIFA, MRP-GTCO, DEEC, and MH-LEACH protocols, respectively. In the HND metric, GDRL-MRA performs excellently in scenarios with 200 nodes or less, only slightly lagging behind EECRAIFA in high-density scenarios with 250 and 300 nodes, but its HND value remains at a high level and the trend is stable. Overall, GDRL-MRA shows an average improvement of approximately 16.1%, 33.7%, and 51.9% compared to MRP-GTCO, DEEC, and MH-LEACH protocols, respectively. In the NFT metric, GDRL-MRA performs similarly to MRP-GTCO, and both significantly outperform DEEC and MH-LEACH. MH-LEACH and DEEC, due to frequent clustering operations and inefficient routing strategies, lead to faster network energy consumption. Although MRP-GTCO and EECRAIFA improve performance through heuristic algorithm optimization, they perform poorly in energy balancing, resulting in the premature death of some nodes. GDRL-MRA, by utilizing heuristic algorithms for rational cluster head election and combining graph reinforcement learning for dynamic load balancing decisions, achieves significant improvements over comparative algorithms in core metrics such as FND and HND. The results demonstrate that GDRL-MRA can effectively balance network energy distribution, prevent premature node failures and energy black holes, and possess superior load balancing capabilities and network stability.

[0106] Employing the same inventive concept as the graph reinforcement learning-based wireless sensor network routing method, this embodiment also provides a corresponding graph reinforcement learning-based wireless sensor network routing device, including: The training dataset acquisition module is used to acquire the model training dataset, in which the samples are wireless sensor networks. The undirected graph construction module is used to model wireless sensor networks as undirected graphs. The graph nodes of the undirected graph include sensor nodes and base stations, and the adjacency matrix for adding self-connections is determined based on the undirected graph. The graph data construction module is used to determine the feature vectors of each graph node based on the physical attributes of each sensor node and the base station; the feature vectors of each graph node and the adjacency matrix with self-connection are used to form the graph data. The training module is used to train the graph reinforcement learning-based wireless sensor network routing model based on graph data, resulting in the trained graph reinforcement learning-based wireless sensor network routing model. The graph reinforcement learning-based wireless sensor network routing model includes graph convolutional networks and dual-depth Q-networks. During training, the graph convolutional network performs global feature aggregation and encoding on the graph data to obtain high-dimensional node embedding vectors for routing tasks. The dual-depth Q-network includes an evaluation network and a target network. The evaluation network generates Q-values ​​based on the high-dimensional node embedding vectors. These Q-values ​​are then processed by an action masking mechanism to obtain masked Q-values. The optimal action is determined based on the masked Q-values. The target network calculates the Q-value of the optimal action and generates target labels based on the Q-value of the optimal action, combined with the reward value and discount factor. The target labels are used to calculate the loss. The network weights of the graph convolutional network and the dual-depth Q-network are updated based on the loss. The path generation module is used to model the wireless sensor network to be processed as an undirected graph and determine the graph data based on the undirected graph. Any node to be planned in the wireless sensor network to be processed is marked as the current planning node. The graph data is input into the trained wireless sensor network routing model based on graph reinforcement learning. The graph data passes through the graph convolutional network and the evaluation network in sequence to obtain the next hop node of the current planning node. The next hop node is updated to the current planning node. This process is repeated until the next hop node is the base station, thus obtaining a deterministic multi-hop transmission path.

[0107] The wireless sensor network routing device based on graph reinforcement learning in this embodiment has the same inventive concept as the wireless sensor network routing method based on graph reinforcement learning described above. Therefore, the specific implementation of this device can be found in the embodiment section of the wireless sensor network routing method based on graph reinforcement learning described above, and its technical effects correspond to the technical effects of the above method, so it will not be repeated here.

[0108] The above descriptions are merely various embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A routing method for wireless sensor networks based on graph reinforcement learning, characterized in that, include: Obtain the model training dataset, wherein the samples in the model training dataset are wireless sensor networks; The wireless sensor network is modeled as an undirected graph, the graph nodes of which include sensor nodes and base stations, and an adjacency matrix for adding self-connections is determined based on the undirected graph. Based on the physical attributes of each sensor node and base station, the feature vector of each graph node is determined; the feature vector of each graph node and the adjacency matrix with added self-connection constitute graph data. The graph reinforcement learning-based wireless sensor network routing model is trained based on the graph data to obtain the trained graph reinforcement learning-based wireless sensor network routing model; the graph reinforcement learning-based wireless sensor network routing model includes a graph convolutional network and a dual-depth Q-network. During training, the graph convolutional network performs global feature aggregation and encoding on the graph data to obtain a high-dimensional node embedding vector for routing tasks; the dual-depth Q-network includes an evaluation network and a target network. The evaluation network generates Q-values ​​based on the high-dimensional node embedding vectors, and the Q-values ​​are processed by an action masking mechanism to obtain masked Q-values. The optimal action is determined based on the Q-value after masking; the target network calculates the Q-value of the optimal action and generates a target label based on the Q-value of the optimal action combined with the reward value and discount factor; the target label is used to calculate the loss; and the network weights of the graph convolutional network and the dual-depth Q-network are updated based on the loss. The wireless sensor network to be processed is modeled as an undirected graph, and graph data is determined based on the undirected graph. Any node to be planned in the wireless sensor network is denoted as the current planning node. The graph data is input into the trained wireless sensor network routing model based on graph reinforcement learning. The graph data is sequentially passed through a graph convolutional network and an evaluation network to obtain the next-hop node of the current planning node. The next-hop node is updated to the current planning node, and the process is repeated until the next-hop node is a base station, thus obtaining a deterministic multi-hop transmission path.

2. The method as described in claim 1, characterized in that, The high-dimensional node embedding vector is: in, For high-dimensional node embedding vectors, To add self-connected adjacency matrices, For degree matrix, for Activation function These are the feature vectors of each graph node. This is the weight matrix of the first layer of a graph convolutional network (GCN). This is the weight matrix of the second layer GCN in the graph convolutional network.

3. The method as described in claim 1, characterized in that, The state space of the dual-depth Q-network is: the high-dimensional node embedding vector.

4. The method as described in claim 1, characterized in that, The action space of the dual-depth Q-network is: in, In order to be in time step t The space of motion In order to be in time step t Select node for current node The action that serves as the next hop node. Indicates time step t The action of the current node selecting a base station as the next-hop node.

5. The method as described in claim 1, characterized in that, The reward for the dual-deep Q network is: in, In order to be in time step t The reward For combined reward value; When k=1, The remaining energy factor; when k=2, The distance progress factor; when k=3, Energy consumption factor; in, Next hop node The remaining energy, Next hop node initial energy, This represents the distance between the current node and the base station. This represents the distance between the next-hop node and the base station. For communication range, The energy consumed from the current node to the next hop node. is a coefficient.

6. The method as described in claim 1, characterized in that, The Q value after the mask is: in, The Q value after masking. To evaluate the Q-value generated by the network, In order to be in time step t The state space, In time step t The action, To evaluate the weight of the network, the actions in the effective action set are: the next hop node is a node that is alive and has a communication relationship with the current node within the communication range.

7. The method as described in claim 1, characterized in that, The target label is: in, For target tags, In order to be in time step t The reward As a discount factor, The Q value represents the optimal action.

8. The method as described in claim 1, characterized in that, The loss is: in, For loss, As expected, Target tags Evaluate the Q-value generated by the network. In order to be in time step t The state space, During the action at time step t, To evaluate the weights of the network.

9. A wireless sensor network routing device based on graph reinforcement learning, characterized in that, include: The training dataset acquisition module is used to acquire the model training dataset, wherein the samples in the model training dataset are wireless sensor networks; An undirected graph construction module is used to model the wireless sensor network as an undirected graph, wherein the graph nodes of the undirected graph include sensor nodes and base stations, and to determine the adjacency matrix for adding self-connections based on the undirected graph; The graph data construction module is used to determine the feature vector of each graph node based on the physical attributes of each sensor node and the base station; the feature vector of each graph node and the adjacency matrix with added self-connection constitute the graph data. The training module is used to train the graph reinforcement learning-based wireless sensor network routing model based on the graph data to obtain the trained graph reinforcement learning-based wireless sensor network routing model; the graph reinforcement learning-based wireless sensor network routing model includes a graph convolutional network and a dual-depth Q-network. During training, the graph convolutional network performs global feature aggregation and encoding on the graph data to obtain a high-dimensional node embedding vector for routing tasks; the dual-depth Q-network includes an evaluation network and a target network. The evaluation network generates Q-values ​​based on the high-dimensional node embedding vectors, and the Q-values ​​are processed by an action masking mechanism to obtain masked Q-values. The optimal action is determined based on the Q-value after masking; the target network calculates the Q-value of the optimal action and generates a target label based on the Q-value of the optimal action combined with the reward value and discount factor; the target label is used to calculate the loss; and the network weights of the graph convolutional network and the dual-depth Q-network are updated based on the loss. The path generation module is used to model the wireless sensor network to be processed as an undirected graph and determine the graph data based on the undirected graph. Any node to be planned in the wireless sensor network to be processed is designated as the current planning node. Graph data is input into the trained wireless sensor network routing model based on graph reinforcement learning. The graph data is sequentially passed through a graph convolutional network and an evaluation network to obtain the next-hop node of the current planning node. The next-hop node is updated to the current planning node. This process is repeated until the next-hop node is a base station, thus obtaining a deterministic multi-hop transmission path.

Images