Underwater wireless sensor network path sensing routing method based on deep reinforcement learning

Abstract

The invention relates to an underwater wireless sensor network path sensing routing method based on deep reinforcement learning, which comprises the following steps that: firstly, a node constructs and periodically updates a transmission preference model based on local and neighbor node interaction information; secondly, deploying a deep reinforcement learning model at each underwater sensor node to perform distributed routing strategy learning; and finally, generating a global guide vector by the sink node according to the routing path information of the received data packet, reversely spreading the global guide vector to the source node, fusing the global guide vector with a local transmission preference vector of the node to generate a guide reward, optimizing the deep reinforcement learning model, and updating a routing strategy. According to the method, the problems of difference and complexity of underwater transmission tasks can be solved, and the network data transmission efficiency and the overall service quality are improved in combination with local preference and global guidance while the node online learning is kept to adapt to the dynamic underwater environment.

Description

Technical Field

[0001] This invention relates to a path-aware routing method for underwater wireless sensor networks based on deep reinforcement learning, belonging to the field of underwater acoustic wireless sensor network communication support technology. Background Technology

[0002] With the rise of the underwater Internet of Things (IoT), underwater wireless sensor networks have become a core component of the IoT, widely used in marine environmental monitoring, resource exploration, target tracking and positioning, and other fields, constituting a key infrastructure for the research, construction, and development of the ocean. In underwater data transmission, routing protocols are responsible for ensuring the correct transmission of data from source nodes to destination nodes, making them a crucial component. Given the diverse needs of marine applications, transmission tasks exhibit significant differences and complexities. Therefore, designing multi-objective routing schemes for underwater applications has become a key technical issue. Currently, representative multi-objective routing methods in underwater wireless sensor networks are mainly divided into two categories: optimization-based methods and learning-based methods. Traditional optimization-based routing typically relies on predefined models and static weight configurations, making it difficult to adapt to the dynamic characteristics of the underwater environment; while learning-based methods achieve autonomous optimization of routing strategies through continuous interaction between nodes and the environment. However, current learning-based routing schemes, while maintaining distributed learning, do not yet possess global perception and more fine-grained multi-objective optimization capabilities.

[0003] To design routing schemes for multi-target underwater wireless sensor networks, researchers both domestically and internationally have proposed various routing schemes, and relevant literature is as follows: 1. In 2018, Muhammad Faheem et al. proposed a novel QoS-aware evolutionary routing protocol, QERP, in their paper "QERP: Quality-of-Service (QoS) Aware Evolutionary Routing Protocol for Underwater Wireless Sensor Networks." QERP successfully leverages highly reliable link quality information between communication nodes (CHs) in a greedy manner, leading to successful transmission to the receiver. Furthermore, due to its QoS-aware shortest path selection mechanism, it effectively reduces data path loops, network latency, and energy consumption.

[0004] 2. In 2020, Fang et al. proposed a Q-learning-assisted ant colony routing protocol for Underwater Acoustic Sensor Networks (UWSNs) in their paper "QLACO: Q-learning Aided Ant Colony Routing Protocol for Underwater Acoustic Sensor Networks." This protocol utilizes artificial ants to determine a globally optimal route. Furthermore, they designed reward functions related to latency and lifespan in the Q-learning technique. QLACO improves the lifespan and packet delivery rate of UWSNs while reducing energy consumption.

[0005] 3. In 2019, Li et al. proposed a Reinforcement Learning-Based Routing Protocol (RCAR) for Congestion-Avoided Underwater Acoustic Sensor Networks in their paper "RCAR". The reward function in RCAR reinforcement learning considers congestion and energy factors to make adequate routing decisions. A dynamic virtual routing pipeline with a variable radius is introduced to accelerate the algorithm's convergence speed. The RCAR protocol optimizes training convergence speed, routing energy efficiency, and end-to-end latency.

[0006] 4. In 2024, Liu et al., in their paper "CLORP: Cross-Layer Opportunistic Routing Protocol for Underwater Sensor Networks Based on Multiagent Reinforcement Learning," proposed CLORP, a MARL-based underwater sensor network protocol, to address the problems of low data transmission rates, high latency, and limited energy in underwater sensor networks. This protocol combines the decision-making capabilities of deep reinforcement learning with the idea of ​​opportunistic routing, sequentially selecting a set of suitable neighbors as potential forwarding nodes. In designing the MARL reward function, the protocol incorporates cross-layer information to design two reward functions for successful and failed packet transmissions, thereby improving the performance of the routing protocol. The CLORP routing algorithm improves data transmission rates, reduces network latency, and increases network lifetime.

[0007] 5. In 2024, Wang et al. proposed a Multi-Agent Reinforcement Learning-Based Routing Protocol for Underwater Wireless Sensor Networks With Value of Information (MARV) in their paper "Multi-Agent Reinforcement Learning-Based Routing Protocol for Underwater Wireless Sensor Networks With Value of Information". In order to distinguish the differences in transmission requirements, information value (VoI) is introduced to characterize the importance of data and reflect the requirements for real-time characteristics. The proposed protocol performs well in the transmission of different data. Summary of the Invention

[0008] The technical problem this invention aims to solve is that existing underwater routing protocols are ill-suited to complex dynamic environments and cannot meet diverse transmission requirements. This invention proposes a path-aware differential routing method for underwater wireless sensor networks based on deep reinforcement learning, achieving synergy between node autonomous decision-making and global network optimization. To achieve the above objective, this invention is implemented through the following technical solution: A path-aware routing method for underwater wireless sensor networks based on deep reinforcement learning includes the following steps: Step 1: Construction of underwater sensor node transmission preference model; Each underwater sensor node generates a transmission preference vector based on the interaction information between itself and its neighboring nodes within the sliding time window, and performs an update operation on the preference vector every M decision cycles. Step 2: Distributed routing decision based on deep reinforcement learning; Each underwater sensor node deploys a deep reinforcement learning model, which makes routing and forwarding decisions based on the state information of the node and its neighbors, and stores the resulting interaction experience in a local experience replay buffer; the model samples experience data from the experience replay buffer to update the online network parameters. Step 3: Path-aware global reward and optimized node routing strategy methods; After receiving a data packet, the aggregation node extracts its routing path information and generates a path-level guidance vector that integrates the historical reliability, historical latency, and historical energy consumption of the path. This vector is propagated back along the original path to the source node, which then weights and merges it with its local preference vector to form a global guidance reward. This reward is then used to correct the reward value of the corresponding experience in the local experience replay buffer, thereby guiding the node's routing strategy to optimize towards building a high-quality end-to-end path.

[0009] In step one above, the method for constructing the transmission preference model of the underwater sensor node is as follows: Each node continuously monitors and summarizes its interaction statistics with neighboring nodes within a sliding time window. The data includes packet delivery rate, average latency, and remaining energy. These indicators are mapped to a unified utility function for calculation. The resulting utility values ​​are normalized using the Softmax function, and the output is a stable preference weight. The node transmission preference vector composed of these weights can effectively characterize the differences between nodes in terms of transmission latency, remaining energy, and link quality, thereby enabling each node to dynamically adjust its priority for transmission reliability, latency, and energy efficiency based on its own operating status.

[0010] In step two above, the distributed routing decision method based on deep reinforcement learning is as follows: Each node deploys a deep reinforcement learning model as its local routing agent, forming a fully distributed decision-making architecture; the agent's state space encompasses the link quality φ between the node and its neighbors. i Location information P of neighboring nodes i The key features include the location 'd' of the destination node; its action space is defined as selecting the next-hop forwarding node; its reward function is designed based on local information such as transmission success rate and relative distance to the destination node. Each node will use its decision-making experience ( s t , a t , r t, s t+1 The model parameters are stored in a local experience replay buffer and are updated asynchronously to evolve a routing strategy that adapts to the local environment.

[0011] In step three above, the global guidance reward is calculated as follows: When the aggregation node successfully receives a data packet from the source node, it extracts the complete end-to-end transmission path information from the packet header. Based on this path information, it calculates and generates a path-level guidance vector, which integrates indicators such as historical path reliability, historical latency, and historical energy consumption.

[0012] In step three above, the global guidance reward optimization node routing strategy is as follows: The aggregation node propagates the path-level guidance vector back along the original path to the source node; upon receiving it, the source node weighted and merges it with its local transmission preference vector to generate a guidance reward; based on this, the source node corrects the corresponding historical experience in its local experience replay buffer, specifically by combining the guidance reward with the experience tuple (…). s t , a t , r t, st+1 The original reward r t The field is then weighted and summed, and the resulting experience guides the strategy in subsequent model training to evolve in a direction conducive to building high-quality end-to-end paths.

[0013] By employing the above-mentioned technical means, the beneficial effects of this invention are as follows: Through distributed deep reinforcement learning, nodes can autonomously adapt to the dynamic changes of the underwater acoustic channel without the need for a pre-defined precise network model. A node transmission preference model captures the differentiated transmission needs of nodes in terms of link quality, latency, and energy. An innovative global reward guidance mechanism initiated by the sink node is introduced, feeding back the end-to-end path quality to the source node, effectively correcting its local decisions and achieving synergistic optimization of local efficiency and overall network performance. Attached Figure Description

[0014] Figure 1 This is a schematic diagram of the transmission preference mechanism of the sensor node in this invention; Figure 2 This is a schematic diagram of deep reinforcement learning routing modeling according to the present invention; Figure 3 This is a schematic diagram of the path-aware global reward guidance mechanism of the present invention. Detailed Implementation

[0015] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0016] A path-aware routing method for underwater wireless sensor networks based on deep reinforcement learning, comprising the following steps: Step 1: As Figure 1 The underwater sensor node transmission preference model can be constructed as follows: Each underwater sensor node continuously monitors and summarizes its interaction statistics with neighboring nodes within a sliding time window. This data includes, but is not limited to, packet delivery rate, average end-to-end latency, and the node's remaining energy. Subsequently, the node inputs these heterogeneous metrics into a pre-designed unified utility function for standardized calculation. To further improve the comparability and stability of the vector, the calculated utility values ​​are normalized using a Softmax function, ultimately outputting a set of preference weights that sum to 1. These weights together constitute the node's transmission preference vector. The key to this vector is its ability to dynamically capture the comprehensive state differences of nodes across multiple dimensions such as geographical location, remaining energy, and link quality. This allows each node to autonomously adjust its emphasis on different objectives such as transmission reliability, low latency, and energy efficiency based on its constantly changing operational status.

[0017] Step Two: As Figure 2The deployment of the distributed routing strategy based on deep reinforcement learning involves deploying a deep reinforcement learning model on each underwater sensor node in the network as its local routing agent, thus forming a fully distributed decision-making architecture. The state space of this agent is carefully designed to encompass key features of the underwater environment, such as the link quality φ between the current node and its neighbors. i Location information P of neighboring nodes i The location d of the destination or sink node is also considered. Its action space is directly defined as selecting one of the available neighbor nodes as the next-hop forwarding node. Furthermore, the agent's reward function is primarily designed based on local information such as transmission success rate and relative distance to the destination node. Each node will record the interaction experience generated from each decision in tuples (…). s t , a t , r t, s t+1 The data is stored in its local experience replay buffer in the form of ) and these experiences are sampled asynchronously to update the model parameters, ultimately enabling each node to autonomously evolve an efficient routing strategy that adapts to its local dynamic environment.

[0018] Step 3: This step optimizes local decision-making by introducing a global perspective, specifically including the following two consecutive processes: 3.1 Calculation process of global guidance rewards: See Figure 3 This process is triggered when the sink node successfully receives a data packet from a source sensor node. The sink node first extracts the complete end-to-end transmission path information from the packet header. This information accurately records the sequence of all intermediate nodes the data packet traverses from the source node to the sink node. Subsequently, based on this path information and historical transmission statistics, the sink node calculates and generates a path-level guidance vector. This vector is a comprehensive evaluation metric, encompassing quantitative assessments of multiple dimensions, including historical path reliability, historical latency, and historical energy consumption.

[0019] 3.2 Strategy optimization execution process: Following the previous process, the aggregation node propagates the calculated path-level guidance vector back to the source node of the data packet in the opposite direction to the original transmission path. Upon receiving this guidance vector, the source node weighted and merges it with the local transmission preference vector generated in step one to generate a more instructive guidance reward. Subsequently, the source node uses this globally feedback guidance reward to correct the corresponding historical experience in its local experience replay buffer. Specifically, this involves combining the guidance reward with the local reward in the original experience tuple.r t The fusion is performed using a weighted summation method, and the empirical tuple is then updated. s t , a t , r t, s t+1 The "reward" field in the model. In this way, during the subsequent training of the model, these corrected historical experiences rich in global information will guide the deep reinforcement learning model's strategy to continuously optimize and evolve in a direction that is conducive to forming a high-quality end-to-end transmission path.

Claims

1. A path-aware routing method for underwater wireless sensor networks based on deep reinforcement learning, characterized in that: Includes the following steps: Step 1: Construction of underwater sensor node transmission preference model; Each underwater sensor node generates a transmission preference vector based on the interaction information between itself and its neighboring nodes within the sliding time window, and performs an update operation on the preference vector every M decision cycles. Step 2: Distributed routing decision based on deep reinforcement learning; Each underwater sensor node deploys a deep reinforcement learning model, which makes routing and forwarding decisions based on the state information of the node and its neighbors, and stores the resulting interaction experience in a local experience replay buffer; the model samples experience data from the experience replay buffer to update the online network parameters. Step 3: Path-aware global reward and optimized node routing strategy methods; After receiving a data packet, the aggregation node extracts its routing path information and generates a path-level guidance vector that integrates the historical reliability, historical latency, and historical energy consumption of the path. This vector is propagated back along the original path to the source node, which then weights and fuses it with its local transmission preference vector to form a global guidance reward. This reward is then used to correct the reward value of the corresponding experience in the local experience replay buffer, thereby guiding the node's routing strategy to optimize towards building a high-quality end-to-end path.

2. The underwater wireless sensor network path-aware routing method based on deep reinforcement learning according to claim 1, characterized in that, In step one, the method for constructing the transmission preference model of the underwater sensor node is as follows: Each node continuously monitors and summarizes the interaction statistics between itself and its neighboring nodes within the sliding time window. The data includes packet delivery rate, average delay, and remaining energy. The interaction information is mapped to a unified utility function for calculation. The obtained utility value is normalized by the Softmax function, and the output is a stable preference weight. The node transmission preference vector, composed of these weights, can effectively characterize the differences between nodes in terms of transmission latency, remaining node energy, and link quality, thereby enabling each node to dynamically adjust its priority for transmission reliability, latency, and energy efficiency based on its own operating status.

3. The underwater wireless sensor network path-aware routing method based on deep reinforcement learning according to claim 1, characterized in that, The distributed routing decision-making method based on deep reinforcement learning in step two is as follows: Each node deploys a deep reinforcement learning model as its local routing agent, forming a fully distributed decision-making architecture; the state space of the agent covers the link quality φ between the node and its neighbors. i Location information P of neighboring nodes i Key features such as the location d of the target node; Its action space is defined as selecting the next-hop forwarding node; Its reward function is designed based on local information such as transmission success rate and relative distance to the destination node, and each node will use its decision-making experience ( s t , a t , r t, s t+1 The model parameters are stored in a local experience replay buffer and are updated asynchronously to evolve a routing strategy that adapts to the local environment.

4. The underwater wireless sensor network path-aware routing method based on deep reinforcement learning according to claim 3, characterized in that, In step two, the deep reinforcement learning model is a dual-deep Q-network.

5. The underwater wireless sensor network path-aware differential routing method based on deep reinforcement learning according to claim 1, characterized in that, In step three, the calculation of the path-level guidance vector is based on at least one of the following: path historical reliability, path historical delay, and path historical energy consumption.

6. The underwater wireless sensor network path-aware routing method based on deep reinforcement learning according to claim 1, characterized in that, The global guidance reward calculation method in step three is as follows: When the aggregation node successfully receives the data packet from the source node, it extracts the complete end-to-end transmission path information from the header of the data packet; based on the path information, it calculates and generates a path-level guidance vector, which integrates indicators such as path historical reliability, historical latency, and historical energy consumption.

7. The underwater wireless sensor network path-aware routing method based on deep reinforcement learning according to claim 6, characterized in that, The global guidance reward optimization node routing strategy method in step three is as follows: the aggregation node propagates the path-level guidance vector back along the original path to the source node; after receiving it, the source node performs weighted fusion with the output of the local preference model to generate a guidance reward; Accordingly, the source node corrects the corresponding historical experience in its local experience replay buffer, specifically by modifying the bootstrap reward and the experience tuple ( s t , a t , r t, s t+1 The original reward r t Calculate the weighted sum and update the field; The experience gained from this correction guides the strategy in subsequent model training to evolve in a direction that is conducive to building high-quality end-to-end paths.

Images