Hybrid dynamic UWSNs routing protocol optimization method and system based on DDQN

Abstract

A hybrid dynamic UWSNs routing protocol optimization method and system based on DDQN relates to the field of dynamic topology UWSNs routing optimization. This method addresses the existing problems of dynamic UWSNs, such as difficulty in establishing routing links and poor stability due to network sparsity, node mobility, and high positioning difficulty, while active routing protocols and opportunistic routing protocols face problems such as poor delivery rates and high energy consumption, respectively. The method includes: designing a forwarding target evaluation mechanism based on the DDQN protocol framework, introducing a dual Q network mechanism into the protocol; combining active unicast forwarding and passive opportunistic forwarding into one, using unicast forwarding. If unicast fails, it is converted to passive opportunistic forwarding. If, within a predetermined time window, a node detects that its selected next-hop node has successfully forwarded the data packet, it is rewarded; otherwise, the node is penalized to complete the optimization. This method is also applicable to fields such as node optimization.

Description

Technical Field

[0001] The present invention relates to the technical field of dynamic topology UWSNs routing optimization, and in particular to a hybrid dynamic UWSNs routing protocol optimization method and system based on DDQN. Background Art

[0002] Due to the mobility of nodes in UWSNs under dynamic topologies, packet forwarding links are often time-sensitive and unstable. Furthermore, the difficulty in positioning caused by node mobility also makes routing decisions lack convenient auxiliary conditions. In this case, the main routing forwarding strategies have changed to several aspects: first, based on depth information, considering that the aggregation nodes are mainly deployed on the water surface, data packets are preferentially transmitted to shallower nodes; second, AUVs are used as data mules. Compared with ordinary nodes, AUVs have more flexible size requirements and are more maneuverable. Combined with inertial navigation technology and path planning technology, they can take on the role of underwater aggregation nodes; third, active movement is relied upon for void repair. Generally, UWSN nodes are equipped with pressure chambers for easy recovery, so they can adjust their own depth. This condition can be used to allow void nodes to attempt void repair.

[0003] In response to the above content, the following protocol is proposed in the existing design: the Depth-Based Routing (DBR) protocol, which does not rely on positioning information and only obtains the depth for forwarding decisions through the pressure gauge on the node. Since the target aggregation nodes are mostly deployed on the water surface, nodes in the DBR that are shallower than the previous hop can participate in forwarding.

[0004] The Geographic and Opportunistic Routing with Depth Adjustment Based Topology Control for Communication Recovery (GEDAR) protocol takes into account the role of node ups and downs in void repair, requiring the node itself to be able to adjust the depth according to instructions, and recover the forwarding link by adjusting the depth of the void node. Considering the flooding of DBR forwarding, the existing technology proposes a DBR (Weighting Depth And Forwarding Area Division DBR, WDFAD-DBR) protocol based on weighted forwarding area division, which divides the main forwarding area and forwarding suppression area to reduce redundant transmission and optimize the overall energy consumption of the network. CoutinhoRW L proposes a rotating forwarding priority mechanism to optimize the forwarding priority of opportunistic routing protocols, which helps to balance energy consumption.

[0005] The Distance Vector Based Opportunistic Routing (DVOR) protocol uses a query mechanism to obtain packet distance and hop count information to construct a distance vector, effectively optimizing routing holes and detours. The Cluster Based Depth Source Selection Routing (CBDS2R) protocol incorporates communication link quality into routing selection, resulting in improved communication reliability.

[0006] Taking into account the dynamic and time-varying nature of underwater topology, a reinforcement learning-based opportunistic routing (RLOR) protocol has been proposed in the prior art. Before forwarding a packet, neighbor discovery is performed. Neighboring nodes transmit their Q-values ​​back to the source node, and the optimal next-hop forwarder is selected based on the maximum Q-value principle. Simulation results show that the RLOR protocol has a lower energy tax than other opportunistic routing protocols. Considering the similarities between UWSN data forwarding and multi-agent environments, a multi-agent reinforcement learning (MARL)-based underwater sensor network routing protocol has been proposed. This protocol calculates the time and energy costs of communication between agents and trains the network to optimize node routing decisions, thereby optimizing delivery rate and latency. This protocol introduces information value and a multi-level forwarding mechanism. A reward function is designed based on the number of data retransmissions. Nodes use dedicated broadcast packets to calculate specific rewards between neighboring nodes. Subsequently, Q-values ​​are calculated based on these rewards during forwarding, and packets are forwarded to the node with the highest Q-value, achieving excellent latency optimization. Considering the contribution of mobile AUVs to data forwarding, an AUV path is generated based on the Proximal Policy Optimization (PPO) algorithm. Ordinary nodes prioritize forwarding data to the AUV, which then acts as a data mule to ultimately transmit the data to the sink, achieving significant energy savings. The Energy Efficient Depth Based Opportunistic Routing with Void Avoidance (EEDOR-VA) protocol is proposed. This protocol utilizes a complete link exploration mechanism. During the node route exploration phase, nodes forward sink packets to obtain forwarding hop counts. Routing links are then selected based on Q-Learning, optimizing forwarding energy consumption.

[0007] Based on the Energy Efficient Guiding Network Based Routing (EEGNBR) protocol, an opportunistic protocol based on Q-Learning decision-making is designed based on a single-target forwarding mechanism. Before each forwarding, neighbor information is exchanged through a handshake and the optimal forwarding target is selected, achieving good energy-saving effects. To address the redundant transmission problem of opportunistic forwarding, a Reinforcement Learning-based Opportunistic Routing Protocol using Depth Information for Energy-efficient (DROR) protocol is proposed. This protocol trains node holding times based on a reinforcement learning mechanism, thereby better suppressing redundant forwarding in opportunistic forwarding. A hierarchical routing protocol for UWSNs based on node mobility prediction selects more stable and less-latent forwarding targets by considering node mobility. Simulation results show that this protocol can improve delivery rates and reduce end-to-end latency in mobile scenarios.

[0008] Dynamic UWSNs suffer from time-varying topologies and high localization challenges. This disrupts the stability of forwarding links, leading to significant degradation in delivery rates for proactive protocols. While opportunistic forwarding protocols can maintain delivery rates, they incur significant energy costs. The neighbor table in dynamic UWSNs is time-sensitive, and the forwarding potential of the same node can change over time. Traditional Q-Learning decisions are heavily influenced by the V value, making them prone to misleading and detrimental to selecting the optimal forwarding target. Deep Q Networks (DQNs) suffer from overestimation, making training difficult and difficult to converge in the volatile environments of UWSNs. Therefore, this protocol designs a forwarding target evaluation mechanism based on DDQN. In its decision-making, DDQN outputs the "forwarding potential of nodes with this information" rather than "selecting a specific node as the forwarding target." This mitigates the impact of frequent changes in the neighbor table and prioritizes selecting nodes with more stable communication and reliable forwarding paths among currently available neighbors.

[0009] In summary, dynamic UWSNs are characterized by sparse networks and time-varying topologies, leading to unstable forwarding links. Active routing protocols struggle to achieve high delivery rates in this scenario, while the flooding nature of opportunistic forwarding incurs significant energy overhead. Furthermore, node mobility makes it even more difficult to guarantee connectivity in sparse networks. Therefore, it is necessary to design a reasonable redundancy suppression mechanism to conserve energy, and to combine it with a hole repair mechanism to further improve network delivery rates. Summary of the Invention

[0010] The present invention aims to solve the existing problems in the dynamic UWSNs technology, such as the difficulty in establishing routing links and poor stability caused by network sparsity, node mobility and high positioning difficulty, while the active routing protocol and the opportunistic routing protocol respectively face the problems of poor delivery rate and high energy consumption. To solve the above technical problems, the present invention is implemented through the following technical solutions:

[0011] Solution 1: The present invention proposes a hybrid dynamic UWSNs routing protocol optimization method based on DDQN, which includes the following steps:

[0012] Step 1: Design a forwarding target evaluation mechanism based on the DDQN protocol framework. Introduce a dual Q network mechanism into the DDQN protocol to separate behavior decision-making and behavior evaluation.

[0013] Step 2: Combine active unicast forwarding and passive opportunistic forwarding into one. The forwarding process is as follows: first, unicast forwarding is used. If unicast fails, it is converted to passive opportunistic forwarding. If, within a predetermined time window, the node detects that its selected next-hop node has successfully forwarded the data packet, it is rewarded according to the mechanism of the online learning phase; otherwise, if the expected forwarding behavior is not detected, the node is punished, completing the DDQN hybrid dynamic UWSNs routing protocol optimization.

[0014] Furthermore, a preferred embodiment is provided, wherein the reward function is calculated as follows:

[0015]

[0016] where R success and R fail They represent the instant rewards when the forwarding of this hop is successful and when the forwarding of this hop fails; g is the reward offset factor, N choice and N max Respectively represent the number of neighbors of the selected target and the preset maximum number of neighbors; Dep choice and Dep max They represent the depth of the selected target and the maximum deployment depth respectively; E choice and E initial Used to represent the remaining energy of the target and the initial energy of the node; D choice and R represent the distance between the target and the node and the maximum communication distance respectively; V choice and V max They represent the relative moving speed between the target and the node and the maximum relative moving speed of the node respectively; α1, α2, α3, α4, and α5 represent the weights of each influencing factor, satisfying α1+α2+α3+α4+α5=1.

[0017] Furthermore, a preferred implementation is provided, wherein step 2 also includes the step of performing an opportunistic broadcast hole repair mechanism on the unicast forwarding protocol.

[0018] Furthermore, a preferred embodiment is provided, wherein the opportunistic broadcast hole repair mechanism requires that any node that is not designated as a forwarding target but has a depth shallower than the sender is allowed to participate in opportunistic forwarding.

[0019] Furthermore, a preferred implementation is provided, in which the DDQN protocol in step 1 further includes three steps: neighbor discovery, cluster formation, and data forwarding.

[0020] Furthermore, a preferred embodiment is provided, wherein step 2 also includes the steps of depth adjustment and cavity repair mechanism of channel axis transmission.

[0021] Solution 2: A hybrid dynamic UWSNs routing protocol optimization system based on DDQN, the system comprising:

[0022] The dual-Q network mechanism module is used to design a forwarding target evaluation mechanism based on the DDQN protocol. The dual-Q network mechanism is introduced into the DDQN protocol to separate behavior decision-making and behavior evaluation.

[0023] The hybrid forwarding module is used to combine active unicast forwarding and passive opportunistic forwarding into one. Its forwarding process is as follows: first, unicast forwarding is used. If unicast fails, it is converted to passive opportunistic forwarding. If, within a predetermined time window, the node detects that its selected next-hop node has successfully forwarded the data packet, it is rewarded according to the mechanism of the online learning phase; conversely, if the expected forwarding behavior cannot be detected, the node is punished, completing the DDQN hybrid dynamic UWSNs routing protocol optimization.

[0024] Solution 3: A computer device includes a memory and a processor, wherein the memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes any one of the methods described in Solution 1.

[0025] Solution 4: Computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of any one of the methods in Solution 1 are implemented.

[0026] The present invention is beneficial in that:

[0027] In response to the problem that dynamic topology UWSNs use active routing with poor delivery rate and opportunistic routing with high energy consumption, the present invention proposes a hybrid dynamic UWSNs routing protocol optimization method based on DDQN. The algorithm combines the advantages of active unicast forwarding and passive opportunistic forwarding, giving priority to unicast transmission in forwarding to reduce energy consumption, and converting to opportunistic forwarding for routing repair after unicast fails. In order to improve the success rate of unicast transmission, the algorithm trains the next-hop routing forwarding target selection strategy based on DDQN, taking into account the distance, speed, depth and energy factors of itself and its neighbors, and avoiding the selection of forwarding targets with poor reliability. At the same time, in response to the problem that sparse networks have poor connectivity and forwarding holes are sometimes unrepairable, a hole repair mechanism based on depth adjustment and channel axis transmission is proposed. When a node faces a hole that cannot be repaired by flooding forwarding alone, the node actively adjusts its own depth until it reaches the channel axis for long-distance transmission or a neighbor successfully forwards the data packet during this process. Through simulation experiments, the proposed protocol outperforms the unicast protocol in terms of delivery rate and the opportunistic protocol in terms of energy consumption, and significantly improves the data delivery rate in sparse deployment cases.

[0028] The present invention should also be applicable to the fields of node optimization and sparse deployment. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 Schematic diagram of DDQN decision and update according to the eleventh embodiment.

[0030] Figure 2 This is a schematic diagram of the experience replay mechanism described in Implementation 11.

[0031] Figure 3 This is a schematic diagram of DDQN forwarding priority calculation according to the eleventh embodiment.

[0032] Figure 4 This is a schematic diagram of neighbor failure caused by neighbor node movement as described in the eleventh embodiment.

[0033] Figure 5 This is a schematic diagram of the opportunistic broadcast repair mechanism described in Implementation 11.

[0034] Figure 6 This is the ocean sound speed profile curve and the corresponding sound line diagram described in embodiment 11.

[0035] Figure 6 In the figure, (a) is a schematic diagram of the sound velocity profile curve, and (b) is a schematic diagram of the sound line.

[0036] Figure 7 Schematic diagram of the depth adjustment and channel axis transmission cavity repair mechanism described in embodiment eleven.

[0037] Figure 8 This is a schematic diagram of the DDQN routing protocol data packet structure described in the eleventh embodiment.

[0038] Figure 9 This is a flowchart of the DDQN routing protocol described in the eleventh embodiment.

[0039] Figure 10 This is a schematic diagram of the training loss described in the eleventh embodiment.

[0040] Figure 10 In the figure, (a) is a diagram of Training Loss, and (b) is a diagram of Decision Reward.

[0041] Figure 11 This is a statistical diagram of the data packet delivery rate described in the eleventh embodiment.

[0042] Figure 12 This is a statistical diagram of end-to-end delay according to the eleventh embodiment.

[0043] Figure 13 This is a statistical diagram of energy tax as described in Implementation Method 11.

[0044] Figure 14 This is a schematic diagram of the impact of moving speed on the protocol delivery rate described in the eleventh embodiment.

[0045] Figure 15 Schematic diagram of the impact of mobile speed on protocol end-to-end delay described in embodiment 11.

[0046] Figure 16 This is a schematic diagram of the impact of moving speed on the protocol energy tax described in the eleventh embodiment. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the implementation methods of this application clearer, the technical solutions in the implementation methods of this application will be clearly and completely described below in combination with the drawings in the implementation methods of this application. Obviously, the described implementation methods are only part of the implementation methods of this application, not all of the implementation methods.

[0048] Implementation 1: This implementation provides a hybrid dynamic UWSNs routing protocol optimization method based on DDQN, the method comprising the following steps:

[0049] Step 1: Design a forwarding target evaluation mechanism based on the DDQN protocol framework. Introduce a dual Q network mechanism into the DDQN protocol to separate behavior decision-making and behavior evaluation.

[0050] Step 2: Combine active unicast forwarding and passive opportunistic forwarding into one. The forwarding process is as follows: first, unicast forwarding is used. If unicast fails, it is converted to passive opportunistic forwarding. If, within a predetermined time window, the node detects that its selected next-hop node has successfully forwarded the data packet, it is rewarded according to the mechanism of the online learning phase; otherwise, if the expected forwarding behavior is not detected, the node is punished, completing the DDQN hybrid dynamic UWSNs routing protocol optimization.

[0051] Implementation method 2. This implementation method further limits the hybrid dynamic UWSNs routing protocol optimization method based on DDQN described in implementation method 1. Step 2 also includes the step of designing a reward function based on the number of neighbors, depth, residual energy, and distance and relative speed of the forwarding target to the node.

[0052] Implementation method 3: This implementation method further limits the hybrid dynamic UWSNs routing protocol optimization method based on DDQN described in implementation method 1. The calculation method of the reward function is:

[0053]

[0054] where R success and R fail They represent the instant rewards when the forwarding of this hop is successful and when the forwarding of this hop fails; g is the reward offset factor, N choice and N max Respectively represent the number of neighbors of the selected target and the preset maximum number of neighbors; Dep choice and Dep max They represent the depth of the selected target and the maximum deployment depth respectively; E choice and E initial Used to represent the remaining energy of the target and the initial energy of the node; D choice and R represent the distance between the target and the node and the maximum communication distance respectively; V choice and V max They represent the relative moving speed between the target and the node and the maximum relative moving speed of the node respectively; α1, α2, α3, α4, and α5 represent the weights of each influencing factor, satisfying α1+α2+α3+α4+α5=1.

[0055] Implementation method 4: This implementation method further limits the DDQN-based hybrid dynamic UWSNs routing protocol optimization method described in implementation method 1. Step 2 also includes the step of performing an opportunistic broadcast hole repair mechanism on the unicast forwarding protocol.

[0056] Implementation method 5. This implementation method further limits the DDQN-based hybrid dynamic UWSNs routing protocol optimization method described in implementation method 1. The opportunistic broadcast hole repair mechanism requires that any node that is not designated as a forwarding target but has a shallower depth than the sender is allowed to participate in opportunistic forwarding.

[0057] Implementation method 6: This implementation method further limits the DDQN-based hybrid dynamic UWSNs routing protocol optimization method described in implementation method 1. In step 1, the DDQN protocol further includes three steps: neighbor discovery, cluster formation, and data forwarding.

[0058] Implementation method seven: This implementation method further limits the DDQN-based hybrid dynamic UWSNs routing protocol optimization method described in implementation method one. Step 2 also includes the steps of depth adjustment and channel axis transmission hole repair mechanism.

[0059] Implementation 8: This implementation proposes a hybrid dynamic UWSNs routing protocol optimization system based on DDQN, the system comprising:

[0060] The dual-Q network mechanism module is used to design a forwarding target evaluation mechanism based on the DDQN protocol. The dual-Q network mechanism is introduced into the DDQN protocol to separate behavior decision-making and behavior evaluation.

[0061] The hybrid forwarding module is used to combine active unicast forwarding and passive opportunistic forwarding into one. Its forwarding process is as follows: first, unicast forwarding is used. If unicast fails, it is converted to passive opportunistic forwarding. If, within a predetermined time window, the node detects that its selected next-hop node has successfully forwarded the data packet, it is rewarded according to the mechanism of the online learning phase; conversely, if the expected forwarding behavior cannot be detected, the node is punished, completing the DDQN hybrid dynamic UWSNs routing protocol optimization.

[0062] Implementation method 9. This implementation method proposes a computer device, including a memory and a processor, wherein the memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes the method described in any one of implementation methods 1 to 7.

[0063] Embodiment 10: This embodiment provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of the method described in any one of Embodiments 1 to 7 are implemented.

[0064] Implementation 11: This implementation provides an example, which is used to explain the above implementations 1 to 10. Specifically, the example is as follows:

[0065] See also Figures 1 to 16 To illustrate this embodiment, the method described in this embodiment includes the following steps:

[0066] In value-based reinforcement learning algorithms, evaluating the current situation and behavior is crucial for guiding algorithmic decision-making and promoting network convergence. As a value-based algorithm, DQN utilizes deep neural networks to learn a criticism function, or critic, to adaptively adjust its evaluation strategy in a dynamic environment, thereby guiding the agent to adaptively optimize its behavior choices. DDQN separates behavior decision-making from behavior evaluation by introducing a dual Q-network mechanism: the action network determines the next action, while the goal network assesses the value of that action. This separation of behavior selection and value assessment makes DDQN's Q values ​​smoother and more stable, helping to reduce errors caused by overestimation. This not only improves the accuracy of the policy but also enhances the stability of the learning process, making training easier to converge.

[0067] The agent in the DDQN framework relies on the behavior network to generate action decisions, while the target network is responsible for evaluating actions. These two networks are structurally identical and remain independent during training. Periodically copying the behavior network's parameters to the target network ensures that the target network's weights are not frequently updated. This delayed update strategy provides a relatively stable Q-value reference for training, enhancing the stability of the entire learning process, mitigating overfitting and overestimation, and helping to avoid routing loops in situations where nodes are dynamically moving and network topology is frequently changing.

[0068] Another advantage of DDQN is its experience replay mechanism, which allows the agent to learn from past experiences and improves the efficiency of sample data utilization. Specifically, this mechanism maintains an experience buffer to store experiences generated during the agent's interaction with the environment. Each experience typically consists of a state, an action, an immediate reward, and the subsequent new state. During training, the agent does not rely on real-time or continuous experience for learning. Instead, it randomly samples a small batch of experiences from this replay buffer. This random sampling method breaks the sequential relationship between experiences, helps reduce correlation between samples, prevents overfitting, and helps alleviate the impact of the dynamic neighbor table on decision-making, making the learning process more stable and effective.

[0069] The experience replay mechanism has a wide range of applications in deep reinforcement learning, not limited to online training of intelligent agents. In research on routing in underwater sensor networks, this mechanism can also be used as a powerful tool for network pre-training. Given that each routing decision may involve substantial energy costs, the experience replay mechanism allows nodes to be pre-trained by storing experience gained in other scenarios in an experience buffer. This allows nodes to more quickly identify optimal or suboptimal action strategies during actual training. This pre-training not only provides the agent with a more solid foundation for decision-making but also helps accelerate the convergence of the training process, thereby reducing energy consumption and improving overall network performance in practical applications.

[0070] The DDQN-based unicast forwarding mechanism operates as a decision-making agent. Its decision state is based on the information of each neighbor node in the neighbor table and local information, and the output of the decision is the forwarding priority of the current neighbor. A node's forwarding potential is determined by its residual energy, depth, and number of neighbors. To account for the impact of node mobility on communication link stability, neighbor distance and movement speed are also factored into the generation of forwarding priorities. Generally speaking, nodes with higher residual energy can handle more forwarding tasks and should be given higher priority. Nodes with shallower depth are more likely to be close to sink nodes and should therefore be given higher priority. A greater number of neighbors means more forwarding targets, which increases the probability of finding the optimal forwarding target and should therefore be given higher priority. Furthermore, neighbor information in dynamic topologies is often time-sensitive. Generally, nodes with greater distances and faster movement speeds have more rapidly changing information and a greater likelihood of failure, and therefore should be given lower forwarding priorities.

[0071] After calculating the priorities of all neighbors, the node selects the highest priority target for forwarding and then waits for the target's response. If the target forwards the packet within a certain period of time, it indicates that the current forwarding has been successful. Otherwise, the node enters opportunistic broadcast repair mode and sends again.

[0072] Among them, the reward function is designed as follows:

[0073] To ensure that DDQN selects unicast forwarding targets with good and stable forwarding performance, this chapter designs a reward function based on the number of neighbors, depth, residual energy, and distance and relative speed of the forwarding target. The design principle of this reward function is that nodes with more neighbors, shallower depth, and higher residual energy should have higher forwarding priority. Furthermore, considering link stability, nodes that are too far away are likely to have their neighbor tables invalidated due to the lag of neighbor information, and relatively faster nodes are more likely to leave communication range. Therefore, such nodes should have their forwarding priority lowered. The specific reward function calculation is shown in Equation (5-1).

[0074]

[0075] where R success and R fail They represent the immediate rewards when the forwarding is successful and when the forwarding fails; g is the reward offset factor, which is mainly used to ensure that the reward when the forwarding fails is always negative and the reward when the forwarding is successful is always positive, thereby avoiding the generation of passive strategies; isSink is the identifier of whether the selected target is a sink node. Since the sink node is the end point of the forwarding link, when the node forwards to the sink, it indicates that the current forwarding is completed. Therefore, a larger reward should be given to encourage the node to prioritize the sink as the communication target; N choice and N max Respectively represent the number of neighbors of the selected target and the preset maximum number of neighbors (used for normalization); Dep choice and Dep max They represent the depth of the selected target and the maximum deployment depth respectively; E choice and E initial Mainly used to represent the remaining energy of the target and the initial energy of the node; D choice and R represent the distance between the target and the node and the maximum communication distance respectively; V choice and V max They represent the relative moving speed between the target and the node and the maximum relative moving speed of the node respectively; α1, α2, α3, α4, and α5 represent the weights of each influencing factor, satisfying α1+α2+α3+α4+α5=1.

[0076] The reward function design shown in Equation (5-1) ensures that every successful forwarding attempt by a node, regardless of whether it can select the optimal forwarding target, will receive a positive feedback reward, which can prevent negative decisions from being made during training. Any failed forwarding attempt, regardless of whether it can make the energy distribution in the network more balanced, will also result in a penalty for the node, which can prevent excessive decisions such as random forwarding and wasting energy in order to balance energy consumption. This reward function design promotes efficient and rational routing selection training for intelligent agents in UWSNs, which is conducive to optimizing the communication efficiency and energy utilization of the network.

[0077] The link repair mechanism based on opportunistic forwarding is designed as follows:

[0078] In dynamic topology environments, node mobility often leads to poor forwarding link reliability, which has a significant impact on unicast forwarding protocols. To compensate for the deterioration in unicast forwarding delivery rates in mobile topologies, this algorithm proposes an opportunistic broadcast hole repair mechanism. When a node fails to receive a response from the target within a certain period of time after sending a unicast message, it considers unicast forwarding a failure and adjusts the forwarding mode to opportunistic broadcast. First, the receiving destination in the routing packet header is changed to a broadcast address, the forwarding flag is set to repair mode, and the packet is rebroadcast. In repair mode, any node that was not previously designated as a forwarding target but has a shallower depth than the sender is allowed to participate in opportunistic forwarding.

[0079] Considering the serious deterioration of forwarding energy overhead caused by redundant links, this mechanism refers to the redundant link suppression based on holding time in the WDFAD protocol and introduces the neighbor forwarding priority to calculate the node's packet holding time.

[0080]

[0081] Where T hold Indicates the data packet holding time; T tran represents the maximum propagation delay, which is composed of underwater acoustic propagation delay and data packet interpretation delay; P ne represents the normalized forwarding priority; N represents the number of candidate neighbors; R represents the maximum communication distance; Dep pkt Indicates the depth difference between the sender and this node; represents the maximum depth difference between the candidate neighbor and the current node; β is a coefficient used to balance the impact of the two depth differences; v water is the speed of underwater sound propagation, generally 1500m / s; L pkt is the data packet length; S is the data rate.

[0082] If a node detects a forwarding of the same packet by a lower-priority node before the hold time expires, indicating that the higher-priority node has already repaired the data, the node abandons repairing the packet. If no other forwarding is detected after the hold time expires, the node has the highest forwarding potential in the current neighborhood and can therefore perform a broadcast repair of the packet.

[0083] To further mitigate redundancy and save energy, this mechanism incorporates a forwarding mode switching mechanism. When a node first receives a packet in forwarding repair mode, it first attempts to send the packet to its current best forwarding neighbor in non-repair mode before resuming forwarding. Only if unicast forwarding fails does it resume broadcasting in repair mode.

[0084] This embodiment also includes a cavity repair mechanism based on depth adjustment and channel axis transmission. In a deep-sea environment, the convergence characteristics of sound waves provide an advantageous factor for sound propagation - the channel axis. The channel axis is mainly composed of two parts: the surface channel and the deep-sea channel. In some waters, there may be more than one channel axis. The surface channel refers to the shallow water area that is stirred by factors such as waves, so that the water at this depth is mixed evenly, with a more uniform or even positive gradient sound wave velocity distribution, forming a channel that is conducive to sound propagation. The deep-sea channel refers to the fact that due to the constant unevenness of the ocean water area, the transmission of sound waves can be continuously refracted and reflected, causing the sound waves to converge at certain depths. The sound waves in the convergence area are almost unaffected by reflection loss and can communicate over very long distances. The channel axis is the superposition of the two parts of the surface channel and the deep-sea channel, thereby forming a propagation channel with very small reflection loss and transmission loss at a certain depth.

[0085] If the accuracy requirement is not high, the depth of the vocal tract axis can be estimated approximately by the sound velocity curve. Figure 5 、 Figure 6 The figure shows a 4,000-meter ocean sound speed profile curve and its corresponding sound line diagram drawn based on open source data from the Internet. It can be seen from the figure that the sound line appears as a straight line at about 600 meters, and the fluctuation between 500 meters and 1000 meters is very small. Therefore, it can be roughly assumed that the sound channel axis depth presented by the data is approximately between 400 meters and 800 meters.

[0086] Leveraging the excellent propagation properties of the channel axis and the advantages of nodes proactively adjusting their depth, this study proposes a hole repair mechanism based on depth adjustment and channel axis transmission. Under this mechanism, when a node lacks shallower neighbors or is completely isolated, it proactively descends or ascends to a depth close to the channel axis and broadcasts data over long distances at this specific depth. Thanks to the excellent acoustic propagation properties of the channel axis, even nodes at a similar depth but distant can capture packets from the hole node, allowing them to relay the packets.

[0087] The data packet is designed as follows:

[0088] In order to save energy, each data packet forwarding carries the sending node’s own information. Any node that monitors the data packet can maintain its own neighbor table based on the information in the packet header. The data packet is sent to the detailed structure as follows: Figure 8 The transmission information part is used to describe the forwarding status of the current data packet, the node information part is used by any listening node to maintain the neighbor table, and the data payload part is the data that the node wants to transmit.

[0089] Furthermore, relying solely on data communication to maintain neighbor information can easily lead to outdated neighbor information when the network is idle. Therefore, to ensure the accuracy and timeliness of neighbor information, it is essential to regularly broadcast data packets containing the node's own information to update the neighbor table. This periodic maintenance operation can mitigate the problem of outdated neighbor information caused by network inactivity or node movement, providing nodes with continuously updated neighbor dynamics and ensuring the smooth operation of the entire network.

[0090] The DDQN routing protocol workflow is as follows Figure 9 The core operation of the protocol is divided into three key phases: neighbor discovery, cluster formation, and data forwarding. Neighbor discovery is a prerequisite for network initialization. During the data forwarding phase, nodes make routing decisions based on the DDQN algorithm to determine the optimal next-hop forwarder for data packets.

[0091] Whenever a node (usually the cluster head) prepares to forward a packet to a sink, it first constructs a candidate forwarding set based on its current neighbor list. This forwarding candidate set is primarily based on the residual energy and depth of neighboring nodes. During this process, the node first sorts its neighbors by residual energy, eliminating those that cannot support at least one transmission and reception of a packet. The node then further eliminates nodes deployed deeper than itself, ensuring that forwarding prioritizes nodes in shallower water depths.

[0092] If the candidate set obtained by the node after the above screening process is empty, it means that the current node lacks a forwarding path to the shallow layer, that is, the node is in a "hole" state. At this time, the node will start the hole repair process and actively adjust its own depth to find forwarding opportunities: nodes closer to the water surface try to float up, while deeper nodes try to move to the depth of the sound channel axis. Once a suitable neighbor node is successfully encountered, the node can continue to perform the forwarding task; if the adjustment fails to encounter a neighbor node, the node will directly transmit to the aggregation node through cross-domain electromagnetic wave communication on the water surface, or attempt long-distance broadcasting on the sound channel axis. Any node at the depth of the sound channel axis is eligible to continue forwarding the data packet, thereby completing the repair of the hole link.

[0093] During a node's data forwarding process, forwarding success can be determined by the continuity of forwarding behavior. Specifically, if a forwarding operation succeeds, the source node should be able to detect further forwarding activity for the same packet by the next-hop node. This mechanism eliminates the need for dedicated ACKs to confirm forwarding success, helping to conserve energy and channel resources. If a node detects that its selected next-hop node has successfully forwarded the packet within a predetermined time window, it is rewarded according to the mechanism used in the online learning phase. Conversely, if the expected forwarding behavior is not detected, the node is penalized. In this way, nodes continuously adjust and optimize their forwarding strategies based on feedback during data forwarding, improving the efficiency and reliability of data transmission across the entire network.

[0094] In order to verify the optimization performance of the proposed HR-DDQN routing protocol under dynamic topology, the protocol was simulated in NS3. The detailed simulation parameters are shown in Table 4.1.

[0095] Table 5.1 HR-DDQN algorithm simulation parameters

[0096] Parameter name value Deployment Area 5000m×5000m×5000m Vocal channel axis depth 1000m Channel axis range 800m~1200m Deployment method random Convergence node coordinates (2500,2500,0) Total number of nodes 20, 40, 60, 80, 100, 120, 140, 160 Movement Speed Random 0 to 5m / s, 1, 3, 5m / s Maximum transmission distance 1500m Data rate 800bps Data length 50 bytes Carrier frequency 10kHz Full charge energy 100000J Receiving power consumption 1w Transmit power calculation 200w Idle energy consumption <![CDATA[1×10 -5 in]]> Mobile energy consumption 77.76J / m Simulation time 12000s The number of packets generated by each source node 1000 Number of source nodes 1 Mobile Model RandomWalk2dMobilityModel

[0097] Figure 10 The loss function and decision reward of DDQN during training. Figure 10 In (a), the loss function gradually decreases with increasing update steps and then remains near a small value, indicating that DDQN gradually converges during updates. The fluctuations in the curve at the end are due to parameter copying of the target network. Because the target network is updated later than the behavior network, the loss increases during parameter copying. Large outliers are caused by the fact that in random training scenarios, valid links may not exist, resulting in most decisions not providing effective feedback, thus increasing the loss. However, this situation is relatively rare, so these losses can converge again during subsequent training. Figure 10(b) shows a screenshot of the rewards generated by node decisions during training. As the number of decisions increases, the proportion of nodes receiving positive rewards for making correct decisions increases, indicating that the model gradually grasps the principles of decision-making. Due to the dynamic topology and random deployment of training scenarios, node forwarding can sometimes be affected by force majeure. For example, in random scenarios, a forwarding node's neighbors may move far outside the forwarding area. This can lead to transmission failures even after achieving a certain level of performance during training, inevitably resulting in poor decisions in some situations. Therefore, the punitive reward does not disappear after convergence.

[0098] Figure 11 The statistical results of the simulated delivery rates are presented. As can be seen from the figure, the proposed protocol achieves excellent delivery rates across a wide range of node counts, and even significantly outperforms several other algorithms when the number of nodes is small. This is due to the proposed protocol's implementation of a hole repair mechanism based on depth adjustment and channel axis transmission. When a node loses a forwarding link, it proactively adjusts the depth to repair the forwarding link.

[0099] Figure 12 The simulated end-to-end delay statistics show that the proposed protocol performs well in terms of end-to-end delay, achieving the lowest delay in many scenarios. This is because the proposed protocol prioritizes proactive forwarding. In this mode, nodes do not need to spend extra time to suppress redundant links, thus optimizing end-to-end delay performance. Because additional time is required to adjust the depth when the number of nodes is small, the delay initially decreases and then slightly increases with the number of nodes. As the number of nodes increases, the number of neighbors to compare when making forwarding decisions also increases. Due to link stability concerns, nodes that are too far away are generally not selected. While this measure improves delivery rates, it also increases the number of forwarding hops, resulting in a slightly increasing delay trend. EEDOR-VA exhibits the longest delay in the figure above, especially when the number of nodes is small, where its delay is significantly greater than that of other algorithms. This is because EEDOR-VA requires a complete multi-hop link establishment before forwarding. Although this measure can effectively save energy when connectivity is good, when the number of nodes is small and the mobility is strong, the establishment and maintenance of communication links are extremely random. Therefore, EEDOR-VA incurs a huge delay cost in these situations.

[0100] Figure 13The following table shows the energy tax of packet forwarding in simulations. The proposed algorithm achieves a good energy tax in various scenarios. When the number of nodes is small, poor connectivity triggers more active depth adjustments, resulting in additional energy consumption. In this case, the energy tax of the protocol is high. However, as the number of nodes increases, the required movement distance and frequency of hole repair decrease, reducing the additional energy consumption. Furthermore, the prioritized unicast forwarding effectively suppresses redundant links, significantly reducing the energy tax compared to other opportunistic protocols under good connectivity. In this figure, WDFAD-DBR and EEDOR-VA exhibit significant energy taxes when the number of nodes is small. This is because, under poor connectivity, relying solely on passive neighbor discovery broadcasts is not only ineffective for connectivity repair but also incurs an energy cost comparable to that of active movement. Furthermore, at this node density, the delivery rates of these two algorithms are inherently poor, and the energy consumption of neighbor discovery, when averaged over each successfully received packet, becomes a significant energy tax.

[0101] Figure 14 Figure 1 shows the impact of node mobility speed on the protocol delivery rate in the NS3 simulation. As shown in the figure, this protocol, the DBR protocol, and the WDFAD-DBR protocol are not significantly affected by mobility speed and exhibit relatively stable overall performance. However, the DROR protocol and the EEDOR-VA protocol exhibit a decreasing delivery rate trend. Due to the opportunistic broadcast and active mobility repair mechanisms employed in this protocol, the delivery rate remains relatively good. The DBR protocol, the WDFAD-DBR protocol, and the DROR protocol all employ opportunistic routing mechanisms, maintaining a relatively high delivery rate. Only the DROR protocol exhibits a certain downward trend due to the interference of neighbor table timeliness in its decision-making. The EEDOR-VA protocol, which employs an active unicast forwarding mode, deteriorates with increasing node speed, reducing the timeliness of its neighbor table. Consequently, the probability of selecting the optimal routing node decreases, leading to a significant decrease in the delivery rate.

[0102] Figure 15Figure 1 shows the impact of node mobility on the protocol's end-to-end delay in the NS3 simulation. As shown in the figure, as node mobility increases, the delays of this protocol, DBR, WDFAD-DBR, and DROR all change slightly. Only this protocol and DROR show a slight increase. This is because both protocols require a certain amount of neighbor information for routing decisions. Node mobility degrades the timeliness of the neighbor table, disrupting the efficiency of protocol decisions and, to a certain extent, increasing the frequency of link repairs, leading to a slight increase in delay. In the EEDOR-VA protocol, node mobility introduces significant randomness into the lifetime of complete links, forcing the protocol to perform multiple link switches and even re-explore links during forwarding, which consumes a significant amount of time. As a result, the overall delay significantly increases compared to the previous case of random node speed. However, as mobility increases, the node's ability to carry data packets improves, which helps to reduce the number of forwarding hops, resulting in a slight downward trend in delay.

[0103] Figure 16 Figure 1 shows the impact of node mobility on the protocol energy tax in the NS3 simulation. As shown in the figure, the forwarding energy tax for this protocol, the WDFAD-DBR protocol, the DROR protocol, and the EEDOR-VA protocol all increases with increasing node mobility, while the energy tax for the DBR protocol decreases. Increased node mobility reduces the timeliness of neighbor information and the efficiency of neighbor monitoring. This results in a higher triggering frequency of the link repair mechanism in this protocol, the WDFAD-DBR protocol, the DROR protocol, and the EEDOR-VA protocol, resulting in additional energy consumption. For the DBR protocol, increased node mobility means a larger spatial area can be reached for forwarding, increasing the probability that packets will be forwarded to the current optimal relay node, helping to reduce the total number of forwarding hops. Consequently, the energy tax decreases to a certain extent.

[0104] In summary, the proposed protocol significantly optimizes the data delivery rate for sparsely deployed nodes in dynamic UWSNs. Because forwarding attempts are initiated with a single target, fewer redundant links are generated, resulting in a lower packet forwarding energy tax. Furthermore, since proactive single-target forwarding attempts do not require additional packet holding time, the proposed algorithm also demonstrates excellent performance in terms of end-to-end packet forwarding latency.

[0105] This embodiment focuses on studying the optimization of routing protocols for dynamic topology UWSNs in terms of energy consumption and delivery rate. Depending on the mechanism, dynamic topology UWSNs routing protocols often face the challenges of short link life or excessive redundant forwarding, which leads to problems of low forwarding delivery rate or high forwarding energy consumption. To address these problems, this embodiment proposes a hybrid UWSNs routing protocol based on DDQN. This algorithm combines the advantages of active unicast forwarding and passive opportunistic forwarding. During forwarding, unicast transmission is prioritized to reduce energy consumption. After unicast fails, it switches to opportunistic forwarding for route repair. In order to improve the success rate of unicast transmission, this algorithm trains the next-hop routing forwarding target selection strategy based on DDQN, taking into account the distance, speed, depth and energy factors of itself and its neighbors, and avoiding the selection of forwarding targets with poor reliability. At the same time, to address the problem that forwarding holes are sometimes unrepairable due to poor connectivity in sparse networks, a hole repair mechanism based on depth adjustment and channel axis transmission is proposed. When a node faces a hole that cannot be repaired by flooding alone, it proactively adjusts its depth until it reaches the channel axis for long-distance transmission or a neighbor successfully forwards the packet. Simulation experiments demonstrate that the proposed protocol outperforms unicast protocols in terms of delivery rate and opportunistic protocols in terms of energy consumption, significantly improving data delivery rates in sparse deployment scenarios.

[0106] Those skilled in the art will understand that the above description is only a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of the present disclosure may be combined or coupled in various ways, even if such a combination or coupling is not explicitly described in the present disclosure. It is not intended to limit the present invention. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art may still modify the technical solutions described in the aforementioned embodiments or make equivalent substitutions for some of the technical features therein. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention shall be included in the scope of protection of the present invention.

[0107] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they are aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, the present invention is intended to include such changes and modifications as fall within the scope of the claims and their equivalents.

Claims

1. A hybrid dynamic UWSNs routing protocol optimization method based on DDQN is characterized by: The method comprises the following steps: Step 1: Design a forwarding target evaluation mechanism based on the DDQN protocol framework. Introduce a dual Q network mechanism into the DDQN protocol to separate behavior decision-making and behavior evaluation. Step 2: Combine active unicast forwarding and passive opportunistic forwarding into one. The forwarding process is as follows: first, unicast forwarding is used. If unicast fails, it is converted to passive opportunistic forwarding. If, within a predetermined time window, the node detects that its selected next-hop node has successfully forwarded the data packet, it is rewarded according to the mechanism of the online learning phase; otherwise, if the expected forwarding behavior is not detected, the node is punished, completing the DDQN hybrid dynamic UWSNs routing protocol optimization.

2. The hybrid dynamic UWSNs routing protocol optimization method based on DDQN according to claim 1 is characterized in that: Step 2 also includes the step of designing a reward function based on the number of neighbors, depth, residual energy, and distance and relative speed of the forwarding target to the node.

3. The hybrid dynamic UWSNs routing protocol optimization method based on DDQN according to claim 2 is characterized in that: The reward function is calculated as follows: where R success and R fail They represent the instant rewards when the forwarding of this hop is successful and when the forwarding of this hop fails; g is the reward offset factor, N choice and N max Respectively represent the number of neighbors of the selected target and the preset maximum number of neighbors; Dep choice and Dep max They represent the depth of the selected target and the maximum deployment depth respectively; E choice and E initial Used to represent the remaining energy of the target and the initial energy of the node; D choice and R represent the distance between the target and the node and the maximum communication distance respectively; V choice and V max They represent the relative moving speed between the target and the node and the maximum relative moving speed of the node respectively; α1, α2, α3, α4, and α5 represent the weights of each influencing factor, satisfying α1+α2+α3+α4+α5=1.

4. The hybrid dynamic UWSNs routing protocol optimization method based on DDQN according to claim 1 is characterized in that: Step 2 also includes the step of performing an opportunistic broadcast hole repair mechanism on the unicast forwarding protocol.

5. The hybrid dynamic UWSNs routing protocol optimization method based on DDQN according to claim 1 is characterized in that: The opportunistic broadcast hole repair mechanism requires that any node that is not designated as a forwarding target but has a shallower depth than the sender is allowed to participate in opportunistic forwarding.

6. The hybrid dynamic UWSNs routing protocol optimization method based on DDQN according to claim 1 is characterized in that: In step 1, the DDQN protocol also includes three steps: neighbor discovery, cluster formation, and data forwarding.

7. The hybrid dynamic UWSNs routing protocol optimization method based on DDQN according to claim 5 is characterized in that: Step 2 also includes steps for depth adjustment and cavity repair mechanism for channel axis transmission.

8. A hybrid dynamic UWSNs routing protocol optimization system based on DDQN, characterized by: The system comprises: The dual-Q network mechanism module is used to design a forwarding target evaluation mechanism based on the DDQN protocol. The dual-Q network mechanism is introduced into the DDQN protocol to separate behavior decision-making and behavior evaluation. The hybrid forwarding module is used to combine active unicast forwarding and passive opportunistic forwarding into one. Its forwarding process is as follows: first, unicast forwarding is used. If unicast fails, it is converted to passive opportunistic forwarding. If, within a predetermined time window, the node detects that its selected next-hop node successfully forwards the data packet, it is rewarded according to the mechanism of the online learning phase; conversely, if the expected forwarding behavior is not detected, the node is punished, completing the DDQN hybrid dynamic UWSNs routing protocol optimization.

9. A computer device comprising a memory and a processor, characterized in that A computer program is stored in the memory. When the processor runs the computer program stored in the memory, the processor executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images