Convergecast-based static uwsns routing protocol optimization method and system

Abstract

The application relates to a method and system for optimizing a static UWSNs routing protocol based on convergence guidance, and relates to the field of routing protocol optimization. The method solves the problem of high end-to-end delay and low delivery rate caused by routing detours and routing loops in the existing static topology UWSNs routing protocol. The method comprises the following steps: a forwarding decision model based on Q-Learning is used to establish a forwarding path from a source node to a convergence node; a decision state of information of a node where a data packet is located and a candidate node for forwarding is determined according to whether there is a completely shared parameter between nodes; when the node needs to forward the data packet, the Q value of each neighbor node is calculated, and the neighbor node with the maximum Q value is selected as a next-hop forwarding target; a convergence guidance mechanism is constructed, and the node forwarding decision model is allowed to generate a training reward based on position information of each node in the convergence during a training process, so that a path for gathering data packets to a central node is generated, and optimization of the static UWSNs routing protocol based on convergence guidance is completed.

Description

Technical Field

[0001] This invention relates to the field of static UWSNs routing protocol optimization technology, specifically to a static UWSNs routing protocol optimization method and system based on aggregation guidance. Background Technology

[0002] With economic development and a deeper understanding of the ocean, its strategic, economic, and scientific value has gained increasing attention, leading to a demand for various technological innovations such as marine environmental monitoring and marine resource exploration. Underwater Wireless Sensor Networks (UWSNs) offer significant advantages in continuous underwater monitoring and target detection. However, the reliable operation of UWSNs depends on the efficiency of their routing protocol data collection, but limitations imposed by underwater channel conditions result in lower reliability and efficiency for underwater routing.

[0003] However, while proactive routing protocols offer good energy efficiency in static UWSNs, they suffer from routing detours and even loops due to the single forwarding link, which negatively impacts the delivery rate and end-to-end latency of UWSN data collection. This paper addresses the high-end-to-end latency and low delivery rate issues caused by routing detours and loops in static topology UWSN routing protocols. Summary of the Invention

[0004] This invention aims to address the high-end-to-end latency and low delivery rate issues caused by route detours and loops in existing static topology UWSNs routing protocols. To solve these problems, this invention employs the following technical solution:

[0005] Option 1: This invention proposes a static UWSNs routing protocol optimization method based on convergence guidance, the method comprising the following steps:

[0006] Step 1: Based on the Q-Learning forwarding decision model, establish the optimal forwarding path from the source node to the sink node;

[0007] Step 2: Based on whether there are fully shared parameters among nodes in the static UWSNs routing protocol, the forwarding decision model described in Step 1 is divided into two categories: a decision model with data packets as agents and a decision model with nodes as agents.

[0008] The decision state of the node where the data packet is located and the forwarding candidate node information in the static UWSNs routing protocol is determined. When the node needs to forward the data packet, the Q value of each neighbor node is calculated in turn, and finally the neighbor node with the largest Q value is selected as the next hop forwarding target.

[0009] Step 3: Construct a convergence guidance mechanism. The convergence guidance mechanism allows nodes to generate training rewards based on the location information of each node during the training process of the forwarding decision model of Q-Learning, generate paths for data packets to converge to the central node, and complete the static UWSNs routing protocol optimization based on convergence guidance.

[0010] Furthermore, a preferred implementation is provided in which all nodes in the Q-Learning-based forwarding decision model described in step 1 share a set of reinforcement learning parameters.

[0011] Furthermore, a preferred implementation is provided in which the data packet in step 2 is implemented through a decision model with nodes as intelligent agents.

[0012] Furthermore, a preferred embodiment is provided in which, after the static UWSNs routing protocol is deployed, a SAOL routing protocol design step is also included, which includes neighbor discovery, cluster formation, and data forwarding steps.

[0013] Furthermore, a preferred embodiment is provided in which the convergence guidance mechanism described in step 3 includes the steps of designing a reward function and a fixed V value mechanism.

[0014] Furthermore, a preferred embodiment is provided, wherein the reward function design method is as follows:

[0015] R = R sink +R energy +R drift (4-4)

[0016] Where R is the reward of the current decision; R sink It is a convergence factor return; R energy It is an energy factor return; R drift It is a drift factor return.

[0017] Furthermore, a preferred embodiment is provided, wherein the calculation method for the fixed V value mechanism described in step 3 is as follows:

[0018]

[0019] In the formula, V π (s) represents the expected total reward that can be obtained in state s according to the π strategy; Q π (s,a) represents the evaluation of the current decision action, α is the learning rate of the system, and 0 < α ≤ 1.

[0020] Option 2: A static UWSNs routing protocol optimization method based on aggregation guidance, the method comprising the following steps:

[0021] The forwarding path determination module is used to establish the optimal forwarding path from the source node to the sink node based on the Q-Learning forwarding decision model.

[0022] The forwarding decision model construction module is used to classify the forwarding decision model described in the forwarding path determination module into two categories based on the static UWSNs routing protocol and whether the nodes in the static UWSNs routing protocol fully share parameters. These categories include decision models with data packets as agents and decision models with nodes as agents.

[0023] The decision state of the node where the data packet is located and the forwarding candidate node information in the static UWSNs routing protocol is determined. When the node needs to forward the data packet, the Q value of each neighbor node is calculated in turn, and the neighbor node with the largest Q value is selected as the next hop forwarding target.

[0024] The static UWSNs routing protocol optimization module is used to build a convergence guidance mechanism. This mechanism allows nodes to generate training rewards based on the location information of the convergence node during Q-Learning training, creating paths for data packets to converge towards the central node, thus completing the static UWSNs routing protocol optimization based on convergence guidance.

[0025] Option 3: A computer device, including a memory and a processor, characterized in that the memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method described in any one of Options 1.

[0026] Option 4: A computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the method described in any one of Options 1.

[0027] The advantages of this invention are:

[0028] This invention presents a static UWSN routing protocol optimization method based on convergence guidance. It employs a forwarding target selection strategy based on Q-Learning, allowing nodes to gradually learn the forwarding potential of their neighbors through continuous forwarding, thus enabling subsequent forwarding to be completed with lower energy consumption and latency. Simultaneously, to accelerate convergence and reduce the probability of detours and loops during training, a convergence guidance mechanism is proposed. In this mechanism, the location of the convergence node influences the design of the reward function, encouraging nodes to attempt forwarding towards the convergence direction during training. Furthermore, fixing the V value of the convergence node ensures that nodes closer to the convergence point converge earlier, thereby accelerating route training. Simulation results demonstrate that the proposed protocol effectively optimizes route detours and loops, exhibiting excellent performance in improving delivery rate, reducing end-to-end latency, and lowering forwarding energy tax.

[0029] This invention addresses the high latency caused by forwarding detours and low delivery rates caused by forwarding loops in static topology UWSNs routing protocols. It proposes a static UWSNs routing protocol based on convergence guidance, employing an active routing protocol to control forwarding links, using Q-Learning to train decision selection, and introducing a convergence guidance mechanism to accelerate training convergence while optimizing routing detours and loops. This improves the delivery rate and latency performance of the routing protocol in static topology UWSNs networks.

[0030] This invention is also applicable to the fields of marine environmental monitoring and marine resource exploration. Attached Figure Description

[0031] Figure 1 This is a schematic diagram of the anchor node drift described in Implementation Method Eleven.

[0032] Figure 2 This is a schematic diagram illustrating the influence of the fixed V value on forwarding convergence as described in Implementation Method Eleven.

[0033] Figure 3 This is a schematic diagram of the SAQL routing protocol data packet structure as described in Implementation Method Eleven.

[0034] Figure 4 This is a flowchart of the SAQL routing protocol workflow described in Implementation Method Eleven.

[0035] Figure 5 This is a flowchart of the SAQL routing protocol neighbor discovery process described in Implementation Method Eleven.

[0036] Figure 6 This is a flowchart illustrating the data forwarding process of the SAQL routing protocol as described in Implementation Method Eleven.

[0037] Figure 7 This is a schematic diagram illustrating the update of the V value of a forwarding link node in the simulation described in Implementation Method Eleven.

[0038] Figure 8 This is a statistical diagram of the data packet delivery rate as described in Implementation Method Eleven.

[0039] Figure 9 This is a schematic diagram of energy tax statistics as described in Implementation Method Eleven.

[0040] Figure 10 This is a schematic diagram of end-to-end delay statistics as described in Implementation Method Eleven.

[0041] Figure 11 This is a schematic diagram illustrating the impact of the number of source nodes on the delivery rate as described in Implementation Method Eleven.

[0042] Figure 12This is a schematic diagram illustrating the impact of the number of source nodes on energy tax as described in Implementation Method Eleven.

[0043] Figure 13 This is a schematic diagram illustrating the impact of the number of source nodes on end-to-end latency as described in Implementation Method Eleven. Detailed Implementation

[0044] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them.

[0045] Implementation Method 1: This implementation method provides a static UWSNs routing protocol optimization method based on aggregation guidance. The method includes the following steps:

[0046] Step 1: Based on the Q-Learning forwarding decision model, establish the optimal forwarding path from the source node to the sink node;

[0047] Step 2: Based on whether there are fully shared parameters among nodes in the static UWSNs routing protocol, the forwarding decision model described in Step 1 is divided into two categories: a decision model with data packets as agents and a decision model with nodes as agents.

[0048] The decision state of the node where the data packet is located and the forwarding candidate node information in the static UWSNs routing protocol is determined. When the node needs to forward the data packet, the Q value of each neighbor node is calculated in turn, and finally the neighbor with the largest Q value is selected as the next hop forwarding target.

[0049] Step 3: Construct a convergence guidance mechanism. The convergence guidance mechanism allows nodes to generate training rewards based on the location information of each node during the training process of the forwarding decision model of Q-Learning, generate paths for data packets to converge to the central node, and complete the static UWSNs routing protocol optimization based on convergence guidance.

[0050] Implementation Method 2: This implementation method further defines the static UWSNs routing protocol optimization method based on convergence guidance described in Implementation Method 1. In the forwarding decision model based on Q-Learning described in step 1, all nodes share a set of reinforcement learning parameters.

[0051] Implementation Method 3: This implementation method further defines the static UWSNs routing protocol optimization method based on convergence guidance described in Implementation Method 1. In step 2, the data packets are implemented through a decision model with nodes as agents.

[0052] Implementation Method 4: This implementation method further defines the static UWSNs routing protocol optimization method based on aggregation guidance described in Implementation Method 1. After the static UWSNs routing protocol is deployed, it also includes the SAOL routing protocol design step, which includes neighbor discovery, cluster formation, and data forwarding steps.

[0053] Implementation Method 5: This implementation method further defines the static UWSNs routing protocol optimization method based on convergence guidance described in Implementation Method 1. The convergence guidance mechanism in step 3 includes the steps of reward function design and fixed V value mechanism.

[0054] Implementation Method Six: This implementation method further defines the static UWSNs routing protocol optimization method based on convergence guidance described in Implementation Method One. The reward function design method is as follows:

[0055] R = R sink +R energy +R drift (4-4)

[0056] Where R is the reward of the current decision; R sink It is a convergence factor return; R energy It is an energy factor return; R drift It is a drift factor return.

[0057] Implementation Method Seven: This implementation method further defines the static UWSNs routing protocol optimization method based on aggregation guidance described in Implementation Method One. The calculation method for the fixed V-value mechanism in step 3 is as follows:

[0058]

[0059] In the formula, V π (s) represents the expected total reward that can be obtained in state s according to the π strategy; Q π (s,a) represents the evaluation of the current decision action, α is the learning rate of the system, and 0 < α ≤ 1.

[0060] Implementation Method 8: This implementation method proposes a static UWSNs routing protocol optimization method based on convergence guidance. The method includes the following steps:

[0061] The forwarding path determination module is used to establish the optimal forwarding path from the source node to the sink node based on the Q-Learning forwarding decision model.

[0062] The forwarding decision model construction module is used to classify the forwarding decision model described in the forwarding path determination module into two categories based on the static UWSNs routing protocol and whether the nodes in the static UWSNs routing protocol fully share parameters. These categories include decision models with data packets as agents and decision models with nodes as agents.

[0063] The decision state of the node where the data packet is located and the forwarding candidate node information in the static UWSNs routing protocol is determined. When the node needs to forward the data packet, the Q value of each neighbor node is calculated in turn, and finally the neighbor with the largest Q value is selected as the next hop forwarding target.

[0064] The static UWSNs routing protocol optimization module is used to build a convergence guidance mechanism. This mechanism allows nodes to generate training rewards based on the location information of the convergence node during Q-Learning training, creating paths for data packets to converge towards the central node, thus completing the static UWSNs routing protocol optimization based on convergence guidance.

[0065] Implementation Method 8: This implementation method proposes a static UWSNs routing protocol optimization system based on convergence guidance. The system includes:

[0066] The forwarding path determination module is used to establish the optimal forwarding path from the source node to the sink node based on the Q-Learning forwarding decision model.

[0067] The forwarding decision model construction module is used to classify the forwarding decision model described in the forwarding path determination module into two categories based on the static UWSNs routing protocol and whether the nodes in the static UWSNs routing protocol fully share parameters. These categories include decision models with data packets as agents and decision models with nodes as agents.

[0068] The decision state of the node where the data packet is located and the forwarding candidate node information in the static UWSNs routing protocol is determined. When the node needs to forward the data packet, the Q value of each neighbor node is calculated in turn, and finally the neighbor with the largest Q value is selected as the next hop forwarding target.

[0069] The static UWSNs routing protocol optimization module is used to build a convergence guidance mechanism. This mechanism allows nodes to generate training rewards based on the location information of the convergence node during Q-Learning training, creating paths for data packets to converge towards the central node, thus completing the static UWSNs routing protocol optimization based on convergence guidance.

[0070] Implementation Method Nine: This implementation method provides a computer device, including a memory and a processor. The memory stores a computer program. When the processor runs the computer program stored in the memory, the processor executes the method described in any one of Implementation Methods One to Seven.

[0071] Implementation Method 10: This implementation method provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method described in any one of Implementation Methods 1 to 7.

[0072] Implementation Method Eleven: This implementation method provides an example, which is used to explain the above implementation method. The specific example is as follows:

[0073] See Figures 1 to 13 This implementation focuses on optimizing routing protocols in static topology UWSN scenarios. Addressing the issues of high-end-to-end latency and low delivery rate caused by route detours and loops in static topology UWSN routing protocols, this implementation proposes a novel underwater routing algorithm based on convergence guidance and Q-Learning. Forwarding decisions are made using Q-Learning, and a convergence guidance mechanism is introduced to reduce the probability of proactive routing encountering detours or even getting stuck in infinite loops, thereby improving the delivery rate of multi-hop forwarding and reducing end-to-end latency.

[0074] Specifically, it includes:

[0075] 1. Forwarding decision model based on Q-Learning

[0076] In the design scenario of static UWSN routing protocols, forwarding decision models can be divided into two categories based on whether nodes fully share parameters: decision models using data packets as agents and decision models using nodes as agents. When the forwarded data packet is used as the agent, the routing decision is similar to a "maze" model. The current state is the node the data packet reaches, and the action is the node the data packet is about to be delivered to. This model requires all nodes to share a set of reinforcement learning parameters, making its application in underwater scenarios very limited. When the node itself is used as the agent, each node in a UWSN is an agent. In this case, the node's local information and neighbor table represent the agent's state, and the next-hop forwarding target is the action to be decided. This model does not require nodes to share RL parameters beforehand and can adapt to more underwater operating scenarios.

[0077] The SinkAttraction-Q-Learning Routing Protocol (SAQL Routing Protocol) proposed in this embodiment operates with nodes as agents. In the protocol, the node where the data packet is located and the information of forwarding candidate nodes (such as remaining energy, number of neighboring nodes, depth, and location) are the decision states of the agent. When the node needs to forward the data packet, it calculates the Q value of each neighboring node in turn, and finally selects the neighbor with the largest Q value as the next hop forwarding target.

[0078] 2. Construction of forwarding candidate set

[0079] To expedite the decision-making process, neighbors with relatively poor forwarding performance in the neighbor table are first excluded. These neighbors are then used as a backup forwarding set. The Q-value of the neighbor nodes in this backup set is calculated only if all forwarding attempts have failed.

[0080] The construction of the node forwarding candidate set mainly follows two principles: energy and depth.

[0081]

[0082] Where E ne E represents the remaining energy of neighboring nodes. ave is the average remaining energy in the current neighborhood; k is the energy selection coefficient. A larger k results in fewer candidate nodes, making it more suitable for denser networks, while a smaller k results in more candidate nodes, making it more suitable for networks with a relative coefficient. Dep ne Dep represents the depth of neighboring nodes. me This is the depth of the current node. This filtering condition can select nodes with higher energy and shallower depth (convergence nodes are generally deployed on the water surface, and the data forwarding direction is generally towards the water surface) to form a forwarding candidate set, which reduces the action space of the current decision in Q-Learning, helps to increase the probability of selecting nodes with strong forwarding potential during training, and speeds up training convergence.

[0083] 3. Q-function update

[0084] Q-Learning is a value-oriented reinforcement learning algorithm based on the Markov Decision Process (MDP) framework. It aims to evaluate the value of an agent's decision-making attempts and train the agent's action strategy in the current scenario by rewarding or penalizing the agent's actions. The core of Q-Learning is training the agent to learn an action-value function (Q-function). This function assigns a value to each pair of the current state and the agent's actions, representing the expected reward the agent can obtain by performing that action in the current state. The Bellman equation is central to updating the action-value function, describing the iterative relationship between the agent's current state and future states. It reveals the key patterns in estimating the value of one state from another, reflecting the connection between the current decision and future rewards, enabling the agent to continuously adjust its strategy to estimate future rewards and achieve optimal decision-making.

[0085] Like most intelligent algorithms, Q-Learning primarily employs a balance between exploration and exploitation to learn the optimal strategy. In Q-Learning, an agent's decision-making is broken down into two aspects: action evaluation and state evaluation. The evaluation of the agent's action decision is called the Q-function, and the evaluation of the agent's state is called the value function. Their expressions are shown in (4-2).

[0086]

[0087] Equation (4-2) is one representation of the Bellman equation, where V π (s) represents the expected total reward that can be obtained in state s according to the π strategy; Q π R(s,a) represents the reward obtained by maintaining the policy until the final state after performing action a in state s. In other words, the evaluation of the current decision action R(s,a) is the immediate reward of choosing action a in state s. Based on equation (4-2), the method for updating the Q-value in Q-learning can be obtained:

[0088] Q'(s,a)←(1-α)Q(s,a)+α(R(s,a)+γ×maxQ(s',a))(4-3)

[0089] Where Q'(s,a) represents the updated Q-value; α is the system's learning rate, 0 < α ≤ 1. If α = 1, it indicates that the Q-value update depends entirely on the latest decision. When α < 1, the calculation of the new Q-value is influenced by the calculation of past Q-values, which helps to smooth the training process. By continuously adjusting the action evaluation method based on the reward of the current decision and the evaluation of future rewards, the agent can gradually learn which actions to take in a specific state to obtain the best long-term reward, thus learning the optimal strategy for solving problems. That is, in continuous updates, each node gradually learns the current forwarding potential of each neighbor node, which helps to find the optimal forwarding path.

[0090] 4. Aggregation and Guidance Mechanism

[0091] Classical Q-Learning routing algorithms are prone to routing detours or loops, affecting convergence speed. Furthermore, since training is real-time, slower convergence increases network delivery rate, end-to-end latency, and even energy tax costs. Therefore, guidance is needed during training to increase the probability of the agent selecting better nodes, accelerating convergence and reducing energy and latency costs. Convergence guidance mechanisms primarily work in two areas: reward function design and a fixed V-value mechanism.

[0092] 5. Reward Function Design

[0093] The reward function defines the immediate reward or penalty that an agent will receive after taking a certain action in the current state. It is a direct feedback from the environment to the agent's decision and has a significant impact on whether the agent can learn the optimal strategy. In order to optimize the impact of energy consumption balance, forwarding direction and link stability on decision-making, the reward function of the SAQL protocol is calculated as shown in Equation (4-4).

[0094] R = R sink +R energy +R drift (4-4)

[0095] Where R is the reward of the current decision; R sink It is a convergence factor return; R energy It is an energy factor return; R drift It is a drift factor return.

[0096] 6. Convergence Factors

[0097] The convergence factor aims to guide the forwarding direction of the algorithm. Specifically, this factor, based on location information, calculates the forwarding distance achievable by forwarding the data packet to the current candidate node, influencing the node's decision. Generally, the larger this forwarding distance, the closer the forwarding will be to the convergence node, and the closer the resulting forwarding link will be to the shortest forwarding link, thus helping to achieve a smaller end-to-end latency. The convergence guidance reward is calculated as shown in equation (4-5).

[0098]

[0099] Where (x) i ,y i ,z i (x) represents the node coordinates of the current decision; next ,y next ,z next (x) represents the currently estimated coordinates of the next-hop forwarding candidate node; sink ,y sink ,z sink R represents the coordinates of the current node's pre-recorded sink node (or the coordinates of the nearest sink node in the case of multiple sink nodes). Equation (4-5) actually calculates a normalized forwarding distance, representing the degree to which the data packet gets closer to the sink node after being forwarded to the candidate node. When the candidate forwarder is farther from the sink node than itself, R... sink The calculation is negative, which is a penalty for the agent's decision-making and helps to prevent the agent from taking a detour or even getting stuck in a forwarding loop.

[0100] 7. Energy factors

[0101] Underwater nodes are often difficult to charge promptly after deployment, so premature node failure should be avoided as much as possible. When the energy of a forwarding node is low, try to let a nearby node with higher remaining energy take over the forwarding. In cluster mode, the energy consumption of the cluster head node is slightly higher than that of the member nodes. When the next hop forwarder happens to be the cluster head, it often leads to excessive energy consumption of the forwarding cluster head, which is not conducive to long-term maintenance of the current cluster. Therefore, non-cluster head nodes can also be selected into the candidate forwarding set during forwarding, and the final forwarding is determined based on the remaining energy of the node. The calculation method of the energy factor reward function is shown in (4-6).

[0102]

[0103] Where g = -1 represents resident energy consumption. Since nodes consume energy regardless of whether forwarding is successful or not, resident energy consumption is introduced to represent that any action attempt requires energy; E next E i and E initial These represent the remaining energy of the candidate forwarding node, the remaining energy of the decision node, and the initial energy of the node, respectively; EDnext and ED i α1 and α2 represent the energy distribution in the neighborhood of the candidate forwarding node, i.e., the average energy in their respective neighborhoods; α1 and α2 represent the weights of the remaining energy influence and the energy distribution influence, respectively, and satisfy α1+α2=1, α1, α1∈[0,1].

[0104] 8. Drift Factors

[0105] UWSN nodes are affected by changing water currents in water. Even with anchored deployment, nodes will drift within a certain range. When two nodes are far apart, this drift can cause intermittent interruptions in the forwarding link. This disruption not only affects the current transmission but also forces the forwarding node to re-explore new candidate nodes, introducing additional energy consumption. Therefore, a drift factor is introduced to represent the evaluation reward of the time a neighboring node stays within the neighborhood.

[0106]

[0107] Where d max d represents the maximum distance for node communication; d represents the distance between the current candidate forwarding node and the current node, which can be calculated using the Received Signal Strength Indicator (RSSI); h represents the depth difference between the current candidate forwarding node and the current node. The node's own depth information can be obtained through a pressure sensor, while the depth information of neighboring nodes is recorded and shared in the neighbor detection data packets. According to equation (4-7), the greater the distance between two nodes and the larger the depth difference, the more likely the candidate node is to drift out of range, and therefore its drift reward is smaller. Such nodes are relatively less likely to be selected as the next-hop node, which helps to optimize the stability of the forwarding path obtained by the decision.

[0108] 9. Fixed V-value mechanism

[0109] The V-value represents the expected cumulative forwarding reward that a node can achieve while maintaining its current forwarding strategy, and it has a crucial impact on Q-Learning decisions. Based on the conversion relationship between V-values ​​and Q-values ​​as expressed in Equation (4-8), the convergence guidance mechanism sets the V-value of all convergence nodes to always be 0 to guide the direction of packet forwarding by the agent. By fixing the V-value of the convergence nodes to 0, the forwarding nodes closest to the convergence nodes will achieve training convergence faster, and then gradually spread to the packet source nodes, thus forming forwarding paths more quickly.

[0110]

[0111] 10. Data Packet Design

[0112] After UWSN deployment, due to limitations in energy and channel resources, nodes struggle to efficiently maintain neighbor information. To address this, the SAQL protocol adopts an efficient strategy that combines forwarding and information sharing. Each data forward carries local information, allowing any node receiving the data packet to update its neighbor information. The core of this process lies in the designed data packet structure, which not only carries data but also node status information such as battery level and location. The detailed design of the data packet structure is as follows... Figure 3 exhibit.

[0113] When network load is low and inter-node packet forwarding activity is reduced, a situation may arise where a node urgently needs to forward data but its neighbor information is outdated. To address this, the SAQL protocol maintains a low-frequency information broadcast mechanism, periodically updating and maintaining neighbor information through network broadcast packets even when network activity is low. These special broadcast packets have both the destination node address and the next-hop node address set to broadcast addresses, explicitly specifying that the sole purpose of these packets is to enforce neighbor maintenance, ensuring that each node's neighbor information remains valid even when the network is less active, thus guaranteeing network stability and reliability.

[0114] 11. Protocol Workflow

[0115] The SAQL routing protocol's workflow consists of three key phases: neighbor discovery, cluster formation, and data forwarding. In the neighbor discovery phase, nodes ensure mutual recognition and awareness through mandatory local information broadcasts. This phase is particularly crucial during initial network deployment and needs to be repeated periodically in subsequent operation to maintain updated neighbor tables, ensuring timely node information even without data transmission. The cluster formation phase aims to optimize energy consumption and integrate local data through clustering, reducing channel resource consumption. This not only helps maintain coverage areas during network operation but also reduces the possibility of packet collisions to some extent; details are explained in Chapter 3. The data forwarding phase is the core of the protocol, responsible for the efficient forwarding of data packets between network nodes, ensuring smooth information delivery.

[0116] 12. The neighbor discovered

[0117] During the initial deployment phase of UWSNs, nodes lack information about their neighboring nodes, making it impossible to build an effective candidate forwarding set when data transmission is needed. Similarly, if a certain area of ​​the network is inactive for a long time, the node's neighbor information will not be updated, making the established forwarding candidate set unreliable. To address this issue, our proposed solution employs a periodic information sharing mechanism. This mechanism allows all nodes to time their operations according to an internal timer. Once the predetermined neighbor information maintenance cycle is reached, the node pauses its current task, actively sends broadcast packets containing local information, and refreshes its neighbor information based on the packets it hears. The workflow of this phase is as follows: Figure 5 As shown.

[0118] To reduce the energy cost of maintaining neighbor information in high-load network environments, the SAQL protocol adopts a strategy of sharing information along with packet forwarding. This allows each node receiving a packet to directly obtain the data needed to maintain neighbor information from the packet header, ensuring the timeliness of the node's neighbor information. At the same time, it can reduce the frequency of forced neighbor discovery operations, thereby saving energy consumption.

[0119] 13. Data forwarding

[0120] The packet forwarding phase is the core of the routing protocol. In this phase, each node first sends its packet to the cluster head node of its cluster. The cluster head node performs initial deduplication of the data and becomes the sole "source node" in its current neighborhood. It then constructs a forwarding candidate set, where each neighboring node is considered a possible action that the SAQL algorithm might take in the current state. By evaluating the Q-values ​​of each node, a ranking of node forwarding priorities is obtained. Nodes forward data according to this ranking, and the next forwarding of the packet by the destination node serves as a transmission acknowledgment signal.

[0121] When constructing a node forwarding candidate set, two main factors are considered: energy and forwarding history. First, the node searches its neighbor table based on the destination address (i.e., the sink node address) contained in the data packet. If a target node is found, it is prioritized and given the highest priority in the forwarding candidate set. Next, the node calculates the average remaining energy of all nodes in the neighbor table and excludes neighbors whose remaining energy does not meet the forwarding criteria. Finally, the node removes the forwarding node from the previous hop of the data packet and nodes that attempted to forward the packet but failed.

[0122] After the candidate set is constructed, nodes will attempt data forwarding sequentially according to priority. Each time a node sends a data packet to the next-hop target, a local timer is started, the duration of which is calculated as shown in (4-9). During this period, the node needs to monitor the forwarding behavior of the target node. If the target node successfully forwards the data packet, the forwarding is considered successful. Otherwise, it is considered a forwarding failure, and the node will then try other candidate nodes. If the entire candidate set fails to forward the data packet, the node will make a final attempt, considering nodes previously excluded due to power issues as backups in the hope of repairing the forwarding link.

[0123]

[0124] Where R represents the communication radius of the node; v water This represents the velocity of sound waves in the current water area at the node, typically 1500 m / s; L pkt The length of the data packet is represented by ; S represents the data rate of transmission and reception.

[0125] 14. Simulation Analysis

[0126] To verify the performance of the proposed SAQL routing protocol in terms of delivery rate, latency, and energy consumption, the protocol was simulated in NS3. The detailed simulation parameters are shown in Table 4.1.

[0127] Table 4.1 Simulation parameters of SAQL algorithm

[0128]

[0129] Figure 7 The diagram shows the V-value updates for each node on a forwarding link in a simulation. Node 148 is the source node, Node 69 is the first-hop node, Node 56 is the second-hop node, and Node 64 is the third-hop node. The source node converges after approximately two hundred training iterations, and its V-value rapidly converges to around -2.5. Because the source node continuously generates and sends data, its remaining energy decreases, so its V-value continues to slowly decrease after convergence. For each relay node, its V-value also updates rapidly, and after convergence, it slowly decreases as energy is consumed. The closer the node is to the convergence point, the larger its V-value, guiding the selection of the next-hop forwarding target.

[0130] Figure 8The performance of the proposed protocol in packet delivery rate (PDR) is demonstrated in NS3 simulations. Simulation results show that the proposed protocol achieves a top-tier delivery rate regardless of whether the node count is sparse or dense. Furthermore, when the number of nodes is sufficient, the proposed algorithm consistently maintains a delivery rate above 90% with more than 70 nodes, and maintains a delivery rate above 95% even with 100 or more nodes, which is sufficient for application requirements. Although FVBF has a slightly higher PDR with a larger number of nodes, its flooding transmission incurs a significant energy cost; see [link to documentation] for details. Figure 9 .

[0131] Figure 9 The energy tax performance of various algorithms in NS3 simulation tests is presented. Simulation statistics show that the SAQL protocol has the lowest forwarding energy tax in most cases. This is because the proposed protocol adopts a single forwarding link mechanism, where only one target can undertake the forwarding task in each forwarding, thus resulting in a low energy tax. As the number of nodes increases, the number of nodes that can listen to the data packet in each forwarding increases, thus increasing the forwarding energy tax. For QELAR and QLFR, their forwarding delivery rate is low when nodes are sparse. QELAR performs multiple forwarding repairs, while QLFR increases the number of forwarding links, resulting in a high energy tax. For FVBF and ALRP protocols, both are opportunistic forwarding protocols with many redundant links. As the number of nodes increases, the number of nodes participating in forwarding also increases, thus significantly increasing the energy tax.

[0132] Figure 10 The end-to-end latency performance of various protocols in NS3 simulation tests is presented. According to the statistical results, the proposed protocol has the best end-to-end latency. This is because the convergence guidance mechanism can guide nodes to forward data towards the convergence target, helping to shorten the length of the forwarding link, thus resulting in better end-to-end latency. Another protocol with end-to-end latency performance close to this algorithm is the QLFR protocol, which employs a multi-link forwarding mechanism, thus also having a high probability of finding a shorter forwarding path.

[0133] Figure 11The figure illustrates the impact of the number of source nodes simultaneously on protocol delivery rates in NS3 simulation tests. As shown, due to differences in protocol mechanisms and modes, some protocols, such as this protocol, QELAR, and QLFR, require feedback mechanisms to update decision-making strategies, while others do not. To prevent interference from additional factors, the simulation uses the BroadcastMAC protocol, which inherently lacks any handshake or feedback mechanisms, minimizing interference with routing protocols. However, its collision avoidance is also poor. Therefore, as the number of source nodes increases, the delivery rates of all protocols show a downward trend, but overall, this protocol and FVBF still maintain the best delivery rates. The PDR decrease of this protocol is more severe than that of FVBF because this protocol is mainly based on a single communication link, thus its redundancy backup during transmission is not as good as the opportunistic FVBF protocol. Nevertheless, this protocol has better PDR performance than similar protocols such as QELAR and QLFR.

[0134] Figure 12 The impact of the number of source nodes on the protocol energy tax during NS3 simulation testing is demonstrated. As shown in the figure, with the increase of the number of source nodes at the same time, the energy tax of each protocol tends to increase to some extent, but this protocol still maintains the lowest packet energy tax, proving that the protocol has good energy-saving performance. The performance of most protocols changes little, only the QELAR protocol shows serious deterioration. This is mainly because the collision avoidance effect of the MAC protocol used is poor, and QELAR itself has not been optimized for multi-link interference compatibility. Therefore, with the increase of the number of source nodes, its delivery rate decreases significantly, leading to a serious deterioration in its packet delivery rate.

[0135] Figure 13 The figure illustrates the impact of the number of source nodes simultaneously on the end-to-end latency of several protocols in NS3 simulation tests. As shown in the figure, increasing the number of source nodes leads to two trends in the end-to-end latency of several protocols. The original protocol, FVBF, and QELAR show an increasing latency trend, possibly due to the additional time required for link repair after packet collisions. In contrast, HHVBF, QLFR, and ALRP show a relatively unchanged latency trend. This is likely because QLFR involves multi-link transmission, while HHVBF and ALRP's opportunistic forwarding exhibits flooding; therefore, as long as the communication link remains intact, their latency remains relatively stable.

[0136] In summary, the proposed protocol effectively balances delivery rate, end-to-end latency, and forwarding energy tax in static UWSNs. While its delivery rate isn't the highest among all algorithms, it can still quickly reach over 90% under suitable node density. Due to the convergence bootstrapping mechanism, the SAQL protocol tends to select the next-hop node closer to the convergence point, significantly increasing the probability of finding the shortest forwarding link. This effectively optimizes routing detours and even routing loops, reducing end-to-end latency. Finally, in static UWSNs, forwarding links typically maintain their validity for a longer period. Therefore, this protocol's use of a single forwarding link reduces redundant transmissions, thereby lowering routing energy consumption and resulting in a lower forwarding energy tax.

[0137] Those skilled in the art will understand that the above description is merely a preferred embodiment of the present invention, and the features described in the various embodiments and / or claims of this disclosure can be combined or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. This is not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0138] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method for optimizing static UWSNs routing protocol based on convergence guidance, characterized in that, The method comprises the following steps: Step 1, a forwarding decision model based on Q-Learning is used to establish an optimal forwarding path from a source node to a sink node; Step 2, according to whether there is a completely shared parameter between nodes in the static UWSNs routing protocol, the forwarding decision model in step 1 is divided into two categories, including a decision model taking a data packet as an agent and a decision model taking a node as an agent; The decision state of the node where the data packet is located and the forwarding candidate node information in the static UWSNs routing protocol is determined, when the node needs to forward the data packet, the Q value of each neighbor node is calculated in turn, and finally the neighbor node with the maximum Q value is selected as the next hop forwarding target; Step 3, a sink guiding mechanism is constructed, the sink guiding mechanism allows the node to generate a training reward based on the location information of the sink nodes in the training process of the Q-Learning forwarding decision model, generates a path for the data packet to gather to the center node, and completes the optimization of the static UWSNs routing protocol based on sink guiding.

2. The method of claim 1, wherein, All nodes share a set of reinforcement learning parameters in the forwarding decision model based on Q-Learning in step 1.

3. The method of claim 1, wherein, The data packet is implemented by the decision model taking the node as the agent in step 2.

4. The method of claim 3, wherein, The static UWSNs routing protocol also comprises the step of SAOL routing protocol design after deployment, and the SAOL routing protocol design comprises the steps of neighbor discovery, cluster formation and data forwarding.

5. The method of claim 1, wherein, The sink guiding mechanism in step 3 comprises the steps of reward function design and fixed V value mechanism.

6. The method of claim 5, wherein, The reward function design method is: R = R sink + R energy + R drift (4-4) where R is the return of the current decision; R sink is the convergence factor return; R energy is the energy factor return; R drift is the drift factor return.

7. The method of claim 5, wherein, The calculation method of the fixed V value mechanism in step 3 is: where V π (s) represents the expected sum of returns that can be obtained according to the π policy in the s state; Q π (s,a) represents the evaluation of the current decision action, and a is the learning rate of the system, 0 < a < 1.

8. The system for optimizing static UWSNs routing protocol based on convergence guidance, characterized in that, The system comprises: A forwarding path determination module is used to establish an optimal forwarding path from a source node to a sink node based on a forwarding decision model based on Q-Learning; A forwarding decision model construction module is used to divide the forwarding decision model of the forwarding path determination module into two categories, including a decision model taking a data packet as an agent and a decision model taking a node as an agent, according to whether there is a completely shared parameter between nodes in the static UWSNs routing protocol based on the static UWSNs routing protocol; The decision state of the node where the data packet is located and the forwarding candidate node information in the static UWSNs routing protocol is determined, when the node needs to forward the data packet, the Q value of each neighbor node is calculated in turn, and finally the neighbor node with the maximum Q value is selected as the next hop forwarding target; A static UWSNs routing protocol optimization module is used to construct a sink guiding mechanism, the sink guiding mechanism allows the node to generate a training reward based on the location information of the sink nodes in the training process of Q-Learning, generates a path for the data packet to gather to the center node, and completes the optimization of the static UWSNs routing protocol based on sink guiding.

9. Computer device comprising a memory and a processor, characterized in that The memory stores a computer program, and when the processor runs the computer program stored in the memory, the processor executes the method in any one of claims 1-7.

10. A computer readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the method in any one of claims 1-7.

Images