An underwater routing method based on dynamic pareto candidate set constraint Q learning

Abstract

The application relates to the technical field of underwater wireless routing networks, and provides an underwater routing method based on dynamic Pareto candidate set constraint Q learning. The method constructs a cumulative delay gradient taking a sink node as a zero reference, provides hop-by-hop local forwarding guidance for Q learning, and alleviates the reward feedback delay and sparsity problems caused by long propagation delay; a Pareto non-dominated candidate set is constructed based on the predicted residual energy, the cumulative delay, the smoothed noise level and the smoothed queuing delay of neighbors, the action selection of Q learning is constrained, and the maximum space of the next state Q value is maximized; a differentiated epsilon-greedy strategy is used to select the first and second competitive nodes, and a self-adaptive waiting timer is used to realize listening suppression and backup replacement; the Q value updating is only executed in the maximum of the next state Pareto non-dominated candidate set, so that the Q value overestimation tendency is alleviated, and the data packet delivery rate, the end-to-end delay and the energy efficiency are synergistically optimized.

Description

Technical Field

[0001] This invention relates to the field of underwater wireless routing network technology, and in particular to an underwater routing method based on dynamic Pareto candidate set constrained Q-learning. Background Technology

[0002] Underwater wireless sensor networks are a crucial component of the underwater Internet of Things (IoT), typically consisting of moorings or underwater sensor nodes deployed at varying depths, a convergence node located at or near the surface, and a shore-based control center. Mooring nodes collect and upload data hop-by-hop on marine environmental, resource exploration, disaster warning, and equipment operational status data via acoustic links. The convergence node then transmits the data to the shore-based center via satellite, cellular networks, or other multimode communication links. Underwater acoustic communication is characterized by low propagation speed, limited available bandwidth, significant propagation attenuation, and high susceptibility to environmental noise. Mooring nodes may also drift with ocean currents, causing continuous changes in network topology, link quality, and neighbor availability. Consequently, routing decisions directly impact packet delivery rate, end-to-end latency, energy consumption, and network lifetime.

[0003] Existing depth- or location-based heuristic routing typically selects the next hop based on node depth difference, location relationship, or preset forwarding area. While simple to implement, it has high overhead for accurate positioning. Furthermore, it is prone to getting stuck in routing holes when relying solely on depth-decreasing rules. Existing algorithms mostly focus on handling passive recovery after getting stuck in holes, resulting in extremely high additional overhead.

[0004] Furthermore, opportunistic routing attempts to improve reliability by constructing multiple candidate nodes, but in underwater environments with high propagation latency, half-duplex operation, and hidden terminals, multiple candidate contention often leads to timing suppression failures, repeated broadcasts, and severe redundant transmissions and additional energy consumption.

[0005] While reinforcement learning routing can adaptively update strategies based on network feedback, the long acoustic propagation delay underwater means that if the learning process relies primarily on global feedback after data packets travel through multiple hops and finally reach the sink node, the reward information obtained by intermediate nodes will be significantly delayed and sparse, making it difficult to determine in a timely manner whether the current forwarding action is beneficial for the data packet to reach the sink node. Furthermore, the underwater environment exhibits significant non-stationarity; after node drift, link mutations, or node failures, the historical Q-values ​​of some neighbors may have become outdated. If the maximum Q-value calculation is still directly performed on all available actions, the estimation error will be amplified by the maximization operation, resulting in a maximization-induced systematic positive estimation bias that may mislead subsequent routing decisions.

[0006] Underwater routing requires balancing multiple objectives, including energy, latency, noise, queuing delay, and reliability. However, existing algorithms often use fixed thresholds or fixed linear weights when constructing candidate sets, making it difficult to adapt to dynamic environments. This rigid parameter setting prevents the algorithm from dynamically filtering out inferior candidate nodes when faced with complex network conditions. Summary of the Invention

[0007] To address the problems existing in the prior art, this invention provides an underwater routing method based on dynamic Pareto candidate set constrained Q-learning, comprising: A cumulative delay gradient is constructed, which is based on the aggregation node as the zero baseline and increases layer by layer. The buoy node updates the cumulative delay hop by hop based on cross-layer piggybacking and listening mechanism. The buoy node listens to and acquires neighbor information, and updates the neighbor information of neighbor nodes through a sliding window. The neighbor information includes cross-layer information such as predicted remaining energy, cumulative delay, smoothed noise level, and smoothed queuing delay. The sending node constructs a pre-screened candidate set composed of neighboring nodes with a cumulative latency less than itself, constructs a multi-objective evaluation vector based on the neighbor prediction index, generates a Pareto non-dominated candidate set, and uses the Pareto non-dominated candidate set as the action selection space for the current state; when the data packet is forwarded to the next state node, its Pareto non-dominated candidate set is reconstructed based on the latest neighbor state of the next state node, and the next state maximization operation in the Q-value update is restricted to the Pareto non-dominated candidate set; The sending node selects the first competing node from the Pareto non-dominated candidate set according to the differentiated ε-greedy strategy, and selects the second competing node from the candidate set after removing the first competing node if there are remaining candidate nodes. The first competing node forwards the data packet immediately after receiving it, and the second competing node listens silently before the waiting timer expires. If it does not hear the same data packet forwarded by the first competing node, it performs backup forwarding after the waiting timer expires. The sending node updates the corresponding state-action Q value according to the forwarding action actually performed.

[0008] Specifically, when the second competing node When present, the waiting timer is an adaptive waiting timer. It consists of two parts: the basic propagation protection time and the additional time due to the Q-value difference. , In the formula, For maximum communication distance, The speed of sound underwater; Indicates the current state Next, select the second competing node. The state of the next jump action - the action Q value; Indicates the current state The state-action Q-value of the maximum next-hop action under Pareto candidate set constraints. Indicates the current state The state-action Q-value of the minimum next-hop action under Pareto candidate set constraints, where Represents a node Pareto candidate set constrained by Pareto nondominated algorithm; A constant set to prevent the Q value from being zero. .

[0009] Specifically, the node currently holding the data packet Current state Select neighbor nodes The next hop is the current action; when the data packet is successfully forwarded to the node... After that, node Become the next state And restrict the operation of maximizing the state value of the next node to that node. The corresponding Pareto non-dominated candidate set Within this timeframe, the Q value is updated according to the following formula: , In the formula, Let Q be the learning rate; Current state Forward to Instant rewards; Discount factor; For the next state Corresponding node Candidate neighbor nodes, Indicates the state Choose below The state of the next jump action – the action Q value. For nodes Real-time reconstructed Pareto non-dominated candidate set.

[0010] Specifically, the instant reward function According to neighbor nodes The predicted remaining energy, cumulative delay decrease, and queuing delay are constructed, and a positive incentive is given when the predicted remaining energy is higher than the local mean, relative to the current node. Positive incentives are given for nodes with larger cumulative latency reductions, while penalties are imposed on neighboring nodes whose queuing latency is higher than the local mean. The immediate reward function is: , In the formula, Neighboring nodes Predicted remaining energy, This represents the average predicted remaining energy of the current valid candidate neighbors. and These correspond to the maximum and minimum values; For the current node The cumulative delay, Neighboring nodes The cumulative delay of the most recent valid observation; Neighboring nodes Smooth queuing delay, This is the smoothed queuing delay average of the current valid candidate neighbors. and These correspond to the maximum and minimum values; and A constant set to prevent the normalization denominator from being zero. , ; The weights are non-negative and satisfy the following conditions: .

[0011] Specifically, the sending node writes the address of the first competing node into the DATA packet header, and simultaneously writes the address of the second competing node when the second competing node exists, and then broadcasts it through the acoustic channel; the first competing node forwards the data packet immediately after receiving it; when the second competing node exists, it starts a waiting timer after receiving the data packet and silently listens for data packets with the same data packet identifier; If the second competing node detects the same data packet forwarded by the first competing node before the waiting timer expires, it deletes the cached data packet and ends the current action. If the second competing node does not detect the same data packet forwarded by the first competing node when the timer expires, it executes the backup forwarding task.

[0012] Specifically, the cumulative latency of the sink node remains zero, while the cumulative latency of the ordinary buoy nodes is initialized to a finite upper bound. The underwater buoy node obtains the single-hop propagation delay based on the sending and receiving timestamps, or estimates the single-hop propagation delay based on the distance between nodes and the underwater speed of sound. In each update cycle, it calculates the minimum sum of the cumulative delays of its neighbors and the corresponding single-hop propagation delays. The smaller value is taken as its cumulative latency and then piggybacked into the HELLO and DATA packets for hop-by-hop propagation; where the total number of deployed nodes is assumed to be... Maximum communication distance is The speed of sound underwater is ,but When the physical neighbor set is empty, the cumulative delay of a regular buoy node is taken as... .

[0013] Specifically, the multi-objective evaluation vector includes: Energy evaluation indicators , , In the formula, Neighboring nodes Predicted remaining energy, To predict the maximum value of remaining energy in the pre-screened candidate set; Latency evaluation index , , In the formula, For time delay sensitivity adjustment parameters, , For data packets via neighboring nodes The estimated cumulative propagation delay when forwarding to the aggregation node; Noise evaluation indicators , , In the formula, Neighboring nodes The smoothing noise level, and To smooth out the maximum and minimum noise levels in the pre-screened candidate set, A constant set to prevent the noise range from being zero. ; Queuing delay evaluation indicators , , In the formula, Neighboring nodes Smooth queuing delay, This represents the maximum smooth queuing delay in the pre-screened candidate set. constant, .

[0014] Specifically, the predicted remaining energy When the sliding window contains at least two valid observations, the average energy consumption rate of neighbors is calculated based on the energy difference and time difference between adjacent observations, and then linearly extrapolated to the current time using the remaining energy of the most recent observation as a reference. When the sliding window contains only one valid observation, the remaining energy of the most recent observation is directly used as the reference. The smoothed noise level and smooth queuing delay Updated using exponentially weighted moving averages.

[0015] Specifically, the sliding window is independently established according to the neighbor node address and updated in a first-in-first-out manner. Each neighbor node is set with an aging timer. When a neighbor node fails to generate a valid listenable data packet beyond the preset timeout threshold, the neighbor node is marked as unselectable and excluded from the construction range of the pre-screened candidate set and the Pareto non-dominated candidate set until the valid status information of the neighbor node is obtained again.

[0016] Specifically, the sending node selects from the Pareto non-dominated candidate set according to the first exploration probability. The first competing node is obtained by performing ε-greedy selection. When the candidate set contains two or more nodes, the first competing node is removed. The remaining candidate set is then determined by the second exploration probability. The second competing node is obtained by performing ε-greedy selection. ,in Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention utilizes the cumulative delay propagated outward from the sink node to construct a delay gradient oriented towards the sink node. This transforms the sink node directional information, which originally required multiple hops to obtain, into hop-by-hop forwarding guidance that can be used locally by each node. This enables Q-learning to obtain local feedback based on the cumulative delay decrease generated by the candidate next hop, thereby alleviating the reward feedback delay and sparsity problem caused by long underwater propagation delay. At the same time, by handling temporarily disconnected network areas through a finite upper bound, when an effective cumulative delay gradient exists, the data packet selects the next hop only along the strictly decreasing direction of the cumulative delay. This avoids routing loops and reduces the risk of entering routing holes from the candidate construction stage, reducing the waiting, backoff, and additional energy consumption caused by passive hole recovery.

[0017] 2. This invention first excludes backward propagation and loop neighbors using the cumulative delay reduction rule, and then uses the Pareto dominance relationship to perform dynamic screening without fixed weight preference among energy, delay, noise and queuing delay. This can retain non-dominated candidates with advantages on different objectives and avoid the candidate set being too wide, too narrow or inferior candidate residue caused by a single threshold or fixed weight.

[0018] 3. This invention uses the dynamic Pareto non-dominated candidate set as both the constraint action space for Q-learning and the Q-value maximization space for the next state. Before the maximum value operation, it removes dominated nodes that have deteriorated in real time based on the latest multidimensional observations and narrows the candidate range for the maximization operation, thereby alleviating the tendency of Q-value overestimation in non-stationary underwater environments and improving learning stability and routing robustness.

[0019] 4. This invention employs a differentiated exploration probability selection method to choose the first and second competing nodes. The first competing node forwards the data immediately after receiving it, while the second competing node uses an adaptive waiting timer to suppress listening and provide backup. The waiting timer integrates the basic propagation protection time with the Q-value difference additional time and uses a small positive lower bound to avoid division by zero when the Q-value is zero. This allows the higher-quality second competing node to have a shorter additional waiting time, reducing redundant broadcasts, additional energy consumption, and end-to-end delays caused by multiple candidate competition while maintaining the reliability of backup forwarding. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of an underwater wireless sensor network scenario to which this invention is applicable; Figure 2 This is a schematic diagram of the cumulative delay propagation mechanism of the present invention; Figure 3 This is a flowchart of the dynamic Pareto candidate set constraint Q-learning routing process of the present invention; Figure 4 This is a schematic diagram of the dual-competition candidate node forwarding mechanism of the present invention; Figure 5 This is a comparison chart of the data packet delivery rate of the present invention with that of existing algorithms under different numbers of nodes; Figure 6 This is a comparison chart of packet delivery rates under different ablation configurations for the present invention; Figure 7 This is a comparison chart of the energy consumption per unit of successful data packet of the present invention with that of existing algorithms under different numbers of nodes; Figure 8 This is a comparison chart of the energy consumption per unit of successful data packet under different ablation configurations according to the present invention; Figure 9 This is a comparison chart of the average end-to-end latency of the present invention with that of existing algorithms under different numbers of nodes; Figure 10 This is a comparison chart of the average end-to-end delay of the present invention under different ablation configurations; Figure 11 This represents the number of routing hole encounters under different node numbers in this invention. Detailed Implementation

[0021] The embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and should not be construed as limiting the scope of the invention.

[0022] refer to Figure 1The underwater wireless sensor network in this embodiment includes a convergence node located in the central area of ​​the water surface and multiple moored nodes anchored at different depths. The moored nodes drift randomly with ocean currents near their initial deployment positions. Half-duplex underwater acoustic communication is used between the nodes. The convergence node receives data uploaded hop-by-hop by the moored nodes and transmits it to the control center via satellite or shore-based wireless links. Each moored node is equipped with a processor, acoustic modem, power module, data buffer, and environmental sensors, and maintains a neighbor table, sliding window, and Q-value table locally.

[0023] When the network starts up, the aggregation node maintains its accumulated latency at 1. And broadcast HELLO packets at preset intervals. Assume the total number of nodes deployed in the network (including the aggregation node) is... The maximum communication distance between nodes is The speed of sound underwater is The maximum single-hop propagation delay is To handle nodes or network areas temporarily disconnected from the sink node, and to prevent the historical accumulated latency from continuing to grow unbounded after the disconnection, a finite upper bound for the accumulated latency is defined. : .

[0024] Ordinary underwater mooring nodes are all based on Initialization. During each update cycle. At the end, node Collect the accumulated latency obtained from neighboring nodes' broadcasts or passive listening, and update your own accumulated latency according to the minimum sum of the neighboring nodes' accumulated latency and the corresponding single-hop propagation latency: , In the formula, Represents a node Updated cumulative latency Represents a node The set of physical neighbors, Representing neighboring nodes The cumulative latency in the current update cycle, Represents a node and The single-hop acoustic propagation delay between. When When the inner minimum value is considered as Thus making .

[0025] by Figure 2 For example, buoy node 3 obtains the cumulative delay from buoy node 1 and buoy node 2 respectively. and And based on the single-hop propagation delay and Update as follows: , Updated The header of the HELLO and DATA packets subsequently sent by node 3 is written into the data.

[0026] Through daily service forwarding and periodic HELLO interactions between buoy nodes, the zero-latency baseline of the aggregation node propagates layer by layer to distant buoy nodes, forming a layered structure with gradually increasing cumulative latency. This mechanism transforms the path latency information in the direction of the aggregation node into a cumulative latency state that can be obtained locally by each buoy node, allowing nodes to determine the progress of the current hop in the direction of the aggregation node based on the cumulative latency decrease between the current node and the candidate next hop without waiting for the data packet to finally reach the aggregation node. This decrease is then used for the latency component of the immediate reward, thereby providing hop-by-hop local forwarding guidance and mitigating the reward feedback delay and sparsity problems caused by long underwater propagation latency. Ordinary nodes only... The current available sink node has the accumulated time delay gradient in the direction of the sink node; when any buoy node selects its next hop, it only retains the buoy that satisfies the following conditions. Neighbors as pre-screened candidate set This ensures that each effective forwarding link satisfies the requirement of strictly decreasing cumulative delay and avoids the formation of routing loops.

[0027] The DATA header includes at least the packet type, packet ID, source node address, destination node address, first contention node address, optional second contention node address, cumulative delay, remaining energy, queuing delay, and noise level; the HELLO header includes at least the packet type, cumulative delay, remaining energy, queuing delay, and noise level.

[0028] underwater buoy node When sending or forwarding a DATA packet, the address of the first contention node, the address of the second contention node (if any), the accumulated delay, remaining energy, queuing delay, and noise level are written into the packet header. The next-state Q-value information for Q-value updates can be obtained through DATA packet piggybacking or inter-node listening. When a node sends a HELLO packet, the accumulated delay, remaining energy, queuing delay, and noise level are written into the packet header. Due to the broadcast characteristics of the underwater acoustic channel, in addition to the designated receiving node, other neighboring nodes within the communication range can also passively listen to the header information of the DATA and HELLO packets.

[0029] Neighboring nodes determine whether data packets can be correctly decoded based on the signal-to-interference-plus-noise ratio (SIR) of the received signal, thereby obtaining link availability and noise levels. The SIR is jointly determined by transmit power, propagation attenuation, ambient noise, and simultaneously arriving interference signals. Propagation attenuation can be achieved using the Thorp absorption model, and background noise can be achieved using the Urick empirical model, which includes turbulence noise, shipping noise, wind and wave noise, and thermal noise.

[0030] Each daemon node maintains a neighbor table, a sliding window, and a Q-value table locally. The Q-value table records the state-action value of the node under different next-hop actions. Among these, the current node... Select neighbor nodes When used as the next hop, the corresponding state-action Q-value represents the long-term expected value of that forwarding action. This occurs when the packet is successfully forwarded to the node. After that, node The next state becomes the next action space, which is then reconstructed in real time from the Pareto non-dominated candidate set. Given: Each node establishes an independent sliding window based on its neighbor addresses. The sliding window is updated in a first-in, first-out manner, with new records written to the end of the window as they arrive. When the number of records reaches the capacity... The earliest record is then deleted. An aging timer is set for each neighbor. If a neighbor fails to generate a listenable data packet after exceeding the preset timeout threshold, the neighbor is temporarily marked as unselectable or its expired record is deleted until valid information is obtained again.

[0031] The neighbor information used for Pareto evaluation is calculated based on a sliding window. The neighbor information includes at least cross-layer information such as predicted remaining energy, cumulative delays via neighbor forwarding, smoothed noise levels, and smoothed queuing delays.

[0032] Before making a routing decision, the buoy node calculates its neighbors based on the energy difference and time difference between adjacent observations within the sliding window. Average energy consumption rate The predicted remaining energy is obtained by linearly extrapolating from the most recently observed remaining energy to the current time. To compensate for the state lag caused by the long underwater propagation time delay: , In the formula, This represents the number of currently valid observations in the sliding window. For the observation record index, and Neighboring nodes No. The remaining energy and timestamp in each valid observation record. The average energy dissipation rate is only... Calculated in time.

[0033] when At that time, neighboring nodes Predicted remaining energy at the current moment for: , In the formula, and These are the remaining energy and timestamp of the most recent valid observation, respectively. For the current prediction time; when At that time, without extrapolating the trend, directly set .

[0034] Noise levels and queuing delays are smoothed using an exponentially weighted moving average to reduce the impact of transient disturbances and sudden congestion on candidate structures.

[0035] Data packets via neighboring nodes The estimated cumulative propagation delay when forwarding to the aggregation node. From the current node to neighboring nodes Single-hop propagation delay Cumulative delay of the most recent valid observation from a neighboring node Joint decision: , Given the time-varying characteristics of underwater acoustic noise, neighboring nodes Smoothing noise level Updated using exponentially weighted moving average: , In the formula, This is the most recent noise observation value. The EWMA estimate of the noise before the update. This is the noise smoothing coefficient; Because the queuing delay of the buoy node buffer queue is highly volatile, neighboring nodes... Smooth queuing delay Updated using an independent exponentially weighted moving average: , In the formula, This is the most recent queuing delay observation. This is the EWMA estimate of the queuing delay before the update. This is the queuing delay smoothing coefficient; Noise smoothing coefficient With queuing delay smoothing coefficient It can be set according to the degree of environmental noise fluctuation and the degree of business emergencies.

[0036] Current underwater node Traverse the physical neighbors whose aging timers have not expired, and retain only those that meet the criteria. The neighbors are selected so that each effective forwarding proceeds in a strictly decreasing direction of accumulated delay, avoiding routing loops and reducing the risk of entering routing holes from the candidate construction stage, thus constructing a pre-screened candidate set. .

[0037] Neighbor nodes are constructed using four positive metrics: energy, latency, noise, and queuing latency. A multi-objective evaluation vector. Given the different dimensions and optimization directions of each evaluation index, the pre-screened candidate set... Candidate nodes in the dataset are normalized and uniformly mapped to a positive objective function; where , , , , and All are taken from the current pre-screened candidate set. The specific mapping rules are as follows: Energy evaluation indicators for: , In the formula, Neighboring nodes Predicted remaining energy, To predict the maximum value of remaining energy in the pre-screened candidate set; Latency evaluation index for: , In the formula, This is a parameter for adjusting the time delay sensitivity. via neighbor nodes The estimated cumulative propagation delay for continued forwarding to the aggregation node; Noise evaluation indicators for: , In the formula, Neighboring nodes The smoothing noise level, and To smooth out the maximum and minimum noise levels in the pre-screened candidate set, The value should be a constant to avoid a denominator of 0. ; Queuing delay evaluation indicators for: , In the formula, Neighboring nodes Smooth queuing delay, This represents the maximum smooth queuing delay in the pre-screened candidate set. The value should be a constant to avoid a denominator of 0. .

[0038] For any two candidate neighbor nodes and ,when It is not inferior to in any of the evaluation dimensions. And it is better than at least one evaluation dimension. At that time, the judgment Pareto Domination In the pre-screened candidate set Based on this, the neighbors that are not dominated by any other candidate neighbor nodes constitute the Pareto non-dominated candidate set of the current potential skewer node. This set serves both as the constraint action space for subsequent Q-learning and as a constraint on the computational range for maximizing the Q-value in the next state.

[0039] In one implementation, the Pareto non-dominated candidate set The generation process includes: initializing an empty set. ; sequentially from the pre-screened candidate set Extract candidate nodes And compare it with each of the other candidate nodes one by one; if there are other candidate nodes that dominate... Then discard Otherwise join in After the traversal is complete, let .

[0040] Because the quality of underwater acoustic communication links is significantly time-varying, this invention constructs a dual-competition candidate forwarding mechanism to reduce the risk of local hop forwarding failures caused by node drift, link fading, or sudden collisions. When there are at least two selectable nodes in the Pareto non-dominated candidate set, the sending node selects the first competing node from the Pareto non-dominated candidate set. And if there are remaining candidates, select the second competing node. The sending node writes the corresponding address into the DATA packet header and broadcasts it through the acoustic channel. Forward the data packet immediately upon receipt; If the data packet exists, it receives and buffers it, and simultaneously starts a wait timer to silently listen. If Listening before the timer expires If the same data packet is forwarded, the cache is cleared and forwarding is suppressed; if the timer expires and no packet is detected, then... Perform standby forwarding; other unselected neighbors only update the neighbor information they have been listening to without forwarding the packet.

[0041] The first and second competing nodes are selected using a differentiated ε-greedy strategy. The sending node first selects from... China and Israel explore probability Executing ε-greedy selection yields When the candidate set contains two or more nodes, remove them. Then construct a remaining candidate set, and then explore from the remaining candidate set with a higher probability. Executing ε-greedy selection yields ,in .therefore To assume the main role of utilization While maintaining backup reliability, it takes on a stronger exploration role.

[0042] Specifically, for any ε-greedy choice, with The probability is used to select the action with the largest Q-value in the current candidate set. The probability of randomly selecting an action from the candidate set; the second selection only occurs after elimination. Execute on the remaining candidate set, thereby ensuring and For different nodes.

[0043] To suppress redundant transmission and end-to-end latency while ensuring the reliability of backup forwarding, the second contention node employs an adaptive waiting timer. This timer is determined by the base propagation protection time. The difference between the Q value and the time component constitute the additional time, which is based on... In the current Pareto non-dominated candidate set relative to and The position is determined adaptively: the higher the Q value of the second competing node, the shorter its additional wait time.

[0044] When sending node Confirm the first or second competing node After the current hop forwarding is actually completed, the sending node determines the next node based on its neighboring nodes. The predicted remaining energy, cumulative delay decrease, and queuing delay are used to calculate the instant reward. Positive incentives are given when the predicted remaining energy is higher than the local mean; higher positive incentives are given when the cumulative delay decreases; and penalties are imposed when the queuing delay is higher than the local mean. The immediate reward function is: , In the formula, , , and Neighboring nodes The predicted remaining energy and the mean, maximum and minimum values ​​of the current effective candidate neighbors; Accumulate latency for the current node. Neighboring nodes The most recent effective cumulative delay; , , and Neighboring nodes The smooth queuing delay and the mean, maximum and minimum values ​​corresponding to the current valid candidate neighbors; and The value should be a constant to avoid a denominator of 0. , ; The weights are non-negative and their sum is 1.

[0045] To avoid misleading routing decisions by having outdated high Q-values ​​continuously participate in maximization calculations in non-stationary underwater environments, this scheme does not calculate the next-state maximum Q-value on all physical neighbors. Instead, it first reconstructs the Pareto non-dominated candidate set in real time based on the latest energy, accumulated delay, noise, and queuing status, and then performs Q-value maximization only within this set. By narrowing the candidate range for Q-value maximization and excluding dominated nodes with deteriorating real-time states before maximization calculation, the tendency for Q-value overestimation in non-stationary underwater environments is mitigated. (The last sentence appears to be a separate, unrelated statement: "Nodes currently holding data packets...") Current state Select neighbor nodes The next hop is the current action; when the data packet is successfully forwarded to the node... After that, node Become the next state And restrict the operation of maximizing the state value of the next node to that node. The corresponding Pareto non-dominated candidate set Within this timeframe, the Q value is updated according to the following formula: , In the formula, Let Q be the learning rate; Current state Forward to Instant rewards; Discount factor; For the next state Corresponding node Candidate neighbor nodes, Indicates the state Choose below The state of the next jump action – the action Q value. For nodes Real-time reconstructed Pareto non-dominated candidate set. When the node is a convergence node or there are no executable subsequent actions, the subsequent state value is set to 0.

[0046] Each node only updates the Q value corresponding to the actual action performed in its local Q table. When a node... Select node After executing this forwarding and confirming its success, update the local Q table. The next state value is determined based on the Q-value in the Pareto non-dominated candidate set corresponding to the next state. The Q-value of candidate nodes that did not perform local hop forwarding is not updated by this data packet; their broadcast or listening information can be used to refresh neighbor state information. The next state Q-value information used for Q-value updates can be obtained through data packet piggybacking or inter-node listening.

[0047] In this invention, Reflects the current sending node Select the second competing node As a long-term comprehensive value of the next-hop action, after mapping the Q value to waiting priority, a higher Q value corresponds to a shorter additional waiting time, thus enabling faster replacement when the first contention node fails; a lower Q value has a longer additional waiting time to reduce unnecessary contention. Adaptive waiting timer. for: , In the formula, Basic transmission protection time; , , The state-action Q-value corresponding to the second competing node. A constant set to prevent the denominator from being zero. . The closer The shorter the additional waiting time; when Sometimes, The Q-value difference additional term is zero, and the waiting time degenerates to .

[0048] Pareto non-dominated candidate set of the current node In this round, no data forwarding is performed, and a HELLO request may be triggered to wait for neighbor status or cumulative latency gradient updates; when At that time, the only candidate node becomes the first competing node. No second competing node is set; when At that time, the dual competition process shall be followed.

[0049] To prevent duplicate packet forwarding, each neighboring node maintains a short-term duplicate packet buffer based on the source address and packet identifier, and discards the processed packet directly when it detects it.

[0050] by Figure 4For example, node 2 is the current sending node, node 4 is selected as the first competing node, and node 3 is selected as the second competing node. After receiving the data packet, node 4 immediately attempts to forward it; node 3 starts waiting and listening to node 4. If node 4 fails to send due to link fading or collision, node 3 forwards the packet after the timer expires, thereby improving reliability; if node 4 successfully forwards the packet, node 3 suppresses its own forwarding after listening to the same data packet, thereby avoiding redundancy.

[0051] In a set of simulation embodiments, the network area was set to 2000m×2000m×1000m, the number of nodes was set to 10 to 50, the maximum node drift speed was 1m / s, and the maximum communication distance was [not specified]. The underwater speed of sound is 800m. Take 1500m / s as the sliding window capacity. Set the smoothing coefficient to 5. and All values ​​are set to 0.2, with a reward weight of 0.2. The Q-learning rate is set to 0.4, 0.4, and 0.2 respectively. Set to 0.1, discount factor Take 0.9, the probability of the first exploration. Set it to 0.1, the second exploration probability Take 0.5.

[0052] The algorithm of this invention is compared with the traditional algorithm DBR, as well as reinforcement learning routing algorithms such as RLOR, QLFR, and QLRO, and multiple ablation experiments are conducted. Simulation results are presented. Figures 5 to 11 This demonstrates that the method of this invention can maintain a high packet delivery rate and suppress the rapid increase in energy consumption per successful packet and end-to-end latency as node density increases. Ablation results show that removing the dual-competition candidate mechanism increases redundant forwarding or reduces reliability; removing the Pareto constraint expands the candidate space and weakens the constraint on the participation of outdated high-Q values ​​in maximization calculations; and using depth reduction instead of cumulative delay gradients increases the risk of routing holes.

[0053] The above are merely preferred embodiments of the present invention. It should be noted that, for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An underwater routing method based on dynamic Pareto candidate set constrained Q-learning, characterized in that, include: A cumulative delay gradient is constructed, which is based on the aggregation node as the zero baseline and increases layer by layer. The buoy node updates the cumulative delay hop by hop based on cross-layer piggybacking and listening mechanism. The buoy node listens to and acquires neighbor information, and updates the neighbor information of neighbor nodes through a sliding window. The neighbor information includes cross-layer information such as predicted remaining energy, cumulative delay, smoothed noise level, and smoothed queuing delay. The sending node constructs a pre-screened candidate set composed of neighboring nodes with a cumulative latency less than itself, constructs a multi-objective evaluation vector based on the neighbor prediction index, generates a Pareto non-dominated candidate set, and uses the Pareto non-dominated candidate set as the action selection space for the current state; when the data packet is forwarded to the next state node, its Pareto non-dominated candidate set is reconstructed based on the latest neighbor state of the next state node, and the next state maximization operation in the Q-value update is restricted to the Pareto non-dominated candidate set; The sending node selects the first competing node from the Pareto non-dominated candidate set according to the differentiated ε-greedy strategy, and selects the second competing node from the candidate set after removing the first competing node if there are remaining candidate nodes. The first competing node forwards the data packet immediately after receiving it, and the second competing node listens silently before the waiting timer expires. If it does not hear the same data packet forwarded by the first competing node, it performs backup forwarding after the waiting timer expires. The sending node updates the corresponding state-action Q value according to the forwarding action actually performed.

2. The underwater routing method based on dynamic Pareto candidate set constrained Q-learning according to claim 1, characterized in that, When the second competing node When present, the waiting timer is an adaptive waiting timer. It consists of two parts: the basic propagation protection time and the additional time due to the Q-value difference. ， In the formula, For maximum communication distance, The speed of sound underwater; Indicates the current state Next, select the second competing node. The state of the next jump action - the action Q value; Indicates the current state The state-action Q-value of the maximum next-hop action under Pareto candidate set constraints. Indicates the current state The state-action Q-value of the minimum next-hop action under Pareto candidate set constraints, where Represents a node Pareto candidate set constrained by Pareto nondominated algorithm; A constant set to prevent the Q value from being zero. .

3. The underwater routing method based on dynamic Pareto candidate set constrained Q-learning according to claim 1, characterized in that, The node currently holding the data packet Current state Select neighbor nodes The next hop is the current action; when the data packet is successfully forwarded to the node... After that, node Become the next state And restrict the operation of maximizing the state value of the next node to that node. The corresponding Pareto non-dominated candidate set Within this timeframe, the Q value is updated according to the following formula: ， In the formula, Let Q be the learning rate; Current state Forward to Instant rewards; Discount factor; For the next state Corresponding node Candidate neighbor nodes, Indicates the state Choose below The state of the next jump action – the action Q value. For nodes Real-time reconstructed Pareto non-dominated candidate set.

4. The underwater routing method based on dynamic Pareto candidate set constrained Q-learning according to claim 3, characterized in that, The instant reward function According to neighbor nodes The predicted remaining energy, cumulative delay decrease, and queuing delay are constructed, and a positive incentive is given when the predicted remaining energy is higher than the local mean, relative to the current node. Positive incentives are given for nodes with larger cumulative latency reductions, while penalties are imposed on neighboring nodes whose queuing latency is higher than the local mean. The immediate reward function is: ， In the formula, Neighboring nodes Predicted remaining energy, This represents the average predicted remaining energy of the current valid candidate neighbors. and These correspond to the maximum and minimum values; For the current node The cumulative delay, Neighboring nodes The cumulative delay of the most recent valid observation; Neighboring nodes Smooth queuing delay, This is the smoothed queuing delay average of the current valid candidate neighbors. and These correspond to the maximum and minimum values; and A constant set to prevent the normalization denominator from being zero. , ; The weights are non-negative and satisfy the following conditions: .

5. The underwater routing method based on dynamic Pareto candidate set constrained Q-learning according to claim 1, characterized in that, The sending node writes the address of the first competing node into the DATA packet header, and when the second competing node exists, it simultaneously writes the address of the second competing node and broadcasts it through the acoustic channel; the first competing node forwards the data packet immediately after receiving it; when the second competing node exists, it starts a waiting timer after receiving the data packet and silently listens for data packets with the same data packet identifier. If the second competing node detects the same data packet forwarded by the first competing node before the waiting timer expires, it deletes the cached data packet and ends the current action. If the second competing node does not detect the same data packet forwarded by the first competing node when the timer expires, it executes the backup forwarding task.

6. The underwater routing method based on dynamic Pareto candidate set constrained Q-learning according to claim 1, characterized in that, The cumulative latency of the sink node remains zero, while the cumulative latency of the ordinary buoy nodes is initialized to a finite upper bound. ; The underwater buoy node obtains the single-hop propagation delay based on the sending and receiving timestamps, or estimates it based on the distance between nodes and the underwater speed of sound. In each update cycle, it calculates the minimum sum of the cumulative delays of its neighbors and their corresponding single-hop propagation delays. The smaller value is taken as its cumulative latency and then piggybacked into the HELLO and DATA packets for hop-by-hop propagation; where the total number of deployed nodes is assumed to be... Maximum communication distance is The speed of sound underwater is ,but When the physical neighbor set is empty, the cumulative delay of a regular buoy node is taken as... .

7. The underwater routing method based on dynamic Pareto candidate set constrained Q-learning according to claim 1, characterized in that, The multi-objective evaluation vector includes: Energy evaluation indicators , ， In the formula, Neighboring nodes Predicted remaining energy, To predict the maximum value of remaining energy in the pre-screened candidate set; Latency evaluation index , ， In the formula, For time delay sensitivity adjustment parameters, , For data packets via neighboring nodes The estimated cumulative propagation delay when forwarding to the aggregation node; Noise evaluation indicators , ， In the formula, Neighboring nodes The smoothing noise level, and To smooth out the maximum and minimum noise levels in the pre-screened candidate set, A constant set to prevent the noise range from being zero. ; Queuing delay evaluation indicators , ， In the formula, Neighboring nodes Smooth queuing delay, This represents the maximum smooth queuing delay in the pre-screened candidate set. A constant set to prevent the denominator from being zero. .

8. The underwater routing method based on dynamic Pareto candidate set constrained Q-learning according to claim 7, characterized in that, The predicted remaining energy When the sliding window contains at least two valid observations, the average energy consumption rate of neighbors is calculated based on the energy difference and time difference between adjacent observations, and then linearly extrapolated to the current time using the remaining energy of the most recent observation as a reference. When the sliding window contains only one valid observation, the remaining energy of the most recent observation is directly used as the reference. The smoothed noise level and smooth queuing delay Updated using exponentially weighted moving averages.

9. The underwater routing method based on dynamic Pareto candidate set constrained Q-learning according to claim 1, characterized in that, The sliding window is independently established according to the neighbor node address and updated in a first-in-first-out manner. Each neighbor node is set with an aging timer. When a neighbor node fails to generate a valid listenable data packet after exceeding the preset timeout threshold, the neighbor node is marked as unselectable and excluded from the construction range of the pre-screened candidate set and the Pareto non-dominated candidate set until the valid status information of the neighbor node is obtained again.

10. The underwater routing method based on dynamic Pareto candidate set constrained Q-learning according to claim 1, characterized in that, Sending nodes from the Pareto non-dominated candidate set according to the first exploration probability The first competing node is obtained by performing ε-greedy selection. ; When the candidate set contains two or more nodes, the first competing node is removed. The remaining candidate set is then determined by the second exploration probability. The second competing node is obtained by performing ε-greedy selection. ,in .

Images