Reinforcement learning route optimization method considering dynamic underwater acoustic environment and node mobility
By using reinforcement learning-based routing optimization methods, combined with dynamic environment awareness and node mobility, the routing decisions of underwater networks are optimized, solving the problems of link interruption and uneven energy consumption in dynamic environments, and achieving efficient data delivery and extended network lifetime.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2026-04-03
- Publication Date
- 2026-08-04
AI Technical Summary
Existing underwater routing protocols struggle to cope with the challenges posed by dynamic underwater acoustic environments and node mobility, leading to link interruptions, uneven energy consumption, and shortened network lifetime. Existing intelligent optimization algorithms also struggle to achieve real-time and efficient path optimization in complex, time-varying scenarios.
A reinforcement learning-based route optimization method is adopted, which combines dynamic environment awareness and node mobility. Through an ε-greedy policy and a multi-objective reward function, the network topology connection status is updated in real time to optimize routing decisions, and control overhead is reduced through distributed learning.
It achieves efficient data delivery of underwater sensor networks in dynamic environments, optimizes energy consumption and latency for multiple objectives, and improves the robustness and lifetime of the network.
Smart Images
Figure CN121985392B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of underwater acoustic communication and network routing technology, and in particular to a reinforcement learning routing optimization method that considers dynamic underwater acoustic environment and node mobility, for realizing dynamic, efficient and reliable data forwarding in underwater acoustic sensor networks. Background Technology
[0002] Underwater acoustic sensor networks are core infrastructures for marine environmental monitoring, resource exploration, and disaster early warning. Because they communicate through underwater acoustic channels, which inherently possess characteristics such as high latency, narrow bandwidth, strong time-varying nature, and high bit error rate, the design of routing protocols for underwater networks faces significant challenges. Traditional underwater routing protocols, such as static path-based routing, location-based routing, or depth-based routing, typically rely on pre-defined rules or simplified network models. These methods often exhibit insufficient adaptability when facing dynamically changing underwater acoustic propagation conditions, node location drift caused by ocean currents and biological activity, and the limited energy supply of nodes. Specifically, static routing strategies cannot cope with link interruptions caused by channel quality fluctuations or node movement; while decisions relying solely on distance or depth ignore energy balance and link stability, easily leading to network energy depletion and shortening overall lifetime.
[0003] In recent years, to improve network adaptability, some studies have introduced intelligent optimization algorithms or mechanisms based on network state feedback. However, most of these methods focus on a single optimization objective (such as minimizing hop count or energy consumption) or rely on the collection and processing of global information. This can lead to considerable control overhead and communication latency in dynamic, distributed deep-water environments, making it difficult to achieve real-time, efficient path optimization. Furthermore, existing solutions generally lack the ability to systematically model and learn the coupling relationship between underwater environmental dynamics and node mobility, resulting in difficulties in achieving optimal performance in routing decisions under complex, time-varying scenarios.
[0004] Therefore, there is an urgent need for an intelligent routing method capable of autonomously learning and adapting to the dynamic underwater environment and node mobility. This method needs to comprehensively perceive multi-dimensional network states, evaluate link quality and stability online, and make distributed decisions based on multi-objective optimization principles (such as balancing data delivery rate, end-to-end latency, and network energy consumption). Reinforcement learning technology, due to its ability to optimize policies through trial and error in unknown environments, offers a highly promising solution to these problems. However, directly applying reinforcement learning to underwater routing still faces challenges such as complex state space design, difficulty in accurately characterizing multi-objective trade-offs in the reward function, and low learning efficiency in dynamic environments. Summary of the Invention
[0005] To address the shortcomings of existing technologies, this invention proposes a reinforcement learning-based route optimization method that considers dynamic underwater acoustic environments and node mobility.
[0006] The specific technical solution is as follows:
[0007] A reinforcement learning-based route optimization method considering dynamic underwater acoustic environments and node mobility includes the following steps:
[0008] Step 1: Initialize the parameters of the underwater sensor network;
[0009] Step 2: Monitor and update underwater environmental parameters and node locations in real time; based on node locations and channel conditions, evaluate the quality and stability of inter-node links to dynamically update the network topology connection status.
[0010] Step 3: When data needs to be transmitted, the agent within the node constructs a multi-dimensional state vector for reinforcement learning, and adopts an ε-greedy strategy to balance utilization and exploration, selects routing actions, and generates routing decisions;
[0011] Step 4: Execute the routing action. If the data packet transmission fails, record the failure information and proceed to Step 6; otherwise, calculate the multi-target reward value; update the Q-value table based on the transmission result and the multi-target reward value to optimize the routing decision.
[0012] Step 5: Monitor the performance indicators of the underwater sensor network in real time, and adaptively adjust the reinforcement learning parameters of the agent based on the network performance indicators and underwater environmental parameters;
[0013] Step 6: Determine if the termination iteration condition is met. If not, repeat steps 2-6 using the adaptively adjusted agent. If yes, end the iteration and output the adaptive routing model.
[0014] Furthermore, in step one, parameter initialization includes: initial deployment of underwater sensor network nodes, initial configuration of underwater environmental parameters, setting of communication range, and initialization of reinforcement learning agents within each node;
[0015] The underwater environmental parameters include: sound speed propagation conditions, noise level parameters, and ocean current speed.
[0016] Furthermore, the multidimensional state vector includes: the node's own state, the neighboring node's state, channel conditions, mobility prediction results, and data packet characteristics; these parameters with different dimensions are normalized and unified into state feature values to obtain the multidimensional state vector.
[0017] The node's own state includes the node's remaining energy; the neighbor node's state includes the neighbor node's energy level and the distance between nodes; the channel conditions include the signal-to-noise ratio and channel quality; the mobility prediction results include the node's relative moving speed and the predicted link stability value; and the data packet characteristics include data packet priority.
[0018] Furthermore, in step three, the agent adopts an ε-greedy strategy, based on the current multidimensional state vector S. t From the independent Q-value table constructed for each node based on the state discretization method, select the optimal or exploratory action A. t The process of determining the optimal next-hop forwarding node for data packets, i.e., selecting a route, and generating a route decision, is as follows:
[0019] Generate a random number rand in the interval [0,1]. If rand < ε, where ε is the current exploration rate, enter exploration mode. The agent will randomly select a node from the list of neighboring nodes as the next hop.
[0020] If rand ≥ ε, then the agent enters exploitation mode, queries the Q-value table, and selects the value in the current state S. t The action that yields the maximum Q value is the one that uses existing experience to select what is currently considered the optimal path;
[0021] As the number of training steps t increases, the exploration rate decays exponentially.
[0022] Furthermore, the multi-objective reward value is calculated by comprehensively considering data packet delivery success rate, energy consumption, transmission delay, and link stability.
[0023] Furthermore, in step four, the Q-value table is updated using the Q-learning algorithm combined with an experience replay mechanism, specifically through the following operations:
[0024] (1) Storage: Establish an experience replay buffer in the node; at each time step, store the interaction tuple of the agent into the experience replay buffer. When the experience replay buffer is full, the earliest data is overwritten using the first-in-first-out principle.
[0025] (2) Sampling: Randomly select a batch of samples from the experience buffer for training;
[0026] (3) Update: Calculate the update amount of Q value according to the Bellman equation, and iteratively update the corresponding value in the Q value table.
[0027] Furthermore, in step five, the network performance metrics include: packet delivery rate, end-to-end latency, network energy consumption, and node lifetime. A performance gradient-based dynamic parameter adjustment mechanism is used to adaptively adjust the reinforcement learning parameters of the agent, specifically through the following operations:
[0028] (1) Learning rate α adjustment: When the underwater sensor network detects that the variance of the data packet delivery rate exceeds the preset threshold, or the change in the channel quality index exceeds the set threshold, it indicates that a sudden change has occurred in the underwater environment, and the learning rate α of other nodes is increased; conversely, when the performance of the underwater sensor network tends to be stable, the learning rate α is gradually reduced to ensure convergence accuracy.
[0029] (2) Adjustment of discount factor γ: When the remaining energy of a node is greater than the preset upper limit of energy, the value of γ is increased; when the remaining energy of a node is less than the preset lower limit of energy, the value of γ is decreased, so that the agent is more inclined to choose the action with the least immediate energy consumption.
[0030] Furthermore, in step six, the termination iteration condition includes the following three logical judgments:
[0031] (1) Task completion status: Determine whether the data buffer of the current node is empty and whether there are no more data packets to be forwarded in the underwater sensor network;
[0032] (2) Network survival status: Determine whether the remaining energy of the current node is lower than the death threshold for maintaining basic communication, or whether the number of surviving nodes in the underwater sensor network is lower than the minimum number for maintaining connectivity; the surviving node is the node with remaining energy greater than the death threshold;
[0033] (3) Reachability of the convergence node: Determine whether there is a valid path to the surface convergence node under the current network topology connection state, that is, whether there is at least one neighbor node whose Q value is greater than the preset unreachable penalty value.
[0034] The iteration terminates if any one of the three logical conditions is met; otherwise, the iteration continues.
[0035] Furthermore, in step six, when the iteration ends, a mapping relationship between underwater environmental parameters and routing decisions is established. The routing decisions are then optimized again through historical data analysis combined with pattern recognition technology to form an adaptive routing selection model output.
[0036] A reinforcement learning route optimization system considering dynamic underwater acoustic environment and node mobility, used to implement the reinforcement learning route optimization method considering dynamic underwater acoustic environment and node mobility, includes: a state input module, a reinforcement learning agent, an environment interaction module, a multi-objective reward value calculation module, and a learning update module;
[0037] The state input module is used to construct a multi-dimensional state vector based on the current state of the node;
[0038] The reinforcement learning agent is used to determine the optimal next-hop forwarding node using an ε-greedy strategy;
[0039] The environmental interaction module is used to monitor underwater environmental parameters in real time, as well as to perform routing actions and provide feedback on transmission results.
[0040] The multi-objective reward value calculation module is used to calculate the multi-objective reward value based on the transmission results;
[0041] The learning update module is used to update the Q-value table and store the agent's interaction tuples in the experience replay buffer.
[0042] The beneficial effects of this invention are:
[0043] (1) This invention enables underwater sensor networks to intelligently cope with time-varying channel and node mobility challenges by combining reinforcement learning with dynamic environment perception through an adaptive routing optimization mechanism. While significantly improving the data delivery rate, it achieves a multi-objective optimization balance of energy consumption, latency and stability.
[0044] (2) This invention reduces control overhead and enhances the robustness and adaptability of underwater sensor networks in dynamic underwater acoustic environments through distributed online learning, effectively extending the overall network lifespan.
[0045] (3) The method of the present invention is universal and can be widely applied to various underwater monitoring and communication scenarios, and has important practical value and broad application prospects. Attached Figure Description
[0046] Figure 1 This is a flowchart of a reinforcement learning route optimization method that considers dynamic underwater acoustic environment and node mobility in an embodiment of the present invention.
[0047] Figure 2 This is a schematic diagram illustrating the state construction, decision-making, and learning update process of a reinforcement learning agent in an embodiment of the present invention. Detailed Implementation
[0048] The present invention will be described in detail below with reference to the accompanying drawings and preferred embodiments. The objectives and effects of the present invention will become clearer as a result. The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0049] like Figure 1 and Figure 2 As shown, a reinforcement learning-based route optimization method considering dynamic underwater acoustic environment and node mobility is presented. The entire optimization process is continuously performed in a dynamic underwater environment, achieving online learning and adaptive optimization of the routing strategy. The method includes the following steps:
[0050] S1: Initialize the parameters of the underwater sensor network, including: initial deployment of underwater sensor network nodes (hereinafter referred to as nodes), initial configuration of underwater environmental parameters, setting of communication range, and initialization of reinforcement learning agents in each node.
[0051] Node deployment considers three-dimensional spatial distribution; each node contains a reinforcement learning agent (hereinafter referred to as the agent). Underwater environmental parameters include dynamic factors such as sound speed propagation conditions, noise level parameters, and ocean current speed. Among them, ocean current speed changes periodically with time, noise level considers random disturbances from biological and environmental noise, and sound speed is adjusted according to changes in sound speed propagation conditions (such as water temperature).
[0052] S2: At each time step, monitor the dynamic changes of the underwater environment in real time, update the underwater environment parameters, and predict and update the location information of all nodes based on the node mobility model.
[0053] Preferably, the node mobility model takes into account the influence of ocean currents and random disturbances.
[0054] S3: Based on the updated current location of the nodes and channel conditions, evaluate the quality and stability of the links between nodes to dynamically update the network topology connection status. If the current node has data to send, proceed to S4; otherwise, wait for the next time step to return and proceed to S2.
[0055] Preferably, the link quality assessment comprehensively considers multipath propagation effects, Doppler frequency shift, and time-varying attenuation characteristics; among which, the influence of factors such as distance, relative speed, and depth difference between nodes on signal transmission is calculated, and an underwater acoustic channel model is used to predict the quality and stability of the link between nodes.
[0056] S4: The agent constructs a multidimensional state vector for reinforcement learning. This multidimensional state vector includes: the node's own state (remaining energy), neighboring node states (neighboring node energy levels, distance between nodes), channel conditions (signal-to-noise ratio, channel quality), mobility prediction results (relative node speed, predicted link stability), and data packet characteristics (data packet priority). Normalization is then applied to unify parameters of different dimensions into state feature values, forming a multidimensional state vector representing the multidimensional state. This multidimensional state vector integrates multidimensional information to comprehensively reflect the dynamic characteristics of the underwater sensor network.
[0057] S5: The agent employs an ε-greedy strategy (exploration rate decays over time) to achieve a balance between utilizing the known optimal strategy and exploring new strategies (the strategy refers to the rules for choosing actions to balance exploration and utilization), selecting a routing action and generating a routing decision; that is, based on the current multidimensional state vector, it selects the optimal or exploratory action from an independent Q-value table constructed for each node based on a state discretization method (mapping continuous states to discrete space), determining the optimal next-hop forwarding node for the data packet. The Q-value is short for the state-action value function, representing the estimated expected cumulative reward that the agent can obtain by following the current strategy (or the optimal strategy) after performing action a in a specific state s.
[0058] The agent, based on the current multidimensional state vector S t Choose the optimal or exploratory action A t The specific process is as follows: First, a random number rand is generated in the interval [0,1]. If rand < ε (current exploration rate), then the exploration mode is entered, and the agent will randomly select a node from the list of neighboring nodes as the next hop; this mechanism ensures that the agent has the opportunity to discover potential paths that currently have low Q values but may become better with changes in the environment. Figure 2 The "random action selection" branch in the algorithm. If rand ≥ ε, then the agent enters the exploitation mode, queries the Q-value table, and selects the action in the current state S. t The action with the maximum Q value, i.e. This means using existing experience to select what is currently considered the optimal path. As the number of training steps *t* increases, the exploration rate is adjusted according to an exponential decay formula, thus achieving a smooth transition from "broad-based exploration" to "precise utilization." The exponential decay formula is as follows:
[0059]
[0060] In the formula, This represents the minimum exploration rate (usually set to 0.01 or 0.1), ensuring that there is still a small amount of exploration available later. This represents the initial exploration rate (usually set to 1.0).
[0061] S6: Execute the routing action. The node attempts to transmit data packets. If the transmission fails, the failure information is recorded, and execution jumps to S9; otherwise, the multi-objective reward value is calculated based on the transmission result. The multi-objective reward value R comprehensively considers data packet delivery success rate, energy consumption, transmission delay, and link stability. Its value is defined as the weighted sum of each sub-objective. The expression of the multi-objective reward function is as follows:
[0062]
[0063] In the formula, R delivery For delivery rewards, a positive fixed reward is given if the data packet successfully reaches the next hop or the sink node (example: +100); if the transmission fails (i.e., packet loss), a negative penalty is given (example: -50); C energy The energy penalty is the energy consumed in this transmission, used to suppress high-energy-consuming paths; T delay The delay penalty is the sum of the propagation delay and processing delay during transmission, used to optimize end-to-end latency; L stability The link stability reward is calculated based on Doppler frequency shift and link duration prediction. If the predicted link duration is greater than the data packet transmission time, a positive reward is given; otherwise, a negative penalty is given to optimize stability. w1, w2, w3, and w4 are weighting coefficients that can be dynamically adjusted according to network conditions and application requirements (such as some applications being insensitive to latency but sensitive to energy consumption) to achieve a multi-objective optimization balance of delivery rate, energy consumption, and latency.
[0064] S7: Based on the data packet transmission results and multi-objective reward values, update the agent's Q-value table to optimize routing decisions. The Q-value table is learned and updated using a Q-learning algorithm combined with an experience replay mechanism. The experience replay mechanism stores historical state-action-reward tuples (used to update the Q-value table) to improve learning efficiency and stability. The optimized routing decisions will guide subsequent routing decisions, forming a closed loop of continuous learning and optimization.
[0065] The learning process (the entire process of iterative optimization of agent parameters) includes the calculation of the current Q value, the estimation of the target Q value, and the iterative update of the Q value. The immediate reward and the future reward are balanced by a discount factor. The target Q value is the immediate reward plus the discounted maximum future Q value. The difference between the current Q value and the target Q value is calculated and iteratively updated using the learning rate.
[0066] To address the training instability issue caused by the strong correlation of underwater channel data samples, this invention introduces an empirical replay mechanism, the specific implementation steps of which are as follows:
[0067] (1) Storage: Establish an experience replay buffer (Buffer) with a capacity of N in the node. At each time step, store the agent's interaction tuple (i.e., the historical state-action-reward tuple). The data is stored in the experience replay buffer. When the experience replay buffer is full, the earliest data is overwritten using the first-in, first-out (FIFO) principle.
[0068] (2) Sampling: When updating the Q-value table, the samples at the current time are not used directly, but a batch of samples (Mini-batch) is randomly selected from the experience buffer for training.
[0069] (3) Update: The update amount of the Q value is calculated according to the Bellman Equation, and the expression is as follows:
[0070]
[0071] In the formula, Indicates the learning rate. This represents the discount factor.
[0072] Iteratively update the corresponding value in the Q-value table using the update amount of the Q-value.
[0073] Beneficial effects: This mechanism breaks the correlation of data sequences, making the learning process more stable and effectively addressing the training oscillation problem caused by sudden changes in the underwater environment.
[0074] S8: Real-time monitoring of underwater sensor network performance metrics, adaptively adjusting the agent's reinforcement learning parameters based on network performance metrics and underwater environmental parameters. Network performance metrics include: packet delivery rate, end-to-end latency, network power consumption, and node lifetime.
[0075] Specifically, a performance gradient-based dynamic parameter adjustment mechanism is used to adaptively adjust the reinforcement learning parameters of the agent, which is achieved through the following operations:
[0076] (1) Learning rate α adjustment: When the underwater sensor network detects a sharp fluctuation in the "data packet delivery rate" (variance exceeding the preset threshold) or a significant decrease in the "channel quality index", it indicates a sudden change in the environment. At this time, the agent automatically increases the learning rate α of other nodes (example: from 0.1 to 0.5) so that the agent can quickly forget the old Q value and adapt to the new environment; conversely, when the performance of the underwater sensor network tends to stabilize, the learning rate α is gradually reduced to ensure convergence accuracy.
[0077] (2) Adjustment of discount factor γ: The discount factor is adjusted according to the "remaining energy" of the node. When the node has sufficient energy (i.e., greater than the preset energy upper limit, for example: greater than 80%), the value of γ is increased (close to 1) to encourage the agent to focus on long-term path optimization; when the node has insufficient energy (i.e., less than the preset energy lower limit, for example: less than 20%), the value of γ is decreased to make the agent more inclined to choose the action with the least immediate energy consumption, thereby extending the node's survival time.
[0078] S9: Determine if the termination iteration condition is met. If not, activate the closed-loop feedback mechanism, that is, use the agent with adaptively adjusted reinforcement learning parameters to repeat S2-S9 to form closed-loop optimization. If yes, establish the mapping relationship between underwater environmental parameters and routing decisions, and optimize the routing decisions again through historical data analysis and pattern recognition technology (such as cluster analysis of historical data) to form an adaptive routing selection model output (that is, summarize the rule that "a certain type of node should be selected first under a certain type of ocean current pattern").
[0079] The termination iteration conditions include the following three logical judgments (real-time routing terminates if any one of them is satisfied, otherwise it continues):
[0080] (1) Task completion status: Determine whether the data buffer of the current node is empty and whether there are no more data packets to be forwarded in the underwater sensor network.
[0081] (2) Network survival status: Determine whether the remaining energy of the current node is lower than the "death threshold" (i.e., the energy level that cannot maintain basic communication), or whether the number of surviving nodes (nodes with remaining energy greater than the death threshold) in the underwater sensor network is lower than the minimum number required to maintain connectivity.
[0082] (3) Sink Node Reachability: Determine whether there is a valid path to the sink node (i.e., whether there is at least one neighbor node whose Q value is greater than the preset unreachable penalty value) under the current network topology connection state.
[0083] Preferably, combining pattern recognition technology in the process of optimizing routing decisions can improve the accuracy and efficiency of routing decisions.
[0084] To implement the reinforcement learning route optimization method that considers dynamic underwater acoustic environment and node mobility, this embodiment also proposes a reinforcement learning route optimization system that considers dynamic underwater acoustic environment and node mobility, including: a state input module, a reinforcement learning agent, an environment interaction module, a multi-objective reward value calculation module, and a learning update module.
[0085] The state input module is used to construct a multi-dimensional state vector based on the current state of the node; the reinforcement learning agent is used to determine the optimal next-hop forwarding node using an ε-greedy strategy; the environment interaction module is used to monitor underwater environment parameters in real time, execute routing actions, and provide feedback on transmission results; the multi-objective reward value calculation module is used to calculate multi-objective reward values based on the transmission results; and the learning update module is used to update the Q-value table and store the agent's interaction tuples in the experience replay buffer.
[0086] The routing strategy is continuously improved through the aforementioned closed-loop feedback mechanism. System parameters and strategies are dynamically adjusted based on network performance changes and environmental conditions, enhancing the adaptability and robustness of the routing system. Furthermore, system performance is periodically evaluated and algorithm parameters are optimized (to prevent exceptionally poor performance). The main purpose of periodic evaluation and optimization is to prevent strategy degradation and escape from local optima. Underwater environments are highly non-stationary; during long-term operation, the agent may get trapped in local optima (e.g., consistently choosing a stable but extremely slow path). By setting an evaluation period (e.g., every 100 data packets transmitted constitutes one period), the system calculates the average delivery rate and average latency for that period. If the performance metrics fall below 80% of the historical average, a "reset mechanism" is triggered, specifically including: ① Resetting the exploration rate: forcing the ε value back to its initial high value, compelling the agent to re-explore ignored paths; ② Cleaning the experience pool: removing "expired" samples with long timestamps from the experience replay buffer to prevent outdated environmental data from misleading current decision-making training.
[0087] This invention constructs a multi-dimensional state representation that integrates environmental dynamics, node status, and topology information. It designs a multi-objective reward function that considers delivery success rate, energy consumption, transmission delay, and link stability. Furthermore, it employs a distributed learning framework with mobility prediction capabilities, enabling each node in the underwater sensor network to independently and adaptively select the optimal forwarding path. Compared to traditional and existing adaptive routing technologies, this invention significantly reduces energy consumption and transmission delay in underwater sensor networks while maintaining a high data delivery rate, effectively extending the network's lifespan. It is particularly suitable for large-scale, long-term, and highly dynamic complex underwater monitoring applications.
[0088] The reinforcement learning-based routing optimization method considering dynamic underwater acoustic environments and node mobility described above is merely one specific embodiment of the present invention. The present invention is equally applicable to any underwater network routing optimization method utilizing reinforcement learning mechanisms, dynamic environment perception, and mobility prediction, and does not constitute a limitation thereof. Any modifications, equivalent substitutions, improvements, and extensions made to the node mobility model, multi-dimensional state vector construction method, multi-objective reward function design, learning algorithm, and system architecture within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
[0089] It will be understood by those skilled in the art that the above descriptions are merely preferred examples of the invention and are not intended to limit the invention. Although the invention has been described in detail with reference to the foregoing examples, those skilled in the art can still modify the technical solutions described in the foregoing examples or make equivalent substitutions for some of the technical features. All modifications and equivalent substitutions made within the spirit and principles of the invention should be included within the scope of protection of the invention.
Claims
1. A reinforcement learning-based route optimization method considering dynamic underwater acoustic environment and node mobility, characterized in that, Includes the following steps: Step 1: Initialize the parameters of the underwater sensor network; Step 2: Monitor and update underwater environmental parameters and node locations in real time; based on node locations and channel conditions, evaluate the quality and stability of inter-node links to dynamically update the network topology connection status. Step 3: When data needs to be transmitted, the agent within the node constructs a multi-dimensional state vector for reinforcement learning, and adopts an ε-greedy strategy to balance utilization and exploration, selects routing actions, and generates routing decisions; The multidimensional state vector includes: the node's own state, the neighboring node's state, channel conditions, mobility prediction results, and data packet characteristics; these parameters with different dimensions are normalized and unified into state feature values to obtain the multidimensional state vector. The node's own state includes the node's remaining energy; the neighbor node's state includes the neighbor node's energy level and the distance between nodes; the channel conditions include the signal-to-noise ratio and channel quality; the mobility prediction results include the node's relative moving speed and the predicted link stability value; and the data packet characteristics include data packet priority. Step 4: Execute the routing action. If the data packet transmission fails, record the failure information and proceed to Step 6; otherwise, calculate the multi-target reward value; update the Q-value table based on the transmission result and the multi-target reward value to optimize the routing decision. Step 5: Monitor the performance indicators of the underwater sensor network in real time, and adaptively adjust the reinforcement learning parameters of the agent based on the network performance indicators and underwater environmental parameters; In step five, network performance metrics include: packet delivery rate, end-to-end latency, network energy consumption, and node lifetime. A performance gradient-based dynamic parameter adjustment mechanism is used to adaptively adjust the reinforcement learning parameters of the agent, specifically through the following operations: (1) Learning rate α adjustment: When the underwater sensor network detects that the variance of the data packet delivery rate exceeds the preset threshold, or the change in the channel quality index exceeds the set threshold, it indicates that a sudden change has occurred in the underwater environment, and the learning rate α of other nodes is increased; conversely, when the performance of the underwater sensor network tends to be stable, the learning rate α is gradually reduced to ensure convergence accuracy. (2) Discount factor γ adjustment: When the remaining energy of a node is greater than the preset upper limit of energy, the value of γ is increased; when the remaining energy of a node is less than the preset lower limit of energy, the value of γ is decreased, so that the agent is more inclined to choose the action with the least immediate energy consumption. Step 6: Determine if the termination iteration condition is met. If not, repeat steps 2-6 using the adaptively adjusted agent. If yes, end the iteration and output the adaptive routing model.
2. The reinforcement learning route optimization method considering dynamic underwater acoustic environment and node mobility according to claim 1, characterized in that, In step one, parameter initialization includes: initial deployment of underwater sensor network nodes, initial configuration of underwater environmental parameters, setting of communication range, and initialization of reinforcement learning agents in each node. The underwater environmental parameters include: sound speed propagation conditions, noise level parameters, and ocean current speed.
3. The reinforcement learning route optimization method considering dynamic underwater acoustic environment and node mobility according to claim 1, characterized in that, In step three, the agent adopts an ε-greedy strategy, based on the current multidimensional state vector S. t From the independent Q-value table constructed for each node based on the state discretization method, select the optimal or exploratory action A. t The process of determining the optimal next-hop forwarding node for data packets, i.e., selecting a route, and generating a route decision, is as follows: Generate a random number rand in the interval [0,1]. If rand < ε, where ε is the current exploration rate, enter exploration mode. The agent will randomly select a node from the list of neighboring nodes as the next hop. If rand ≥ ε, then the agent enters exploitation mode, queries the Q-value table, and selects the value in the current state S. t The action that yields the maximum Q value is the one that uses existing experience to select what is currently considered the optimal path; As the number of training steps t increases, the exploration rate decays exponentially.
4. The reinforcement learning route optimization method considering dynamic underwater acoustic environment and node mobility according to claim 1, characterized in that, The multi-objective reward value is calculated by comprehensively considering data packet delivery success rate, energy consumption, transmission delay, and link stability.
5. The reinforcement learning route optimization method considering dynamic underwater acoustic environment and node mobility according to claim 1, characterized in that, In step four, the Q-value table is updated using the Q-learning algorithm combined with an experience replay mechanism, specifically through the following operations: (1) Storage: Establish an experience replay buffer in the node; at each time step, store the interaction tuple of the agent into the experience replay buffer. When the experience replay buffer is full, the earliest data is overwritten using the first-in-first-out principle. (2) Sampling: Randomly select a batch of samples from the experience buffer for training; (3) Update: Calculate the update amount of Q value according to the Bellman equation, and iteratively update the corresponding value in the Q value table.
6. The reinforcement learning route optimization method considering dynamic underwater acoustic environment and node mobility according to claim 1, characterized in that, In step six, the termination iteration conditions include the following three logical judgments: (1) Task completion status: Determine whether the data buffer of the current node is empty and whether there are no more data packets to be forwarded in the underwater sensor network; (2) Network survival status: Determine whether the remaining energy of the current node is lower than the death threshold for maintaining basic communication, or whether the number of surviving nodes in the underwater sensor network is lower than the minimum number for maintaining connectivity; the surviving node is the node with remaining energy greater than the death threshold; (3) Reachability of the convergence node: Determine whether there is a valid path to the surface convergence node under the current network topology connection state, that is, whether there is at least one neighbor node whose Q value is greater than the preset unreachable penalty value. The iteration terminates if any one of the three logical conditions is met; otherwise, the iteration continues.
7. The reinforcement learning route optimization method considering dynamic underwater acoustic environment and node mobility according to claim 1, characterized in that, In step six, when the iteration ends, a mapping relationship between underwater environmental parameters and routing decisions is established. The routing decisions are then optimized again by combining historical data analysis with pattern recognition technology to form an adaptive routing selection model output.
8. A reinforcement learning route optimization system considering dynamic underwater acoustic environment and node mobility, used to implement the reinforcement learning route optimization method considering dynamic underwater acoustic environment and node mobility as described in any one of claims 1-7, characterized in that, include: The module includes a status input module, a reinforcement learning agent, an environment interaction module, a multi-objective reward value calculation module, and a learning update module. The state input module is used to construct a multi-dimensional state vector based on the current state of the node; The reinforcement learning agent is used to determine the optimal next-hop forwarding node using an ε-greedy strategy; The environmental interaction module is used to monitor underwater environmental parameters in real time, as well as to perform routing actions and provide feedback on transmission results. The multi-objective reward value calculation module is used to calculate the multi-objective reward value based on the transmission results; The learning update module is used to update the Q-value table and store the agent's interaction tuples in the experience replay buffer.