A Topology Optimization Method for Underwater Acoustic Sensor Networks Based on Reinforcement Learning and Genetic Algorithms
By combining reinforcement learning and genetic algorithms to optimize the topology of underwater acoustic sensor networks, we have achieved coordinated optimization of network coverage, latency, and throughput. This solves the problem of difficulty in coordinating global exploration and local optimization, and improves the efficiency and adaptability of topology optimization.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHEJIANG UNIV
- Filing Date
- 2026-03-18
- Publication Date
- 2026-06-30
AI Technical Summary
Existing methods for optimizing the topology of underwater acoustic sensor networks struggle to coordinate global exploration with local optimization, resulting in difficulties in balancing network coverage, latency, and throughput. This leads to insufficient performance, especially in complex underwater environments and under high load conditions.
By combining reinforcement learning and genetic algorithms, multi-objective collaborative optimization of network coverage, latency, and throughput is achieved through initializing the topological population, genetic evolution, and reinforcement learning optimization. Gradient-driven fine-tuning is performed using the PPO algorithm, and the search process is balanced through adaptive mutation rate and elite retention mechanisms, combined with multi-objective constraints.
It significantly improves the Pareto optimality of network deployment schemes, has cross-scenario migration capabilities, rapid deployment and dynamic adjustment, and is suitable for topology optimization under complex underwater environments and high load conditions.
Smart Images
Figure CN121882175B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of topology optimization technology for underwater acoustic sensor networks, and more specifically to a topology optimization method for underwater acoustic sensor networks based on reinforcement learning and genetic algorithms. Background Technology
[0002] With the rapid growth in demand for marine exploration, environmental monitoring, and underwater communication, underwater acoustic sensor networks have become a crucial means of marine observation due to their wide coverage, flexible deployment, and low cost. However, limited by the long propagation delay, narrow bandwidth, high energy consumption, and spatial constraints of underwater acoustic channels, designing a network topology that can ensure network connectivity, coverage, and throughput while effectively reducing end-to-end latency has become a challenge in current research and engineering practice.
[0003] Traditional multi-objective topology optimization methods are mostly based on heuristic or metaheuristic algorithms, such as GA-based multi-objective optimization frameworks. Genetic algorithms have good global exploration capabilities in high-dimensional search spaces, but often suffer from insufficient accuracy due to premature convergence and difficulty in balancing multi-objective conflicts. On the other hand, reinforcement learning (RL) techniques excel at gradient-driven local fine-tuning of individual solutions, which can further improve performance based on generated candidate solutions, but their global search efficiency and the quality of the initial solutions are highly dependent. Therefore, there is an urgent need for a method that organically combines the discrete global exploration of genetic algorithms with the continuous local optimization of reinforcement learning to overcome the shortcomings of existing technologies that are prone to local optima or difficulty in balancing multi-objective conflicts when using only a single strategy, thereby achieving coordinated optimization of network coverage, latency, and throughput under complex underwater environments and high load conditions.
[0004] Therefore, overcoming the shortcomings of existing underwater acoustic sensor network topology optimization methods that make it difficult to coordinate global exploration and local optimization, and providing an underwater acoustic sensor network topology optimization method based on reinforcement learning and genetic algorithms to achieve multi-objective collaborative optimization of network coverage, end-to-end latency and throughput is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention
[0005] In view of this, the present invention provides a method for optimizing the topology of underwater acoustic sensor networks based on reinforcement learning and genetic algorithms, aiming to solve the above-mentioned technical problems.
[0006] To achieve the above objectives, the present invention adopts the following technical solution:
[0007] A method for optimizing the topology of underwater acoustic sensor networks based on reinforcement learning and genetic algorithms includes the following steps:
[0008] S1. Initialize the UASN topology population: Randomly generate an initial population containing P candidate topologies. Each topology encodes the three-dimensional coordinate positions of N underwater sensor nodes to obtain the initial population dataset.
[0009] S2. Based on the initial population dataset obtained in S1, evaluate the population and select elites: calculate multiple performance target values for each topology in the population, perform non-dominated sorting and crowding distance calculation on the population, identify the Pareto front, and select the top-ranked topology as the elite solution set based on the crowding distance.
[0010] S3. Based on the elite solution set obtained in S2, perform genetic evolution and reinforcement learning optimization:
[0011] S31. Based on the elite solution set, apply genetic operators to generate a new generation of population;
[0012] S32. Input each elite topology in the first Pareto front identified in S2 into the RL optimization module. The RL optimization module is based on the PPO algorithm. It generates node position fine-tuning actions according to the state feature vector of the topology. After executing the actions, it evaluates the performance of the new topology, calculates the reward and updates the experience replay buffer. When the new topology is better than or equal to the original elite topology in a Pareto sense, it replaces the original elite topology in the new generation population generated in S3 to obtain the optimized new generation population.
[0013] S4. Verify the optimization effect by repeating S2-S3 within the set maximum number of generations G to extract the Pareto optimal solution set of the final population; each topology in the solution set represents an optimized node deployment scheme.
[0014] S5. Training and Iterating the RL Policy: During the RL optimization process in S3, the parameters of the reinforcement learning policy network are updated periodically using the state, action, reward, new state and action log probability data stored in the experience replay buffer. Through the feedback mechanism, the topology optimization effect is fed back to the RL policy network to continuously improve the policy network's decision-making ability at local fine adjustment point positions.
[0015] S6. Based on the improved policy network, achieve coordinated optimization of network coverage, latency and throughput under complex underwater environments and high load conditions.
[0016] Furthermore, in S1, the topology must satisfy the surface convergence node reachability constraint, the minimum node spacing constraint, and the deployment space location constraint.
[0017] Furthermore, in S1, the surface convergence node reachability constraint requires that at least one underwater node be located within the communication radius of the surface convergence node. Internal; Minimum node spacing constraint requires that the Euclidean distance between any two nodes is not less than The deployment spatial constraints require all nodes to be deployed within a specified 3D cuboid region, with the surface convergence nodes fixed within it. Planar and underwater nodes must be full ,in The spatial coordinates of the underwater node. The maximum coordinates of the constrained region.
[0018] Furthermore, in S2, the plurality of performance target values include network coverage, average end-to-end latency, and network throughput.
[0019] Furthermore, in S3, the genetic evolution portion employs an adaptive mutation rate that decays linearly. Balancing search and refinement, among which Let g be the initial mutation rate, g be the number of generations passed, and G be the maximum number of generations. The elite solution set is directly retained to the next generation, and the parent generation is selected through a binary tournament method based on probability. A single-point crossover is performed, followed by applying a Gaussian mutation perturbation to the child coordinate components.
[0020] Furthermore, in S4, the RL optimization module uses the PPO algorithm to perform gradient-driven fine-tuning of the elite topology generated by the genetic algorithm to collaboratively optimize coverage, latency, and throughput. When the warm-up generation and update interval conditions are met and the amount of data in the experience buffer reaches the threshold, the parameters of the policy network and value network are periodically updated to maintain the method's adaptability to complex underwater environments and high-load scenarios.
[0021] Furthermore, after S1 to S5, the trained policy network and value network parameters are saved, and the Pareto optimal solution set is extracted from the final population obtained in S4 as the output to support the rapid deployment of topology in different marine scenarios.
[0022] Compared to existing technologies, this invention utilizes a collaborative mechanism that integrates global exploration using genetic algorithms with local optimization using reinforcement learning. It autonomously generates optimized topologies based on the multi-objective characteristics of network coverage, latency, and throughput, significantly improving overall performance while satisfying three-dimensional deployment constraints. This method balances the search process using an adaptive decay mutation strategy and an elite retention mechanism, effectively avoiding premature convergence. Furthermore, the reinforcement learning module performs gradient-driven fine-tuning of elite solutions, significantly expanding the quality of the Pareto solution set. The trained policy network possesses cross-scenario transfer capabilities, greatly shortening the deployment cycle in new environments and supporting rapid topology reconstruction under dynamic disturbance conditions. This achieves adaptive collaboration and efficient deployment of multi-objective topology optimization in complex underwater environments. Attached Figure Description
[0023] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.
[0024] Figure 1 This is a schematic diagram of the process for optimizing the topology of an underwater acoustic sensor network according to the present invention.
[0025] Figure 2 This is a schematic diagram of the structure of a reinforcement learning module according to the present invention. Detailed Implementation
[0026] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0027] See appendix Figures 1 to 2 This invention discloses a method for optimizing the topology of underwater acoustic sensor networks based on reinforcement learning and genetic algorithms, comprising the following steps:
[0028] (1) Initialize the topology population: Randomly generate an initial population containing P candidate topologies. Each topology encodes the three-dimensional coordinates of N underwater sensor nodes and satisfies the surface convergence node reachability constraint, minimum node spacing constraint and deployment space constraint.
[0029] Surface convergence node reachability constraints require at least one node to be located within its communication radius. The minimum node spacing constraint requires that the Euclidean distance between any two nodes is not less than [a certain value]. Deployment space constraints limit all nodes to being located in Three-dimensional region, and surface convergence nodes are fixed at Consider optimizing the node distribution topology within a rectangular region. , , The dimensions are the length, width, and height of the cuboid region;
[0030] (2) Population evaluation and elite selection: Calculate the network coverage C(s), average end-to-end delay D(s), and network throughput T(s) for each topology. Identify the Pareto front through non-dominated ranking and congestion distance calculation, and select the top-ranked ... The elite topologies constitute the elite solution set E;
[0031] (3) Optimization of genetic evolution and reinforcement learning: The elite solution set E is directly retained to the next generation, and the parent generation is selected through a binary tournament with probability. Perform single-point crossover with adaptive mutation rate Gaussian mutation is performed to generate offspring, where Let g be the initial mutation rate, g be the number of generations, and G be the maximum number of generations; simultaneously, the first Pareto front of the current generation will be included. The input reinforcement learning module generates position fine-tuning actions based on node density distribution, link connectivity matrix, and objective function gradient information. After executing the actions, the reward function is calculated. In the formula, The difference between the old and new coverage rates, The difference in latency between the old and new systems, The difference between the old and new throughput, weighted Reflecting the priority of multiple objectives, if the new topology has a better Pareto value than the original topology, then the network parameters are replaced and updated.
[0032] Elite Analysis Directly retained to the next generation; adaptive mutation rate With algebra Linear decay achieves a balance between early global exploration and later local fine-tuning; the Gaussian mutation perturbation formula is... ,in Based on solution space dimension calibration, The adaptive mutation rate is defined as follows: x is a gene value of the current individual, and x' represents the new gene value after mutation. The adaptive mutation rate adjusts with each generation. It is typically maintained at a larger value in the early stages to enhance global exploration, and gradually decreased in later stages to refine local searches.
[0033] The PPO module updates network parameters when three indicators—warm-up algebra, update interval, and experience buffer—reach thresholds; the reward function is designed as follows: In the formula, The difference between the old and new coverage rates, The difference in latency between the old and new systems, The difference between the old and new throughput, weighted It reflects the priority of multiple objectives and is set according to the requirements of the scenario and environment; the state feature vector includes node density distribution, link connectivity matrix and objective function gradient information;
[0034] (4) Iterative optimization and solution set extraction: Repeat steps (2)-(3) until the maximum number of generations G is reached, and extract the Pareto optimal solution set of the final population. And save the trained policy network parameters and value network parameters;
[0035] Pareto optimal solution set Includes non-dominant topology schemes, each offering different trade-offs in coverage, latency, and throughput; saves the trained policy network parameters. With value network parameters It supports rapid deployment in new scenarios;
[0036] The RL optimization module uses the PPO algorithm to fine-tune the elite topology generated by the genetic algorithm in a gradient-driven manner to coordinate the optimization of coverage, latency and throughput. When the warm-up generation and update interval conditions are met and the amount of empirical buffer data reaches the threshold, the parameters of the policy network and value network are periodically updated to maintain the method's adaptability to complex underwater environments and high-load scenarios.
[0037] (5) Policy network training: During the RL optimization process, the parameters of the reinforcement learning policy network are updated periodically using the state, action, reward, new state and action log probability data stored in the experience replay buffer; the topology optimization effect is fed back to the RL policy network through the feedback mechanism, continuously improving the decision-making ability of the policy network when adjusting the local fine adjustment point position, and improving the topology optimization efficiency and overall performance of the system under multi-objective constraints.
[0038] The policy network update uses the PPO-Clip algorithm: in The advantage function is used; the value network update employs mean squared error loss. in For cumulative rewards.
[0039] in, It represents the ratio of the probability of choosing the same action under the same state with the new and old strategies; This is the advantage function, used to measure the performance of actions relative to the average strategy; This is a hyperparameter used to limit the magnitude of policy updates. The objective function, by minimizing the values between undated and pruned terms, effectively prevents excessive policy shifts during updates, ensuring the stability of the training process.
[0040] in, Indicates by parameters The defined critic function is the estimate of the value function in state st, where Rt is the true cumulative reward obtained in that state. This loss function drives the critic network to gradually approach the true reward, thereby providing a more accurate assessment of state value and a reliable benchmark for advantage function estimation and policy update.
[0041] (6) Based on the improved strategy network, the network coverage, latency and throughput are optimized in a coordinated manner under complex underwater environment and high load conditions.
[0042] The invention will now be described in detail with reference to the accompanying drawings:
[0043] like Figure 1 As shown, the underwater acoustic sensor network topology optimization process of this invention can be divided into three main stages: initialization, evolution-evaluation iteration, and result output. The system first generates an initial population that satisfies the three-dimensional deployment constraints, and simultaneously initializes the experience replay buffer D and the PPO policy-value network parameters. , Then, the generational cycle begins: in each generation, three metrics are calculated for each topology: network coverage, average end-to-end latency, and throughput. The Pareto front is then identified using non-dominated sorting and congestion distance. Subsequently, the elite set is retained, and a binary tournament, single-point crossover, and Gaussian perturbation with linearly decaying mutation rates are performed to expand the offspring and maintain population size. After completing the genetic operations, the system extracts the first Pareto front of the current generation. Each elite topology is fed into a reinforcement learning fine-tuning module for continuous refinement of node coordinates. If the improved topology is Pareto-dominant, the original topology is directly replaced and the interaction samples are written to the buffer. When the number of generations reaches a set upper limit G, the algorithm saves the trained policy-value network and extracts the set of non-dominant Pareto optimal solutions from the final population. This is provided as a network deployment solution.
[0044] like Figure 2 As shown, the reinforcement learning module comprises five sub-units: feature extraction, policy inference, environment interaction, reward evaluation, and network update. The module first encodes topological features such as node coordinates, connectivity density, and objective function gradients into state vectors, which are then input into the policy network to obtain node fine-tuning actions. After these actions are applied to the topology, three metrics—coverage gain, latency improvement, and throughput enhancement—are calculated immediately and applied according to… Generate rewards, where, The difference between the old and new coverage rates, The difference in latency between the old and new systems, The difference between the old and new throughput, weighted It reflects the priority of multiple objectives and is set according to the requirements of the scenario and environment.
[0045] The system writes the quintuple of <state-action-reward-new state-action probability> into buffer D, and updates the policy network and value network when the warm-up algebra, update interval, and data volume thresholds are met, achieving stable convergence. The buffer is then pruned to its maximum capacity to maintain freshness. With this online closed-loop learning mechanism, the policy can continuously accumulate optimization experience in different topologies and transfer it to new environments, significantly improving the convergence speed and solution set quality in complex underwater high-load scenarios.
[0046] This invention proposes a topology optimization method for underwater acoustic sensor networks based on a combination of reinforcement learning and genetic algorithms. By organically integrating the global discrete exploration of the genetic algorithm with the local continuous fine-tuning of reinforcement learning, it achieves coordinated optimization of network coverage, end-to-end latency, and throughput under multi-objective constraints. This method maintains population diversity in the high-dimensional topology search space and effectively avoids premature convergence. Simultaneously, it utilizes a reinforcement learning module to perform gradient-driven fine-tuning of the elite solutions selected by the genetic algorithm, significantly improving the Pareto optimality of the network deployment scheme. Compared with traditional single-element heuristics or pure RL methods, this invention achieves faster convergence and obtains better solutions in complex underwater channels and high-load scenarios, while possessing good adaptability and reusability. It is particularly suitable for applications requiring rapid deployment and real-time adjustment, such as marine environmental monitoring, seabed exploration, and underwater communication, and has broad engineering promotion value and application prospects.
[0047] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.
[0048] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for optimizing the topology of underwater acoustic sensor networks based on reinforcement learning and genetic algorithms, characterized in that, Includes the following steps: S1. Initialize the UASN topology population: Randomly generate an initial population containing P candidate topologies. Each topology encodes the three-dimensional coordinate positions of N underwater sensor nodes to obtain the initial population dataset. S2. Based on the initial population dataset obtained in S1, evaluate the population and select elites: calculate multiple performance target values for each topology in the population, perform non-dominated ranking and crowding distance calculation on the population, identify the Pareto front, and select the top-ranked elites based on the crowding distance. Each topology is used as the elite solution set. In each generation, the network coverage, average end-to-end latency, and throughput of each topology are calculated, and the Pareto front is identified through non-dominated sorting and congestion distance. Then, the elite solution set is retained, and a binary tournament, single-point crossover, and Gaussian perturbation with linearly decaying mutation rate are performed to expand the offspring and maintain the population size. After completing the genetic operations, the system extracts the first Pareto front of the current generation. ; S3. Based on the elite solution set obtained in S2, perform genetic evolution and reinforcement learning optimization: S31. Based on the elite solution set, apply genetic operators to generate a new generation of population; S32. Input each elite topology in the first Pareto front identified in S2 into the RL optimization module. The RL optimization module is based on the PPO algorithm. It generates node position fine-tuning actions according to the state feature vector of the topology. After executing the actions, it evaluates the performance of the new topology, calculates the reward and updates the experience replay buffer. When the new topology is better than or equal to the original elite topology in a Pareto sense, it replaces the original elite topology in the new generation population generated in S31 to obtain the optimized new generation population. S4. Verify the optimization effect by repeating S2-S3 within the set maximum number of generations G to extract the Pareto optimal solution set of the final population; each topology in the solution set represents an optimized node deployment scheme. S5. Training and Iterating the RL Policy: During the RL optimization process in S3, the parameters of the reinforcement learning policy network are updated periodically using the state, action, reward, new state and action log probability data stored in the experience replay buffer. Through the feedback mechanism, the topology optimization effect is fed back to the RL policy network to continuously improve the policy network's decision-making ability at local fine adjustment point positions. S6. Based on the improved policy network, achieve coordinated optimization of network coverage, latency and throughput under complex underwater environments and high load conditions.
2. The underwater acoustic sensor network topology optimization method based on reinforcement learning and genetic algorithm according to claim 1, characterized in that, In S1, the topology must satisfy the surface convergence node reachability constraint, minimum node spacing constraint, and deployment space location constraint.
3. The underwater acoustic sensor network topology optimization method based on reinforcement learning and genetic algorithm according to claim 2, characterized in that, In S1, the surface convergence node reachability constraint requires that at least one underwater node be located within the communication radius of the surface convergence node. Internal; Minimum node spacing constraint requires that the Euclidean distance between any two nodes is not less than The deployment spatial constraints require all nodes to be deployed within a specified 3D cuboid region, with the surface convergence nodes fixed within it. Planar and underwater nodes must be full ,in The spatial coordinates of the underwater node. The maximum coordinates of the constrained region.
4. The underwater acoustic sensor network topology optimization method based on reinforcement learning and genetic algorithm according to claim 1, characterized in that, In S2, the plurality of performance target values include network coverage, average end-to-end latency, and network throughput.
5. The underwater acoustic sensor network topology optimization method based on reinforcement learning and genetic algorithm according to claim 1, characterized in that, In S3, the genetic evolution part uses an adaptive mutation rate that decays linearly. Balancing search and refinement, among which Let g be the initial mutation rate, g be the number of generations passed, and G be the maximum number of generations. The elite solution set is directly retained to the next generation, and the parent generation is selected through a binary tournament method based on probability. A single-point crossover is performed, followed by applying a Gaussian mutation perturbation to the child coordinate components.
6. The underwater acoustic sensor network topology optimization method based on reinforcement learning and genetic algorithm according to claim 1, characterized in that, In S3, the RL optimization module uses the PPO algorithm to perform gradient-driven fine-tuning of the elite topology generated by the genetic algorithm to collaboratively optimize coverage, latency, and throughput. When the warm-up generation and update interval conditions are met and the amount of data in the experience buffer reaches the threshold, the parameters of the policy network and value network are periodically updated to maintain the method's adaptability to complex underwater environments and high-load scenarios.
7. The underwater acoustic sensor network topology optimization method based on reinforcement learning and genetic algorithm according to claim 1, characterized in that, After S1 to S5, the trained policy network and value network parameters are saved, and the Pareto optimal solution set is extracted from the final population obtained in S4 as the output to support the rapid deployment of topology in different marine scenarios.