A method and system for autonomous underwater vehicle path planning and data collection based on adaptive curriculum learning

Abstract

This application relates to the fields of autonomous underwater vehicle (AUV) control technology, path planning, and deep reinforcement learning applications, and particularly to a method and system for AUV path planning and data acquisition based on adaptive curriculum learning. The method includes constructing a multi-dimensional difficulty parameterized coefficient that incorporates macroscopic wave and local turbulence difficulty coefficients, dividing the learning environment into multi-stage progressive courses; establishing a Markov decision process integrating multi-dimensional constraints and a multi-objective comprehensive reward function; constructing a dual-channel joint neural network using a dual-delay deep deterministic policy gradient algorithm, iteratively updating network parameters with an experience replay mechanism; adaptively switching course stages based on a dual criterion of sliding window success rate and minimum training rounds per stage, inheriting parameters to complete progressive training, and outputting the final planning strategy. This invention significantly improves the data acquisition efficiency and navigation safety of the vehicle under extreme sea conditions.

Description

Technical Field

[0001] This application relates to the fields of autonomous underwater vehicle control technology, path planning and deep reinforcement learning application technology, and in particular to a method and system for autonomous underwater vehicle path planning and data acquisition based on adaptive curriculum learning. Background Technology

[0002] As ocean exploration deepens, autonomous underwater vehicles (AUVs) play a central role in ocean data acquisition missions. However, when performing large-scale data acquisition tasks, AUVs often face extreme sea states (such as giant waves and strong time-varying eddies). The extreme sea state flow fields driven by partial differential equations (PDEs) exhibit high temporal nonstationarity and spatial nonuniformity, which not only cause severe fluctuations in the angle of attack and sideslip angle of the AUV, but also introduce significant uncertainties in environmental state transitions. This poses a major challenge to the AUV's adaptive flow field coping capabilities.

[0003] For the problem of two-dimensional path planning and data acquisition of AUVs under extreme sea conditions, existing technical solutions include: ① Traditional planning and control methods based on accurate analytical modeling: such as proportional-integral-derivative (PID) control, sliding mode control (SMC), and model predictive control (MPC). This method establishes accurate hydrodynamic and flow field partial differential dynamic equations, and calculates the speed and heading of the vehicle by combining classical control theory, realizing trajectory tracking and basic navigation under known and determined flow field disturbances; ② Conventional deep reinforcement learning (DRL) methods: such as a curriculum-free control strategy based on the dual-delay deep deterministic policy gradient (TD3) algorithm. This method models the planning task as a Markov decision process (MDP), directly placing the agent in the target high-difficulty sea state environment for "trial and error" interaction and policy network optimization; ③ Reinforcement learning methods that introduce fixed-stage curriculum learning (Fixed-CL): To alleviate the training difficulty of conventional reinforcement learning, this method uses human experience to pre-set fixed environmental difficulty increment stages (such as fixed number of training rounds or fixed difficulty step size), guiding the agent to gradually transition from calm waters to complex flow fields.

[0004] The shortcomings of existing technical solutions in handling strongly non-stationary extreme sea states coupled with partial differential equations (PDEs) are mainly reflected in the following aspects: ① Traditional planning and control methods: These methods rely heavily on accurate modeling of hydrodynamics. Faced with real-time evolving strongly non-stationary hydrodynamic disturbances, traditional methods struggle to make accurate predictions, easily leading to model mismatch, control divergence, and planning failure; ② Conventional deep reinforcement learning methods: When conventional DRL agents are directly placed in high-difficulty extreme flow fields for training, they encounter high-variance state transitions and frequent collisions, failing to obtain sparse success rewards. The agents are prone to falling into continuous negative feedback, leading to control policy collapse or stagnation; ③ Fixed-course reinforcement learning methods: Fixed-CL methods rely on human experience to pre-set improvement timing, ignoring the agent's real-time learning progress. Too rapid an increase in difficulty can lead to knowledge gaps and policy collapse; too slow an increase results in low sample efficiency and an inability to adapt to policy fluctuations during training. Summary of the Invention

[0005] The purpose of this application is to overcome the problems of existing reinforcement learning path planning methods being prone to divergence under extreme sea conditions and having poor generalization ability in fixed course learning. In order to provide a method and system for path planning and data acquisition of autonomous underwater vehicles based on adaptive course learning, by introducing a multi-dimensional difficulty parameterization system and an adaptive transformation criterion based on success rate sliding window, the method achieves smooth knowledge transfer of AUVs under strong non-stationary flow fields, high robust obstacle avoidance under extreme sea conditions, and efficient data acquisition.

[0006] To address the aforementioned technical problems, the technical solution provided in this application offers a method for autonomous underwater vehicle path planning and data acquisition based on adaptive curriculum learning, comprising: Step 1: Establish an extreme sea state environment model, construct a multi-dimensional difficulty parameter coefficient including macro wave difficulty coefficient and local turbulence difficulty coefficient, and divide the learning stages into multiple progressive courses from low to high according to the environmental complexity. Each course learning stage continuously increases the environmental complexity by gradually increasing the macro wave difficulty coefficient and local turbulence difficulty coefficient. Step 2: Establish the Markov decision process corresponding to the autonomous underwater vehicle path planning and data acquisition tasks, and construct the state space, action space and multi-objective comprehensive reward function that integrates multi-dimensional constraints respectively; Step 3: Establish a dual-channel joint neural network based on the dual-delay deep deterministic policy gradient algorithm. The dual-channel joint neural network includes an Actor policy network, a dual-path Critic value network, an Actor target network, and a dual-path Critic target network. At the same time, establish an experience playback buffer to store historical samples generated by the interaction between the autonomous underwater vehicle and the course learning environment. Step 4: Randomly extract training samples from the experience replay buffer, calculate the target action value corresponding to the training sample based on the multi-objective comprehensive reward function, use the dual-delay deep deterministic policy gradient algorithm to iteratively update the Actor policy network and the dual-path Critic value network, and synchronously update the Actor target network and the dual-path Critic target network through soft update to obtain the control policy corresponding to the current course learning stage. Step 5: Use a first-in-first-out sliding window to calculate the data acquisition success rate of the autonomous underwater vehicle within the current learning stage, obtain the sliding window success rate, and combine it with the minimum training anti-fluctuation threshold rounds to determine the transition to the learning stage. When the transition condition is met, switch the multi-dimensional difficulty parameterization coefficients to the next learning stage, inherit all network parameters obtained from the current learning stage training, and continue training until all learning stages are completed. Output the final path planning and data acquisition strategy of the autonomous underwater vehicle.

[0007] As an improvement to the above technical solution, step 1 specifically includes: Step 1.1: Establish a macroscopic wave field dynamic model described by two-dimensional shallow water equations to simulate the flow field disturbance of non-stationary ocean wave field; Step 1.2: Establish a local strong turbulence field model described by the Lamb-Oseen vortex model to simulate local high-risk disturbances under extreme sea conditions; Step 1.3: Construct multidimensional difficulty parameterization coefficients: ; in, Indicates the first Multidimensional difficulty parameterization coefficients corresponding to each course learning stage; This indicates the preset macroscopic wave difficulty coefficient, used to control the initial amplitude of the macroscopic wave field; This represents the preset local turbulence difficulty coefficient, used to control the number of random vortices generated and the circulation intensity in a local strong turbulence field; Step 1.4: Establish multiple progressive course learning stages based on the multidimensional difficulty parameterization coefficients. Each course learning stage is arranged in order of increasing environmental complexity. By gradually increasing the macroscopic wave difficulty coefficient and the local turbulence difficulty coefficient, the complexity of extreme sea state environments is continuously changed.

[0008] As an improvement to the above technical solution, step 2 specifically includes: Step 2.1: Establish the Markov decision process corresponding to the autonomous underwater vehicle's path planning and data acquisition tasks; Step 2.2: Construct the state space, which includes the normalized two-dimensional coordinates of the current position of the autonomous underwater vehicle (AUV), the relative two-dimensional distance vectors between the AUV and each dynamic obstacle, the relative two-dimensional coordinate vectors between the AUV and the target sensor node, and the task constraint ratio feature vector. ;in, For the first Penalty item for each autonomous underwater vehicle crossing the boundary indicator mark; The maximum number of decision steps allowed per mission round for an autonomous underwater vehicle; This represents the number of sensor nodes currently in a data overflow state. This indicates the total number of sensor nodes within the mission area of ​​the autonomous underwater vehicle. Step 2.3: Construct the motion space, which includes the normalized linear velocity control quantity. and normalized heading angle control quantity Among them, action vector for: Normalized linear velocity control quantity and normalized heading angle control quantity The range of values ​​is ; Step 2.4: Calculate the actual linear velocity of the autonomous underwater vehicle based on the normalized linear velocity control value. : ; in, The preset minimum linear velocity constraint, The preset maximum linear velocity constraint; Step 2.5: Calculate the actual heading angle of the autonomous underwater vehicle based on the normalized heading angle control value. : ; Step 2.6: Calculate the actual linear velocity of the autonomous underwater vehicle. and the actual heading angle of the autonomous underwater vehicle Together they serve as control inputs for the actuators of autonomous underwater vehicles; Step 2.7: Construct a multi-objective comprehensive reward function that integrates multi-dimensional constraints. The multi-objective comprehensive reward function includes distance-guided reward, boundary violation penalty, data overflow penalty, dynamic collision avoidance penalty, task success trigger reward, and control action energy consumption penalty.

[0009] As an improvement to the above technical solution, step 2.7 specifically includes: Total instant rewards As a multi-objective comprehensive return function Total instant rewards Distance-guided rewards Penalties for Boundary Violations Data overflow penalty Dynamic collision avoidance penalty Successfully completing the task will trigger a reward. and energy consumption penalty for control actions Together constitute; ; Among them, distance guides returns This is used to guide the autonomous underwater vehicle (AUV) towards the target sensor node based on the Euclidean distance between the AUV and the target sensor node, and provides distance guidance reporting. for: ; in, The Euclidean distance between the current two-dimensional absolute coordinates of the autonomous underwater vehicle and the target sensor node. The weight hyperparameters are set to guide the weighting of the preset target distance; Border violation penalties Used to restrict autonomous underwater vehicles from exceeding their pre-defined mission area; boundary violation penalties. for: ; in, For the first Penalty item for each autonomous underwater vehicle crossing the boundary indicator mark. For indexing autonomous underwater vehicles; The preset boundary violation penalty weight hyperparameter; Data overflow penalty This is used to ensure that autonomous underwater vehicles complete data acquisition tasks in a timely manner, and includes a data overflow penalty. for: ; in, This represents the number of sensor nodes currently in a data overflow state. The preset data overflow penalty weight hyperparameter; Dynamic collision avoidance penalty Dynamic collision avoidance penalty to prevent autonomous underwater vehicles from entering within a safe obstacle avoidance distance threshold. for: ; in, This represents the relative distance between the current autonomous underwater vehicle and other autonomous underwater vehicles. The preset safe obstacle avoidance distance threshold; The preset obstacle avoidance penalty weight hyperparameter; Successfully triggering the reward Used to guide the convergence of the global strategy; rewards are triggered upon successful task completion. for: ; in, This is a binary variable indicating the success of a single-round data acquisition task. It takes the value 1 when the task is successful and 0 otherwise. The preset task success constant reward value; Control action energy consumption penalty Energy consumption for controlling the thruster, energy consumption penalty for control actions for: ; in, The function value representing the overall energy consumption of the current control action; These are the preset energy consumption constraint weight hyperparameters.

[0010] As an improvement to the above technical solution, step 4 specifically includes: Step 4.1: Randomly select a predetermined number of historical training samples from the experience replay buffer. The maximum capacity of the experience replay buffer is [missing information]. ; Step 4.2: Based on the target action corresponding to the next state output by the Actor target network, add pruning noise to the target action to obtain the smooth target action: ; in, This represents the target smoothed action obtained after smoothing regularization. This represents the truncation function. For the Actor target network, For the parameters of the Actor target network, Indicates the current state The state transitioned to at the next moment after the action is performed. For the current Actor target network in The action output at the location; It is random noise; The target policy noise truncation boundary; Physical boundary constraints for the action space; Random noise satisfy: ;in, Represents a normal distribution. Indicates the standard deviation of the target policy noise; Step 4.3: Calculate the target action value corresponding to the dual-path Critic target network based on the target smoothing action. : ; in, For total immediate reward; D is the termination status flag for the current step, which is triggered when the task reaches its maximum step size or a hard collision causes the round to end. ,otherwise ; Discount factor; Indicates the index of the Critic target network; For the first A Critic target network; For the first Each Critic target network follows the current strategy in Execution The expected cumulative discounted return value after output; the target action value. The target action value is calculated by using the smaller of the expected cumulative discounted return values ​​output by the first Critic target network and the second Critic target network, in order to reduce the risk of overestimation of the value function. Step 4.4: Establish the dual-path Critic value network loss function based on the target action value. : ; in, The batch size of the extracted training samples; Indicates the ordinal number of the sample; Indicates the value of the target action In the The target value corresponding to each sample; First Critic Value Network; For the first Critic value network to follow the current strategy in Execution The expected cumulative discount return value output later; For the second Critic value network; For the second Critic value network to follow the current strategy in Execution The expected cumulative discount return value output later. and Together they form a dual-path Critic value network; For the first The state corresponding to each sample; For the first The action corresponding to each sample; These are the parameters of the first Critic value network; For the parameters of the second Critic value network; For the reason and The joint parameter vector formed; Step 4.5: Employ the dual-path Critic value network loss function Update the parameters of the first Critic value network and parameters of the second Critic value network The parameters of the Actor policy network are updated using a delayed gradient strategy. The parameters of the Actor target network are updated using the Polyak averaging method. Parameters of the first Critic target network Parameters of the second Critic target network .

[0011] As an improvement to the above technical solution, step 4.5 specifically includes: Parameters of the First Critic Value Network The update formula is expressed as: ; in, The preset value network learning rate; express right The gradient operator is obtained; Parameters of the Second Critic Value Network The update formula is expressed as: ; in, express right The gradient operator is obtained; The loss function of the Actor policy network is expressed as: ; in, Represents the Actor policy network. Indicates the current Actor policy network in The action output at the location; For the first Critic value network to follow the current strategy in Placement and execution The expected cumulative discount return value output later; Parameters of the Actor Policy Network The update formula is expressed as: ; in, The preset policy network learning rate; express right The gradient operator is obtained; The Actor target network, the first Critic target network, and the second Critic target network all use preset soft update parameters. Polyak averaging updates are performed to smoothly transfer the parameters of the currently trained network to the corresponding target network parameters, thereby maintaining the stability of the dual-channel joint neural network training process and suppressing overestimation of the value function. Parameters of the first Critic target network The update formula is expressed as: ; Parameters of the second Critic target network The update formula is expressed as: ; Parameters of the Actor target network The update formula is expressed as: .

[0012] As an improvement to the above technical solution, step 5 specifically includes: Step 5.1: Establish a first-in-first-out sliding window of length W, based on the actual cumulative data collected by the autonomous underwater vehicle at the end of each training round. Determine whether the data collection task has been completed in the current training round, and calculate the corresponding first training round. Successful indicator binary variable for round data acquisition task , among which, when Not less than the preset single-round data collection threshold hour, The value is 1 if it is set to 1, otherwise the value is 0. Step 5.2: According to Calculate the sliding window success rate corresponding to the current learning stage of the course: ; in, For the first Success rate of sliding windows in each course learning stage; This is the latest training round number; among them, the sliding window success rate is used to represent the proportion of training rounds in which the data acquisition task was successfully completed within the current sliding window out of all training rounds, in order to evaluate the degree of adaptation of the autonomous underwater vehicle to the environment of the current course learning stage. Step 5.3: Obtain the first Sliding window success rate at each course learning stage , No. Cumulative number of training rounds in each course learning phase and the Minimum number of training anti-fluctuation threshold rounds for each course learning phase Set a success rate advancement threshold When continuous Simultaneous satisfaction and At that time, the first Multidimensional difficulty parameterization coefficients corresponding to each course learning stage Switch to the Multidimensional difficulty parameterization coefficients corresponding to each course learning stage This is to achieve adaptive transitions between different stages of the course learning process; among them, This is the preset number of consecutive compliance checks; Step 5.4: In the... The course learning phase inherits from the first... The parameters of the first Critic value network obtained from each course learning phase training are... Parameters of the second Critic value network Parameters of the Actor policy network Parameters of the first Critic target network Parameters of the second Critic target network And the parameters of the Actor target network Continue training until all course learning phases are completed, and output the final path planning and data acquisition strategy for the autonomous underwater vehicle.

[0013] To achieve another objective of the present invention, the present invention also provides an autonomous underwater vehicle (AUV) path planning and data acquisition system based on adaptive curriculum learning, used in the aforementioned AUV path planning and data acquisition method based on adaptive curriculum learning. The system includes: an environment modeling and curriculum division module, a decision process modeling module, a dual-channel network construction module, a strategy iterative training module, and an adaptive curriculum conversion module. The environmental modeling and course division module is used to establish an extreme sea state environment model, construct a multi-dimensional difficulty parameter coefficient including macro wave difficulty coefficient and local turbulence difficulty coefficient, and divide the environmental complexity into multiple progressive course learning stages from low to high. Each course learning stage achieves continuous improvement of environmental complexity by gradually increasing the macro wave difficulty coefficient and local turbulence difficulty coefficient. The decision process modeling module is used to establish the Markov decision process corresponding to the path planning and data acquisition tasks of the autonomous underwater vehicle, and to construct the state space, action space and multi-objective comprehensive reward function that integrates multi-dimensional constraints. The dual-channel network construction module is used to establish a dual-channel joint neural network based on the dual-delay deep deterministic policy gradient algorithm. The dual-channel joint neural network includes an Actor policy network, a dual-path Critic value network, an Actor target network, and a dual-path Critic target network. At the same time, an experience playback buffer is established to store historical samples generated by the interaction between the autonomous underwater vehicle and the course learning environment. The strategy iterative training module is used to randomly extract training samples from the experience replay buffer, calculate the target action value corresponding to the training sample based on the multi-objective comprehensive reward function, use the dual-delay deep deterministic policy gradient algorithm to iteratively update the Actor policy network and the dual-path Critic value network, and synchronously update the Actor target network and the dual-path Critic target network through soft update to obtain the control strategy corresponding to the current course learning stage. The adaptive course transition module is used to calculate the data acquisition success rate of the autonomous underwater vehicle (AUV) within the current course learning stage using a first-in-first-out sliding window. The success rate of the sliding window is then combined with the minimum training anti-fluctuation threshold rounds for each stage to determine the course learning stage transition. When the transition condition is met, the multi-dimensional difficulty parameterization coefficients are switched to the next course learning stage. All network parameters obtained from the current course learning stage are inherited and training continues until all course learning stages are completed. Finally, the module outputs the final path planning and data acquisition strategy for the AUV.

[0014] The advantage of this application lies in its ability to effectively overcome the convergence difficulties of training agents directly under extreme sea conditions. In complex ocean data acquisition tasks, the strategy of this invention significantly increases AUV data acquisition by 36.6% and reduces the number of collisions during task execution by 27.8%. Furthermore, in multi-scale generalized sea state tests facing unknown and more challenging levels, the performance fluctuation of this application is significantly lower than that of a fixed strategy, demonstrating extremely strong adaptive robustness. Attached Figure Description

[0015] Figure 1(a) shows the three-dimensional height diagram of the non-stationary wave field at t=25s, obtained based on the solution of the two-dimensional shallow water equation. Figure 1(b) shows the three-dimensional height diagram of the non-stationary wave field at t=50s, obtained based on the two-dimensional shallow water equation. Figure 1(c) shows the three-dimensional height diagram of the non-stationary wave field at t=75s, obtained based on the two-dimensional shallow water equation. Figure 2(a) is a schematic diagram of the complex ocean current field environment in the simple stage (Stage 0); Figure 2(b) is a schematic diagram of the complex ocean current field environment in the intermediate stage (Stage 1); Figure 2(c) is a schematic diagram of the complex ocean current field environment during the difficult stage (Stage 2); Figure 2(d) is a schematic diagram of the complex ocean current field environment under the extreme stage (Stage 3); Figure 3(a) is a schematic diagram of the reinforcement learning policy network architecture of the Direct-TD3 algorithm; Figure 3(b) is a schematic diagram of the reinforcement learning value network architecture of the Direct-TD3 algorithm; Figure 4(a) shows the convergence curves of data acquisition during the training process for different methods; Figure 4(b) shows the convergence curves of the number of collisions during the training process for different methods; Figure 4(c) shows the convergence curves of energy consumption acquisition ratio for different methods during the training process from 0 to 100 rounds. Figure 4(d) shows the energy consumption acquisition ratio convergence curves of different methods during the training process from 100 to 500 rounds. Figure 5(a) is a comparison of the performance evaluation of the final training data collection volume of the TD3 algorithm in different course training processes; Figure 5(b) is a performance evaluation chart of the final training collision count of the TD3 algorithm in different course training processes; Figure 5(c) is a comparison of the final training energy consumption to acquisition ratio performance evaluation of the TD3 algorithm in different course training processes; Figure 6(a) is a comparison of the performance evaluation of the final training data collection volume of different fixed allocation strategies (Fixed-CL) and adaptive strategies (Adaptive-CL); Figure 6(b) is a comparison of the final training collision count performance evaluation of different fixed-CL and adaptive-CL strategies; Figure 6(c) is a comparison of the final training energy consumption acquisition ratio performance evaluation of different fixed allocation strategies (Fixed-CL) and adaptive strategies (Adaptive-CL); Figure 7 Radar chart for comprehensive performance evaluation of each algorithm under unknown extreme sea conditions after training; Figure 8(a) is a comparison of the path planning and data acquisition trajectory of the Baseline algorithm under the same extreme sea state scenario; Figure 8(b) is a comparison of the path planning and data acquisition trajectory of the Fixed-CL algorithm under the same extreme sea state scenario; Figure 8(c) is a comparison of path planning and data acquisition trajectories of the ALP-based CL algorithm under the same extreme sea state scenario; Figure 8(d) is a comparison of the path planning and data acquisition trajectory of the Adaptive-CL algorithm of this application under the same extreme sea state scenario; Figure 9 A flowchart of the autonomous underwater vehicle path planning and data acquisition method based on adaptive curriculum learning provided in this application. Detailed Implementation

[0016] The technical solutions provided in this application are further illustrated below with reference to the embodiments.

[0017] Example 1 This embodiment provides a method for path planning and data acquisition of autonomous underwater vehicles (AUVs) based on adaptive curriculum learning. The core principle lies in jointly utilizing a multi-dimensional, challenging parameterized environment based on physical flow field dynamics modeling and an adaptive reinforcement learning training framework based on dual-condition decision-making. This method enables AUVs to perform two-dimensional path planning and data acquisition decisions under extreme sea conditions coupled with partial differential equations (PDEs). The method is as follows... Figure 9 As shown, firstly, a continuously quantified environmental difficulty coefficient is established for the macroscopic wave field and stochastic local turbulence field driven by partial differential equations (PDEs). First, the highly non-stationary extreme sea state environment is divided into progressive learning stages. Second, within the Markov Decision Process (MDP) framework, a deep reinforcement learning (DRL) agent is used to solve the strategy for continuous actions such as linear velocity and heading of the AUV, and a comprehensive reward function integrating data acquisition guidance, boundary violation, collision avoidance, and energy consumption is designed. Finally, a success rate based on a first-in-first-out sliding window is introduced. Minimum number of training rounds per stage The adaptive transformation criterion with dual simultaneous constraints, while satisfying continuous After passing the initial evaluation, the environmental difficulty is dynamically changed, thereby achieving smooth knowledge transfer and efficient robust convergence of the aircraft control strategy under strong time-varying flow field disturbances. The detailed design process of this invention is as follows: 1. Extreme sea state environment modeling and multi-dimensional difficulty parameterization system: (1) Macroscopic wave field dynamics modeling In this embodiment, the two-dimensional shallow water equations (SWE) are used to simulate the flow field disturbance of the non-stationary wave field. Its complete conserved partial differential equations (PDEs) are expressed in the following form: (1) In the formula, and These represent the horizontal and vertical spatial coordinates of the Cartesian coordinate system in a two-dimensional horizontal plane, used to describe the horizontal coordinates of any location within the AUV's operating area. This represents the (continuous) simulation time variable in the process of solving the partial differential equations of the flow field, which shares the same time axis with the discrete decision-making moments of the agent in the subsequent Markov decision-making process; Indicates the dynamic wave height of the sea surface; Indicates the total water depth of the flow field; It is the acceleration due to gravity; and They represent the flow field at... shaft and Vertical average velocity component in the axial direction. The three-dimensional spatiotemporal evolution of the non-stationary wave field driven by partial differential equations is shown in Figures 1(a)-1(c).

[0018] (2) Modeling of local strong turbulent flow fields To simulate local high-risk disturbances under extreme sea states, this embodiment uses the Lamb-Oseen vortex model to generate randomly distributed tangential disturbance velocities in the flow field. This is used to characterize the strong turbulent properties of the flow field, and its calculation formula is: (2) In the formula, The circulation parameter determines the vortex intensity. Let be the radial distance of any point in space from the center of the vortex. The characteristic vortex core radius of the vortex.

[0019] (3) Parameterized system design for multidimensional environment difficulty To achieve a continuous and smooth quantitative representation of the complexity of simulation environments from calm waters to extreme sea states, this invention defines multidimensional difficulty parameterization coefficients. : (3) In the formula, The preset macro wave difficulty coefficient is a dimensionless scaling factor with a value range of [0, 1], used to control the initial amplitude scaling ratio of the macro wave field. The preset local turbulence difficulty coefficient is a dimensionless coefficient ranging from [0, 1], used to jointly control the number of random vortices generated and the circulation generation intensity of individual vortices in a local strong turbulence field. This is based on a multidimensional difficulty parameterization coefficient. This invention divides the entire reinforcement learning training environment into four progressive course stages, from calm waters to extreme sea conditions, namely Stage 0 to Stage 3. As the course stages increase, the multidimensional difficulty parameterization coefficients... The difficulty increases smoothly according to a set step size. To enable those skilled in the art to more clearly implement this invention, Table 1 provides the multidimensional difficulty parameterization coefficients for each progressive course stage. Specific configuration examples are shown in Figures 2(a)-2(d) for the schematic diagram of the complex ocean current field environment under the four progressive course stages. In the figure, the depth of the background color represents the normalized wave height, and the red spiral line represents the randomly distributed Lamb-Oseen viscous diffusion turbulent vortex, showing the continuous and smooth course variation trend of environmental complexity.

[0020] Table 1. Classification of Marine Environmental Difficulty and Multidimensional Parametric Configuration for Different Course Learning Stages

[0021] 2. Markov control decision modeling based on dual-channel joint neural network: This invention formally models the path planning and data acquisition tasks of AUVs under extreme sea states and complex flow fields as an infinite-level Markov decision process (MDP). Specifically, it encompasses state space and action space design, construction of a multi-objective hierarchical comprehensive reward function, and the derivation of a strategy optimization based on a dual-channel joint neural network. The detailed technical solutions are as follows: (1) Design of state space and action space state space At each discrete decision time The state vector observed by the agent from the simulation environment It characterizes the mapping between the AUV's own motion state and the characteristics of the external environment. Specifically, it includes the following components: Normalized absolute two-dimensional coordinates of the current position of the AUV; The relative two-dimensional distance vectors between the AUV and each known dynamic obstacle; The relative two-dimensional coordinate vectors between the AUV and the target sensor nodes (SNs); and, Task constraint ratio eigenvector These are used to represent the current boundary violation rate and data overflow rate in real time, respectively. For the first Penalty items for each AUV's boundary crossing indicator position, and boundary violation penalties. In For the same variable; This indicates the maximum number of decision steps allowed in a single mission round for an AUV; This indicates the number of sensor nodes that are in a data overflow state at the current decision-making moment; This indicates the total number of sensor nodes (SNs) within the AUV mission area.

[0022] Action space : The continuous action vector output by the policy network Defined as ,in, This represents the normalized continuous control quantity used to control the linear speed of the AUV. This represents the normalized continuous control quantity used to control the AUV's heading angle. In this embodiment, a linear mapping layer is used to convert the network output value into actual physical control commands that directly act on the AUV's physical actuators. (4) (5) In the formula, The actual linear velocity of the AUV during flight. The physical boundary constraint range for linear velocity. The preset minimum linear velocity constraint, The preset maximum linear velocity constraint; The actual heading angle (absolute heading angle relative to the positive x-axis of the global coordinate system) corresponding to the current decision step of the AUV, in radians (rad), and its value range is [ π, π], and the normalized continuous control quantity They exhibit a linear mapping relationship.

[0023] (2) Design of multi-objective comprehensive return function To synergistically optimize the convergence efficiency, collision avoidance performance, mission completion rate, and energy consumption of AUV path planning, this invention constructs a multi-objective hierarchical real-time reward function. Its total instant return The calculation formula is as follows: (6) Total Instant Returns The specific mathematical design for the hierarchical structure of the various sub-objective reward items is as follows: Distance-guided reward items This is used to guide the AUV to smoothly approach the target sensor node, solving the technical problem of low convergence efficiency in traditional sparse reward algorithms. Its calculation formula is as follows: (7) In the formula, This represents the Euclidean distance between the current two-dimensional absolute coordinates of the AUV and the target sensor node; The preset target distance guides the weight hyperparameters.

[0024] Boundary violation penalties This is used to constrain AUVs to navigate within the safe sea area chart boundaries and prevent them from veering off course. The calculation formula is as follows: (8) In the formula, For the first Penalty item for each AUV's out-of-bounds indicator flag. For AUV index; This is a preset hyperparameter for the boundary violation penalty weight.

[0025] Data overflow penalty Used to ensure the timeliness and integrity of data acquisition tasks and to avoid node buffer overflow, its calculation formula is: (9) In the formula, This represents the amount of data overflow in the current simulation environment, which is also the number of sensor nodes currently in a data overflow state. This is a preset hyperparameter for the data overflow penalty weight.

[0026] Dynamic collision avoidance penalty items A nonlinear truncation function is used to ensure that the AUV is penalized only when it enters a dangerous collision avoidance threshold, thus guaranteeing navigation safety under flow field disturbances. The calculation formula is as follows: (10) In the formula, This represents the relative distance between the current AUV and other AUVs. The preset safe obstacle avoidance distance threshold; This is the preset obstacle avoidance penalty weight hyperparameter.

[0027] The task successfully triggered the reward item. The sparse positive incentive, which guides the convergence of the global policy, is calculated using the following formula: (11) In the formula, A binary variable indicating the success of a single-round data acquisition task; it takes the value 1 when the task is successful and 0 otherwise. This is the preset reward value for a successful task.

[0028] Control action energy consumption penalty item The formula for optimizing the energy consumption of AUV actuators, preventing excessive movement, and extending underwater endurance is as follows: (12) In the formula, The function value representing the overall energy consumption of the current control action; These are the preset energy consumption constraint weight hyperparameters.

[0029] (3) Derivation of dual-channel optimization for TD3 strategy network This invention employs the dual-delay deep deterministic policy gradient (TD3) algorithm as the underlying control decision core. This dual-channel joint neural network architecture includes a parameter... (i.e., the parameter vector consisting of all learnable weights and biases of the Actor policy network) Weighted Actor policy network and two independent parameters (These are parameter vectors consisting of all learnable weights and biases for each of the two independent Critic networks) Weighted Twin Critic Value Network and .

[0030] To effectively mitigate the overestimation of the value function caused by high-variance flow fields in extreme sea states, this embodiment starts with the maximum empirical replay pool capacity of... Experience replay pool The size of the randomly selected micro-batch is Historical samples. The neural network parameter update of this invention follows the following rigorous mathematical derivation steps: Target Action Smoothing Regularization and Noise Pruning: When calculating the next target value, pruning noise is added to the action output by the target policy network to construct a smooth exploration mechanism. (13) (14) In the formula, This represents the target smooth action obtained after smoothing regularization; clip(x, min, max) represents the cutoff function, which restricts the input variable x to the closed interval [min, max]. For the Actor target network, For the current Actor target network in The action output at the location, For the parameters of the Actor target network, This indicates the state at the next moment, that is, the state that the current state s transitions to after performing an action; To satisfy the normal distribution of random noise; The target policy noise truncation boundary; Physical boundary constraints for the action space; Represents a normal distribution; This represents the standard deviation of the target policy noise.

[0031] Dual-channel Critic network target value estimation (Twin Critic mechanism): Two independent target value networks are jointly evaluated, and the minimum value is taken as the updated target. Suppressing overestimation of value in multi-objective environments: (15) In the formula, The total instant return value calculated in step (2) is... Indicates the sequence number of the target value network; Discount factor; For the first A Critic target network; For the first A Critic target network in Q-value estimation of state-action pairs, where Q-value represents the action value function value, which represents the expected cumulative discounted reward that the agent can obtain by following the current policy after performing the action in the current state; For the first A target value network; For the first A target value network in Q-value estimation of state-action pairs. This is the end status flag (done) for the current step. The turn ends when the task reaches its maximum time step or a hard collision occurs. ,otherwise .

[0032] Value network parameter update (Critic loss function minimization): from size Randomly sample a batch of samples from the experience replay buffer, and minimize the loss function by minimizing the mean squared error. To update the weight parameters of the two Critic networks : (16) in, The batch size of the extracted training samples. Indicates the ordinal number of the sample. This represents the target value calculated according to formula (15). The specific value at the j-th sample; and These respectively represent the Twin Critic value network mentioned above. , For the same group of networks, in the... Sample The output value at that point, i.e., the two Critic networks on this point. Q-value estimation of state-action pairs.

[0033] Therefore, the joint parameter vector of the Critic network By the specified learning rate Evolution: (17) (18) in, This represents the preset value network learning rate. Represent the loss function respectively For parameters , The gradient operator is obtained; This represents the joint parameter vector formed by the parameters of the two Critic networks.

[0034] Delayed Policy Gradient Update (Actor Network Update): This invention employs a delayed update mechanism, meaning that the Actor network only performs a gradient update after the Critic network update mechanism has been triggered several times. This is achieved through the deterministic policy gradient theorem, utilizing the first-channel value network... Gradients guide the policy network parameters The optimization update is equivalent to the objective optimization loss function. It manifests as: (19) in, This indicates the first time the data was drawn from the experience playback buffer. The state corresponding to each sample; Indicates the current Policy network in state The action output at the location; This indicates that the first channel Critic network has this effect. Q-value estimation of state-action pairs.

[0035] (20) in, This represents the preset policy network learning rate. Represents the loss function For parameters The gradient operator is obtained.

[0036] Soft update of target network parameters (Polyak averaging): After the update is complete, the preset soft update parameters are used. The weights of the current training network are smoothly transferred to the corresponding target network to maintain the dynamic stability of the entire dual-channel joint neural network training system. (twenty one) (twenty two) (twenty three) in, and These represent the parameters of the first and second Critic target networks, respectively, and the parameters of the currently trained network. , Distinguish between phases; This represents the parameters of the Actor target network, compared to the parameters of the currently trained network. Distinguish between them.

[0037] The complete topology of the strategy and value interaction flow of the dual-channel joint neural network is shown in Figure 3(a) and Figure 3(b), which demonstrates the end-to-end mapping process from multidimensional continuous state space input to actuator action output. The value network adopts a dual-channel joint network design to effectively alleviate the overestimation of the value function.

[0038] 3. Adaptive course conversion criteria based on sliding window success rate and anti-fluctuation mechanism: To monitor the agent's adaptability to the current flow field in real time, this invention abandons the fixed step size and innovatively introduces a dual-condition judgment anti-fluctuation adaptive advancement criterion: (1) Evaluation index definition and sliding window success rate The length is set as First-in-first-out sliding window, defining the first Binary variable of round execution result This embodiment extracts the actual cumulative data collection volume of AUV at the end of the round. and compared it with the preset single-round data acquisition threshold constant. The mathematical expression for nonlinear comparison is as follows: (twenty four) In the formula, The preset single-round success threshold is a hard boundary indicator used to measure whether an AUV has successfully completed the basic data collection task under the current sea state.

[0039] Current stage Sliding window success rate Defined as the percentage of successful rounds within a window: (25) (2) Dual-condition judgment and stage adaptive transition logic To prevent accidental successes caused by high-variance flow field disturbances from leading to blind advancements, environmental difficulty advancements (i.e., stages) are implemented. (must be simultaneous and consecutive) The following dual constraint conditions must be met: (26) In the formula: The preset success rate threshold is used for advancement. For intelligent agents at the current stage The actual cumulative number of training rounds already completed; This is the minimum number of training rounds to prevent fluctuations during this phase, in order to ensure sufficient exploration data; This indicates the number of times the condition is met consecutively, used to filter out evaluation fluctuations caused by environmental randomness. This embodiment only applies to consecutive... The environment complexity parameter is only triggered when all evaluations meet the dual conditions of "performance fitness" and "temporal training sufficiency" mentioned above. A substantial, step-by-step upgrade and replacement.

[0040] Example 2 This embodiment provides an autonomous underwater vehicle (AUV) path planning and data acquisition system based on adaptive curriculum learning, used to implement the AUV path planning and data acquisition method based on adaptive curriculum learning in Embodiment 1. It includes: an environment modeling and curriculum division module, a decision process modeling module, a dual-channel network construction module, a strategy iteration training module, and an adaptive curriculum conversion module. The environmental modeling and course division module is used to establish an extreme sea state environment model, construct a multi-dimensional difficulty parameter coefficient including macro wave difficulty coefficient and local turbulence difficulty coefficient, and divide the environmental complexity into multiple progressive course learning stages from low to high. Each course learning stage achieves continuous improvement of environmental complexity by gradually increasing the macro wave difficulty coefficient and local turbulence difficulty coefficient. The decision process modeling module is used to establish the Markov decision process corresponding to the path planning and data acquisition tasks of the autonomous underwater vehicle, and to construct the state space, action space and multi-objective comprehensive reward function that integrates multi-dimensional constraints. The dual-channel network construction module is used to establish a dual-channel joint neural network based on the dual-delay deep deterministic policy gradient algorithm. The dual-channel joint neural network includes an Actor policy network, a dual-path Critic value network, an Actor target network, and a dual-path Critic target network. At the same time, an experience playback buffer is established to store historical samples generated by the interaction between the autonomous underwater vehicle and the course learning environment. The strategy iterative training module is used to randomly extract training samples from the experience replay buffer, calculate the target action value corresponding to the training sample based on the multi-objective comprehensive reward function, use the dual-delay deep deterministic policy gradient algorithm to iteratively update the Actor policy network and the dual-path Critic value network, and synchronously update the Actor target network and the dual-path Critic target network through soft update to obtain the control strategy corresponding to the current course learning stage. The adaptive course transition module is used to calculate the data acquisition success rate of the autonomous underwater vehicle (AUV) within the current course learning stage using a first-in-first-out sliding window. The success rate of the sliding window is then combined with the minimum training anti-fluctuation threshold rounds for each stage to determine the course learning stage transition. When the transition condition is met, the multi-dimensional difficulty parameterization coefficients are switched to the next course learning stage. All network parameters obtained from the current course learning stage are inherited and training continues until all course learning stages are completed. Finally, the module outputs the final path planning and data acquisition strategy for the AUV.

[0041] Comparison of algorithm simulation examples and quantization techniques: To verify the effectiveness of this invention, specific digital simulation experiments were conducted. The specific environmental physical parameters and algorithm training hyperparameter settings in this invention are detailed in Table 2.

[0042] Table 2: Specific Environmental Physical Parameters and Algorithm Training Hyperparameter Settings

[0043] To fully verify the technical advantages of this invention, this simulation comparison experiment introduced four different control strategies and learning methods. ① Baseline-less method: No curriculum mechanism is introduced; the agent is directly trained in the most challenging extreme sea state environment using conventional reinforcement learning. ② Fixed-CL method: This is a traditional static phased curriculum that is completely detached from the agent's actual learning performance. It mechanically increases the environmental difficulty in stages based solely on a fixed set ratio of rounds (including 1:1:1:1, 4:3:2:1, and 1:2:3:4). ③ ALP-based adaptive method: This is a continuous adaptive curriculum baseline that dynamically samples the current training difficulty proportionally by statistically analyzing the absolute change in the agent's business indicators (i.e., absolute learning progress) within different difficulty ranges. ④ The adaptive course learning method (Adaptive-CL) proposed in this invention is a staged adaptive course that prevents fluctuations. It uses a sliding window to evaluate the success rate of the current stage of AUV in real time. Only when the dual conditions of "performance adaptability" and "temporal training sufficiency" are met will the tiered advancement of environmental difficulty be automatically triggered.

[0044] Figures 4(a)-4(d) show the convergence curves of various performance indicators of different algorithms during the training process. Due to the high energy consumption caused by uncertain exploration in the early stage of training, the energy consumption acquisition ratio (Figures 4(c) and 4(d)) is plotted and magnified in segments to clearly show the low-energy and efficient convergence characteristics of the algorithm in the middle and later stages. It can be seen from the content that the data acquisition amount of the method of this invention (Adaptive-CL) first stabilizes above 60 with minimal fluctuation, the number of collisions stabilizes at a low level of about 4 after 300 rounds, and the energy consumption acquisition ratio stabilizes below 4.8W in the middle and later stages. The overall convergence speed and stability are significantly better than the control method.

[0045] Figures 5(a)-5(c) show the final training performance evaluation comparison of the reinforcement learning algorithm in different course training processes. It can be seen that the method of this invention achieves an average data collection of 68.97 times (variance of only 4.13) in the final stage, which is a significant improvement of 36.6% compared with the method without a course baseline (Baseline, 50.50 times). At the same time, the number of collisions is reduced by 27.8% compared with the Baseline. The average performance and stability of energy consumption collection ratio are both optimal.

[0046] Figures 6(a)-6(c) show a comparison of the final training performance evaluation of different fixed-course strategies (Fixed-CL) and the adaptive strategy of this invention. It can be seen that the performance of the fixed-course method is highly dependent on the manually preset stage ratio, while the method of this invention can comprehensively outperform all fixed-CL strategies in terms of data acquisition volume, safety (average collision count of only 2.6 times) and energy efficiency without manual adjustment.

[0047] Figure 7 The radar charts show the comprehensive performance evaluation of each algorithm under unknown extreme sea conditions after training. It can be seen that the method of this invention forms the largest envelope area across the four dimensions of data acquisition volume, security, energy efficiency, and stability, exhibiting the best overall comprehensive performance.

[0048] Figures 8(a)-8(d) show a comparison of path planning and data acquisition trajectories of different algorithms under the same extreme sea state scenario. As can be seen from the trajectory diagrams, the method of this invention (Figure 8(d)) can adaptively plan a better and safer path when facing strong random flow field disturbances and dense obstacles, maximizing data acquisition coverage while strictly ensuring the navigation safety of AUVs.

[0049] As can be seen from the above detailed description of this application, this application has at least the following advantages: 1) An adaptive course conversion criterion, incorporating both "sliding window success rate evaluation" and "minimum number of training rounds" (AND logic), is introduced into the flow field disturbance environment to dynamically replace the AND parameters upon which the flow field calculation depends. This is the core innovation of this invention in overcoming training fluctuations.

[0050] 2) The independent state space in the traditional DRL is extended to include dynamic velocity mapping under macroscopic wave and microscopic vortex disturbances, and a continuous penalty term (boundary ratio, specific distance overflow attenuation and energy consumption function mapping) specifically for the physical working characteristics of underwater AUVs is coupled into the reward function.

[0051] 3) A continuous counter is introduced into the state transition logic, stipulating that a substantial upgrade in environmental difficulty is triggered only when both the "sliding window success rate" and "temporal training sufficiency" indicators simultaneously meet the threshold for consecutive evaluations. This feature is specifically designed to filter out transient successes caused by high-variance PDE flow field interference, and to forcibly block strategy convergence and divergence caused by "blind upgrades" by adding temporal steady-state verification.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions to the technical solutions of the present invention do not depart from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A method for path planning and data acquisition of an autonomous underwater vehicle based on adaptive curriculum learning, characterized in that, include: Step 1: Establish an extreme sea state environment model, construct a multi-dimensional difficulty parameter coefficient including macro wave difficulty coefficient and local turbulence difficulty coefficient, and divide the learning stages into multiple progressive courses from low to high according to the environmental complexity. Each course learning stage continuously increases the environmental complexity by gradually increasing the macro wave difficulty coefficient and local turbulence difficulty coefficient. Step 2: Establish the Markov decision process corresponding to the autonomous underwater vehicle path planning and data acquisition tasks, and construct the state space, action space and multi-objective comprehensive reward function that integrates multi-dimensional constraints respectively; Step 3: Establish a dual-channel joint neural network based on the dual-delay deep deterministic policy gradient algorithm. The dual-channel joint neural network includes an Actor policy network, a dual-path Critic value network, an Actor target network, and a dual-path Critic target network. At the same time, establish an experience playback buffer to store historical samples generated by the interaction between the autonomous underwater vehicle and the course learning environment. Step 4: Randomly extract training samples from the experience replay buffer, calculate the target action value corresponding to the training sample based on the multi-objective comprehensive reward function, use the dual-delay deep deterministic policy gradient algorithm to iteratively update the Actor policy network and the dual-path Critic value network, and synchronously update the Actor target network and the dual-path Critic target network through soft update to obtain the control policy corresponding to the current course learning stage. Step 5: Use a first-in-first-out sliding window to calculate the data acquisition success rate of the autonomous underwater vehicle within the current learning stage, obtain the sliding window success rate, and combine it with the minimum training anti-fluctuation threshold rounds to determine the transition to the learning stage. When the transition condition is met, switch the multi-dimensional difficulty parameterization coefficients to the next learning stage, inherit all network parameters obtained from the current learning stage training, and continue training until all learning stages are completed. Output the final path planning and data acquisition strategy of the autonomous underwater vehicle.

2. The method for path planning and data acquisition of autonomous underwater vehicles based on adaptive curriculum learning according to claim 1, characterized in that, Step 1 specifically includes: Step 1.1: Establish a macroscopic wave field dynamic model described by two-dimensional shallow water equations to simulate the flow field disturbance of non-stationary ocean wave field; Step 1.2: Establish a local strong turbulence field model described by the Lamb-Oseen vortex model to simulate local high-risk disturbances under extreme sea conditions; Step 1.3: Construct multidimensional difficulty parameterization coefficients: ； in, Indicates the first Multidimensional difficulty parameterization coefficients corresponding to each course learning stage; This indicates the preset macroscopic wave difficulty coefficient, used to control the initial amplitude of the macroscopic wave field; This represents the preset local turbulence difficulty coefficient, used to control the number of random vortices generated and the circulation intensity in a local strong turbulence field; Step 1.4: Establish multiple progressive course learning stages based on the multidimensional difficulty parameterization coefficients. Each course learning stage is arranged in order of increasing environmental complexity. By gradually increasing the macroscopic wave difficulty coefficient and the local turbulence difficulty coefficient, the complexity of extreme sea state environments is continuously changed.

3. The method for path planning and data acquisition of autonomous underwater vehicles based on adaptive curriculum learning according to claim 1, characterized in that, Step 2 specifically includes: Step 2.1: Establish the Markov decision process corresponding to the autonomous underwater vehicle's path planning and data acquisition tasks; Step 2.2: Construct the state space, which includes the normalized two-dimensional coordinates of the current position of the autonomous underwater vehicle (AUV), the relative two-dimensional distance vectors between the AUV and each dynamic obstacle, the relative two-dimensional coordinate vectors between the AUV and the target sensor node, and the task constraint ratio feature vector. ;in, For the first Penalty item for each autonomous underwater vehicle crossing the boundary indicator mark; The maximum number of decision steps allowed per mission round for an autonomous underwater vehicle; This represents the number of sensor nodes currently in a data overflow state. This indicates the total number of sensor nodes within the mission area of ​​the autonomous underwater vehicle. Step 2.3: Construct the motion space, which includes the normalized linear velocity control quantity. and normalized heading angle control quantity Among them, action vector for: Normalized linear velocity control quantity and normalized heading angle control quantity The range of values ​​is ; Step 2.4: Calculate the actual linear velocity of the autonomous underwater vehicle based on the normalized linear velocity control value. : ； in, The preset minimum linear velocity constraint, The preset maximum linear velocity constraint; Step 2.5: Calculate the actual heading angle of the autonomous underwater vehicle based on the normalized heading angle control value. : ； Step 2.6: Calculate the actual linear velocity of the autonomous underwater vehicle. and the actual heading angle of the autonomous underwater vehicle Together they serve as control inputs for the actuators of autonomous underwater vehicles; Step 2.7: Construct a multi-objective comprehensive reward function that integrates multi-dimensional constraints. The multi-objective comprehensive reward function includes distance-guided reward, boundary violation penalty, data overflow penalty, dynamic collision avoidance penalty, task success trigger reward, and control action energy consumption penalty.

4. The method for path planning and data acquisition of autonomous underwater vehicles based on adaptive curriculum learning according to claim 3, characterized in that, Step 2.7 specifically includes: Total instant rewards As a multi-objective comprehensive return function Total instant rewards Distance-guided rewards Penalties for Boundary Violations Data overflow penalty Dynamic collision avoidance penalty Successfully completing the task will trigger a reward. and energy consumption penalty for control actions Together constitute; ； Among them, distance guides returns This is used to guide the autonomous underwater vehicle (AUV) towards the target sensor node based on the Euclidean distance between the AUV and the target sensor node, and provides distance guidance reporting. for: ； in, The Euclidean distance between the current two-dimensional absolute coordinates of the autonomous underwater vehicle and the target sensor node. The weight hyperparameters are set to guide the weighting of the preset target distance; Border violation penalties Used to restrict autonomous underwater vehicles from exceeding their pre-defined mission area; boundary violation penalties. for: ； in, For the first Penalty item for each autonomous underwater vehicle crossing the boundary indicator mark. For indexing autonomous underwater vehicles; The preset boundary violation penalty weight hyperparameter; Data overflow penalty This is used to ensure that autonomous underwater vehicles complete data acquisition tasks in a timely manner, and includes a data overflow penalty. for: ； in, This represents the number of sensor nodes currently in a data overflow state. The preset data overflow penalty weight hyperparameter; Dynamic collision avoidance penalty Dynamic collision avoidance penalty to prevent autonomous underwater vehicles from entering within a safe obstacle avoidance distance threshold. for: ； in, This represents the relative distance between the current autonomous underwater vehicle and other autonomous underwater vehicles. The preset safe obstacle avoidance distance threshold; The preset obstacle avoidance penalty weight hyperparameter; Successfully triggering the reward Used to guide the convergence of the global strategy; rewards are triggered upon successful task completion. for: ； in, This is a binary variable indicating the success of a single-round data acquisition task. It takes the value 1 when the task is successful and 0 otherwise. The preset task success constant reward value; Control action energy consumption penalty Energy consumption for controlling the thruster, energy consumption penalty for control actions for: ； in, The function value representing the overall energy consumption of the current control action; These are the preset energy consumption constraint weight hyperparameters.

5. The method for path planning and data acquisition of autonomous underwater vehicles based on adaptive curriculum learning according to claim 4, characterized in that, Step 4 specifically includes: Step 4.1: Randomly select a predetermined number of historical training samples from the experience replay buffer. The maximum capacity of the experience replay buffer is [missing information]. ; Step 4.2: Based on the target action corresponding to the next state output by the Actor target network, add pruning noise to the target action to obtain the smooth target action: ； in, This represents the target smoothed action obtained after smoothing regularization. This represents the truncation function. For the Actor target network, For the parameters of the Actor target network, Indicates the current state The state transitioned to at the next moment after the action is performed. For the current Actor target network in The action output at the location; It is random noise; The target policy noise truncation boundary; Physical boundary constraints for the action space; Random noise satisfy: ;in, Represents a normal distribution. Indicates the standard deviation of the target policy noise; Step 4.3: Calculate the target action value corresponding to the dual-path Critic target network based on the target smoothing action. : ； in, For total immediate reward; D is the termination status flag for the current step, which is triggered when the task reaches its maximum step size or a hard collision causes the round to end. ,otherwise ; Discount factor; Indicates the index of the Critic target network; For the first A Critic target network; For the first Each Critic target network follows the current strategy in Execution The expected cumulative discounted return value after output; the target action value. The target action value is calculated by using the smaller of the expected cumulative discounted return values ​​output by the first Critic target network and the second Critic target network, in order to reduce the risk of overestimation of the value function. Step 4.4: Establish the dual-path Critic value network loss function based on the target action value. : ； in, The batch size of the extracted training samples; Indicates the ordinal number of the sample; Indicates the value of the target action In the The target value corresponding to each sample; First Critic Value Network; For the first Critic value network to follow the current strategy in Execution The expected cumulative discount return value output later; For the second Critic value network; For the second Critic value network to follow the current strategy in Execution The expected cumulative discount return value output later. and Together they form a dual-path Critic value network; For the first The state corresponding to each sample; For the first The action corresponding to each sample; These are the parameters of the first Critic value network; For the parameters of the second Critic value network; For the reason and The joint parameter vector formed; Step 4.5: Employ the dual-path Critic value network loss function Update the parameters of the first Critic value network and parameters of the second Critic value network The parameters of the Actor policy network are updated using a delayed gradient strategy. The parameters of the Actor target network are updated using the Polyak averaging method. Parameters of the first Critic target network Parameters of the second Critic target network .

6. The method for path planning and data acquisition of autonomous underwater vehicles based on adaptive curriculum learning according to claim 5, characterized in that, Step 4.5 specifically includes: Parameters of the First Critic Value Network The update formula is expressed as: ； in, The preset value network learning rate; express right The gradient operator is obtained; Parameters of the Second Critic Value Network The update formula is expressed as: ； in, express right The gradient operator is obtained; The loss function of the Actor policy network is expressed as: ； in, Represents the Actor policy network. Indicates the current Actor policy network in The action output at the location; For the first Critic value network to follow the current strategy in Placement and execution The expected cumulative discount return value output later; Parameters of the Actor Policy Network The update formula is expressed as: ； in, The preset policy network learning rate; express right The gradient operator is obtained; The Actor target network, the first Critic target network, and the second Critic target network all use preset soft update parameters. Polyak averaging updates are performed to smoothly transfer the parameters of the currently trained network to the corresponding target network parameters, thereby maintaining the stability of the dual-channel joint neural network training process and suppressing overestimation of the value function. Parameters of the first Critic target network The update formula is expressed as: ； Parameters of the second Critic target network The update formula is expressed as: ； Parameters of the Actor target network The update formula is expressed as: 。 7. The method for path planning and data acquisition of autonomous underwater vehicles based on adaptive curriculum learning according to claim 1, characterized in that, Step 5 specifically includes: Step 5.1: Establish a first-in-first-out sliding window of length W, based on the actual cumulative data collected by the autonomous underwater vehicle at the end of each training round. Determine whether the data collection task has been completed in the current training round, and calculate the corresponding first training round. Successful indicator binary variable for round data acquisition task , among which, when Not less than the preset single-round data collection threshold hour, The value is 1 if it is set to 1, otherwise the value is 0. Step 5.2: According to Calculate the sliding window success rate corresponding to the current learning stage of the course: ； in, For the first Success rate of sliding windows in each course learning stage; This is the latest training round number; among them, the sliding window success rate is used to represent the proportion of training rounds in which the data acquisition task was successfully completed within the current sliding window out of all training rounds, in order to evaluate the degree of adaptation of the autonomous underwater vehicle to the environment of the current course learning stage. Step 5.3: Obtain the first Sliding window success rate at each course learning stage , No. Cumulative number of training rounds in each course learning phase and the Minimum number of training anti-fluctuation threshold rounds for each course learning phase Set a success rate advancement threshold When continuous Simultaneous satisfaction and At that time, the first Multidimensional difficulty parameterization coefficients corresponding to each course learning stage Switch to the Multidimensional difficulty parameterization coefficients corresponding to each course learning stage This is to achieve adaptive transitions between different stages of the course learning process; among them, This is the preset number of consecutive compliance checks; Step 5.4: In the... The course learning phase inherits from the first... The parameters of the first Critic value network obtained from each course learning phase training are... Parameters of the second Critic value network Parameters of the Actor policy network Parameters of the first Critic target network Parameters of the second Critic target network And the parameters of the Actor target network Continue training until all course learning phases are completed, and output the final path planning and data acquisition strategy for the autonomous underwater vehicle.

8. A path planning and data acquisition system for autonomous underwater vehicles based on adaptive curriculum learning, used to implement the path planning and data acquisition method for autonomous underwater vehicles based on adaptive curriculum learning as described in any one of claims 1-7, characterized in that, include: The system includes modules for environmental modeling and curriculum division, decision process modeling, dual-channel network construction, strategy iterative training, and adaptive curriculum transformation. The environmental modeling and course division module is used to establish an extreme sea state environment model, construct a multi-dimensional difficulty parameter coefficient including macro wave difficulty coefficient and local turbulence difficulty coefficient, and divide the environmental complexity into multiple progressive course learning stages from low to high. Each course learning stage achieves continuous improvement of environmental complexity by gradually increasing the macro wave difficulty coefficient and local turbulence difficulty coefficient. The decision process modeling module is used to establish the Markov decision process corresponding to the path planning and data acquisition tasks of the autonomous underwater vehicle, and to construct the state space, action space and multi-objective comprehensive reward function that integrates multi-dimensional constraints. The dual-channel network construction module is used to establish a dual-channel joint neural network based on the dual-delay deep deterministic policy gradient algorithm. The dual-channel joint neural network includes an Actor policy network, a dual-path Critic value network, an Actor target network, and a dual-path Critic target network. At the same time, an experience playback buffer is established to store historical samples generated by the interaction between the autonomous underwater vehicle and the course learning environment. The strategy iterative training module is used to randomly extract training samples from the experience replay buffer, calculate the target action value corresponding to the training sample based on the multi-objective comprehensive reward function, use the dual-delay deep deterministic policy gradient algorithm to iteratively update the Actor policy network and the dual-path Critic value network, and synchronously update the Actor target network and the dual-path Critic target network through soft update to obtain the control strategy corresponding to the current course learning stage. The adaptive course transition module is used to calculate the data acquisition success rate of the autonomous underwater vehicle (AUV) within the current course learning stage using a first-in-first-out sliding window. The success rate of the sliding window is then combined with the minimum training anti-fluctuation threshold rounds for each stage to determine the course learning stage transition. When the transition condition is met, the multi-dimensional difficulty parameterization coefficients are switched to the next course learning stage. All network parameters obtained from the current course learning stage are inherited and training continues until all course learning stages are completed. Finally, the module outputs the final path planning and data acquisition strategy for the AUV.

Images