Underwater vehicle path planning method

Abstract

The invention relates to an underwater vehicle path planning method, and aims to realize rapid and efficient path planning in a dynamic flow field environment. According to the algorithm, the size of a propeller of the underwater vehicle is automatically adjusted, ocean current force is utilized, the advancing route of the underwater vehicle is optimized, energy consumption is reduced, and the navigational speed is increased. Different from a traditional path planning method, the method disclosed by the invention particularly considers the obvious influence of ocean current in the ocean environment on the motion of the underwater vehicle, and carries out deep research on the Karman vortex street flow field environment. An AUV simulation environment based on an AUV kinetic model is designed, and a dynamic or static Karman vortex street flow field is added in the environment. An in-orbit reinforcement learning algorithm based on a near-end strategy algorithm and expectation regression is provided and designed, the strategy update amplitude is constrained by using the near-end strategy algorithm, and convergence of a value network is accelerated by using the expectation regression. Propeller power control and path planning under ocean current assistance can be realized in an underwater complex flow field environment.

Description

Technical Field

[0001] The present invention designs a path planning algorithm, specifically relates to a fast path planning method that autonomously adjusts the size of an AUV thruster in a dynamic flow field environment, thereby utilizing the force of ocean currents. Background Art

[0002] Reinforcement Learning (RL) is a machine learning method that emphasizes learning decision-making strategies through rewards and penalties obtained during environmental interactions. Its history can be traced back to the 1950s, and early research laid the foundation for modern reinforcement learning. Learning from interaction has profoundly influenced the development history of reinforcement learning, which requires constantly dealing with the balance between exploration and exploitation, and tends to conduct goal-oriented learning from interaction.

[0003] Different from the path planning tasks in land and air environments, the submersible in the marine environment is greatly affected by the ocean currents, and the submersible needs to choose the best path according to the surrounding ocean current environment to bypass the countercurrent area. On the other hand, the underwater submersible has a larger range of activities, but the power source it carries is limited, so how to reasonably allocate the use of power and how to reasonably use the ocean current environment to achieve the power-assisted mission has important practical value.

[0004] Ocean currents tend to be consistent on a large scale, but in offshore waters or in areas with complex seabed terrain, environmental factors such as obstacles can cause underwater turbulence. When fluid flows over seabed obstacles, alternating rotating vortex streets are formed downstream. This phenomenon is called a Karman vortex street, which can greatly interfere with the movement of passing submersibles. Summary of the invention

[0005] The present invention aims at the problem of difficult trajectory control of submersibles in complex ocean current fields. Starting from the underlying modeling of the submersible, various physical parameters of the submersible are designed, and the control components of the submersible, such as thrusters and fins, are modeled. Numerical simulation of Karman vortex street current data is carried out. On this basis, a three-dimensional environment based on time-varying ocean current data is established to provide environmental support for algorithm path planning. The algorithm receives position information and ocean current information from the environment, and outputs the attitude angle of the AUV for control. While improving the efficiency of policy gradient updates, it also has good anti-interference ability for time-varying ocean currents. Secondly, the expected regression method is used to weight the importance of training data. Expected regression focuses on estimating specific quantiles of conditional distributions rather than the mean. By assigning different weights to the positive and negative deviations of the quantiles, the overestimation problem of the comment network can be alleviated.

[0006] A submersible path planning method, the technical solution is as follows:

[0007] The first step is to model the submersible.

[0008] (1) When an AUV performs a task underwater, it is affected by the combined effects of gravity, buoyancy, drag, and attitude angle. Under the influence of the combined force, it changes its own acceleration and angular acceleration, which in turn affects its own velocity and angular velocity, and ultimately affects its own position and attitude. The dynamics of the AUV are modeled and motion analysis is performed. This involves the conversion between the inertial coordinate system and the AUV coordinate system.

[0009] (2) Constructing the kinematic model of AUV. The kinematic model of AUV is shown in Formula 1, where represents the acceleration and angular acceleration in the AUV coordinate system.

[0010]

[0011] Where M, C and D represent inertia, Coriolis force and damping force respectively, and g(p) represents the restoring force caused by gravity and buoyancy. It is a function of position p, taking into account the torque and balance force generated by the AUV in water due to the difference in gravity and buoyancy. And τ represents all external forces and torques that affect the motion of the AUV. These forces and torques jointly affect the dynamic behavior of the AUV. Formula 1 defines represents the acceleration caused by the ocean current in the AUV's coordinate system. We only focus on the effects of ocean current on planar motion and ignore the effects in vertical or other rotational directions. The above involves calculating the acceleration and angular acceleration of the AUV in its own coordinate system. The model takes into account various forces, including inertia, Coriolis force, damping force, as well as gravity and buoyancy. These forces are combined to form a comprehensive model of the AUV's motion.

[0012] (3) After obtaining the acceleration of the AUV, the Runge-Kutta method is used to update the position and attitude of the model. The new velocity is equal to the current velocity plus the acceleration multiplied by the time interval. Compared with the Newton method, the Runge-Kutta method can calculate the acceleration more accurately and exhibits higher accuracy and stability.

[0013] The AUV constructed by the above equations is underactuated and is able to adjust its direction under the influence of dynamics and ocean currents.

[0014] The second step is to model the Karman vortex flow field

[0015] (1) Establish a dynamic model of the submersible to define the basic characteristics of the ocean current. Use the Newton-Euler equation to describe the forces and equilibrium state of the submersible in the fluid, and parameterize the ocean current characteristics through experimental or literature data. Establish the 6-DOF dynamic equation of the submersible, and initialize and adjust it according to the fluid parameters in the actual ocean environment.

[0016] (2) Use the lattice Boltzmann method (LBM) to simulate ocean currents. The fluid domain of the ocean current is discretized into multiple grid points, the particle distribution function at each grid point is defined, and the evolution of the fluid is simulated through collision and migration rules. In actual operation, it is necessary to reasonably divide the lattice structure of the LBM, and set boundary conditions and initial conditions according to the characteristics of the ocean current to ensure the accuracy and stability of the simulation.

[0017] (3) Introducing obstacles into the flow field to simulate the Karman vortex street. Fixed obstacles are set in the LBM simulation domain, and the vortex street formation process is controlled by adjusting parameters such as the shape and size of the obstacles and the speed of the ocean current.

[0018] (4) Dynamic ocean currents are introduced to simulate flow field motion. The initial state of the flow field is assumed to be that the liquid molecules are stationary, and then ocean currents are gradually introduced. By setting the flow velocity and flow direction functions that change with time, the temporal and spatial changes of ocean currents are simulated.

[0019] (5) As time goes by, vortex structures begin to form behind the obstacle. These vortices are caused by the interaction between the viscosity and inertial forces of the fluid. The vortex structure keeps changing, oscillating up and down, and finally forming a stable Karman vortex street. This dynamic evolution not only affects the path of the submersible, but may also affect its energy consumption and stability.

[0020] The third step is to design the details of the reinforcement learning algorithm.

[0021] (1) Use the policy gradient algorithm to reduce variance. The update formula of the policy gradient will subtract the baseline function, thereby reducing the fluctuations introduced by randomness. In addition, the advantage function technique is used to further reduce the variance by calculating the advantage value of the action.

[0022] (2) Use the shearing technique to improve update efficiency and stability. By calculating the ratio of the new and old strategies, the weight of historical data in the current strategy update is adjusted to ensure the validity of the off-track data. At the same time, in order to avoid excessive policy changes during the update process, the shearing technique is used to limit the amplitude of the policy update to prevent the policy from updating too quickly and causing instability.

[0023] (3) Update the value network using expected regression. The value network is trained by replaying data from the experience pool and its parameters are updated regularly so that it can adapt to dynamic changes in the environment. The technique of predicting state values ​​by fitting the value network effectively reduces the variance in the return estimate.

[0024] (4) Conduct path planning experiments in complex ocean flow fields. The velocity field, pressure field and other information of the flow field are used as inputs to the reinforcement learning algorithm, and the policy network outputs control instructions for the submersible based on these inputs. At the same time, the value network estimates the long-term return of the submersible under different states to help the policy network make better decisions. During the experiment, as time goes by, the vortex structure will continue to change, and the algorithm will adjust the strategy based on the real-time flow field information to ensure that the submersible can efficiently complete the path planning task in a complex flow field. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 It is the specific implementation process of the reinforcement learning path planning algorithm in the specific implementation mode of the present invention;

[0026] Figure 2 It is a time-varying Karman vortex street environment in a specific embodiment of the present invention;

[0027] Figure 3 It is a schematic diagram of the expected regression loss function in a specific implementation manner of the present invention. DETAILED DESCRIPTION

[0028] In order to make the technical solution of the present invention clearer, the present invention is further described below in conjunction with the accompanying drawings. Figure 1 This is a specific implementation process of a path planning algorithm for a submersible in a Karman vortex street environment based on a PPO algorithm and using expected regression. The present invention is specifically implemented in the following steps:

[0029] The first step is to model the submersible.

[0030] (1) Model the rudder and stern plane. The rudder and stern plane generate resistance in the direction of motion. In addition, the model's motion is mainly affected by forces generated in the lateral and vertical directions. The rolling force generated by the rudder enables the ship to move left or right. The vertical force generated by the stern plane controls the rise or sink of the ship. These two forces are generated by the rudder and stern plane changing the direction of the water flow at the corresponding control angles.

[0031]

[0032] where k r and k s denote the scale coefficient, ρ denotes the density of water, u, v and w denote the linear velocity, S r and S s It represents the force-bearing area of ​​the stern rudder and the stern plane.

[0033] (2) Modeling the thruster. The thruster affects the thrust and rolling moment in the direction of motion. The thrust calculation formula is as follows:

[0034]

[0035] Where D t is the diameter of the propeller, K0 and K max is the corresponding coefficient.

[0036] (3) Lattice Boltzmann method (LBM) modeling flow field. This paper uses the lattice Boltzmann method (LBM) to simplify Boltzmann's original gas dynamics idea by reducing the number of particles and restricting them to grid points. For the two-dimensional model, the particles are restricted to propagate in nine possible directions, including standing still. These velocities are called microscopic velocities and are expressed as Represents, where i=0,...,8.

[0037] For each particle on the grid, a discrete probability distribution function is associated Or abbreviated as It describes the probability of propagation in a particular direction.

[0038] The macroscopic fluid density can be defined as the sum of the microscopic particle distribution functions:

[0039]

[0040] Therefore, the macroscopic speed is the distribution function f i Weighted micro velocity Average value:

[0041]

[0042] For simulating single-phase flow, this chapter uses the Bhatnagar-Gross-Krook (BGK) collision model, whose equilibrium distribution Defined by the following formula:

[0043]

[0044] Weight ω i remains constant over time. The key steps of LBM are the propagation and collision processes, which are given by the following equations, where

[0045]

[0046] in is the equilibrium distribution, and τ is considered as the time to relax to the local equilibrium. c is the velocity of the particle, which is a constant. Δt is the time step, which is a small time interval. The formula describes the one-step update rule in the Lattice Boltzmann method, which simulates the movement and collision of particles. It divides the change of the distribution function into two parts: the part on the left side of the equal sign is due to the free flow of particles, and the part on the right side of the equal sign is due to the collision between particles.

[0047] Using the above method, a time-varying Karman vortex street environment in the obstacle flow field is created. The ocean currents that vary with time are as follows: Figure 2 shown.

[0048] (4) The policy gradient algorithm optimizes the policy based on the advantage function, which depends on accurate policy evaluation. The value function is updated by minimizing the error between the reward and the predicted state value. In order to speed up the convergence of the value network and alleviate the problem of overestimation of the network, the expected regression is used to update the value function.

[0049]

[0050] The symbol 1(u<0) represents an indicator function. It is used to indicate whether a condition is true (value 1) or false (value 0). When u is 0.1, 0.5, and 0.9, The function graph of Figure 3 When τ is set to 0.5, it corresponds to the most commonly used mean square error.

[0051] The second step is to model the Karman vortex street flow field.

[0052] (1) Establish a dynamic model of the submersible to define the basic characteristics of the ocean current, such as velocity, direction, and depth distribution. Through these parameters, the impact of the ocean current on the motion of the submersible can be accurately simulated. The key to this step is to build a mathematical model that can accurately reflect the behavior of the submersible based on the interaction of various forces and fluid dynamics.

[0053] (2) Use the lattice Boltzmann method (LBM) to simulate ocean currents. LBM simplifies the continuous motion of the fluid into the propagation of microscopic particles on the grid by discretizing the Boltzmann equation. For a two-dimensional model, particles propagate in nine possible directions, including a stationary state. LBM implements the propagation and collision process of the fluid through local update rules. Using LBM, complex flow phenomena can be efficiently simulated and fluid dynamic characteristics can be captured at a detailed level.

[0054] (3) Introducing obstacles into the flow field to simulate the Karman vortex street. This vortex phenomenon is caused by the separation of the boundary layer due to the viscosity of the fluid. When the fluid flows over the obstacle, the boundary layer formed on the surface of the object due to the viscosity effect will separate at a certain critical point, causing the fluid to form a periodic vortex. By simulating this phenomenon, the influence of the vortex on the motion of the submersible can be studied, including its stability and trajectory changes.

[0055] (4) The initial state of the flow field is assumed to be that the liquid molecules are stationary, and then the ocean currents are gradually introduced. Over time, the flow field evolves from a symmetric state to an asymmetric state. This process simulates the real ocean environment, in which the flow field conditions change over time. By observing the changes in the flow field, the impact of the fluid flow characteristics on the submersible can be analyzed, and the control strategy of the submersible can be adjusted according to these changes.

[0056] (5) As time goes by, vortex structures begin to form behind the obstacle. These vortices are caused by the interaction between the viscosity and inertial forces of the fluid. The vortex structure keeps changing, oscillating up and down, and finally forming a stable Karman vortex street. This dynamic evolution not only affects the path of the submersible, but may also affect its energy consumption and stability.

[0057] (6) The simulation data was recorded and the alternating vortex structure of the fluid behind the obstacle was observed. This phenomenon became more obvious over time and the flow field was introduced into the AUV simulation environment.

[0058] The third step is to design the details of the reinforcement learning algorithm.

[0059] (1) Use the policy gradient algorithm to reduce variance. The policy gradient theorem is a policy optimization method used in reinforcement learning. It directly optimizes the policy parameters and improves the expected return of the policy through gradient ascent.

[0060] (2) Use clipping techniques to improve update efficiency and stability. The policy gradient method has an inevitable defect that is common to the on-track policy algorithm, that is, the update speed is slow because online resampling is required every time the parameters are updated. After a large number of samples are utilized and updated, the data will be discarded. The next update requires sampling again, resulting in low data utilization. Therefore, the concept of off-track policy is introduced, and the use of importance sampling of experience can greatly improve the update efficiency. Importance sampling technology is introduced to use another distribution to approximate the sampling distribution.

[0061] (3) Use expected regression to update the value network. Use expected regression to update the value network. The core idea of ​​expected regression is to reduce variance by fitting a value network to predict the expected return for each state. Specifically, the value network will continuously adjust its parameters based on past experience data, making its predictions of returns for different states more accurate. In this way, the policy gradient algorithm can use smoother and more stable value estimates when updating the policy, rather than relying on high-variance return sampling.

[0062] (4) Conduct path planning experiments in complex ocean current fields. Use Karman vortex streets to generate complex autonomous underwater vehicle path planning scenarios. Use the time-varying ocean currents in actual scenarios and consider the time-varying ocean current information in the above reinforcement learning algorithm, so that the AUV has the adaptive ability to resist ocean current interference.

Claims

1. In a submersible path planning method, it is characterized in that: The method includes: (1) Dynamic modeling of the submersible, including but not limited to the combined effects of gravity, buoyancy, drag and attitude angle; (2) Construct a kinematic model of the submersible and calculate the acceleration and angular acceleration in the submersible coordinate system through the dynamic equations of the underactuated submersible based on Newton's second law; (3) Use the Runge-Kutta method to update the position and attitude of the submersible; (4) Modeling of the Karman vortex flow field, including velocity, direction, and depth distribution, and simulation of ocean currents using the lattice Boltzmann method (LBM); (5) Introducing obstacles to simulate the Karman vortex street phenomenon; (6) Design reinforcement learning-based algorithms, including policy gradient algorithm and expectation regression method, to optimize the path planning of the submersible.

2. A submersible vehicle path planning method according to claim 1, characterized in that: The submersible modeling further includes: (1) Model the rudder and stern plane to generate drag and rolling forces; (2) Modeling thrusters to generate thrust and rolling moments; (3) Calculate the forces and moments generated by the rudder and thruster and substitute them into the dynamic equations of the underactuated submersible. (4) Use the BGK collision model to define the equilibrium distribution, model the Karman vortex street flow field distribution, and simulate the movement of particles through propagation and collision processes; (5) Consider the hydrodynamic characteristics of the submersible at different depths and speeds; (6) Simulate the dynamic response of a submersible in a complex marine environment.

3. A submersible vehicle path planning method as claimed in claim 1, characterized in that: The reinforcement learning algorithm further comprises: (1) Use the policy gradient algorithm to optimize the control strategy of the submersible; (2) Use the expected regression method to update the value network to reduce the problem of overestimation; (3) Calculate the value network loss through the expected regression loss function. (4) Using time-varying ocean current data for strategy optimization; (5) Verify the effectiveness and robustness of the algorithm through simulation experiments.

4. A submersible vehicle path planning method as claimed in claim 1, characterized in that: Design a control device for a submersible, the device comprising: (1) a thruster control unit, used to adjust the thrust of the thruster according to the output of the reinforcement learning algorithm; (2) An attitude control unit, used to adjust the attitude angle of the submersible according to the output of the reinforcement learning algorithm.

Images