Enhanced AUV path planning method and system based on ocean spatio-temporal data driving
By constructing a three-dimensional ocean simulation environment and improving reinforcement learning algorithms, combined with soft behavior critical algorithms, the problems of data deviation and processing pressure in AUV path planning are solved, and more efficient ocean spatiotemporal data-driven path planning is achieved, improving the execution efficiency of AUV and reliability of simulation environments.
Patent Information
- Application Number
- CN202410847650.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-27
- Publication Date
- 2024-11-26
AI Technical Summary
When the existing AUV path planning method processes complex and changeable ocean spatiotemporal data, it is difficult to find the optimal path, resulting in low execution efficiency and high energy consumption. The modeling data of the existing methods are biased from the real data, which lacks practical value. .
By using ocean spatiotemporal data to build a three-dimensional ocean simulation environment, improve reinforcement learning algorithms, design adaptive state space and reward functions, preprocess the ocean spatiotemporal data, and use soft behavior criticism algorithm (SAC) for path planning to form ocean spatiotemporal space-time Data-driven path planning algorithm.
It realizes a more realistic marine simulation environment, reduces the processing pressure of the algorithm, improves the efficiency and practical value of path planning, and can better deal with data processing problems in complex marine environments.
Smart Images

Figure CN119024868A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of AUV path planning, and in particular to an enhanced AUV path planning method and system driven by ocean spatiotemporal data. Background Art
[0002] With the application of the Internet of Things and information technology in the ocean field, a large amount of ocean spatiotemporal data has been generated and collected. The processing and mining of these big data are crucial to human development of marine resources. As an important part of ocean exploration, autonomous underwater vehicles (AUVs) have shown great potential and broad prospects, and their technological development has attracted much attention. However, the complex and changeable ocean spatiotemporal data poses a challenge to the processing and analysis capabilities of AUVs. For the most basic path planning tasks, AUVs need to process and analyze massive spatiotemporal data and find the optimal path to complete the task. The optimal path can not only effectively improve the execution efficiency of AUVs, but also reduce the energy consumption of AUVs. Therefore, it is necessary to study the AUV path planning scheme driven by spatiotemporal big data.
[0003] A complete path planning solution usually consists of two parts: environment modeling and path planning algorithm:
[0004] Environmental modeling plays a vital role in ocean research and simulation, and is generally divided into two-dimensional modeling and three-dimensional modeling. Two-dimensional environmental modeling is simple and fast in simulation, but it is limited to one plane. Therefore, more and more studies have begun to focus on three-dimensional environmental modeling. In addition to the expansion of spatial dimensions, many scholars have also introduced a variety of models such as ocean currents and seabeds to simulate the impact on AUV path planning. Common three-dimensional environmental modeling methods include grid method, visibility graph method and topological method. The grid method divides the environment into several squares and represents the environmental information of the area in units of squares. This modeling method can only decompose the three-dimensional motion of the AUV into two motion planes, and has high requirements on the resolution of the environment. The visibility graph method converts the path planning problem into a mathematical geometry problem. The topological method divides the planning space into subspaces with consistent attributes and constructs a topological network. It can find a topological path from the starting point to the end point on the topological network, and derive a geometric path from the topological path. The modeling data of these existing traditional methods are generated by numerical predictions and mathematical functions, which means that there is a certain deviation between them and the real ocean spatiotemporal data;
[0005] The goal of path planning is to find the optimal path. Common path planning algorithms include traditional algorithms, graphical methods, intelligent bionic algorithms, and deep reinforcement learning algorithms. Traditional methods include simulated annealing algorithms, artificial potential field methods, etc. These methods are simple to describe but are prone to fall into local optimality. Graphical methods provide basic modeling methods, mainly C-space method, grid method, and Voronoi diagram method. These methods are easy to model but lack search capabilities. People have proposed intelligent bionic algorithms that imitate natural organisms, such as ant colony algorithms and genetic algorithms. Although they have good global optimization capabilities and are easy to implement with computers, they are computationally intensive and inefficient. These common path planning methods do not take into account the large amount of spatiotemporal data generated in the marine environment, resulting in relatively idealized algorithms and lack of practical value.
[0006] Therefore, there is an urgent need for an enhanced AUV path planning method and system driven by ocean spatiotemporal data to solve the above problems. Summary of the invention
[0007] The purpose of the present invention is to provide an enhanced AUV path planning method and system based on ocean spatiotemporal data driving, which can help AUV make full use of spatiotemporal data and improve decision-making and planning capabilities.
[0008] In order to achieve the above object, the present invention is implemented through the following technical solutions:
[0009] A method for enhanced AUV path planning based on ocean spatiotemporal data driving includes the following steps:
[0010] S1: Environmental modeling using ocean spatiotemporal data;
[0011] S2: Improve the reinforcement learning algorithm based on the environment model established in step S1;
[0012] S3: Based on the improvement of the reinforcement learning algorithm in step S2, an ocean spatiotemporal data path planning algorithm is established.
[0013] Preferably, the step S1 specifically comprises:
[0014] A three-dimensional ocean simulation environment is constructed by measuring actual ocean time-space data, and the three-dimensional ocean simulation environment is constructed by using a regional ocean model system to perform three-dimensional primitive equation calculations.
[0015] Preferably, the step S2 is specifically:
[0016] Based on the three-dimensional ocean simulation environment established in step S1, the reinforcement learning algorithm is improved, specifically by preprocessing the ocean spatiotemporal data through state space and reward functions.
[0017] Preferably, the state space is specifically set as follows:
[0018] Consider the AUV as a point mass, and the position of the AUV at time t is P t (x,y,z), which is represented as follows:
[0019] P t (x,y,z)=(P xt ,P yt ,P zt ),
[0020] Collision detection is used to limit the range of AUV activities, specifically:
[0021] When the AUV touches the boundary, P e Set to 1 to retrain, otherwise, P e Set to 0 for normal exploration, the state space Obs of the AUV at time t t , which is expressed as follows:
[0022] Obs t =(P xt ,P yt ,P zt ,P e )
[0023]
[0024] Preferably, the reward function setting includes: a target point approach reward, a target point arrival reward, a step size penalty reward and a collision penalty reward;
[0025] The target point proximity reward is set to r t , the final reward for approaching the target point is set as:
[0026] r dis =β+(1+β)(d last -d current ),β∈[0,1],
[0027] Among them, β is a hyperparameter, d last is the distance between the AUV and the target point at the last moment, d current is the distance between the AUV and the target point at the current moment;
[0028] The target point arrival reward is set to r goal , when the AUV reaches the target point, it is G done =1, the reward for reaching the target point is expressed as:
[0029] r goal =50*G done ;
[0030] The step penalty reward is set to r step, the step penalty reward is expressed as:
[0031] r step = -1.5*n;
[0032] The collision penalty reward is set to r collision , and set the judgment parameter I done , the collision penalty reward is expressed as:
[0033] r collision =-50*I done ;
[0034] The final rewards under the ocean spatiotemporal data are:
[0035] r total =r dis +r goal +r step +r collision .
[0036] Preferably, the step S3 is specifically as follows: an AUV path planning algorithm based on a soft behavior critical algorithm expresses the preprocessed data as: the real-time position P(x t ,y t ,z t ) and collision mark P et ,Right now:
[0037] S t =[x t ,y t ,z t ,P et ];
[0038] Input preprocessed data to get action A t , reward R t and the next state S t , and stored in the experience pool, the experience pool is expressed as:
[0039] Pool = [S1, A1, R1, S2 ...];
[0040] Define a different state function V π , V π In addition to the reward function, each time step change of also contains an entropy reward, expressed as:
[0041] V π (s) = E Tπ [∑γ′(R(s t ,a t ,s t+1 )+αH(π(·|s t )))||||s0=s,a0=a],
[0042] Except for the initial step, the value function Q π The variation also includes an entropy bonus, which is:
[0043]
[0044] Build V π and Q π The relationship between them is expressed as:
[0045] V π (s) = E Tπ [Q π (s,a)]+αH(π(|s t ))
[0046] Combining the above two equations, we can get V π and Q π The Bellman equation is:
[0047] Q π (s,a)=E s′ P[R(s,a,s′)+γV π (s′)]
[0048] V π (s) = E aπ [Q(s,a)+αlogπ(a|s)].
[0049] An enhanced AUV path planning based on ocean spatiotemporal data drive, including:
[0050] Data modeling module, used to: use ocean spatiotemporal data for environmental modeling;
[0051] Model improvement module, used to: improve the reinforcement learning algorithm based on the established environment model;
[0052] The model output module is used to establish an ocean spatiotemporal data path planning algorithm based on the improvement of the reinforcement learning algorithm.
[0053] Compared with the prior art, the beneficial effects of the present invention are:
[0054] 1. In order to solve the problem that the existing AUV ocean simulation environment is not realistic and the data is generated by formulas, a reliable three-dimensional ocean simulation environment is constructed through the Regional Ocean Model System (ROMS) using real measurement data such as ocean current information and seabed topography.
[0055] 2. An adaptive state space and reward function are designed to pre-process the ocean big data. This design transforms the input of the algorithm from massive spatiotemporal data into a four-dimensional vector, which greatly reduces the processing pressure of the algorithm.
[0056] 3. Based on the SAC framework, a path planning algorithm driven by ocean spatiotemporal data is proposed. The algorithm inputs preprocessed spatiotemporal data to support the path planning of AUV. The data is mined and analyzed through three coupled networks, thus overcoming the processing difficulties of ocean big data. BRIEF DESCRIPTION OF THE DRAWINGS
[0057] Figure 1 It is a flow chart of the method of the present invention.
[0058] Figure 2 It is a ROMS simulation flow chart of the present invention.
[0059] Figure 3 It is the ocean environment modeling simulation diagram of the present invention.
[0060] Figure 4 It is a schematic diagram of the path planning algorithm driven by ocean spatiotemporal data of the present invention.
[0061] Figure 5 It is a schematic diagram of the system structure of the present invention. DETAILED DESCRIPTION
[0062] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the application equally.
[0063] In the present invention, terms such as "upper", "lower", "left", "right", "front", "back", "vertical", "horizontal", "side", "bottom", etc. indicate directions or positional relationships based on the directions or positional relationships shown in the accompanying drawings. They are relational words determined only for the convenience of describing the structural relationships of the various parts or elements of the present invention, and do not specifically refer to any part or element in the present invention and should not be understood as limitations on the present invention.
[0064] In the present invention, terms such as "fixed connection", "connected", "connection", etc. should be understood in a broad sense, indicating that it can be fixedly connected, integrally connected or detachably connected; it can be directly connected or indirectly connected through an intermediate medium. Relevant scientific research or technical personnel in this field can determine the specific meanings of the above terms in the present invention according to specific circumstances, and they should not be understood as limiting the present invention.
[0065] Example:
[0066] By introducing real ocean spatiotemporal data, a reinforcement learning AUV path planning scheme (OSDP) driven by spatiotemporal big data is proposed. Figure 1As shown, a reinforced AUV path planning method driven by ocean spatiotemporal data includes the following steps:
[0067] S1: Environmental modeling using ocean spatiotemporal data;
[0068] S2: Improve the reinforcement learning algorithm based on the environment model established in step S1;
[0069] S3: Based on the improvement of the reinforcement learning algorithm in step S2, an ocean spatiotemporal data path planning algorithm is established.
[0070] Among them, ocean spatiotemporal data is used for environmental modeling:
[0071] In order to overcome the problem of unrealistic data in marine environment modeling, a three-dimensional marine simulation environment was constructed using the actual measured marine spatiotemporal data, narrowing the gap with the actual application scenario. This environment can effectively improve the reliability of the simulation environment and provide real data support for the path planning algorithm.
[0072] In order to match the real scene, this embodiment uses the Regional Ocean Model System (ROMS) to calculate the three-dimensional primitive equation. The system can import real ocean spatiotemporal data to generate a simulation environment. ROMS is an ocean model with surface freedom, terrain following, internal and external model splitting and other technologies. It can be used to calculate the three-dimensional nonlinear baroclinic primitive equation. It has complete functional modules, including fluid mechanics module, biogeochemistry module, sediment module, wave module, sea ice module and data assimilation module. The calculation process of the ROM simulation is as follows: Figure 2 As shown;
[0073] In order to ensure the authenticity of the simulation environment as much as possible, this embodiment uses the real spatiotemporal data of part of the Mariana Trench as the basis for ROMS simulation. The specific location of ocean data collection is as follows: Figure 2 As shown, first, the seafloor topography and other data are input into ROMS, then the ocean current, temperature, salinity and other data in January 2019 are set as the initial field, and finally, the temperature and salinity records in February 2017 are added as the boundary field, and the weather data in February 2019 are added as the forcing field;
[0074] The ocean big data is processed by ROMS to generate a three-dimensional ocean environment model. Considering the display effect, the ocean area is divided into a grid form. The points represent the real three-dimensional ocean current information, which is synthesized by three vectors: longitude, latitude and verticality. Each position of the ocean current is represented by an arrow. The direction of the arrow indicates the direction of the ocean current, and the length of the arrow indicates the size of the ocean current. For clear display, only the current simulation data of each position is retained later.
[0075] The environment model established in step S1 improves the reinforcement learning algorithm:
[0076] Based on the established 3D ocean simulation environment, the reinforcement learning algorithm was adaptively modified to preprocess the ocean spatiotemporal data through state space and reward functions, where:
[0077] State space setup:
[0078] Due to the rapid development of marine information technology, AUV can easily obtain a large amount of marine spatiotemporal data. If all this data is directly input, it will be a great challenge to the calculation and processing capabilities of the algorithm. Therefore, a state space is established to pre-process the data. The state space is the input of the neural network and directly determines the amount of calculation of the network. In a complex marine environment, the motion trajectory of the AUV is affected by its own dynamics and the external force of the ocean. The position change of the AUV can directly reflect the marine environmental information. Therefore, directly taking the position change of the AUV as the main part of the state space instead of directly inputting a large amount of environmental data can greatly reduce the dimension of the state space and reduce the amount of calculation of the neural network. In this embodiment, the AUV is regarded as a particle and only its position is considered. It is assumed that the position of the AUV at time t is P t (x,y,z), which can be expressed as follows:
[0079] P t (x,y,z)=(P xt ,P yt ,P zt )
[0080] In addition, in order to limit the activity range of the AUV, a collision detection method is used to respond to the mission range. Compared with the area information directly imported into the environment, this method only requires a collision flag bit P e , which greatly compresses the redundant environmental data and helps the AUV to learn. When the AUV touches the boundary, P e Set to 1 to retrain, otherwise P e Set to 0 for normal exploration. Therefore, the state space is designed as a four-dimensional continuous vector, including the AUV position and the collision flag P e , the state space Obs of AUV at time t t It can be expressed as:
[0081] Obs t =(P xt ,P yt ,P zt ,P e )
[0082]
[0083] The combination of the ocean currents in the environment and the dynamics of the AUV determines the change of the AUV's position. Therefore, the Obs before and after can indirectly reflect the change information of the ocean environment, thereby realizing the preprocessing of some ocean spatiotemporal data;
[0084] Reward function settings:
[0085] The state space setting has preprocessed some ocean spatiotemporal data. However, due to the complex ocean environment, the AUV needs to face a large amount of redundant data and sparse rewards. In order to improve the efficiency of the algorithm, a reward function is designed according to the ocean environment data. The reward function used in this embodiment is divided into the following parts: target point approach reward, target point arrival reward, step penalty reward and collision penalty reward;
[0086] The target point is close to the reward r t :The most basic task of AUV is to approach the target point. Therefore, the simplest design method is to give a fixed positive reward when the distance to the target point decreases, and a fixed negative reward when the distance increases. In order to avoid AUV taking a detour to obtain a larger reward, the reward is set according to the distance change between the previous and next moments. Therefore, the final reward for approaching the target point is set as:
[0087] r dis =β+(1+β)(d last -d current ),β∈[0,1]
[0088] β is a hyperparameter, which is obtained at each step, d last is the distance between the AUV and the target point at the last moment, d current is the distance between the AUV and the target point at the current moment;
[0089] Reward for reaching the target point r goal : It represents the final reward obtained by the AUV after completing the task. If the target point reward is too large, the AUV will only learn to reach the target point, which is not conducive to finding the optimal path. Set the judgment parameter G done , if the AUV reaches the target point, G done =1, therefore, the target point arrival reward used in this embodiment is expressed as:
[0090] r goal =50*G done
[0091] Step penalty reward r step:It requires the AUV to reach the target point with the least number of steps. This embodiment adopts the simplest penalty method, that is, a negative value is rewarded for each step. The reward implicitly uses the ocean current, which helps the AUV learn how to use the ocean current to move. The absolute value of the step penalty reward must be greater than the reward for approaching the target point. Therefore, each step n of the AUV has a penalty reward. The step penalty reward is set as:
[0092] r step =-1.5*n
[0093] Collision Penalty Reward r collision :The position of the AUV needs to be limited to a certain range. Therefore, when the AUV touches the boundary, it needs to be punished. This is the collision penalty reward. In the experiment, it was found that if the collision reward is too large, the AUV will wander in place. This is because the AUV finds that every time it approaches the boundary, the value function becomes smaller, causing it to be unwilling to leave the center position. The judgment parameter I is set done If the AUV hits the boundary, then I done , the collision penalty reward is recorded as:
[0094] r collision =-50*I done
[0095] Therefore, the final reward under the ocean spatiotemporal data is:
[0096] r total =r dis +r goal +r step +r collision .
[0097] Based on the improvement of the reinforcement learning algorithm in step S2, an ocean spatiotemporal data path planning algorithm is established:
[0098] The spatiotemporal data of the ocean environment has been preprocessed by the state space and reward function. In order to better mine and analyze the ocean data, an ocean spatiotemporal data driven path planning algorithm OSDP is proposed. The framework of the scheme is as follows Figure 4 As shown in FIG. 1 , the AUV path planning environment constructed in this embodiment is a continuous state space and discrete action space scenario, which is characterized by a lot of redundant information and is difficult to be effectively processed by traditional methods. Therefore, this embodiment proposes an AUV path planning algorithm based on the soft behavior criticism (SAC) algorithm. The algorithm uses the preprocessed ocean data as the input of the neural network. The preprocessed data can be expressed as the real-time position P(x t ,y t ,z t ) and collision mark P et , respectively expressed as:
[0099] S t =[x t ,y t ,z t ,P et ]
[0100] By inputting the preprocessed data, we can get action A t , reward R t and the next state S t , these data are stored in the experience pool, these training data are stored in the experience pool, and sampled learning is performed in subsequent learning to improve data utilization. The experience pool is expressed as:
[0101] Pool = [S1, A1, R1, S2 ...]
[0102] The algorithm designs three coupled networks to deeply process and analyze the data, mainly including the action network, Q network and target Q network. Since the optimal function of the acquisition strategy is modified, the derivation of its Bellman equation is also changed accordingly. First, a slightly different state function V is defined. π , V π In addition to the reward function, each time step change of also contains an entropy reward, expressed as:
[0103] V π (s) = E Tπ [∑γ′(R(s t ,a t ,s t+1 )+αH(π(·|s t )))||||s0=s,a0=a]. (11)
[0105] Except for the initial step, the value function Q π The change also includes an entropy bonus, calculated as:
[0106]
[0107] With these definitions, we can build V π and Q π The relationship between them is expressed as:
[0108] V π (s) = E Tπ [Q π (s,a)]+αH(π(|s t ))
[0109] Combining the above two equations, we can get V π and Q π The Bellman equation is calculated as follows:
[0110] Q π (s,a)=E s′ P[R(s,a,s′)+γV π (s′)]
[0111] V π (s) = E aπ [Q(s,a)+αlogπ(a|s)]
[0112] Compared with common reinforcement learning methods, OSDP inputs pre-processed spatiotemporal data into a neural network and deeply processes the data through three coupled networks. This gives the algorithm stronger data processing capabilities and helps AUVs better cope with complex marine environments.
Claims
1. A method for enhanced AUV path planning based on ocean spatiotemporal data, characterized in that: The following steps are involved: S1: Environmental modeling using ocean spatiotemporal data; S2: Improve the reinforcement learning algorithm based on the environment model established in step S1; S3: Based on the improvement of the reinforcement learning algorithm in step S2, an ocean spatiotemporal data path planning algorithm is established.
2. According to claim 1, a method for enhanced AUV path planning based on ocean spatiotemporal data driving, characterized in that: The step S1 is specifically as follows: A three-dimensional ocean simulation environment is constructed by measuring actual ocean time-space data, and the three-dimensional ocean simulation environment is constructed by using a regional ocean model system to perform three-dimensional primitive equation calculations.
3. According to claim 1, a method for enhanced AUV path planning based on ocean spatiotemporal data driving, characterized in that: The step S2 is specifically as follows: Based on the three-dimensional ocean simulation environment established in step S1, the reinforcement learning algorithm is improved, specifically, the ocean spatiotemporal data is preprocessed through the state space and reward function.
4. According to claim 3, a method for enhanced AUV path planning based on ocean spatiotemporal data driving, characterized in that: The state space is specifically set as follows: Consider the AUV as a point mass, and the position of the AUV at time t is P t (x, y, z), expressed as follows: P t (x,y,z)=(P xt ,P yt ,P zt ), Collision detection is used to limit the range of AUV activities, specifically: When the AUV touches the boundary, P e Set to 1 to retrain, otherwise, P e Set to 0 for normal exploration, the state space Obs of the AUV at time t t , which is expressed as follows: Obs t =(P xt ,P yt ,P zt ,P e ) 5. The enhanced AUV path planning method based on ocean spatiotemporal data driving according to claim 3 is characterized in that: The reward function setting includes: target point approach reward, target point arrival reward, step size penalty reward and collision penalty reward; The target point proximity reward is set to r t , the final reward for approaching the target point is set as: r dis =β+(1+β)(d) last -d current) , β∈[0,1], Among them, β is a hyperparameter, d last is the distance between the AUV and the target point at the last moment, d current is the distance between the AUV and the target point at the current moment; The target point arrival reward is set to r goal , when the AUV reaches the target point, it is G done =1, the reward for reaching the target point is expressed as: r goal =50*G done ; The step penalty reward is set to r step , the step penalty reward is expressed as: r step =-1.5*n; The collision penalty reward is set to r collision , and set the judgment parameter I done , the collision penalty reward is expressed as: r collision =-50*I done ; The final rewards under the ocean spatiotemporal data are: r total =r dis +r goal +r step +r collision 。 6. The enhanced AUV path planning method based on ocean spatiotemporal data driving according to claim 1 is characterized in that: The step S3 is specifically as follows: based on the AUV path planning algorithm of the soft behavior critical algorithm, the preprocessed data is expressed as: the real-time position P(x t ,y t , z t ) and collision mark P et ,Right now: S t =[x t ,y t ,z t ,P et ]; Input preprocessed data to get action A t , reward R t and the next state S t , and stored in the experience pool, the experience pool is expressed as: Pool=[S1, A1, R1, S2...]; Define a different state function V π , V π In addition to the reward function, each time step change of also contains an entropy reward, expressed as: V π (s)=E Tπ [∑γ′(R(s t ,a t ,s t+1 )+αH(π(·|s t )))|s0=s,a0=a], Except for the initial step, the value function Q π The variation also includes an entropy bonus, which is: Build V π and Q π The relationship between them is expressed as: V π (s)=E Tπ [Q π (s, a)]+αH(π(|s t )) Combining the above two equations, we can get V π and Q π The Bellman equation is: Q π (s, a)=E s′ P[R(s, a, s′)+γV π (s′)] V π (s)=And aπ [Q(s,a)+αlogπ(a|s)]。 7. An enhanced AUV path planning system driven by ocean spatiotemporal data, characterized in that: include: Data modeling module, used to: use ocean spatiotemporal data for environmental modeling; Model improvement module, used to: improve the reinforcement learning algorithm based on the established environment model; The model output module is used to establish an ocean spatiotemporal data path planning algorithm based on the improvement of the reinforcement learning algorithm.
