Unmanned ship complex path planning method and device based on reinforcement learning algorithm

Abstract

The application provides a complex path planning method and device for an unmanned ship based on a reinforcement learning algorithm, and the algorithm comprises the following steps: S1, performing inflation expansion on a pre-planned coastline contour to establish a time-space change sea current map environment model; S2, adopting an actor-critic reinforcement learning algorithm, combining a safety reward function and an energy consumption reward function to train different critics respectively, and comprehensively evaluating an evaluation value function to obtain an actor-multi-critic reinforcement learning algorithm, and the actor-multi-critic reinforcement learning algorithm generates an optimal path after multiple rounds of training. The application analyzes and utilizes the time-space change characteristics of sea currents to plan a collision-free navigation path with less energy consumption between a starting point and an ending point, so that the unmanned ship can better adapt to long-distance and long-time sea tasks, energy consumption is saved, and the effect of prolonging the endurance time is achieved.

Description

[Technical Field]

[0001] This invention relates to the technical field of unmanned surface vessel (USV) path planning, and particularly to a method and apparatus for complex path planning of USVs based on reinforcement learning algorithms in spatiotemporally changing ocean current environments. [Background Technology]

[0002] With the increasing human activity in the ocean, unmanned surface vessels (USVs) are demonstrating their advantages in marine scientific research and maritime search and rescue. Path planning is a crucial step for USVs in completing tasks such as marine biological sampling and seabed mapping. High-quality navigation paths ensure that USVs can safely and quickly reach their targets and respond rapidly to environmental changes. In recent years, actor-evaluator reinforcement learning algorithms have been widely used to address path planning problems for mobile robots. This algorithm acquires knowledge through environmental exploration and learns the mapping relationship from state to behavior through repeated trials. For path planning problems, reinforcement learning has the advantages of requiring less prior environmental information, having low dependence on sensor accuracy, and strong generalization ability. However, basic actor-evaluator reinforcement learning algorithms suffer from the drawback of relying on a single path quality evaluation criterion, making it difficult to achieve optimal results in multi-objective optimization, complex models and parameters, and globally optimal planning.

[0003] In long-distance ocean navigation missions, the small size and limited payload of unmanned surface vessels (USVs) make energy supply a significant factor restricting their range. Furthermore, tidal forces cause periodic variations in water current velocity at the same location on the sea surface, and significant differences in current velocity at different locations at the same time. The energy consumption of the vessel is largely influenced by its relative speed to the current. To reduce the impact of relative current velocity on ocean navigation missions, a complex path planning method and device for USVs based on reinforcement learning algorithms is proposed. [Summary of the Invention]

[0004] The purpose of this invention is to solve the problems in the prior art and propose a complex path planning method and device for unmanned surface vessels based on reinforcement learning algorithms. This method can generate a collision-free navigation path with energy saving as the goal in a spatiotemporally changing ocean current environment. This path connects the starting point and the ending point of the mission.

[0005] To achieve the above objectives, this invention proposes a complex path planning method for unmanned surface vessels based on reinforcement learning algorithms, comprising the following steps:

[0006] S1. Appropriately expand the pre-planned coastline outline to establish a time-space changing ocean current map environment model;

[0007] S2. An actor-evaluator reinforcement learning algorithm is adopted, which combines a safety reward function and an energy consumption reward function to train different evaluators respectively, and the evaluation value function is comprehensively evaluated to obtain an actor-multi-evaluator reinforcement learning algorithm. The actor-multi-evaluator reinforcement learning algorithm is used to calculate the energy consumption and safety cost of the unmanned surface vessel at each time step, and a collision-free energy-optimal path is generated after multiple rounds of training.

[0008] Preferably, in step S1, ocean current maps of the mission area at different times are drawn using a global high-resolution coastline dataset and an ocean current dataset. When processing the coastline dataset, the coastline outline is appropriately expanded based on the geometry and motion parameters of the unmanned surface vessel.

[0009] As a preferred method, interpolation is used to obtain the ocean current velocity at any given time.

[0010] As a preferred approach, the global high-resolution coastline dataset uses polygons to represent the coastline outline. When expanding the coastline outline, the expanded coastline outline is obtained by calculating the expanded coordinates of the vertices of all polygons.

[0011] Preferably, in step S2, during the training of the actor, the evaluation value functions of each evaluator are summed to obtain a comprehensive evaluation.

[0012] Preferably, in step S2, each evaluator and actor corresponds to a neural network for training.

[0013] Preferably, in step S2, the state space of the unmanned surface vessel is defined as... Where x and y represent the coordinates of the unmanned surface vessel. Let u and v be the directions of travel of the unmanned surface vessel, u and v be the components of the ocean current velocity in the x and y directions, t be the cumulative time since the start of the mission, and define the action space as the change in the travel angle.

[0014] Preferably, in step S2, the security reward function is defined as follows:

[0015]

[0016] Where, k safe For the safety factor, d thresh d is the safe distance threshold. obs E represents the distance between the unmanned surface vessel and the nearest obstacle. L To offset the value and prevent the reward from being too small, L is the distance between the unmanned surface vessel and the destination, and k L Used to measure how sensitive the reward value is to distance L.

[0017] Preferably, in step S2, the energy consumption return function is defined as follows:

[0018]

[0019] Where, k e The energy consumption coefficient is ρ, which represents the density of seawater, and C is the energy consumption coefficient. D A and B represent the drag coefficient and the area of ​​force application, respectively. This represents the velocity vector of the unmanned surface vessel relative to the ocean current. d represents the ground velocity vector of the unmanned surface vessel. step This represents the distance traveled by the unmanned surface vessel (USV) at each time step; the relationship between the USV's ground velocity and relative velocity is: in This is the ocean current velocity vector, calculated from an ocean current dataset.

[0020] This invention also proposes a complex path planning device for unmanned surface vessels (USVs) based on reinforcement learning algorithms. The device includes a storage unit, a sensing unit, and a computing processing unit. The storage unit stores maps, ocean current information, and computer execution instructions. The sensing unit acquires the real-time position and direction of motion of the device. The computing processing unit executes the computer execution instructions stored in the storage unit, enabling it to perform the aforementioned complex path planning method for USVs based on reinforcement learning algorithms. This allows for path pre-planning in a simulated environment, while simultaneously receiving information from the sensing unit in real-world navigation conditions to calibrate the navigation path.

[0021] The beneficial effects of this invention are as follows: The path planning method of this invention fully considers the more realistic marine environment, namely, the spatiotemporal variation characteristics of ocean currents, which are not only spatially variable but also temporally variable. By analyzing and utilizing the spatiotemporal variation characteristics of ocean currents, a less energy-consuming path is planned between the starting point and the destination, thereby enabling unmanned surface vessels (USVs) to better adapt to long-distance, long-endurance maritime missions, saving energy consumption and extending endurance. Furthermore, this invention addresses the shortcomings of basic actor-evaluator reinforcement learning algorithms in terms of a single path quality evaluation standard by proposing an actor-multi-evaluator algorithm. This improved algorithm incorporates both energy consumption and safety costs of navigation into the evaluation criteria, enhancing the algorithm's applicability.

[0022] The features and advantages of the present invention will be described in detail through embodiments and in conjunction with the accompanying drawings. [Attached Image Description]

[0023] Figure 1 This is a schematic diagram of the coastline expansion algorithm of the present invention;

[0024] Figure 2 This is a schematic diagram of the network structure of the actuator part of the present invention;

[0025] Figure 3 This is a schematic diagram of the network structure of the evaluator part of the present invention;

[0026] Figure 4 This is a diagram of the actuator-multi-evaluator algorithm architecture of the present invention.

Detailed Implementation Methods

[0027] The present invention provides a method for complex path planning of unmanned surface vessels based on reinforcement learning algorithms, comprising the following steps:

[0028] S1. Establish a temporal-spatial variation ocean current map environment model: Use global high-resolution coastline datasets and ocean current datasets to draw ocean current maps of the task area at different times.

[0029] Due to the inherent errors in map data and the dynamic changes in coastlines caused by tidal forces, the coastline contour is appropriately expanded after obtaining the coastline data, based on the unmanned surface vessel's (USV) geometry and motion parameters. This helps the USV avoid potential collisions such as bottoming out or running aground. In the coastline dataset, the coastline contour is represented by polygons; therefore, the expanded coastline contour can be obtained simply by using geometric calculations to determine the positions of all polygon vertices after expansion.

[0030] Secondly, ocean current data is discrete in time, but actual ocean current changes are continuous. This invention uses interpolation to estimate the ocean current velocity at any given time. Let a point at time τ... i and time τ i+1 The ocean current velocity vector is and Then the point at time τ i +t ocean current speed for

[0031]

[0032] S2. Using an actor-multi-evaluator reinforcement learning algorithm, calculate the energy consumption and safety cost of the unmanned surface vessel's navigation at each time step. After multiple rounds of training, generate a collision-free, energy-optimal path. This algorithm specifically includes the following steps:

[0033] The expression for calculating the energy consumption of an unmanned surface vessel (USV) is as follows:

[0034]

[0035] Where ρ represents the density of seawater, C D A and B represent the drag coefficient and the area of ​​force application, respectively. This represents the velocity vector of the unmanned surface vessel relative to the ocean current. d represents the ground velocity vector of the unmanned surface vessel. step This represents the distance traveled by the unmanned surface vessel (USV) at each time step. The relationship between the USV's ground velocity and relative velocity is:

[0036]

[0037] in It is the ocean current velocity vector, which can be calculated from an ocean current dataset.

[0038] The basic elements of reinforcement learning include agent, environment, state, action, reward, and policy. The agent is the subject performing the task; the environment is the object the agent interacts with; the state is some or all of the observations the agent receives from the environment at a given moment; the action is the behavior the agent can take, which transitions the agent from the original state to a new state; the reward is used to evaluate the quality of the action and directly defines the goal of reinforcement learning; the policy is the mapping from state to action, determining the action the agent should take when facing a given state. It is the core of reinforcement learning, and the goal of the policy is to minimize the cumulative reward throughout the entire task execution process.

[0039] In the maritime path planning task based on reinforcement learning, the intelligent agent refers to the unmanned surface vessel (USV), the environment is the mission sea area, and the state space of the USV is defined as follows: Where x and y represent the coordinates of the unmanned surface vessel. Let $\mathbf{v}$ be the direction of travel for the unmanned surface vessel (USV), $\mathbf{v}$ be the components of the ocean current velocity in the x and y directions, and $\mathbf{t}$ be the cumulative time since the start of the mission. The USV is assumed to travel at a constant ground velocity. For navigation, the action space can be set as the change in navigation angle, for example, A = [-75°, -60°, -45°, ..., 45°, 60°, 75°]. When setting the reward function, in addition to considering energy consumption, the safety of the path also needs to be considered, that is, avoiding collisions with obstacles. First, the energy consumption reward function R is determined by equation (1). energy =k e E, further define the security reward function.

[0040]

[0041] Where, k safe For the safety factor, d thresh d is the safe distance threshold. obs E represents the distance between the unmanned surface vessel and the nearest obstacle. L To offset the value and prevent the reward from being too small, L is the distance between the unmanned surface vessel and the destination, and k L This is used to measure the sensitivity of the reward value to distance L. The above formula shows that when the unmanned surface vessel is at a distance d from the nearest obstacle... obs After the distance is less than the safety distance threshold, the safety reward function and d obs The relationship is linear, when k safe When d is negative obs The smaller the value, the smaller the reward, which serves as a penalty for increasing the probability of the unmanned surface vessel colliding.

[0042] This invention modifies the basic actor-judge reinforcement learning algorithm, training different judges using a safety reward function and an energy reward function respectively, and combining the evaluation functions V of the two judges during the actor training part. safe (s) and V energy (s) Summation is performed as a comprehensive evaluation. The above algorithm is called the actor-multi-evaluator algorithm, where each evaluator and actor corresponds to a neural network for training. The path generated after training is the energy-optimal path given by the algorithm in a spatiotemporally varying ocean current environment.

[0043] Example 1

[0044] After obtaining the coastline data, the coastline outline is expanded, such as... Figure 1 As shown. Let the expansion distance be r. Considering the impact of seawater on the sandy bottom at the angle between the coastlines, the distance between the virtual vertex arcs is r + δ. The value of δ is adjusted according to the angle between the coastlines. When the angle between AB and BC is greater than or equal to 45°, δ = 0; when the angle between AB and BC is less than 45°, δ = 0.1r. In this embodiment, the angle between AB and BC is greater than 45°, so δ = 0. The broken line ABC expands to abc. Line segments AB and ab, and BC and bc are parallel and perpendicular, with a distance of r between them. ob is the angle bisector, and θ... abm and θ cbm Let a and b represent the angles between line segments ab and cb and the horizontal line, respectively. The goal of the dilation algorithm is to calculate the intersection point b(x). b ,y b First, calculate the included angle θ. abm and θ cbm :

[0045]

[0046] Next, calculate angle θ. obc and Bb

[0047]

[0048] Based on the above derivation, (x) b ,y b ) represents

[0049]

[0050] Applying this algorithm to all vertices of the contour will yield the expanded contour.

[0051] Thanks to advancements in marine science and satellite image processing, ocean current conditions can now be predicted. Based on predicted current data, future ocean current environment models can be constructed. First, the unmanned surface vessel (USV) is trained in a simulation environment. Given the coordinates of the starting and ending points, a network structure for the actor and evaluator is constructed. The actor network takes i-dimensional states as input and outputs n-dimensional probability values, where n is the dimension of the action space. Since each element of the output vector represents the probability of selecting that action, a softmax layer is added to the network for normalization, such as... Figure 2 As shown; the input to the evaluator network is an i-dimensional state, and the output is a value function V, as follows. Figure 3 As shown. This invention involves two path quality evaluation criteria, therefore two evaluator networks need to be established: a safety evaluation evaluator and an energy consumption evaluation evaluator, with outputs being value functions V. safe (s) and V energy (s). When iteratively updating the network parameters, the outputs of the two networks are summed to form the output V of the comprehensive evaluation judge. com (s), taking a discount factor of 0 ≤ γ ≤ 1, then the time difference error of each step is expressed as:

[0052] TD=(r safe +r energy +γV com (s t+1 ))-V com (s t )

[0053] The network parameters are updated using the common backpropagation algorithm, and the loss function for the evaluator part is defined as follows:

[0054]

[0055] The loss function for the actor part is

[0056] E Actor =log(y k,t )×TD

[0057] Where y k,t This is the probability value corresponding to the action obtained after sampling the action based on the network output probability at time t. The gradient is calculated using the loss function, allowing for parameter updates. The actor-multi-evaluator algorithm architecture is as follows: Figure 4 As shown.

[0058] The training program begins each round with the unmanned surface vessel (USV) at the starting position. At each time step, the USV performs an action, changing or maintaining its course and traveling a fixed distance in that direction. If the USV reaches the endpoint or collides with an obstacle, the training round ends, the USV's position is reset to the starting point, and a new round begins. Reaching the endpoint earns a higher reward, while colliding with an obstacle incurs a lower reward.

[0059] After a certain number of training iterations, the algorithm will gradually converge, returning an energy-optimal path connecting the starting and ending points. This path is a pre-planned path obtained from the simulation environment. When the unmanned surface vessel (USV) needs to perform an actual navigation task, it is activated at the appropriate time and location, allowing it to track the generated energy-optimal path. At this time, hardware devices such as GPS and electronic compasses in the sensing unit will output the USV's position and heading information in real time. The computing and processing unit can use linear or nonlinear control methods such as PID control to ensure that the actual navigation route does not deviate. Ultimately, the USV reaches its destination with minimal energy consumption.

[0060] This invention discloses a complex path planning device for unmanned surface vessels (USVs) based on reinforcement learning algorithms. The device, applicable to the USV itself, includes a storage unit, a sensing unit, and a computing processing unit. The storage unit stores maps, ocean current information, and computer-executed instructions. The sensing unit contains sensors such as GPS and an electronic compass, enabling it to obtain the device's real-time position and direction of motion. The computing processing unit executes the computer-executed instructions stored in the storage unit, thereby performing the aforementioned complex path planning method based on reinforcement learning algorithms. This involves pre-planning the path in a simulated environment and simultaneously receiving information from the sensing unit under real-world navigation conditions to calibrate the navigation path.

[0061] The above embodiments are illustrative of the present invention and are not intended to limit the present invention. Any simple modifications to the present invention are within the scope of protection of the present invention.

Claims

1. A method for complex path planning of unmanned surface vessels based on reinforcement learning algorithms, characterized in that: Includes the following steps: S1. Expand the pre-planned coastline outline to establish a time-space changing ocean current map environment model; S2. An actor-evaluator reinforcement learning algorithm is adopted, which combines the safety reward function and the energy reward function to train different evaluators respectively, and the evaluation value function is comprehensively evaluated to obtain an actor-multi-evaluator reinforcement learning algorithm. This actor-multi-evaluator reinforcement learning algorithm generates the optimal path after multiple rounds of training. In step S2, the security reward function is defined as follows: ； Where is the safety coefficient, is the safety distance threshold, and is the distance between the unmanned surface vessel and the nearest obstacle. , where is the offset, and is the distance between the unmanned surface vessel and the destination. Used to measure the reward value against distance The degree of sensitivity; In step S2, the energy consumption reward function is defined as follows: ； in, Energy consumption coefficient Indicates the density of seawater. and These represent the drag coefficient and the area of ​​force application, respectively. This represents the velocity vector of the unmanned surface vessel relative to the ocean current. This represents the ground velocity vector of the unmanned surface vessel. This represents the distance traveled by the unmanned surface vessel (USV) at each time step; the relationship between the USV's ground velocity and relative velocity is: ,in This is the ocean current velocity vector, calculated from an ocean current dataset.

2. The complex path planning method for unmanned surface vessels based on reinforcement learning algorithm as described in claim 1, characterized in that: In step S1, ocean current maps of the task area at different times are drawn using the coastline dataset and ocean current dataset. When processing the coastline dataset, the coastline outline is expanded based on the geometry and motion parameters of the unmanned surface vessel.

3. The method for complex path planning of unmanned surface vessels based on reinforcement learning algorithm as described in claim 2, characterized in that: Use interpolation to obtain the ocean current velocity at any given time.

4. The method for complex path planning of unmanned surface vessels based on reinforcement learning algorithm as described in claim 2, characterized in that: The coastline dataset uses polygons to represent coastline outlines. When expanding the coastline outline, the expanded coastline outline is obtained by calculating the expanded coordinates of the vertices of all polygons.

5. The complex path planning method for unmanned surface vessels based on reinforcement learning algorithm as described in claim 1, characterized in that: In step S2, during the training of the actor, the evaluation value functions of each evaluator are summed to form a comprehensive evaluation.

6. The complex path planning method for unmanned surface vessels based on reinforcement learning algorithm as described in claim 1, characterized in that: In step S2, each evaluator and actor corresponds to a neural network for training.

7. The complex path planning method for unmanned surface vessels based on reinforcement learning algorithm as described in claim 1, characterized in that: In step S2, the state space of the unmanned surface vessel is defined as follows: ,in Indicates the coordinates of the unmanned surface vessel. The unmanned surface vessel's (USV) direction of travel. The ocean current velocity in coordinates The directional component, t is the cumulative time since the start of the task, and the action space is defined as the change in the navigation angle.

8. A complex path planning device for unmanned surface vessels based on reinforcement learning algorithms, characterized in that: include: Storage unit, used to store maps and ocean current information as well as computer execution instructions; The sensing unit is used to acquire the real-time position and direction of motion of the device; The computing processing unit is configured to execute computer execution instructions stored in the storage unit, causing the computing processing unit to execute the unmanned surface vessel complex path planning method based on reinforcement learning algorithm as described in any one of claims 1 to 7, to perform path pre-planning through a simulated environment, and to receive information obtained by the sensing unit in real navigation conditions to calibrate the navigation path.

Images