An obstacle avoidance path planning method and system for an intelligent ship

Abstract

The application discloses an obstacle avoidance path planning method and system for an intelligent ship, and relates to the field of path planning.The method comprises the following steps: obtaining obstacle state information within the sensing range of a target intelligent ship and target intelligent ship state information, and constructing a danger degree membership function; using the danger degree membership function to evaluate the collision danger degree of the target intelligent ship and the obstacles within the sensing range; when the collision danger degree is greater than a danger degree threshold, using an improved PPO algorithm according to the target intelligent ship state information and the obstacle state information to obtain an optimal obstacle avoidance strategy and control the target intelligent ship to move. The improved PPO algorithm introduces an angle component, a distance component and a speed component in the evaluation function of the DWA algorithm into the reward function of the traditional PPO algorithm. When the improved PPO algorithm is used to plan a path trajectory, the problems of slow convergence speed, sparse rewards and easy falling into a deadlock area of the traditional PPO algorithm are solved.

Description

Technical Field

[0001] This invention relates to the field of intelligent obstacle avoidance for ships, and in particular to an obstacle avoidance path planning method and system suitable for small, intelligent, all-electric ships in coastal waters. Background Technology

[0002] With the development of emerging coastal economies, and to further protect the marine environment, there is an urgent need for a new generation of small, green, and intelligent vessels in nearshore water transportation, coastal tourism, law enforcement, and nearshore marine resource development. Small, intelligent, all-electric vessels integrate modern information technology, artificial intelligence, new energy sources, and electric propulsion, offering significant advantages such as safety, reliability, energy efficiency, environmental friendliness, and low operating costs. They represent a key direction for the future development of small nearshore vessels. Currently, due to their high levels of intelligence and autonomy, small intelligent vessels have widespread application demand in both military and civilian sectors. However, ensuring the safe, reliable, and economical navigation of intelligent all-electric vessels in the complex nearshore environment remains a critical bottleneck that must be addressed. The waters they navigate are mostly nearshore, facing complex environmental impacts: winding channels, rapid currents, high vessel traffic, intersecting routes, numerous reefs, dams, and estuaries, poor visibility, and complex and variable hydrology. Simultaneously, vessels are significantly affected by wind and waves, and the presence of numerous docked vessels, fishing boats, and other tourist vessels increases traffic density. Therefore, the probability of ship collisions in nearshore waters has increased significantly. A review of 108 ship collision investigation reports from the Maritime Safety Administration in recent years shows that 9% of collisions occurred in nearshore waters, 19% were caused by poor visibility, and 21% were due to ships dragging anchor or colliding with anchored vessels. Therefore, researching intelligent automatic collision avoidance methods is of immeasurable importance for developing a fully automated and intelligent navigation system. Compared to research on unmanned vehicles and drones, research in the field of small intelligent ships started later, but its development prospects are promising and its potential for growth is enormous.

[0003] Existing technologies for obstacle avoidance in complex nearshore waters are often time-consuming, labor-intensive, and inefficient, while also producing poor obstacle avoidance paths. For example, using a single DWA algorithm results in insufficient trajectory foresight during obstacle avoidance; using a single PPO algorithm leads to slow convergence, sparse rewards, and a tendency to get stuck in deadlock regions. Therefore, existing algorithms are not perfect for the application of unmanned surface vessels, and the planned routes are not accurate enough, resulting in poor overall obstacle avoidance performance and low efficiency. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for obstacle avoidance path planning of intelligent ships. Based on the PPO algorithm, the reward function of the PPO algorithm is improved by utilizing the evaluation function of the Dynamic Window (DWA) algorithm. When planning path trajectories using the improved PPO algorithm, the problems of slow convergence speed, sparse rewards, and susceptibility to deadlock in the traditional PPO algorithm are solved.

[0005] To achieve the above objectives, the present invention provides the following solution:

[0006] A method for obstacle avoidance path planning for an intelligent ship, the method comprising:

[0007] Acquire obstacle status information and target smart ship status information within the target smart ship's perception range;

[0008] Based on the target intelligent ship's state information and the obstacle's state information, a hazard membership function is constructed, and the hazard membership function is used to evaluate the collision hazard between the target intelligent ship and the obstacle within the perception range;

[0009] When the collision risk level is less than or equal to the risk level threshold, the target smart ship is controlled to travel along the original route.

[0010] When the collision risk level is greater than the risk level threshold, an improved PPO algorithm is applied based on the current target smart ship state information and the current obstacle state information to obtain the optimal obstacle avoidance strategy for the target obstacle; the target obstacle is an obstacle whose collision risk level is greater than the risk level threshold; the improved PPO algorithm introduces the angle component, distance component, and velocity component from the DWA algorithm evaluation function into the reward function of the traditional PPO algorithm;

[0011] The target intelligent ship is controlled to move based on the optimal obstacle avoidance strategy.

[0012] Optionally, the construction of the hazard membership function based on the target smart ship's state information and the obstacle's state information specifically includes:

[0013] Calculate the distance between the target intelligent ship and the obstacle, the relative speed ratio, the relative hull angle, the nearest encounter distance, and the nearest encounter time;

[0014] Based on the distance, the relative speed ratio, the relative hull angle, the nearest encounter distance, and the nearest encounter time, calculate the following sub-membership functions: distance hazard sub-membership function, relative speed ratio hazard sub-membership function, relative hull angle hazard sub-membership function, nearest encounter distance hazard sub-membership function, and nearest encounter time hazard sub-membership function.

[0015] The final hazard membership function is determined based on the distance hazard sub-membership function, the relative speed ratio hazard sub-membership function, the relative hull angle hazard sub-membership function, the nearest encounter distance hazard sub-membership function, and the nearest encounter time hazard sub-membership function.

[0016] Optionally, the step of applying the improved PPO algorithm based on the current target smart ship state information and the current obstacle state information to obtain the optimal obstacle avoidance strategy for the target obstacle specifically includes:

[0017] Based on the target smart ship state information and the obstacle state information, the DWA algorithm is used to determine the predicted motion trajectory of the target smart ship, and the evaluation function of each predicted motion trajectory is derived.

[0018] The reward function in the improved PPO algorithm is determined based on the evaluation function.

[0019] Based on the current state information of the target smart ship and the current state information of the obstacle, combined with the improved PPO algorithm, the optimal obstacle avoidance strategy for the target obstacle is obtained.

[0020] Optionally, improving the reward function in the improved PPO algorithm based on the evaluation function specifically includes:

[0021] The normal action reward function in the improved PPO algorithm is determined based on the angle component, distance component, and velocity component in the evaluation function; the reward function includes the normal action reward function, the endpoint reward function, and the collision reward function.

[0022] Optionally, before obtaining the optimal obstacle avoidance strategy for the target obstacle based on the current target smart ship state information and the current obstacle state information combined with the improved PPO algorithm, the method further includes:

[0023] Obtain the training dataset for the PPO algorithm. The training dataset includes multiple training subsets. Each training subset includes environmental state sample information at time step t, action information of the target intelligent ship at time step t, reward value at time step t, and environmental state sample information at time step t+1. The environmental state sample information at time step t includes target intelligent ship state sample data and obstacle state sample data at time step t.

[0024] The environmental state sample information of the last time step in the training dataset is input into the Critic network of the improved PPO algorithm to obtain the state estimate of the last time step.

[0025] Calculate the discount reward based on the state estimate of the last time step and all the reward values;

[0026] All environmental state sample information in the training dataset is input into the Critic network to obtain all state estimates; and the Critic network advantage function is determined based on all state estimates and the reward value.

[0027] The parameters of the Critic network are updated according to its loss function; the loss function of the Critic network is determined based on the discounted reward and all the state estimates.

[0028] All the environmental state sample information in the training dataset are input into the new Actor network and the old Actor network of the improved PPO algorithm, respectively, and the ratio of the new policy to the old policy is determined based on the output of the new Actor network and the output of the old Actor network.

[0029] The Actor network loss function is determined based on the ratio of the new and old strategies and the dominance function, and the parameters of the new Actor network are updated according to the Actor network loss function.

[0030] Return to step "Input all the environmental state sample information in the training dataset into the new Actor network and the old Actor network of the improved PPO algorithm respectively", until the number of updates and iterations of the new Actor network reaches the first preset value;

[0031] Assign the parameters of the new Actor network obtained during the last update iteration to the old Actor network;

[0032] Determine whether the current number of training iterations has reached the second preset value; if so, complete the training of the new Actor network, the old Actor network, and the Critic network in the improved PPO algorithm.

[0033] If not, return to step "Obtain the training dataset for the PPO algorithm" until the number of training iterations reaches the second preset value.

[0034] Optionally, obtaining the training dataset for the PPO algorithm specifically includes:

[0035] Obtain environmental status sample information;

[0036] The current environmental state sample information is input into the new Actor network of the improved PPO algorithm to obtain the current output result of the new Actor network. Based on the current output result of the new Actor network, the action information of the target smart ship is determined.

[0037] Based on the current action information of the target intelligent ship and the reward function, determine the reward value corresponding to the current target intelligent ship state sample data and the environmental state sample information of the next time step, and store the current environmental state sample information, the current action information of the target intelligent ship, the current reward value and the environmental state sample information of the next time step in a training subset;

[0038] The environmental state sample information of the next time step is used as the current environmental state sample information, and the process returns to the step "input the current environmental state sample information into the new Actor network of the improved PPO algorithm" until the amount of data in the training dataset reaches a preset number.

[0039] The present invention also provides an obstacle avoidance path planning system for intelligent ships, the system comprising:

[0040] The data acquisition module is used to acquire obstacle status information and target smart ship status information within the perception range of the target smart ship;

[0041] The membership function construction module is used to construct a hazard membership function based on the target intelligent ship's state information and the obstacle's state information, and to use the hazard membership function to evaluate the collision hazard between the target intelligent ship and the obstacle within the perception range;

[0042] The first path planning module is used to control the target smart ship to travel along the original route when the collision risk is less than or equal to the risk threshold.

[0043] The second path planning module is used to apply an improved PPO algorithm to obtain the optimal obstacle avoidance strategy for the target obstacle when the collision risk is greater than the risk threshold. The target obstacle is an obstacle whose collision risk is greater than the risk threshold. The improved PPO algorithm introduces the angle component, distance component, and velocity component from the DWA algorithm evaluation function into the reward function of the traditional PPO algorithm. The module controls the movement of the target intelligent ship based on the optimal obstacle avoidance strategy.

[0044] Optionally, the membership function construction module specifically includes:

[0045] The parameter calculation submodule is used to calculate the distance between the target smart ship and the obstacle, the relative speed ratio, the relative hull angle, the nearest encounter distance, and the nearest encounter time.

[0046] The membership function construction submodule is used to calculate the distance hazard sub-membership function, relative speed ratio hazard sub-membership function, relative hull angle hazard sub-membership function, nearest encounter distance hazard sub-membership function, and nearest encounter time hazard sub-membership function based on the distance, relative speed ratio, relative hull angle, nearest encounter distance, and nearest encounter time, respectively.

[0047] A submodule for constructing a comprehensive membership function is used to determine the final hazard membership function based on the distance hazard sub-membership function, the relative speed ratio hazard sub-membership function, the relative hull angle hazard sub-membership function, the nearest encounter distance hazard sub-membership function, and the nearest encounter time hazard sub-membership function.

[0048] Optionally, the second path planning module specifically includes:

[0049] The evaluation function determination submodule is used to determine the predicted motion trajectory of the target intelligent ship based on the target intelligent ship state information and the obstacle state information, using the DWA algorithm, and to derive the evaluation function for each predicted motion trajectory.

[0050] The reward function improvement submodule is used to determine the reward function in the improved PPO algorithm based on the evaluation function.

[0051] The path planning submodule is used to obtain the optimal obstacle avoidance strategy for the target obstacle based on the current state information of the target smart ship and the current state information of the obstacle, combined with the improved PPO algorithm.

[0052] Optionally, the reward function improvement submodule specifically includes:

[0053] The normal action reward function in the improved PPO algorithm is determined based on the angle component, distance component, and velocity component in the evaluation function; the reward function includes the normal action reward function, the endpoint reward function, and the collision reward function.

[0054] According to specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0055] This invention provides a method and system for obstacle avoidance path planning for intelligent ships. Based primarily on the Proximal Policy Optimization (PPO) algorithm, it improves the reward function of the PPO algorithm by incorporating the evaluation function of the Dynamic Window Algorithm (DWA). The improved PPO algorithm solves for the optimal obstacle avoidance path of the intelligent ship in complex nearshore environments. This not only solves the problems of slow convergence speed, sparse rewards, and susceptibility to deadlock in traditional PPO algorithms, but also addresses the shortcomings of traditional dynamic window algorithms, such as insufficient foresight, unstable solutions, poor obstacle avoidance performance for dynamic obstacles, and susceptibility to local optima (selecting the best path for the next step rather than the globally optimal path each time). Attached Figure Description

[0056] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0057] Figure 1 Here is a flowchart of an obstacle avoidance path planning method for an intelligent ship provided in Embodiment 1 of the present invention;

[0058] Figure 2 Here is a simplified diagram of the algorithm structure of the improved PPO algorithm provided in Embodiment 1 of the present invention;

[0059] Figure 3 This is a model for the field of ship safety provided in Embodiment 1 of the present invention;

[0060] Figure 4 This is a training framework diagram of the improved PPO algorithm provided in Embodiment 1 of the present invention;

[0061] Figure 5 This is a schematic diagram showing the limitation range of the objective function of the PPO algorithm provided in Embodiment 1 of the present invention. Detailed Implementation

[0062] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0063] In the maritime field, small intelligent vessels suitable for nearshore environments can provide comprehensive high-end technical equipment for the tourism industry and marine resource development. Autonomous obstacle avoidance technology is one of the key technologies for realizing ship intelligence. To address the shortcomings of traditional DWA algorithms in obstacle avoidance, such as insufficient trajectory foresight, excessive course changes, low safety, and inability to perform random obstacle avoidance in complex environments, this paper proposes an intelligent ship obstacle avoidance path planning algorithm based on deep reinforcement learning (PPO) and the dynamic window method, leveraging the advantages of deep reinforcement learning in continuous long-term decision-making problems. The DWA algorithm is introduced and improved upon the PPO algorithm, enabling the improved algorithm to adapt to environmental changes and improve the safety and efficiency of intelligent ships in surface obstacle avoidance path planning.

[0064] The purpose of this invention is to provide a method and system for obstacle avoidance path planning of intelligent ships. Based primarily on the Proximal Policy Optimization (PPO) algorithm, it improves the reward function of the PPO algorithm by incorporating the evaluation function of the Dynamic Window Algorithm (DWA). Through program training, it automatically solves for the optimal obstacle avoidance path of the intelligent ship in complex nearshore environments. This not only significantly improves upon the shortcomings of traditional single dynamic window algorithms, such as insufficient foresight, unstable solutions, poor obstacle avoidance performance for dynamic obstacles, and susceptibility to local optima (selecting the best path for the next step rather than the globally optimal path each time), but also solves the problems of slow convergence speed, sparse rewards, and susceptibility to deadlock in traditional PPO algorithms.

[0065] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0066] Example 1

[0067] The obstacle avoidance path planning method of this invention is based on the traditional Proximal Policy Optimization (PPO) algorithm, and combines the method of describing the ship's motion with linear velocity and angular velocity and the evaluation function in the Dynamic Window (DWA) algorithm to form an obstacle avoidance path planning method based on deep reinforcement learning and dynamic windows.

[0068] like Figure 1 As shown, this embodiment provides an obstacle avoidance path planning method for an intelligent ship, the method comprising:

[0069] S1: Obtain obstacle status information and target smart ship status information within the target smart ship's perception range.

[0070] The intelligent ship perceives and collects relevant data about its surrounding environment through a sensor system. The system, which includes LiDAR, millimeter-wave radar, and surround-view cameras, is used to perceive the environment, acquire real-time point cloud data and image information, and perform information preprocessing.

[0071] The core controller of the intelligent ship receives raw data from sensors and performs data fusion and related processing. The core processor of the intelligent ship processes the sensor system data to make it readable by the processor. A multi-source data fusion algorithm is used to perform coordinate transformations on point cloud data obtained from radar sensors and image data obtained from camera sensors, including millimeter-wave radar coordinates, 3D world coordinates, camera coordinates, image coordinates, and pixel coordinates, fusing data from both temporal and spatial perspectives. Then, a coordinate system incorporating map information and a model of the ship's motion is established. Combined with the processed environmental information, the position coordinates, speed, and direction of motion of obstacle vessels within the intelligent ship's perception range are determined.

[0072] S2: Construct a hazard membership function based on the target intelligent ship's state information and the obstacle's state information, and use the hazard membership function to evaluate the collision hazard between the target intelligent ship and the obstacle within the perception range.

[0073] Based on the acquired environmental information, the collision risk of other obstacles to the intelligent ship at the current moment is evaluated and analyzed. A risk membership function is constructed, using the distance between the ship and the obstacle (which could be an approaching ship) as the main parameters, the relative speed ratio, relative hull angle, nearest encounter distance, and nearest encounter time as the primary considerations to assess the ship's collision risk. A risk threshold is determined based on the intelligent ship's hull structure, deceleration performance, and steering performance. If the calculated collision risk exceeds the risk threshold, subsequent procedures are executed; otherwise, the ship continues on its original course.

[0074] Assume there are N obstruction ships within the detectable range around this ship, where the relevant data for the i-th obstruction ship is: the speed ratio K between this ship and the i-th obstruction ship. i The relative hull angle θ between this ship and the i-th obstacle ship i Recently, we will encounter a distance from DCPA. i， Recently, it will be close to TCPA. i The corresponding membership functions are U Ki U θi U Di U DCPAi U TCPAi All belong to [0,1], where i = 0, 1, 2, ..., N. Their corresponding weights are: the membership function weights a (velocity ratio). K The relative hull angle membership function weight a θ The weights of the membership function for the distance between the two ships, a D The nearest occurrence is the distance membership function weight a. DCPA Recently, it will be related to the time membership function weight a. TCPA , where a K +a θ +aD +a DCPA +a TCPA =1. Considering the magnitude of the influence of each factor on the model, set a. K =0.1, a θ =0.1, a D =0.5, a DCPA =0.15, a TCPA =0.15.

[0075] Risk level (CR) of the obstacle vessel relative to the main vessel i It can be calculated using formula (1):

[0076] CR i =a K U Ki +a θ U θi +a D U D +a DCPA U DCPAi +a TCPA U TCPAi

[0077] Specifically, step S2 includes:

[0078] S21: Calculate the distance between the target smart ship and the obstacle, the relative speed ratio, the relative hull angle, the nearest encounter distance, and the nearest encounter time.

[0079] S22: Calculate the following sub-membership functions based on the distance, relative speed ratio, relative hull angle, nearest encounter distance, and nearest encounter time: distance hazard sub-membership function, relative speed ratio hazard sub-membership function, relative hull angle hazard sub-membership function, nearest encounter distance hazard sub-membership function, and nearest encounter time hazard sub-membership function.

[0080] Dangerous membership function for distance between two ships:

[0081] When judging the danger posed by an approaching vessel to a ship from a distance perspective, the smaller the distance between the two vessels, the greater the danger posed by the obstructing vessel. Therefore, the danger membership function U of the distance between the two vessels is... D for

[0082]

[0083] In the formula, D is the distance between the two ships, D1 is the latest avoidance distance, and D2 is the distance at which avoidance measures can be taken.

[0084] Relative speed ratio danger membership function:

[0085] The relative speed hazard membership function between the vessel and the obstacle vessel should increase with the increase of the relative speed ratio, and the collision hazard level posed by the same relative speed ratio will also differ depending on the encounter angle. Therefore, the relative speed ratio hazard function U K for

[0086]

[0087] In the formula, K≥0, a is the meeting angle and a∈(0°,180°).

[0088] Relative angle danger membership function:

[0089] The collision risk varies depending on the angle at which an obstructing vessel approaches the vessel. Based on the collision damage inflicted on the vessel at different angles, at the same speed, the damage assessment is 0.2924 for a collision occurring at the bow section; 0.1832 for a collision occurring at the bow section; and 0.0731 for a collision occurring at the bow section. Therefore, for ease of calculation, in the encounter situation, the most dangerous situation is when the relative hull angle of the obstructing vessel is (0°, 5°)U(355°, 360°), and the danger membership function value at this point is taken as 0.8; in the crossing situation, the relatively dangerous situation is when the relative hull angle of the obstructing vessel is (5°, 112.5°)U(247.5°, 355°), and the danger membership function value at this point is taken as 0.5; in the overtaking situation, the safest situation is when the relative hull angle of the obstructing vessel is (112.5°, 247.5°), and the danger membership function value at this point is taken as 0.2. Thus, the danger membership function U of the relative hull angle of the obstructing vessel is obtained. θ for:

[0090]

[0091] In the formula, θ is the hull angle of the vessel relative to the obstacle vessel, and θ∈(0°, 360°).

[0092] Membership function for nearest proximity danger:

[0093] DCPA hazard membership function U DCPA for

[0094]

[0095] In the formula, DCPA is the nearest encounter distance at the current moment, d1 represents the minimum safe distance at which the approaching vessel can safely encounter the vessel, and d2 is the absolute safe encounter distance between the two vessels.

[0096] The most recent association with the time-risk membership function:

[0097] The danger membership function of TCPA is U. TCPA for

[0098]

[0099] in

[0100]

[0101]

[0102] In the formula, D1 represents the latest avoidance distance, D2 represents the distance at which avoidance measures can be taken, and V R This indicates the relative speed between the obstacle vessels.

[0103] S23: Determine the final hazard membership function based on the distance hazard sub-membership function, the relative speed ratio hazard sub-membership function, the relative hull angle hazard sub-membership function, the nearest encounter distance hazard sub-membership function, and the nearest encounter time hazard sub-membership function.

[0104] S3: When the collision risk level is less than or equal to the risk level threshold, control the target smart ship to travel along the original route.

[0105] S4: When the collision risk level is greater than the risk level threshold, an improved PPO algorithm is applied based on the current target smart ship state information and the current obstacle state information to obtain the optimal obstacle avoidance strategy for the target obstacle; the target obstacle is an obstacle whose collision risk level is greater than the risk level threshold; the improved PPO algorithm introduces the angle component, distance component, and velocity component from the DWA algorithm evaluation function into the reward function of the traditional PPO algorithm; the target smart ship is controlled to move based on the optimal obstacle avoidance strategy.

[0106] The action space of the PPO algorithm is defined based on the description of ship motion using linear velocity and angular velocity in the DWA algorithm. Then, based on this, a ship safety domain model is constructed and a state space containing information about the ship and the nearest obstacle is determined by combining the maximum detection range of the intelligent ship. Finally, the reward function of the PPO algorithm is improved by combining the evaluation function of the DWA algorithm related to the distance to the obstacle, ship speed, and deviation between the heading angle and the azimuth angle of the target.

[0107] By combining the dynamic window characteristics and evaluation function of the DWA algorithm, the action state space and reward function of the PPO algorithm are improved. The improved algorithm is used to construct a collision avoidance decision system. Real-time data is input into the decision system to obtain the optimal collision avoidance strategy under the current state. The control strategy is transmitted to the intelligent ship motion control system, which then controls the ship to plan the route.

[0108] Based on the dynamic window sampling method for velocity in the DWA algorithm, an action space for the fusion algorithm was designed, including angular velocity ω and linear velocity v, which describe the ship's motion direction and velocity. Considering ship kinematics and practical requirements, the ranges of angular velocity ω and linear velocity v were normalized. Specifically, angular velocity ω ∈ [-1, 1], and linear velocity v ∈ [0, 1]. The action space a is defined as: a = (w, v).

[0109] Among them, such as Figure 2 As shown, step S4 specifically includes:

[0110] S41: Based on the intelligent ship's navigation status information (the target intelligent ship's status information and the obstacle's status information), the DWA algorithm is used to determine the predicted motion trajectory of the target intelligent ship, and the evaluation function of each predicted motion trajectory is derived.

[0111] The state space represents the intelligent agent ( Figure 2 The environmental information perceived by the ship motion model (as described in the text) is the basis for the agent to make decisions and evaluate its long-term benefits. A reasonable state space design can ensure the convergence of the PPO algorithm and improve its performance. In this study, the state space mainly consists of obstacle states S. env And the ship's motion state S os It consists of two parts.

[0112] Obstacle status information S env This mainly includes information such as the status of obstacles and approaching ships. Among these, the distances from the ship to surrounding vessels and obstacles are the most intuitive and important indicators reflecting the current environmental information. In practice, environmental information during ship navigation is obtained through radar and sensor detection; therefore, the design should include... Figure 3 The model shown is for the field of ship safety. Where d... sensor To represent the maximum detection range of the sensing system, the black area at the center of the region indicates the ship's position, while other black areas represent the positions of obstacles. By constructing a ship safety domain, the positional information of multiple obstacles and approaching ships within the ship's domain can be accurately obtained.

[0113] Before training, the algorithm needs to determine the nodes of the neural network's input and output layers, and the number of nodes in the input layer cannot be changed during training. However, the number of obstacles and approaching vessels varies dynamically across different areas of ship safety. Given the numerous obstacles in the complex nearshore environment, increasing the initial number of input layer nodes to reflect information on all obstacles and approaching vessels would make the entire network excessively large and complex, significantly increasing the training time and reducing convergence. Therefore, only the information of the nearest and most dangerous obstacle or approaching vessel is considered. Thus, the obstacle state information S... env The definition is shown in the following formula.

[0114] Senv =[x T ,y T ,v T ,θ T ,d T ,Δθ T ]

[0115] Where (x) T y T ), v T θ T These represent the position, velocity, and direction of motion of the nearest obstacle in the two-dimensional plane, respectively, d T Let Δθ be the distance from the ship to the obstacle. T It is the angle difference between the direction of the obstacle's movement and the bow direction of the ship.

[0116] Ship motion status information S os This mainly includes information such as the ship's position, speed, heading, and distance and bearing between the ship and the target point. Ship motion status information S os The definition is shown in the following formula.

[0117] S os =[x o ,y o ,v o ,w o ,θ o ,d goal ,θ goal ]

[0118] Where (x) o y o ), v o w o θ0 represents the position, linear velocity, angular velocity, and direction of motion of the ship in the two-dimensional plane, respectively, and d goal θ is the distance between the ship and the destination. goal The destination and the ship's bearing.

[0119] S42: Determine the reward function in the improved PPO algorithm based on the evaluation function. Specifically, this includes:

[0120] The normal action reward function in the improved PPO algorithm is determined based on the angle component, distance component, and velocity component in the evaluation function; the reward function includes the normal action reward function, the endpoint reward function, and the collision reward function.

[0121] Traditional PPO algorithms, when used for obstacle avoidance path planning, only include a positive reward for reaching the destination, a negative reward for colliding with an obstacle, and a negative reward for taking a step in their reward function. The agent receives no positive feedback for actions taken before reaching the destination. In complex environments, the agent may fail to learn due to the sparse reward function. Therefore, this invention focuses on the control objective of dynamic obstacle avoidance for intelligent ships. Combining the evaluation function from the DWA algorithm, it designs a suitable reward function to effectively assess the current state of the intelligent ship, guiding the agent to make correct collision avoidance decisions, ensuring the algorithm's convergence, and improving its performance.

[0122] The reward the agent receives at each time step is divided into three parts: normal action reward R o Finish line bonus R goal Collision Reward R col Reward function R:

[0123]

[0124] Normal action reward R o The reward function is defined as the reward obtained by the intelligent ship during one time step of movement without collision or reaching the destination. To address the sparsity problem of rewards in fusion algorithms, the reward function is improved based on the evaluation function in the DWA algorithm. The improved reward function for normal action is defined as follows.

[0125]

[0126] Among them, R head Corresponding to the angle component Heading(v,w) in the DWA evaluation function, this reward value is inversely proportional to the angle difference between the intelligent ship's motion direction and the destination direction; the larger the angle difference, the lower the reward value. head The definition is shown in the following formula.

[0127]

[0128] R dis The distance component Distance(v, w) in the DWA evaluation function is proportional to the distance between the smart ship and the nearest obstacle; the farther the ship is from the obstacle, the higher the reward. dis The definition is shown in the following formula.

[0129]

[0130] Where, d obs This indicates the distance between the obstacle and the target smart ship.

[0131] R velCorresponding to the velocity component (Velocity(v, w)) in the evaluation function of DWA, this reward value is proportional to the speed of the intelligent ship; the faster the speed, the higher the reward value. vel The definition is shown in the following formula.

[0132]

[0133] Where v is the current speed of the target smart ship, v max That is the maximum speed that the target intelligent ship can achieve.

[0134] Finish line reward R goal The reward is defined as the distance between the smart ship's current position and the destination. When the distance between the smart ship and the destination is less than d... sensor When the value is 2, the finish line reward is 100; otherwise, the finish line reward is 0. goal The definition is shown in the following formula.

[0135]

[0136] Collision Bonus R col Defined as a reward given based on the distance between the smart ship's current position and the nearest obstacle or oncoming ship. When the distance between the smart ship and the nearest obstacle or oncoming ship is less than d... sensor When the value is 2, the finish line reward is -100; otherwise, the finish line reward is 0. col The definition is shown in the following formula.

[0137]

[0138] S43: Based on the current target smart ship state information and the current obstacle state information, combined with the improved PPO algorithm, the optimal obstacle avoidance strategy for the target obstacle is obtained.

[0139] An improved algorithm (DWA-PPO) is used to construct an intelligent ship collision avoidance decision system. By inputting perceived environmental information and the ship's motion state information, the optimal collision avoidance speed combination (v, w) is obtained at the current moment.

[0140] In this invention, an improved PPO algorithm is applied for obstacle avoidance path planning of intelligent ships. When applying the improved PPO algorithm in practice, the neural network within the PPO algorithm needs to be trained. Through the training process, the network parameters reach relatively optimal values, which improves the accuracy of path planning using the PPO algorithm. Therefore, in actual path planning, the Actor network and Critic network in the PPO algorithm need to be trained in advance, such as... Figure 2 and 4 As shown:

[0141] (1.1) First, the Actor-new network interacts with the environment, inputting environmental information s into the Actor-new network to obtain the mean μ and variance σ (meaning the distribution of actions) describing the normal distribution. Then, a combination of actions (v, w) is randomly sampled from this normal distribution and input into the environment to obtain the reward r and the next state s. _ Then store {s, a, r, s} _ Then input s_ into the actor-new network and repeat step (1.1) until a certain amount of {s, a, r, s_} is stored.

[0142] (1.2) Input the environmental information s_ obtained from the last step of step (1.1) into the critic-NN network to obtain the estimated state value V(s_), i.e., V_. Then calculate the discounted reward: R(t)=r(t)+γ*r(t+1)+γ 2 *r(t+2)+…+γ T-t-1 *r(T-1)+γ T-t *V_ yields the discount reward sequence R = [R(1), ..., R(T)], where T is the last time step. t is defined as a time step variable, and t can take values ​​of 1, 2, 3, ..., T.

[0143] (1.3) The environmental information s from the experience pool is combined and input into the critic-NN network to calculate the evaluation state value of all states, and then the advantage value is calculated using the following formula.

[0144]

[0145] δ t =r t +γV(s t+1 )-V(s t )

[0146] Where γ represents the discount reward coefficient, and λ is just a coefficient, both belonging to [0,1].

[0147] (1.4) Then, the loss function c_loss of the critic-NN network is calculated using the following formula, and then the parameters of the critic-NN network are updated by backpropagation.

[0148]

[0149] V(s t ) indicates that the state s t The output value after being input into the critic network represents the state s. t The assessment.

[0150] (1.5) Input all the s combinations stored in the experience pool into the actor-old and actor-new networks to obtain the expected variance parameters μ1, σ1 and μ2, σ2 of the corresponding normal distributions, respectively. Then construct two normal distributions Normal1 and Normal2. Then input all the stored actions into action sequences into the normal distributions Normal1 and Normal2 to obtain the probability distributions corresponding to each action, namely prob1 and prob2. Then divide prob2 by prob1 to obtain the ratio of the old and new policies.

[0151] As training progresses, the policy entropy between the two policies increases. Therefore, it's necessary to determine the update step size. If the update step size is inappropriate, the policy reward function corresponding to the updated parameters will decrease, leading to progressively worse learning and ultimately preventing the algorithm from converging. The range of change g between the action output probabilities of the old and new policy networks can be expressed by the following formula. t If (θ) is restricted to a certain area, this problem can be solved.

[0152]

[0153] Here, θ represents the parameters of the policy network π.

[0154] (1.6) Calculate the loss function a_loss of the actor-new-NN network according to the following formula, and then backpropagate to update the parameters of the actor-new-NN network.

[0155]

[0156] Where ε is a hyperparameter, typically set to 0.2; For the dominance function, when When this action is better than the average action, the probability of choosing this action is increased; when If the probability of this action is lower than the average action, then the probability of selecting this action should be reduced. However, the probability distribution of actions obtained by the network cannot deviate too much. Therefore, the network truncates the probability distribution at 1+ε and 1-ε to limit the magnitude of policy updates. The objective function is limited to the following range: Figure 5 As shown, Figure 5 (a) and Figure 5 (b) Explains these two situations separately. Figure 5 L CLIP (θ) represents the objective function of the PPO algorithm, i.e., the loss function.

[0157] (1.7), repeat steps (1.5) and (1.6)T. updateAfter this, the loop ends, and the weight parameters of the actor-new-NN network are updated to those of the actor-old-NN network.

[0158] (1.8), repeat steps (1.1)-(1.7) until the maximum number of training iterations T is reached. max .

[0159] In summary, before obtaining the optimal obstacle avoidance strategy for the target obstacle based on the current target smart ship state information and the current obstacle state information combined with the improved PPO algorithm, the following steps are also included:

[0160] (1) Obtain the training dataset of the PPO algorithm, wherein the training dataset includes multiple training subsets; each training subset includes environmental state sample information at time step t, action information of the target intelligent ship at time step t, reward value at time step t, and environmental state sample information at time step t+1; the environmental state sample information at time step t includes target intelligent ship state sample data at time step t and obstacle state sample data at time step t.

[0161] Specifically, the training dataset for the PPO algorithm includes:

[0162] (i) Obtain environmental status sample information.

[0163] (ii) Input the current environmental state sample information into the new Actor network of the improved PPO algorithm to obtain the current output result of the new Actor network, and determine the current action information of the target smart ship based on the current output result of the new Actor network.

[0164] (iii) Based on the current action information of the target intelligent ship and the reward function, determine the reward value corresponding to the current state sample data of the target intelligent ship and the environmental state sample information of the next time step, and store the current environmental state sample information, the current action information of the target intelligent ship, the current reward value and the environmental state sample information of the next time step in a training subset.

[0165] (iv) Use the environmental state sample information of the next time step as the current environmental state sample information, and return to step (ii) "input the current environmental state sample information into the new Actor network of the improved PPO algorithm"; until the amount of data in the training dataset reaches a preset number.

[0166] (2) Input the environmental state sample information of the last time step in the training dataset into the Critic network of the improved PPO algorithm to obtain the state estimate of the last time step.

[0167] (3) Calculate the discount reward based on the state estimate of the last time step and all the reward values.

[0168] (4) Input all the environmental state sample information in the training dataset into the Critic network to obtain all state estimates; and determine the Critic network advantage function based on all the state estimates and the reward value.

[0169] (5) Update the parameters of the Critic network according to its loss function; the loss function of the Critic network is based on the discounted reward and all the state estimates V. φ (s t It is certain.

[0170] (6) Input all the environmental state sample information in the training dataset into the new Actor network and the old Actor network of the improved PPO algorithm respectively, and determine the ratio of the new and old strategies based on the output of the new Actor network and the output of the old Actor network.

[0171] (7) Determine the Actor network loss function based on the ratio of the new and old strategies and the advantage function, and update the parameters of the new Actor network according to the Actor network loss function.

[0172] (8) Return to step (6) "Input all the environmental state sample information in the training dataset into the new Actor network and the old Actor network of the improved PPO algorithm respectively" until the number of updates of the new Actor network reaches the first preset value.

[0173] (9) Assign the parameters of the new Actor network obtained in the last update iteration to the old Actor network. The principle of this algorithm is to obtain the parameters from the old policy π through a significance sampling method. θold (a t s t Samples are obtained from [the database] to improve sample utilization. Simultaneously, new strategies are periodically [implemented / implemented]. θ (a t s t The parameters of π are updated to the old policy. θold (a|s)

[0174] (10) Determine whether the current number of training iterations has reached the second preset value; if so, complete the training of the new Actor network, the old Actor network and the Critic network in the improved PPO algorithm.

[0175] (11) If not, return to step (1) "obtain the training dataset of the PPO algorithm" until the number of training iterations reaches the second preset value.

[0176] This invention integrates deep reinforcement learning (PPO) and dynamic window algorithm (DWA). By combining the evaluation function of the DWA algorithm related to the distance to obstacles, ship speed, and deviation between the heading angle and the target's azimuth angle, the reward function of the PPO algorithm is improved. This solves the problems of insufficient foresight in the traditional DWA algorithm and the sparse reward in the traditional PPO algorithm, while improving the convergence speed of the algorithm. The improved algorithm can adapt to environmental changes, increase the distance between the ship and obstacles during collision avoidance, and effectively enhance the collision avoidance capability of intelligent ships in specific water environments.

[0177] For obstacle avoidance path planning tasks of intelligent ships navigating in near-shore waters, the improved algorithm combines the excellent performance of the PPO algorithm for continuous control problems with the advantages of the DWA algorithm, such as low computational complexity, high efficiency, and good real-time obstacle avoidance performance. This allows the improved algorithm to adapt to environmental changes, increase the distance between the ship and obstacles during collision avoidance, effectively enhance the collision avoidance capability of intelligent ships in specific water environments, and improve the overall safety and efficiency of intelligent ships in water surface path planning.

[0178] Example 2

[0179] This embodiment provides an obstacle avoidance path planning system for an intelligent ship, the system comprising:

[0180] The data acquisition module is used to acquire obstacle status information and target smart ship status information within the target smart ship's perception range.

[0181] The membership function construction module is used to construct a hazard membership function based on the target intelligent ship's state information and the obstacle's state information, and to use the hazard membership function to evaluate the collision hazard between the target intelligent ship and obstacles within the perception range.

[0182] Specifically, the membership function construction module includes:

[0183] The parameter calculation submodule is used to calculate the distance between the target smart ship and the obstacle, the relative speed ratio, the relative hull angle, the nearest encounter distance, and the nearest encounter time.

[0184] The membership sub-function construction sub-module is used to calculate the distance hazard sub-membership function, relative speed ratio hazard sub-membership function, relative hull angle hazard sub-membership function, nearest encounter distance hazard sub-membership function, and nearest encounter time hazard sub-membership function based on the distance, relative speed ratio, relative hull angle, nearest encounter distance, and nearest encounter time, respectively.

[0185] A submodule for constructing a comprehensive membership function is used to determine the final hazard membership function based on the distance hazard sub-membership function, the relative speed ratio hazard sub-membership function, the relative hull angle hazard sub-membership function, the nearest encounter distance hazard sub-membership function, and the nearest encounter time hazard sub-membership function.

[0186] The first path planning module is used to control the target smart ship to travel along the original route when the collision risk is less than or equal to the risk threshold.

[0187] The second path planning module is used to apply an improved PPO algorithm to obtain the optimal obstacle avoidance strategy for the target obstacle when the collision risk is greater than the risk threshold. The target obstacle is an obstacle whose collision risk is greater than the risk threshold. The improved PPO algorithm introduces the angle component, distance component, and velocity component from the DWA algorithm evaluation function into the reward function of the traditional PPO algorithm. The module controls the movement of the target intelligent ship based on the optimal obstacle avoidance strategy.

[0188] The second path planning module specifically includes:

[0189] The evaluation function determination submodule is used to determine the predicted motion trajectory of the target intelligent ship based on the target intelligent ship state information and the obstacle state information, using the DWA algorithm, and to derive the evaluation function for each predicted motion trajectory.

[0190] The reward function improvement submodule is used to determine the reward function in the improved PPO algorithm based on the evaluation function.

[0191] Specifically, the reward function improvement submodule includes:

[0192] The normal action reward function in the improved PPO algorithm is determined based on the angle component, distance component, and velocity component in the evaluation function; the reward function includes the normal action reward function, the endpoint reward function, and the collision reward function.

[0193] The path planning submodule is used to obtain the optimal obstacle avoidance strategy for the target obstacle based on the current state information of the target smart ship and the current state information of the obstacle, combined with the improved PPO algorithm.

[0194] The same or similar parts between the various embodiments in this specification can be referred to mutually. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0195] This document uses specific examples to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. Furthermore, those skilled in the art will recognize that, based on the ideas of the present invention, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for obstacle avoidance path planning of an intelligent ship, characterized in that, The method comprises: obtaining the target intelligent ship state information and the obstacle state information within the perception range of the target intelligent ship; constructing a danger degree membership function based on the target intelligent ship state information and the obstacle state information, and evaluating the collision danger degree of the target intelligent ship and the obstacles within the perception range by using the danger degree membership function; when the collision danger degree is less than or equal to the danger degree threshold, controlling the target intelligent ship to move according to the original route; when the collision danger degree is greater than the danger degree threshold, applying an improved PPO algorithm according to the current target intelligent ship state information and the current obstacle state information to obtain the optimal obstacle avoidance strategy of the target obstacle; the target obstacle is the obstacle whose collision danger degree is greater than the danger degree threshold; the improved PPO algorithm introduces the angle component, distance component and velocity component in the DWA algorithm evaluation function into the reward function of the traditional PPO algorithm; controlling the target intelligent ship to move based on the optimal obstacle avoidance strategy; wherein the construction of the danger degree membership function based on the target intelligent ship state information and the obstacle state information specifically comprises: calculating the distance between the target intelligent ship and the obstacle, the relative speed ratio, the relative side angle, the closest approach distance and the closest approach time; calculating the distance danger degree sub-membership function, the relative speed ratio danger degree sub-membership function, the relative side angle danger degree sub-membership function, the closest approach distance danger degree sub-membership function and the closest approach time danger degree sub-membership function based on the distance, the relative speed ratio, the relative side angle, the closest approach distance and the closest approach time; determining the final danger degree membership function based on the distance danger degree sub-membership function, the relative speed ratio danger degree sub-membership function, the relative side angle danger degree sub-membership function, the closest approach distance danger degree sub-membership function and the closest approach time danger degree sub-membership function; wherein the application of the improved PPO algorithm according to the current target intelligent ship state information and the current obstacle state information to obtain the optimal obstacle avoidance strategy of the target obstacle specifically comprises: determining the predicted motion trajectory of the target intelligent ship by using the DWA algorithm according to the target intelligent ship state information and the obstacle state information, and obtaining the evaluation function of each predicted motion trajectory; determining the reward function in the improved PPO algorithm according to the evaluation function; obtaining the optimal obstacle avoidance strategy of the target obstacle by combining the improved PPO algorithm according to the current target intelligent ship state information and the current obstacle state information; wherein the improvement of the reward function in the improved PPO algorithm according to the evaluation function specifically comprises: determining the normal action reward function of the reward function in the improved PPO algorithm according to the angle component, distance component and velocity component in the evaluation function; the reward function comprises the normal action reward function, the end reward function and the collision reward function.

2. The method of claim 1, wherein, Before obtaining the optimal obstacle avoidance strategy of the target obstacle according to the current target intelligent ship state information and the current obstacle state information in combination with the improved PPO algorithm, the method further comprises the following steps: obtaining a training data set of the PPO algorithm, wherein the training data set comprises a plurality of training subsets; each training subset comprises t time step environment state sample information, t time step target intelligent ship action information, t time step reward value and t+1 time step environment state sample information; the t time step environment state sample information comprises t time step target intelligent ship state sample data and t time step obstacle state sample data; inputting the last time step environment state sample information in the training data set into a Critic network of the improved PPO algorithm to obtain a last time step state estimation value; calculating a discounted reward according to the last time step state estimation value and all the reward values; inputting all the environment state sample information in the training data set into the Critic network to obtain all state estimation values; and determining a Critic network advantage function according to all the state estimation values and the reward values; updating parameters of the Critic network according to a Critic network loss function; the Critic network loss function is determined according to the discounted reward and all the state estimation values; inputting all the environment state sample information in the training data set into a new Actor network and an old Actor network of the improved PPO algorithm respectively, and determining a new-old policy ratio according to a new Actor network output and an old Actor network output; determining an Actor network loss function according to the new-old policy ratio and the advantage function, and updating parameters of the new Actor network according to the Actor network loss function; returning to the step of inputting all the environment state sample information in the training data set into the new Actor network and the old Actor network of the improved PPO algorithm until the number of update iterations of the new Actor network reaches a first preset value; assigning the parameters of the new Actor network obtained at the last update iteration to the old Actor network; determining whether the number of current training iterations reaches a second preset value; if yes, completing training of the new Actor network, the old Actor network and the Critic network in the improved PPO algorithm; if no, returning to the step of obtaining the training data set of the PPO algorithm until the number of training iterations reaches the second preset value.

3. The method of claim 2, wherein, The step of obtaining the training data set of the PPO algorithm specifically comprises the following steps: obtaining environment state sample information; inputting the current environment state sample information into a new Actor network of the improved PPO algorithm to obtain a current new Actor network output result, and determining current target intelligent ship action information according to the current new Actor network output result; determine a reward value corresponding to the target intelligent ship state sample data and next time step environment state sample information according to the action information of the target intelligent ship and the reward function, and store the current environment state sample information, the action information of the target intelligent ship, the reward value and the next time step environment state sample information in a training subset; return to the step of inputting the current environment state sample information into the new Actor network of the improved PPO algorithm, taking the next time step environment state sample information as the current environment state sample information, until the amount of data in the training data set reaches a preset amount.

4. An obstacle avoidance path planning system for an intelligent ship, characterized by, The system comprises: a data acquisition module configured to acquire obstacle state information within a perception range of a target intelligent ship and target intelligent ship state information; a membership function construction module configured to construct a danger degree membership function based on the target intelligent ship state information and the obstacle state information, and evaluate a collision danger degree of the target intelligent ship and obstacles within the perception range by using the danger degree membership function; The membership function construction module specifically comprises: a parameter calculation submodule configured to calculate a distance between the target intelligent ship and the obstacles, a relative speed ratio, a relative side angle, a closest encounter distance and a closest encounter time; a membership function construction submodule configured to calculate a distance danger degree sub-membership function, a relative speed ratio danger degree sub-membership function, a relative side angle danger degree sub-membership function, a closest encounter distance danger degree sub-membership function and a closest encounter time danger degree sub-membership function based on the distance, the relative speed ratio, the relative side angle, the closest encounter distance and the closest encounter time, respectively; a comprehensive membership function construction submodule configured to determine a final danger degree membership function based on the distance danger degree sub-membership function, the relative speed ratio danger degree sub-membership function, the relative side angle danger degree sub-membership function, the closest encounter distance danger degree sub-membership function and the closest encounter time danger degree sub-membership function; a first path planning module configured to control the target intelligent ship to move according to an original route when the collision danger degree is less than or equal to a danger degree threshold value; a second path planning module configured to apply an improved PPO algorithm to obtain an optimal obstacle avoidance strategy of a target obstacle according to the current target intelligent ship state information and the current obstacle state information when the collision danger degree is greater than the danger degree threshold value; the target obstacle is an obstacle whose collision danger degree is greater than the danger degree threshold value; the improved PPO algorithm introduces an angle component, a distance component and a speed component in a DWA algorithm evaluation function into a reward function of a traditional PPO algorithm; and the target intelligent ship is controlled to move based on the optimal obstacle avoidance strategy; The second path planning module specifically comprises: The evaluation function determining submodule is configured to determine the predicted motion trajectory of the target intelligent ship by using the DWA algorithm according to the target intelligent ship state information and the obstacle state information, and derive the evaluation function of each predicted motion trajectory. The reward function improving submodule is configured to determine the reward function in the improved PPO algorithm according to the evaluation function. The path planning submodule is configured to derive the optimal obstacle avoidance strategy of the target obstacle according to the current target intelligent ship state information and the current obstacle state information in combination with the improved PPO algorithm. The reward function improving submodule specifically includes: The normal action reward function of the reward function in the improved PPO algorithm is determined according to the angle component, the distance component and the speed component in the evaluation function; and the reward function includes the normal action reward function, the terminal reward function and the collision reward function.

Images