Unmanned ship path optimization method based on reinforcement learning reward and punishment double gradient factors

Abstract

The invention relates to an unmanned ship path optimization method based on reward and punishment double gradient factors of reinforcement learning, and the method comprises the following steps: obtaining a rudder angle # imgabs0 # randomly selected in a set rudder angle motion space ACTIONS, and carrying out the conversion between the rudder angle # imgabs1 # and a course angle # imgabs2 # through a first-order nomoto model; obtaining the linear velocity v at the moment, calculating and updating new horizontal and vertical coordinates x and y after the action is adopted based on the linear velocity v at the moment and the course angle # imgabs3 #, and updating a new state s = (x, y, # imgabs4 #); updating the minimum distance r between the ship intelligent body and the obstacle based on the new state s = (x, y, # imgabs5 #); based on the new state s = (x, y, # imgabs6 #) and the minimum distance r, a reward feedback value reward at the moment is calculated through a pre-constructed unmanned ship reward function; judging whether an obstacle is encountered or not based on the reward feedback value reward; and judging whether to arrive at the destination or not based on the reward feedback value reward. A reward function is optimized through the reward and punishment double-gradient factors, and the path of the unmanned ship intelligent body reaching the destination is optimized through the double factors of the reward gradient and the punishment gradient.

Description

Technical Field

[0001] The present invention relates to the technical field related to reinforcement learning, and in particular to an unmanned ship path optimization method based on reinforcement learning with a dual gradient factor of reward and punishment. Background Art

[0002] The application of reinforcement learning in unmanned ship path planning usually involves building a Markov decision process (MDP), which can describe the interactive process of the unmanned ship's path following control. In this process, the state space defines the current state of the unmanned ship, the action space defines the actions that the unmanned ship can take, and the reward function is used to evaluate the pros and cons of each action. Through continuous trial and error and learning, the unmanned ship can learn how to choose the best action in different states to achieve the predetermined goal.

[0003] The following situations are inevitable for the native reinforcement learning-based autonomous driving of unmanned ships: 1. Since the reward value obtained from interacting with the environment (when no obstacles are encountered) is a fixed value and has no specific gradient relationship with the destination, the ship agent cannot get real-time feedback on the destination. 2. Due to the setting of the native reward function, the ship agent only gets reward feedback after it encounters an obstacle, which makes the path planned by the ship agent too costly when facing a multi-obstacle water environment. Summary of the invention

[0004] The purpose of the present invention is to solve at least one of the deficiencies of the prior art and to provide an unmanned ship path optimization method based on reinforcement learning with a dual gradient factor of rewards and penalties.

[0005] In order to achieve the above object, the present invention adopts the following technical solutions: Specifically, an unmanned ship path optimization method based on reinforcement learning and reward and punishment dual gradient factors is proposed, including the following: Step S1, initializing the state value of the ship intelligent body, wherein the state value includes the horizontal coordinate, vertical coordinate and heading angle of the ship intelligent body on the water surface; Step S2, importing the target scene map, setting a rasterized map based on it, and marking the starting position and the destination; Step S3: Get the randomly selected rudder angle in the set rudder angle action space ACTIONS , the rudder angle is calculated by the first-order nomoto model With heading angle conversion between Step S4: Obtain the linear velocity v at this time, based on the linear velocity v and the heading angle Calculate and update the new horizontal and vertical coordinates x and y after taking action, and update the new state s=(x, y, ); Step S5: Based on the new state s=(x, y, ) Update the minimum distance r between the ship agent and the obstacle; Step S6: Based on the new state s=(x, y, ) and the minimum distance r are calculated through the pre-constructed unmanned ship reward function to obtain the reward feedback value reward at this time; Step S7, judging whether an obstacle is encountered based on the reward feedback value reward, if yes, returning to step S1 to continue execution, if no, turning to step S8; Step S8: determine whether the destination has been reached based on the reward feedback value reward, if yes, end the training, if no, return to step S3 to continue execution.

[0006] Further, specifically, the state value of the ship intelligent body is initialized, where the horizontal and vertical coordinates of the ship intelligent body on the water surface are the starting position coordinates, and the heading angle =0.

[0007] Further, specifically, the rudder angle action space ACTIONS = [-35°, -15°, 0°, 15°, 35°]; With heading angle The conversion formula is:

[0008] in, is the heading angle, is the rudder angle, K and T are the maneuverability index and following index respectively, K = 0.08, T = 10.8, and t is the time parameter at the current moment.

[0009] Further, specifically, based on the linear velocity v and the heading angle Calculate the new horizontal and vertical coordinates x and y after taking action action, including, Calculate by the following formula:

[0010] in, and It is the new horizontal and vertical coordinates after taking action. and are the horizontal and vertical coordinates of the starting time.

[0011] Further, the specific construction process of the pre-constructed unmanned ship reward function includes: Set the vertical gradient variable , Determined based on the real-time ordinate y of the ship agent and the longitudinal height H of the map; Based on the actual obstacle coordinates, set the gradient variable around the obstacle , Based on the real-time minimum distance r between the ship agent and the obstacle; According to the obtained reward and punishment dual gradient factor and the set gradient parameters Construct the unmanned ship reward function, where the reward and punishment dual gradient factor is .

[0012] Further, specifically, the longitudinal gradient variable The calculation formula is as follows:

[0013] Among them, H is the longitudinal height of the map, y is the real-time longitudinal coordinate of the ship agent, and the expression of the longitudinal height H of the map is as follows:

[0014] in, is the ordinate of the end point, is the longitudinal length of the end point, is the ordinate of the ship agent at the starting point, and L is the length of the ship agent.

[0015] Furthermore, considering that there are multiple obstacles in the actual environment, each obstacle corresponds to a gradient variable , Therefore, the gradient variable around the nth obstacle is constructed The decision conditions are as follows:

[0016] in, is the minimum distance between the center point of the nth obstacle and the ship agent; is a gradient hyperparameter, and its value is set manually according to the actual required gradient. The minimum distance between the center point of the nth obstacle and the ship agent The expression is as follows:

[0017] in, and are the horizontal and vertical coordinates of the center point of the nth obstacle, respectively; x and y are the horizontal and vertical coordinates of the real-time center point of the ship intelligent body, respectively.

[0018] Further, specifically, the constructed unmanned ship reward function is:

[0019] in, is the longitudinal gradient variable, is the sum of the reward and penalty values ​​of the gradient variables around n obstacles, and α is the weight coefficient.

[0020] Further, specifically, The value of is set to 10.

[0021] The beneficial effects of the present invention are: The present invention proposes an unmanned ship path optimization method based on reinforcement learning and reward and punishment dual gradient factors. The problem that the ship intelligent agent in the original technology cannot obtain gradient feedback of the distance to the destination in real time is solved. The present invention increases the longitudinal gradient factor, thereby realizing the path optimization of the three-layer gradient drive of the ship and improving the training efficiency of the trial-and-error learning of the intelligent agent.

[0022] Taking into account that the trial-and-error cost of the path planned by traditional ship intelligent agents when facing a multi-obstacle water environment is too high, the present invention proposes an obstacle gradient penalty factor, which enables the ship intelligent agent to reduce unnecessary collision trial-and-error in multi-obstacle situations, thereby achieving the optimal path to avoid obstacles, improving navigation safety and hull protection.

[0023] The reward function is optimized by the above-mentioned dual gradient factors of reward and punishment, and the path optimization of the unmanned ship intelligent body to reach the destination is driven by the dual factors of reward gradient and penalty gradient. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The above and other features of the present disclosure will become more apparent by describing in detail the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings of the present disclosure represent the same or similar elements. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure. For those skilled in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work. In the accompanying drawings: Figure 1 Shown is a flow chart of the unmanned ship path optimization method based on reinforcement learning and reward and punishment dual gradient factors of the present invention. DETAILED DESCRIPTION

[0025] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other without conflict. The same reference numerals used throughout the drawings indicate the same or similar parts.

[0026] Example 1, reference Figure 1The present invention proposes an unmanned ship path optimization method based on a double gradient factor of rewards and punishments based on reinforcement learning, including the following: Step S1, initializing the state value of the ship intelligent body, wherein the state value includes the horizontal coordinate, vertical coordinate and heading angle of the ship intelligent body on the water surface; Step S2, importing the target scene map, setting a rasterized map based on it, and marking the starting position and the destination; Step S3: Get the randomly selected rudder angle in the set rudder angle action space ACTIONS , the rudder angle is calculated by the first-order nomoto model With heading angle conversion between Step S4: Obtain the linear velocity v at this time, based on the linear velocity v and the heading angle Calculate and update the new horizontal and vertical coordinates x and y after taking action, and update the new state s=(x, y, ); Step S5: Based on the new state s=(x, y, ) Update the minimum distance r between the ship agent and the obstacle; Step S6: Based on the new state s=(x, y, ) and the minimum distance r are calculated through the pre-constructed unmanned ship reward function to obtain the reward feedback value reward at this time; Step S7, judging whether an obstacle is encountered based on the reward feedback value reward, if yes, returning to step S1 to continue execution, if no, turning to step S8; Step S8: determine whether the destination has been reached based on the reward feedback value reward, if yes, end the training, if no, return to step S3 to continue execution.

[0027] In this embodiment 1, the reward function is optimized by the above-mentioned reward and penalty dual gradient factors, and the path optimization of the unmanned ship intelligent body to reach the destination is driven by the dual factors of reward gradient and penalty gradient.

[0028] In a preferred embodiment of the present invention, specifically, the state value of the ship intelligent body is initialized, wherein the horizontal coordinate and the vertical coordinate of the ship intelligent body on the water surface are the starting position coordinates, and the heading angle =0.

[0029] In a preferred embodiment of the present invention, specifically, the rudder angle action space ACTIONS = [-35°, -15°, 0°, 15°, 35°]; With heading angle The conversion formula is:

[0030] in, is the heading angle, is the rudder angle, K and T are the maneuverability index and following index respectively, K = 0.08, T = 10.8, and t is the time parameter at the current moment.

[0031] In a preferred embodiment of the present invention, specifically, based on the linear velocity v and the heading angle Calculate the new horizontal and vertical coordinates x and y after taking action action, including, Calculate by the following formula:

[0032] in, and It is the new horizontal and vertical coordinates after taking action. and are the horizontal and vertical coordinates of the starting time.

[0033] In a preferred embodiment of the present invention, the specific construction process of the pre-constructed unmanned ship reward function includes: Set the vertical gradient variable , Determined based on the real-time ordinate y of the ship agent and the longitudinal height H of the map; Based on the actual obstacle coordinates, set the gradient variable around the obstacle , Based on the real-time minimum distance r between the ship agent and the obstacle; According to the obtained reward and punishment dual gradient factor and the set gradient parameters Construct the unmanned ship reward function, where the reward and punishment dual gradient factor is .

[0034] In a preferred embodiment of the present invention, specifically, the longitudinal gradient variable The calculation formula is as follows:

[0035] Among them, H is the longitudinal height of the map, y is the real-time longitudinal coordinate of the ship agent, and the expression of the longitudinal height H of the map is as follows:

[0036] in, is the ordinate of the end point, is the longitudinal length of the end point, is the ordinate of the ship agent at the starting point, and L is the length of the ship agent.

[0037] In a preferred embodiment of the present invention, specifically, considering that there are multiple obstacles in the actual environment, each obstacle corresponds to a gradient variable , Therefore, the gradient variable around the nth obstacle is constructed The decision conditions are as follows:

[0038] in, is the minimum distance between the center point of the nth obstacle and the ship agent; is a gradient hyperparameter, and its value is set manually according to the actual required gradient. When the minimum distance between the ship agent and the obstacle is less than or equal to a gradient value , a gradient penalty is obtained, and the penalty value depends on the minimum distance between the ship and the center point of the obstacle. and the gradient hyperparameters ; In other cases, the weight is 0, that is, no gradient penalty is obtained (or the penalty is 0), which means that it finally returns to the reward function In the calculation formula is 0.

[0039] The minimum distance between the center point of the nth obstacle and the ship agent The expression is as follows:

[0040] in, and are the horizontal and vertical coordinates of the center point of the nth obstacle, respectively; x and y are the horizontal and vertical coordinates of the real-time center point of the ship intelligent body, respectively.

[0041] In a preferred embodiment of the present invention, specifically, the constructed unmanned ship reward function is:

[0042] in, is the longitudinal gradient variable, is the sum of the reward and penalty values ​​of the gradient variables around n obstacles, and α is the weight coefficient.

[0043] As a preferred embodiment of the present invention, specifically, The value of is set to 10.

[0044] In a preferred embodiment of the present invention, the longitudinal gradient variable is given a weight of 10 and is summed with the gradient variable around the obstacle, which is consistent with the actual situation of obstacle avoidance in ship driving. If the value is set too large, the reward value will be too close to the final destination when the destination is almost reached, and the destination will be lost when the destination is almost reached. If the value is set too small, the rewards will be too sparse at the beginning, and the gradient relationship weight of the unmanned ship to the destination will be too small, making the unmanned ship's learning ability poor. After many experiments, the following results are obtained: When the values ​​are set to 5, 10, and 15 respectively, the learning ability of the unmanned ship is A value of 10 works best.

[0045] Although the description of the present invention has been quite detailed and specifically describes several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims in view of the prior art by reference to the appended claims, thereby effectively covering the intended scope of the present invention. In addition, the above description of the present invention is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that have not yet been foreseen may still represent equivalent changes to the present invention.

[0046] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, it should belong to the protection scope of the present invention. Within the protection scope of the present invention, its technical scheme and / or implementation method can have various modifications and changes.

Claims

1. An unmanned ship path optimization method based on a dual gradient factor of rewards and punishments based on reinforcement learning, characterized in that: Includes the following: Step S1, initializing the state value of the ship intelligent body, wherein the state value includes the horizontal coordinate, vertical coordinate and heading angle of the ship intelligent body on the water surface; Step S2, importing the target scene map, setting a rasterized map based on it, and marking the starting position and the destination; Step S3: Get the randomly selected rudder angle in the set rudder angle action space ACTIONS , the rudder angle is calculated by the first-order nomoto model With heading angle conversion between Step S4: Obtain the linear velocity v at this time, based on the linear velocity v and the heading angle Calculate and update the new horizontal and vertical coordinates x and y after taking action, and update the new state s=(x, y, ); Step S5: Based on the new state s=(x, y, ) Update the minimum distance r between the ship agent and the obstacle; Step S6: Based on the new state s=(x, y, ) and the minimum distance r are calculated through the pre-constructed unmanned ship reward function to obtain the reward feedback value reward at this time; Step S7, judging whether an obstacle is encountered based on the reward feedback value reward, if yes, returning to step S1 to continue execution, if no, turning to step S8; Step S8: determine whether the destination has been reached based on the reward feedback value reward, if yes, end the training, if no, return to step S3 to continue execution.

2. The unmanned ship path optimization method based on reinforcement learning and reward and punishment dual gradient factors according to claim 1 is characterized in that: Specifically, the state value of the ship agent is initialized, where the horizontal and vertical coordinates of the ship agent on the water surface are the starting position coordinates, and the heading angle =0.

3. The unmanned ship path optimization method based on reinforcement learning and reward and punishment dual gradient factors according to claim 1 is characterized in that: Specifically, the rudder angle action space ACTIONS = [-35°, -15°, 0°, 15°, 35°]; With heading angle The conversion formula is: ； in, is the heading angle, is the rudder angle, K and T are the maneuverability index and following index respectively, K = 0.08, T = 10.8, and t is the time parameter at the current moment.

4. The unmanned ship path optimization method based on reinforcement learning and reward and punishment dual gradient factors according to claim 1 is characterized in that: Specifically, based on the linear velocity v and the heading angle Calculate the new horizontal and vertical coordinates x and y after taking action action, include, Calculate by the following formula: ； in, and It is the new horizontal and vertical coordinates after taking action. and are the horizontal and vertical coordinates of the starting time.

5. The unmanned ship path optimization method based on reinforcement learning and reward and punishment dual gradient factors according to claim 1 is characterized in that: Specifically, the construction process of the pre-constructed unmanned ship reward function is as follows: include, Set the vertical gradient variable , Determined based on the real-time ordinate y of the ship agent and the longitudinal height H of the map; Based on the actual obstacle coordinates, set the gradient variable around the obstacle , Based on the real-time minimum distance r between the ship agent and the obstacle; According to the obtained reward and punishment dual gradient factor and the set gradient parameters Construct the unmanned ship reward function, where the reward and punishment dual gradient factor is .

6. The unmanned ship path optimization method based on reinforcement learning and reward and punishment dual gradient factors according to claim 5 is characterized in that: Specifically, the vertical gradient variable The calculation formula is as follows: ； Among them, H is the longitudinal height of the map, y is the real-time longitudinal coordinate of the ship agent, and the expression of the longitudinal height H of the map is as follows: ； in, is the ordinate of the end point, is the longitudinal length of the end point, is the ordinate of the ship agent at the starting point, and L is the length of the ship agent.

7. The unmanned ship path optimization method based on reinforcement learning and reward and punishment dual gradient factors according to claim 5 is characterized in that: Specifically, considering that there are multiple obstacles in the actual environment, each obstacle corresponds to a gradient variable , Therefore, the gradient variable around the nth obstacle is constructed The decision conditions are as follows: ； in, is the minimum distance between the center point of the nth obstacle and the ship agent; is a gradient hyperparameter, and its value is set manually according to the actual required gradient. The minimum distance between the center point of the nth obstacle and the ship agent The expression is as follows: ； in, and are the horizontal and vertical coordinates of the center point of the nth obstacle, respectively; x and y are the horizontal and vertical coordinates of the real-time center point of the ship intelligent body, respectively.

8. The unmanned ship path optimization method based on reinforcement learning and reward and punishment dual gradient factors according to claim 5 is characterized in that: Specifically, the constructed unmanned ship reward function is: ； in, is the longitudinal gradient variable, is the sum of the reward and penalty values ​​of the gradient variables around n obstacles, and α is the weight coefficient.

9. The unmanned ship path optimization method based on reinforcement learning and reward and punishment dual gradient factors according to claim 8 is characterized in that: Specifically, The value of is set to 10.

Images