Path Planning Method and Device Based on Reinforcement Learning of RRT* Path Intelligent Guidance for Unmanned Vessels

Abstract

The present invention relates to a path planning method and device for an unmanned ship based on reinforcement learning with intelligent guidance of an RRT path. By using an optimal path path (which may be a local optimum) planned by the RRT* algorithm as a reference for the reinforcement learning of the unmanned ship, and adding this path to the reinforcement learning of the unmanned ship, the horizontal distance Δx between the unmanned ship and path during exploration is calculated t squared, and the calculation result is used as the divisor of a constant N, and finally a reward value R is obtained t , and this reward function is named the "N-distance decay reward equation". By introducing the "N-distance decay reward equation", the exploration time of the unmanned ship during exploration can be reduced, thereby improving the efficiency of the unmanned ship to find the optimal path. And because the RRT* path is used to guide the unmanned ship, it is more likely to reach the optimal path compared to guiding with the RRT path.

Description

Technical Field

[0001] The present invention relates to the technical field of machine learning, and particularly to a path planning method and device for reinforcement learning of an unmanned ship based on RRT path intelligent guidance. Background Art

[0002] In the reinforcement learning application of the unmanned ship's autonomous learning and path planning, by introducing a motion reference coordinate system for the unmanned ship, a kinematic model and a dynamic model of the unmanned ship are established in the corresponding coordinate system, enabling the unmanned ship to perform reinforcement learning in the environment corresponding to the coordinates, and giving certain rewards and punishments during its reinforcement learning process to accelerate its learning efficiency. RRT*(Rapidly-exploring Random Tree Star) adds an optimization process for the path on the basis of RRT. During the process of constructing the random tree, whenever a new node is added to the tree, the algorithm checks the nodes within its neighborhood and tries to find a shorter path from the starting point to the new node. If such a path is found, the algorithm updates the structure of the tree to ensure that the paths in the tree are always optimal (or near-optimal).

[0003] During the process of a reinforcement learning training, the unmanned ship often needs to go through a large number of explorations and spend a lot of time, and by introducing a suitable reward function, the number of explorations and time of the unmanned ship can be effectively reduced. In the prior art, how to find a suitable reward function is a major problem encountered in the current development of unmanned ships.

[0004] While the RRT* algorithm has advantages in the existing path planning algorithms, it also has its defects:

[0005] High computational complexity: Since the RRT* algorithm needs to perform a large number of random samplings and path optimizations during the search process, its computational complexity is relatively high. This may lead to the limitation of the performance of the RRT* algorithm in application scenarios with high real-time requirements.

[0006] Prone to local optimal solutions: The RRT* algorithm may fall into local optimal solutions during the search process, that is, the found path is optimal within a local range, but may not be optimal globally. This may cause the unmanned ship to miss better path choices during navigation.

[0007] Insufficient path smoothness: The path generated by the RRT* algorithm may be relatively jagged and requires further smoothing to improve the feasibility of the path. This may increase the complexity and computational amount of the algorithm.

[0008] Sensitive to parameters: The performance of the RRT* algorithm is greatly affected by its parameter settings. If the parameter settings are improper, it may lead to a decline in the algorithm's performance or the inability to find a feasible path. Therefore, in practical applications, the parameters of the algorithm need to be carefully adjusted and optimized. Summary of the Invention

[0009] The purpose of the present invention is to at least solve one of the deficiencies of the prior art, and provide a path planning method and device based on RRT path intelligent guidance for reinforcement learning of unmanned ships.

[0010] To achieve the above purpose, the present invention adopts the following technical solutions:

[0011] Specifically, a path planning method based on RRT path intelligent guidance for reinforcement learning of unmanned ships is proposed, including the following:

[0012] Step S1, determine the programming software and its associated libraries;

[0013] Step S2, set the state space and action space of the unmanned ship, set the exploration environment for the visualization of the unmanned ship and the motion model of the unmanned ship;

[0014] Step S3, initialize the state and action of the unmanned ship, and collect the exploration state and action of the unmanned ship;

[0015] Step S4, determine whether there is an RRT* path in the exploration environment. If there is no RRT* path, explore through the RRT* algorithm to obtain an RRT* path denoted as path, and add it to the exploration environment and return to step S3 for execution. If there is an RRT* path, go to step S5;

[0016] Step S5, explore through the Sarsa algorithm, create and initialize a Q table, and the Q table is updated as the unmanned ship moves;

[0017] Step S6, when exploring through the Sarsa algorithm, calculate the horizontal distance between the current exploration position and the path path each time, and substitute the horizontal distance into the N-distance decay reward equation for calculation to obtain a reward value, and reward the unmanned ship with the reward value;

[0018] Step S7, determine whether the exploration is completed. If so, go to step S8. If not, return to step S5 for execution;

[0019] Step S8, output the optimal path obtained after the exploration is completed.

[0020] Further, specifically, the determined programming software is Pycharm software, and its associated libraries include pandas, numpy, matolotlib, math, time, sys, tkinter.

[0021] Further, specifically, step S2 includes:

[0022] S2-1. Set the determination program for the unmanned ship. By judging to determine its position in the environment, what rewards will be obtained, and decide whether to return to the starting point to re-explore or continue the current exploration. The positions include normal channels, dangerous areas, and destinations, and the corresponding rewards are m, -M, and M respectively. If in a dangerous area or reaching the destination, return to the starting point to continue exploration and learning; otherwise, continue the current exploration.

[0023] S2-2. Set the motion equation of the unmanned ship. When setting the equation, it is necessary to ensure that the ship can sail normally and there will be no rudder steering.

[0024] Further, specifically, an RRT* path is obtained through exploration by the RRT* algorithm, including:

[0025] S11. Initialize the RRT* algorithm, set its starting point start and ending point goal, and create an RRT search tree T that only contains start, that is, let the search tree T grow from start.

[0026] S12. The RRT* starts random sampling, and uses a random point P in the environment as the candidate expansion direction of the search tree T. P is neither the starting point nor the ending point.

[0027] S13. Node selection of the RRT* algorithm. In the search tree T constructed by the RRT*, find a point closest to point P, denoted as point Q.

[0028] S14. Node expansion of the RRT* algorithm. Starting from point Q, extend a certain distance in the direction of point P to obtain a new node R. This node is the expansion of the search tree T towards the ending point.

[0029] S15. Judge whether the extension path between Q and R collides with obstacles. If a collision occurs, return to S11 for the next iteration.

[0030] S16. Cost calculation of the RRT* algorithm. Calculate the path cost from the starting point to point R, which also includes the length of the path, the time required along this path, or other objective functions.

[0031] S17. Select the parent node. Find a group of nodes Q' near point R and try to use some of them as the parent node of Q. Select the node Q' that can make the total cost from start to Q the lowest as the parent node of Q.

[0032] S18. Path iteration of the RRT* algorithm. Whenever a new node is added to the search tree, the algorithm checks and optimizes the path from the starting point to that node, which means the algorithm does not stop immediately when the first path to the end point is found, but continues to search and optimize the path;

[0033] S19. Iteration process of the RRT* algorithm. Repeat the above steps S11 to S16 until an optimal path that meets the requirements is found or the set maximum number of iterations is reached.

[0034] Further, specifically, step S6 includes:

[0035] S6-1. Obtain the initial state of the unmanned ship. Denote the central coordinates of the unmanned ship in the initial state as (x0, y0), and use path[0] and path[1] to represent the abscissa and ordinate of any point on path respectively;

[0036] S6-2. Obtain the central coordinates (x1, y1) of the unmanned ship during each movement;

[0037] S6-3. When y1 = path[1];

[0038] S6-4. Calculate Δx t = x1 - path[0], and substitute Δx t into the N-distance decay reward equation to obtain the reward R t , and give R t to the unmanned ship.

[0039] Further, specifically, the N-distance decay reward equation is as follows:

[0040]

[0041] Where, R t is the reward value at the current exploration position, Δx t is the horizontal distance between the unmanned ship at the current exploration position and path, and N is a preset constant.

[0042] The present invention also proposes a path planning device for an unmanned ship based on RRT path intelligent guidance reinforcement learning, which is characterized by including:

[0043] Software selection module, used to determine the programming software and its associated libraries;

[0044] Parameter setting module, used to set the state space and action space of the unmanned ship, set the exploration environment for the visualization of the unmanned ship and the motion model of the unmanned ship;

[0045] Initialization module, used to initialize the state and action of the unmanned ship, and collect the exploration state and action of the unmanned ship;

[0046] The first judgment module is used to judge whether there is an RRT* path in the exploration environment. If there is no RRT* path, an RRT* path is explored through the RRT* algorithm, denoted as path, and added to the exploration environment, then return to step S3 for execution. If there is an RRT* path, go to step S5;

[0047] The exploration module is used to explore through the Sarsa algorithm, create and initialize a Q table, and the Q table is updated as the unmanned ship moves;

[0048] The reward value calculation module is used to calculate the horizontal distance between the current exploration position and path each time during the exploration through the Sarsa algorithm, and substitute the horizontal distance into the N-distance decay reward equation for calculation to obtain the reward value, and reward the unmanned ship with the reward value;

[0049] The second judgment module is used to judge whether the exploration is completed. If so, go to step S8; if not, go to step S5 for execution;

[0050] The result output module is used to output the optimal path obtained after the exploration is completed.

[0051] The beneficial effects of the present invention are as follows:

[0052] The present invention proposes a path planning method and device for an unmanned ship based on RRT path intelligent guidance and reinforcement learning. By using an optimal path path (which may be a local optimum) planned by the RRT* algorithm as a reference for the reinforcement learning of the unmanned ship, and adding this path to the reinforcement learning of the unmanned ship, the horizontal distance Δx between the unmanned ship during exploration and path is calculated t squared, and the calculation result is used as the divisor of the constant N, and finally a reward value R is obtained t , and this reward function is named the "N-distance decay reward equation". By introducing the "N-distance decay reward equation", the exploration time of the unmanned ship during exploration can be reduced, thereby improving the efficiency of the unmanned ship to find the optimal path. And because the RRT* path is used to guide the unmanned ship, it is more likely to reach the optimal path compared with the RRT path for guidance. Description of the Drawings

[0053] By elaborating on the embodiments shown in conjunction with the drawings, the above and other features of the present disclosure will become more obvious. The same reference numerals in the drawings of the present disclosure represent the same or similar elements. Obviously, the drawings in the following description are only some embodiments of the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts. In the drawings:

[0054] Figure 1The figure shows the flowchart of the path planning method for the intelligent guidance of an unmanned ship based on RRT path reinforcement learning according to the present invention;

[0055] Figure 2 The figure shows the schematic diagram of the "N - distance decay reward equation" in the present invention;

[0056] Figure 3 The figure shows the schematic diagram of obtaining an RRT* path through exploration by the RRT* algorithm in the present invention. Specific Embodiments

[0057] The following will clearly and completely describe the concept, specific structure and technical effects generated by the present invention in combination with embodiments and drawings to fully understand the purpose, solution and effects of the present invention. It should be noted that, without conflict, the embodiments in the present application and the features in the embodiments can be combined with each other. The same reference numerals used throughout the drawings indicate the same or similar parts.

[0058] Embodiment 1, referring to Figure 1 , the present invention proposes a path planning method for the intelligent guidance of an unmanned ship based on RRT path reinforcement learning, including the following:

[0059] Step S1, determine the programming software and its associated libraries;

[0060] Step S2, set the state space and action space of the unmanned ship, set the visual exploration environment of the unmanned ship and the motion model of the unmanned ship;

[0061] Step S3, initialize the state and action of the unmanned ship, and collect the exploration state and action of the unmanned ship;

[0062] Step S4, determine whether there is an RRT* path in the exploration environment. If there is no RRT* path, explore through the RRT* algorithm to obtain an RRT* path denoted as path, add it to the exploration environment and return to step S3 for execution. If there is an RRT* path, go to step S5;

[0063] Step S5: Explore through the Sarsa algorithm, create and initialize a Q-table (the Q-table is a table that records the states and actions of the agent, that is, which actions are available to the agent in a certain state, the corresponding Q-values of each action, and the agent has a probability of ε (this value is between 0 and 1) according to the greedy policy to take the action with the largest Q-value, and a probability of 1 - ε to randomly select from the available actions. The update rule of the Q-table is as follows: Q(s,a)←Q(s,a)+α[r+γmaxa′Q(s′,a′)-Q(s,a)]. α is the learning rate, which controls the influence degree of new information on the old Q-value. r is the immediate reward obtained after taking action a in state s. γ is the discount factor, which represents the current value of future rewards. s′ is the new state transferred to after taking action a, and maxa′Q(s′,a′) is the maximum Q-value that may be obtained in the new state). The Q-table is updated as the unmanned ship moves;

[0064] Step S6: When exploring through the Sarsa algorithm, calculate the horizontal distance between the current exploration position and the path path each time of exploration, and substitute the horizontal distance into the N-distance decay reward equation for calculation to obtain the reward value, and reward the unmanned ship with the reward value;

[0065] Step S7: Determine whether the exploration is completed (before the exploration, an iteration number can be set artificially. Each time before the iteration, it is judged whether the iteration number has been reached. If so, the iteration ends and the exploration is completed. Generally speaking, the larger the iteration number, the better the result obtained by the exploration, but there will always be a critical number, and the result will no longer change after exceeding this number). If so, go to Step S8; if not, go to Step S5 to execute;

[0066] Step S8: Output the optimal path obtained after the exploration is completed.

[0067] In this Embodiment 1, the optimal path path (which may be a local optimum) planned by an RRT* algorithm is used as a reference for the reinforcement learning of the unmanned ship, and this path is added to the reinforcement learning of the unmanned ship. By calculating the square of the horizontal distance Δx between the unmanned ship and path during the exploration t as the divisor of the constant N, and finally obtaining a reward value R t , and naming this reward function as the "N-distance decay reward equation". By introducing the "N-distance decay reward equation", the exploration time of the unmanned ship can be reduced during the exploration, thereby improving the efficiency of the unmanned ship to find the optimal path. And because the RRT* path is used to guide the unmanned ship, it is more likely to reach the optimal path compared to using the RRT path to guide.

[0068] As a preferred embodiment of the present invention, specifically, the determined programming software is Pycharm software and its associated libraries including pandas, numpy, matolotlib, math, time, sys, tkinter.

[0069] As a preferred embodiment of the present invention, specifically, step S2 includes

[0070] S2-1, setting the determination program of the unmanned ship, determining its position in the environment through judgment, obtaining what kind of reward, and deciding whether to return to the starting point to re-explore or continue the current exploration. The positions include normal waterways, dangerous areas, and destinations, and the corresponding rewards are m, -M, and M respectively. If in a dangerous area or reaching the destination, return to the starting point to continue exploration and learning, otherwise continue the current exploration;

[0071] S2-2, setting the motion equation of the unmanned ship. When setting the equation, it is necessary to ensure that the ship can sail normally and there will be no rudder hitting.

[0072] Refer to Figure 3 , as a preferred embodiment of the present invention, specifically, an RRT* path is obtained through exploration by the RRT* algorithm, including

[0073] S11, initializing the RRT* algorithm, setting the starting point start and the ending point goal for it, and creating an RRT search tree T that only contains start, that is, making the search tree T grow from start;

[0074] S12, the RRT* starts random sampling, using a random point P in the environment as the candidate expansion direction of the search tree T, and P is neither the starting point nor the ending point;

[0075] S13, node selection of the RRT* algorithm. In the search tree T constructed by the RRT*, find a point closest to the P point, denoted as the Q point;

[0076] S14, node expansion of the RRT* algorithm. Starting from the Q point, extend a certain distance in the direction of the P point to obtain a new node R, and this node is the expansion of the search tree T towards the ending point direction;

[0077] S15, determine whether the extension path between Q and R collides with obstacles. If a collision occurs, return to S11 for the next iteration;

[0078] S16, cost calculation of the RRT* algorithm, calculating the path cost from the starting point to the R point, which also includes the length of the path, the time required along this path, or other objective functions;

[0079] S17. Select the parent node, find a set of nodes Q' near point R, and try to use some of these nodes as the parent node of Q. Select the node Q' that can minimize the total cost from start to Q as the parent node of Q;

[0080] S18. Path iteration of the RRT* algorithm. Whenever a new node is added to the search tree, the algorithm checks and optimizes the path from the starting point to this node, which means the algorithm does not stop immediately when finding the first path to the end point, but continues to search and optimize the path;

[0081] S19. Iteration process of the RRT* algorithm. Repeat the above steps S11 to S16 until an optimal path that meets the requirements is found or the set maximum number of iterations is reached.

[0082] In this preferred embodiment, the RRT* path is constructed in the above manner. The progressive optimization of the RRT* algorithm. The RRT* algorithm is a process of progressive optimization. As the number of iterations increases, the found path will get closer and closer to the global optimal solution.

[0083] Refer to Figure 2 , as a preferred embodiment of the present invention, specifically, the step S6 includes

[0084] S6-1. Obtain the initial state of the unmanned ship, record the center coordinates (x0, y0) of the unmanned ship in the initial state, and use path[0] and path[1] to represent the abscissa and ordinate of any point on path respectively;

[0085] S6-2. Obtain the center coordinates (x1, y1) of the unmanned ship during each movement;

[0086] S6-3. When y1 = path[1];

[0087] S6-4. Calculate Δx t = x1 - path[0], substitute Δx t into the N-distance decay reward equation to obtain the reward R t , and give R t to the unmanned ship.

[0088] As a preferred embodiment of the present invention, specifically, the N-distance decay reward equation is specifically as follows:

[0089]

[0090] Among them, R t is the reward value at the current exploration position, Δx t is the horizontal distance between the unmanned ship at the current exploration position and path, and N is a preset constant.

[0091] In this preferred embodiment, considering that the original technical basis only sets the following rewards for the unmanned ship: the reward value for reaching the destination is "M" (in order to give a relatively large reward to the unmanned ship, the value of M can be taken from 500 to 1000, which is a recommended value), the reward value for reaching the dangerous area is "-M", and the reward value for normal navigation (without reaching the end point and the dangerous area) is "m" (in order to prevent the phenomenon of the unmanned ship "scoring points", the value of m is usually set relatively small, such as taking a value within 1 to 10). However, doing so will result in a single reward for the unmanned ship during normal navigation, and the time required for exploration is very long.

[0092] Based on this, this strategy uses a method of using the RRT* path to guide the reinforcement learning of the unmanned ship. The optimal path path (which may be a local optimum) planned by the RRT* algorithm is used as a reference for the reinforcement learning of the unmanned ship, and this path is added to the reinforcement learning of the unmanned ship. By calculating the horizontal distance Δx between the unmanned ship and path during exploration t squared, and using this calculation result as the divisor of a constant N (the value of N should not be too large, as it is easy to cause the phenomenon of the unmanned ship "scoring points", nor should it be too small, as it is easy to result in no distinction in the rewards of the unmanned ship in various states, that is, the reward function is ineffective. The specific value of N can be determined according to the size of the exploration space. Generally, it is selected within the interval [1, 10]), and finally a reward value R is obtained t We name this reward function the "N-distance decay reward equation". By introducing the "N-distance decay reward equation", the unmanned ship can reduce the exploration time during exploration, thereby improving the efficiency of the unmanned ship to find the optimal path.

[0093] The overall steps of the method of the present invention in actual application are as follows:

[0094] S1. We use the Pycharm software for programming. The libraries that need to be imported are pandas, numpy, matolotlib, math, time, sys, tkinter, etc.;

[0095] S2. We set the state space and action space of the unmanned ship, and visualize the exploration environment of the unmanned ship. For the environment, it includes a dangerous area, a starting point, a destination, the unmanned ship and a waterway, etc., as well as the motion model and motion equation of the unmanned ship;

[0096] S2-1. Set the determination program of the unmanned ship. By judging, determine its position in the environment, what rewards it will obtain, and decide whether to return to the starting point to re-explore or continue the current exploration (the positions are the normal waterway, the dangerous area, and the destination, and the corresponding rewards are m, -M, and M respectively. If in the dangerous area or reaching the destination, return to the starting point to continue the exploration and learning, otherwise continue the current exploration)

[0097] S2-2. Set the motion equation of the unmanned ship. When setting the equation, it is necessary to ensure that the ship can sail normally and will not turn the rudder too quickly, causing the ship to capsize and result in losses.

[0098] S3. Initialize the state and actions of the unmanned ship, and collect the exploration state and actions of the unmanned ship.

[0099] S4. Determine whether the RRT* path has been introduced into the environment.

[0100] S4-1. If the RRT* path has not been introduced into the environment, add the code corresponding to RRT*, plan a path path for the unmanned ship, draw it in the visualization path, and store the points on this path. Then return to S3.

[0101] S4-2. If the RRT* path already exists in the environment, add the code corresponding to the Sarsa algorithm, create a Q-table to store the state and actions of the unmanned ship, as well as the corresponding Q-Value.

[0102] S5. Add the reward function "N-distance decay reward equation" so that the horizontal distance Δx from the path is calculated at each step when the unmanned ship is exploring. t , substitute it into the reward function "N-distance decay reward equation" to obtain R. t , with R t give a reward to the unmanned ship.

[0103] The most important part of this strategy is to introduce the "N-distance decay reward equation". The specific implementation steps of this reward function are as follows:

[0104] S5-1. According to the initial state of the unmanned ship, obtain the center coordinates (x0, y0) of the unmanned ship in the initial state; associate the horizontal and vertical coordinates of path with the indices [0][1] respectively, so that path[0] and path[1] can be used to represent the horizontal and vertical coordinates of path respectively.

[0105] S5-2. Obtain the center coordinates (x1, y1) of the unmanned ship at each movement.

[0106] S5-3. When y1 = path[1];

[0107] S5-4. Calculate Δx t = x1 - path[0], substitute Δx t into the "N-distance decay reward equation" to obtain the reward R t , and give R t to the unmanned ship.

[0108] S6. Determine whether learning has been completed or the set number of iterations has been reached.

[0109] S6-1, if the determination is no, then return to S4-2;

[0110] S6-2, if the determination is yes, then end the loop and enter S7;

[0111] S7, output the optimal path obtained through reinforcement learning.

[0112] The present invention also proposes a path planning device based on the reinforcement learning of the RRT path intelligent guidance unmanned ship, including:

[0113] A software selection module, used to determine the programming software and its associated libraries;

[0114] A parameter setting module, used to set the state space and action space of the unmanned ship, set the exploration environment for the visualization of the unmanned ship and the motion model of the unmanned ship;

[0115] An initialization module, used to initialize the state and action of the unmanned ship, and collect the exploration state and action of the unmanned ship;

[0116] A first judgment module, used to judge whether there is an RRT* path in the exploration environment. If there is no RRT* path, then explore through the RRT* algorithm to obtain an RRT* path denoted as path, and add it to the exploration environment and return to step S3 for execution. If there is an RRT* path, then go to step S5;

[0117] An exploration module, used to explore through the Sarsa algorithm, create and initialize a Q table, and the Q table is updated as the unmanned ship moves;

[0118] A reward value calculation module, used to calculate the horizontal distance between the current exploration position and the path path each time during the exploration through the Sarsa algorithm, and substitute the horizontal distance into the N-distance decay reward equation for calculation to obtain the reward value, and reward the unmanned ship with the reward value;

[0119] A second judgment module, used to judge whether the exploration is completed. If so, then go to step S8. If not, then go to step S5 for execution;

[0120] A result output module, used to output the optimal path obtained after the exploration is completed.

[0121] In addition, in each embodiment of the present invention, the various functional modules can be integrated in a processing module, or each module can exist physically alone, or two or more modules can be integrated in one module. The above integrated modules can be implemented in the form of hardware or in the form of software functional modules.

[0122] When the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, to implement all or part of the processes in the above-described embodiment methods of the present invention, it can also be completed by a computer program instructing relevant hardware. The computer program can be stored in a computer-readable storage medium. When the computer program is executed by a processor, the steps of the above-described various method embodiments can be implemented. Among them, the computer program includes computer program code, and the computer program code can be in the form of source code, object code, executable file, or some intermediate form, etc. The computer-readable medium can include: any entity or system, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disc, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, RandomAccess Memory), electrical carrier signal, telecommunication signal, and software distribution medium, etc., that can carry the computer program code.

[0123] Although the description of the present invention has been quite detailed and particularly describes several of the described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but rather should be regarded as providing a broad interpretation of these claims in light of the prior art by reference to the appended claims, thereby effectively covering the intended scope of the present invention. In addition, the present invention is described above in terms of embodiments foreseeable by the inventor for the purpose of providing a useful description, and non-substantive modifications to the present invention that are not currently foreseeable may still represent equivalent modifications of the present invention.

[0124] As described above, these are only the preferred embodiments of the present invention. The present invention is not limited to the above-described implementation manners. As long as it achieves the technical effects of the present invention by the same means, it should fall within the protection scope of the present invention. Within the protection scope of the present invention, various different modifications and variations can be made to its technical solutions and / or implementation manners.

Claims

1. A path planning method based on the reinforcement learning of the RRT* path intelligent guidance for an unmanned ship, characterized in that, including the following: Step S1, determine the programming software and its associated libraries; Step S2, set the state space and action space of the unmanned ship, set the exploration environment for the visualization of the unmanned ship and the motion model of the unmanned ship; Step S3, initialize the state and action of the unmanned ship, and collect the exploration state and action of the unmanned ship; Step S4, determine whether there is an RRT* path in the exploration environment. If there is no RRT* path, explore through the RRT* algorithm to obtain an RRT* path denoted as path, and add it to the exploration environment and return to Step S3 for execution. If there is an RRT* path, go to Step S5; Step S5, explore through the Sarsa algorithm, create and initialize a Q table, and the Q table is updated as the unmanned ship moves; Step S6, when exploring through the Sarsa algorithm, calculate the horizontal distance between the current exploration position and the path path each time, and substitute the horizontal distance into the N-distance attenuation reward equation for calculation to obtain the reward value, and reward the unmanned ship with the reward value; Step S7, determine whether the exploration is completed. If so, go to Step S8. If not, return to Step S5 for execution; Step S8, output the optimal path obtained after the exploration is completed; Specifically, Step S6 includes: S6-1, obtain the initial state of the unmanned ship, record the center coordinates (x0, y0) of the unmanned ship in the initial state, and use path[0] and path[1] to represent the abscissa and ordinate of any point on path respectively; S6-2, obtain the center coordinates (x1, y1) of the unmanned ship during each movement; S6-3, when y1 = path[1]; S6-4, Calculate = x1 - path[0], and substitute into the N-distance decay reward equation to obtain the reward , and give to the unmanned ship; Specifically, the N-distance attenuation reward equation is specifically as follows: Among them, is the reward value at the current exploration position, is the horizontal distance between the unmanned ship at the current exploration position and the path, and N is a preset constant.

2. The path planning method based on RRT* path intelligent guidance for reinforcement learning of unmanned ships according to claim 1, wherein, Specifically, the determined programming software is Pycharm software and its associated libraries include pandas, numpy, matolotlib, math, time, sys, tkinter.

3. The path planning method based on the reinforcement learning of the RRT*-path-guided unmanned ship according to claim 1, wherein, Specifically, Step S2 includes: S2-1, set the decision-making program of the unmanned ship, determine its position in the environment through judgment, obtain what kind of reward, and decide whether to return to the starting point to re-explore or continue the current exploration. The positions include normal waterways, dangerous areas and destinations, and the corresponding rewards are m, -M and M respectively. If in a dangerous area or reaching the destination, return to the starting point to continue exploration and learning. Otherwise, continue the current exploration; S2-2, set the motion equation of the unmanned ship. When setting the equation, it is necessary to ensure that the ship can sail normally and there will be no excessive rudder operation.

4. The path planning method based on the reinforcement learning of the RRT*-path intelligent guidance unmanned ship according to claim 1, characterized in that, Specifically, obtaining an RRT* path through the RRT* algorithm includes: S11, initialize the RRT* algorithm, set its starting point start and ending point goal, and create an RRT search tree T that only contains start, that is, make the search tree T grow from start; S12, the RRT* starts random sampling, and uses a random point P in the exploration environment as the candidate expansion direction of the search tree T, and P is neither the starting point nor the ending point; S13, Node selection of the RRT* algorithm. In the search tree T constructed by RRT*, find a point closest to point P, denoted as point Q; S14, Node expansion of the RRT* algorithm. Starting from point Q, extend a certain distance in the direction of point P to obtain a new node R. This node is the expansion of the search tree T towards the end point; S15, Determine whether the extended path between Q and R collides with an obstacle. If a collision occurs, return to S11 for the next iteration; S16, Cost calculation of the RRT* algorithm. Calculate the path cost from the start point to point R, which also includes the length of the path, the time required along this path, or other objective functions; S17, Select the parent node. Find a set of nodes Q' near point R and try to use some of these nodes as the parent node of Q. Select the node Q' that can minimize the total cost from start to Q as the parent node of Q; S18, Path iteration of the RRT* algorithm. Whenever a new node is added to the search tree, the algorithm checks and optimizes the path from the start point to this node. This means that the algorithm does not stop immediately when finding the first path to the end point, but continues to search and optimize the path; S19, Iteration process of the RRT* algorithm. Repeat the above steps S11 to S16 until an optimal path that meets the requirements is found or the set maximum number of iterations is reached.

5. A path planning device for an unmanned ship based on the path intelligent guidance of RRT* and reinforcement learning, characterized in that, Steps of applying the path planning method for the RRT*-path-intelligent-guided unmanned ship reinforcement learning described in any one of claims 1-4, including: Software selection module, used to determine the programming software and its associated libraries; Parameter setting module, used to set the state space and action space of the unmanned ship, set the exploration environment for the visualization of the unmanned ship and the motion model of the unmanned ship; Initialization module, used to initialize the state and action of the unmanned ship, and collect the exploration state and action of the unmanned ship; First judgment module, used to judge whether there is an RRT* path in the exploration environment. If there is no RRT* path, explore through the RRT* algorithm to obtain an RRT* path denoted as path, add it to the exploration environment, and return to step S3 for execution. If there is an RRT* path, go to step S5; Exploration module, used to explore through the Sarsa algorithm, create and initialize a Q-table, and the Q-table is updated as the unmanned ship moves; Reward value calculation module, used to calculate the horizontal distance between the current exploration position and path path each time during the exploration through the Sarsa algorithm, and substitute the horizontal distance into the N-distance decay reward equation for calculation to obtain the reward value, and reward the unmanned ship with the reward value; Second judgment module, used to judge whether the exploration is completed. If so, go to step S8. If not, go to step S5 for execution; Result output module, used to output the optimal path obtained after the exploration is completed.

Images