RRT path intelligent guidance unmanned ship reinforcement learning-based path planning method and device

Abstract

The invention relates to a path planning method and device for intelligently guiding unmanned ship reinforcement learning based on an RRT path, and the method comprises the steps: taking an optimal path (possibly local optimum) planned through an RRT * algorithm as a reference for the unmanned ship reinforcement learning, and adding the path into the unmanned ship reinforcement learning. The square of the horizontal distance delta xt between the unmanned ship and the path during exploration is calculated, the calculation result serves as a divisor of a constant N, a reward value Rt is finally obtained, and a reward function is named as an N-distance attenuation reward equation. By introducing an N-distance attenuation reward equation, the exploration time of the unmanned ship during exploration can be shortened, and the efficiency of finding the optimal path by the unmanned ship is improved. And the RRT * path is used for guiding the unmanned ship, so that compared with the RRT path, the RRT * path is more likely to achieve the optimal path.

Description

Technical Field

[0001] The present invention relates to the technical field related to machine learning, and in particular to a path planning method and device based on RRT path intelligent guided unmanned ship reinforcement learning. Background Art

[0002] In the reinforcement learning application of unmanned ships autonomously learning to plan paths, by introducing the unmanned ship motion reference coordinate system, the unmanned ship kinematic model and dynamic model are established in the corresponding coordinate system, so that the unmanned ship can perform reinforcement learning in the environment corresponding to the coordinates, and give certain rewards and penalties in the process of its reinforcement learning to speed up its learning efficiency. RRT* (Rapidly-exploring Random Tree Star) adds a path optimization process based on RRT. In the process of building a random tree, whenever a new node is added to the tree, the algorithm checks the nodes in its neighborhood and tries to find a shorter path from the starting point to the new node. If such a path is found, the algorithm updates the structure of the tree to ensure that the path in the tree is always optimal (or close to optimal).

[0003] In the process of reinforcement learning training, unmanned ships often need to go through a lot of exploration and spend a lot of time. By introducing a suitable reward function, the number and time of exploration of unmanned ships can be effectively reduced. In the existing technology, how to find a suitable reward function is a major problem encountered in the development of unmanned ships.

[0004] While the RRT* algorithm has advantages over existing path planning algorithms, it also has its own shortcomings:

[0005] High computational complexity: Since the RRT* algorithm requires a large amount of random sampling and path optimization during the search process, its computational complexity is high. This may limit the performance of the RRT* algorithm in application scenarios with high real-time requirements.

[0006] Easy to fall into local optimal solution: The RRT* algorithm may fall into the local optimal solution during the search process, that is, the path found is optimal in the local range, but may not be optimal in the global range. This may cause the unmanned ship to miss better path options during navigation.

[0007] Insufficient path smoothness: The path generated by the RRT* algorithm may be jagged and requires further smoothing to improve the feasibility of the path. This may increase the complexity and computational effort of the algorithm.

[0008] Sensitive to parameters: The performance of the RRT* algorithm is greatly affected by its parameter settings. If the parameters are not set properly, the performance of the algorithm may degrade or fail to find a feasible path. Therefore, in practical applications, the parameters of the algorithm need to be carefully adjusted and optimized. Summary of the invention

[0009] The purpose of the present invention is to solve at least one of the deficiencies of the prior art and to provide a path planning method and device based on RRT path intelligent guided unmanned ship reinforcement learning.

[0010] In order to achieve the above object, the present invention adopts the following technical solutions:

[0011] Specifically, a path planning method based on RRT path intelligent guided unmanned ship reinforcement learning is proposed, including the following:

[0012] Step S1, determining programming software and its associated libraries;

[0013] Step S2, setting the state space and action space of the unmanned ship, setting the visualization exploration environment of the unmanned ship and the unmanned ship motion model;

[0014] Step S3, initializing the state and action of the unmanned ship, and collecting the exploration state and action of the unmanned ship;

[0015] Step S4, determine whether there is an RRT* path in the exploration environment. If there is no RRT* path, use the RRT* algorithm to explore and obtain an RRT* path, record it as the path, and add it to the exploration environment and return to step S3 for execution. If there is an RRT* path, go to step S5;

[0016] Step S5: Explore through the Sarsa algorithm, create and initialize a Q table, and the Q table is updated as the unmanned ship moves;

[0017] Step S6: When exploring by the Sarsa algorithm, the horizontal distance between the current exploration position and the path is calculated for each exploration, and the horizontal distance is brought into the N-distance decay reward equation to calculate a reward value, and the unmanned ship is rewarded with the reward value;

[0018] Step S7, determine whether the exploration is completed, if yes, go to step S8, if no, go to step S5 for execution;

[0019] Step S8: Output the optimal path obtained after the exploration is completed.

[0020] Further, specifically, the determined programming software is Pycharm software and its associated libraries include pandas, numpy, matolotlib, math, time, sys, and tkinter.

[0021] Further, specifically, the step S2 includes:

[0022] S2-1, set the judgment procedure of the unmanned boat, determine the location of the unmanned boat in the environment, what kind of reward it will get, and decide whether to return to the starting point to explore again or continue the current exploration. The location includes the normal channel, the danger zone and the destination. The corresponding rewards are m, -M and M respectively. If it is in the danger zone or reaches the destination, it returns to the starting point to continue exploration and learning, otherwise it continues the current exploration;

[0023] S2-2, set the motion equation of the unmanned ship. When setting the equation, it is necessary to ensure that the ship can sail normally and will not steer.

[0024] Further, specifically, an RRT* path is obtained by exploring through the RRT* algorithm, including:

[0025] S11, initialize the RRT* algorithm, set the starting point start and the end point goal for it, and create an RRT search tree T containing only start, that is, let the search tree T grow from start;

[0026] S12, RRT* starts random sampling to explore a random point P in the environment as a candidate expansion direction of the search tree T, where P is neither the starting point nor the end point;

[0027] S13, node selection of the RRT* algorithm, find a point closest to point P in the search tree T constructed by RRT*, recorded as point Q;

[0028] S14, node expansion of the RRT* algorithm, starting from point Q, extending a certain distance in the direction of point P to obtain a new node R, which is the extension of the search tree T toward the end point;

[0029] S15, determining whether the extended path between Q and R collides with an obstacle, if so, returning to S11 and performing the next iteration;

[0030] S16, cost calculation of the RRT* algorithm, calculating the path cost from the starting point to point R, which also includes the length of the path, the time required along the path, or other objective functions;

[0031] S17, select the parent node, find a group of nodes Q' near point R, and try to use some of them as the parent nodes of Q, and select the node Q' that can minimize the total cost from start to Q as the parent node of Q;

[0032] S18, path iteration of the RRT* algorithm. Whenever a new node is added to the search tree, the algorithm checks and optimizes the path from the starting point to that node. This means that the algorithm does not stop immediately when it finds the first path to the end point, but continues to search and optimize the path.

[0033] S19, the iteration process of the RRT* algorithm, repeating the above steps S11 to S16 until an optimal path that meets the requirements is found, or the set maximum number of iterations is reached.

[0034] Further, specifically, the step S6 includes:

[0035] S6-1, obtain the initial state of the unmanned ship, record the initial state of the unmanned ship center coordinates (x0, y0), and use path[0] and path[1] to represent the horizontal coordinate and vertical coordinate of any point on the path respectively;

[0036] S6-2, obtaining the center coordinates (x1, y1) of the unmanned ship during each movement;

[0037] S6-3, when y1 = path[1];

[0038] S6-4, calculation of Δx t =x1-path[0], Δx t Substitute into the N-distance decay reward equation to obtain the reward R t , and R t Give unmanned ships.

[0039] Further, specifically, the N-distance decay reward equation is as follows:

[0040]

[0041] Among them, R t is the reward value of the current exploration position, Δx t is the horizontal distance between the unmanned boat and the path at the current exploration position, and N is a preset constant.

[0042] The present invention also proposes a path planning device based on RRT path intelligent unmanned ship reinforcement learning, which is characterized by comprising:

[0043] a software selection module for determining programming software and its associated libraries;

[0044] The parameter setting module is used to set the state space and action space of the unmanned ship, the visualization exploration environment of the unmanned ship and the motion model of the unmanned ship;

[0045] Initialization module, used to initialize the state and action of the unmanned ship and collect the exploration state and action of the unmanned ship;

[0046] The first judgment module is used to judge whether there is an RRT* path in the exploration environment. If there is no RRT* path, an RRT* path is obtained by exploring through the RRT* algorithm and recorded as the path path, and the exploration environment is added to return to step S3 for execution. If there is an RRT* path, the process goes to step S5;

[0047] An exploration module, used to explore by using a Sarsa algorithm, create and initialize a Q table, and the Q table is updated as the unmanned ship moves;

[0048] A reward value calculation module is used to calculate the horizontal distance between the current exploration position and the path path each time when exploring through the Sarsa algorithm, and bring the horizontal distance into the N-distance decay reward equation to calculate the reward value, and reward the unmanned ship with the reward value;

[0049] The second judgment module is used to judge whether the exploration is completed, if so, go to step S8, if not, go to step S5 for execution;

[0050] The result output module is used to output the optimal path obtained after the exploration is completed.

[0051] The beneficial effects of the present invention are:

[0052] The present invention proposes a path planning method and device based on RRT path intelligent guidance unmanned ship reinforcement learning. The optimal path (which may be a local optimum) planned by an RRT* algorithm is used as a reference for unmanned ship reinforcement learning, and this path is added to the unmanned ship reinforcement learning. The horizontal distance Δx between the unmanned ship and the path during exploration is calculated. t The square of the calculation result is used as the divisor of the constant N, and finally a reward value R is obtained. t , and named this reward function "N-distance decay reward equation". By introducing the "N-distance decay reward equation", the exploration time of the unmanned ship can be reduced during exploration, thereby improving the efficiency of the unmanned ship in finding the optimal path. And because the RRT* path is used to guide the unmanned ship, it is more likely to reach the optimal path than the RRT path. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] The above and other features of the present disclosure will become more apparent by describing in detail the embodiments shown in the accompanying drawings. The same reference numerals in the accompanying drawings of the present disclosure represent the same or similar elements. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure. For those skilled in the art, other accompanying drawings can be obtained based on these accompanying drawings without creative work. In the accompanying drawings:

[0054] Figure 1It is a flow chart of the path planning method of the present invention based on the RRT path intelligent guided unmanned ship reinforcement learning;

[0055] Figure 2 Shown is a schematic diagram of the "N-distance decay reward equation" in the present invention;

[0056] Figure 3 It is a schematic diagram showing the principle of obtaining an RRT* path through exploration by the RRT* algorithm in the present invention. DETAILED DESCRIPTION

[0057] The following will be combined with the embodiments and drawings to clearly and completely describe the concept, specific structure and technical effects of the present invention, so as to fully understand the purpose, scheme and effect of the present invention. It should be noted that the embodiments in this application and the features in the embodiments can be combined with each other without conflict. The same reference numerals used throughout the drawings indicate the same or similar parts.

[0058] Example 1, reference Figure 1 The present invention proposes a path planning method based on RRT path intelligent guided unmanned ship reinforcement learning, including the following:

[0059] Step S1, determining programming software and its associated libraries;

[0060] Step S2, setting the state space and action space of the unmanned ship, setting the visualization exploration environment of the unmanned ship and the unmanned ship motion model;

[0061] Step S3, initializing the state and action of the unmanned ship, and collecting the exploration state and action of the unmanned ship;

[0062] Step S4, determine whether there is an RRT* path in the exploration environment. If there is no RRT* path, use the RRT* algorithm to explore and obtain an RRT* path, record it as the path, and add it to the exploration environment and return to step S3 for execution. If there is an RRT* path, go to step S5;

[0063] Step S5, explore through the Sarsa algorithm, create and initialize a Q table (the Q table is a table that records the state and action of the agent, that is, what actions are available for the agent in a certain state, the Q value corresponding to each action, and the agent has a probability of ε to take the action with the largest Q value according to the ε value of the greedy strategy (the value is between 0 and 1), and has a probability of 1-ε to randomly select from the optional actions. The update rule of the Q table is as follows: Q(s,a)←Q(s,a)+α[r+γmaxa′Q(s′,a′)-Q(s,a)]α is the learning rate, which controls the influence of new information on the old Q value, r is the immediate reward obtained after taking action a in state s, γ is the discount factor, which indicates the current value of future rewards, s′ is the new state transferred to after taking action a, and maxa′Q(s′,a′) is the maximum Q value that may be obtained in the new state), and the Q table is updated as the unmanned ship moves;

[0064] Step S6: When exploring by the Sarsa algorithm, the horizontal distance between the current exploration position and the path is calculated for each exploration, and the horizontal distance is brought into the N-distance decay reward equation to calculate a reward value, and the unmanned ship is rewarded with the reward value;

[0065] Step S7, determine whether the exploration is completed (before the exploration, an iteration number can be set manually. Before each iteration, it will be determined whether the iteration number has been reached. If it has been reached, the iteration ends and the exploration is completed. Generally speaking, the greater the iteration number, the better the exploration result. However, there will always be a critical number, and the result will not change after exceeding this number). If yes, go to step S8, otherwise go to step S5 for execution;

[0066] Step S8: Output the optimal path obtained after the exploration is completed.

[0067] In this embodiment 1, an optimal path (possibly a local optimum) planned by the RRT* algorithm is used as a reference for unmanned ship reinforcement learning, and this path is added to the unmanned ship reinforcement learning. By calculating the horizontal distance Δx between the unmanned ship and the path during exploration, t The square of the calculation result is used as the divisor of the constant N, and finally a reward value R is obtained. t , and named this reward function "N-distance decay reward equation". By introducing the "N-distance decay reward equation", the exploration time of the unmanned ship can be reduced during exploration, thereby improving the efficiency of the unmanned ship in finding the optimal path. And because the RRT* path is used to guide the unmanned ship, it is more likely to reach the optimal path than the RRT path.

[0068] As a preferred embodiment of the present invention, specifically, the determined programming software is Pycharm software and its associated libraries include pandas, numpy, matolotlib, math, time, sys, and tkinter.

[0069] As a preferred embodiment of the present invention, specifically, step S2 includes:

[0070] S2-1, set the judgment procedure of the unmanned boat, determine the location of the unmanned boat in the environment, what kind of reward it will get, and decide whether to return to the starting point to explore again or continue the current exploration. The location includes the normal channel, the danger zone and the destination. The corresponding rewards are m, -M and M respectively. If it is in the danger zone or reaches the destination, it returns to the starting point to continue exploration and learning, otherwise it continues the current exploration;

[0071] S2-2, set the motion equation of the unmanned ship. When setting the equation, it is necessary to ensure that the ship can sail normally and will not steer.

[0072] Reference Figure 3 As a preferred embodiment of the present invention, specifically, an RRT* path is obtained by exploring through the RRT* algorithm, including:

[0073] S11, initialize the RRT* algorithm, set the starting point start and the end point goal for it, and create an RRT search tree T containing only start, that is, let the search tree T grow from start;

[0074] S12, RRT* starts random sampling to explore a random point P in the environment as a candidate expansion direction of the search tree T, where P is neither the starting point nor the end point;

[0075] S13, node selection of the RRT* algorithm, find a point closest to point P in the search tree T constructed by RRT*, recorded as point Q;

[0076] S14, node expansion of the RRT* algorithm, starting from point Q, extending a certain distance in the direction of point P to obtain a new node R, which is the extension of the search tree T toward the end point;

[0077] S15, determining whether the extended path between Q and R collides with an obstacle, if so, returning to S11 and performing the next iteration;

[0078] S16, cost calculation of the RRT* algorithm, calculating the path cost from the starting point to point R, which also includes the length of the path, the time required along the path, or other objective functions;

[0079] S17, select the parent node, find a group of nodes Q' near point R, and try to use some of them as the parent nodes of Q, and select the node Q' that can minimize the total cost from start to Q as the parent node of Q;

[0080] S18, path iteration of the RRT* algorithm. Whenever a new node is added to the search tree, the algorithm checks and optimizes the path from the starting point to that node. This means that the algorithm does not stop immediately when it finds the first path to the end point, but continues to search and optimize the path.

[0081] S19, the iteration process of the RRT* algorithm, repeating the above steps S11 to S16 until an optimal path that meets the requirements is found, or the set maximum number of iterations is reached.

[0082] In this preferred embodiment, the RRT* path is constructed in the above manner, and the RRT* algorithm is gradually optimized. The RRT* algorithm is a gradual optimization process. As the number of iterations increases, the found path will get closer and closer to the global optimal solution.

[0083] Reference Figure 2 As a preferred embodiment of the present invention, specifically, step S6 includes:

[0084] S6-1, obtain the initial state of the unmanned ship, record the initial state of the unmanned ship center coordinates (x0, y0), and use path[0] and path[1] to represent the horizontal coordinate and vertical coordinate of any point on the path respectively;

[0085] S6-2, obtaining the center coordinates (x1, y1) of the unmanned ship during each movement;

[0086] S6-3, when y1 = path[1];

[0087] S6-4, calculation of Δx t =x1-path[0], Δx t Substitute into the N-distance decay reward equation to obtain the reward R t , and R t Give unmanned ships.

[0088] As a preferred embodiment of the present invention, specifically, the N-distance decay reward equation is as follows:

[0089]

[0090] Among them, R t is the reward value of the current exploration position, Δx t is the horizontal distance between the unmanned boat and the path at the current exploration position, and N is a preset constant.

[0091] In this preferred embodiment, considering that the original technical basis is to set the rewards for the unmanned ship only: the reward value for reaching the destination is "M" (in order to give the unmanned ship a larger reward, the value of M can be taken between 500 and 1000, which is the recommended value), the reward value for reaching the danger zone is "-M", and the reward value for normal navigation (not reaching the end point and the danger zone) is "m" (in order to prevent the unmanned ship from "brushing points", the m value is usually set to a relatively small value, such as a value between 1 and 10), but doing so will result in a single reward for the unmanned ship during normal navigation, and a long time will be required for exploration.

[0092] Based on this, this strategy uses a method of using RRT* path to guide unmanned ship reinforcement learning, takes an optimal path (possibly a local optimum) planned by the RRT* algorithm as a reference for unmanned ship reinforcement learning, and adds this path to the unmanned ship reinforcement learning. By calculating the horizontal distance Δx between the unmanned ship and the path during exploration, t The square of the calculated result is used as the divisor of the constant N (the value of N should not be too large, otherwise the unmanned ship may "brush points", nor should it be too small, otherwise the rewards of the unmanned ship in each state will not be differentiated, that is, the introduction of the reward function is invalid. The specific value of N can be determined according to the size of the exploration space. Generally, it is selected in the interval [1, 10]), and finally a reward value R is obtained. t We named this reward function "N-distance decay reward equation". By introducing the "N-distance decay reward equation", the exploration time of the unmanned ship can be reduced during exploration, thereby improving the efficiency of the unmanned ship in finding the optimal path.

[0093] The overall steps of the method of the present invention in practical application are as follows:

[0094] S1, we use Pycharm software for programming, and the libraries that need to be imported include pandas, numpy, matolotlib, math, time, sys, tkinter, etc.;

[0095] S2, we set the state space and action space of the unmanned ship and visualize the unmanned ship exploration environment, which includes the danger zone, starting point, destination, unmanned ship and waterway, as well as the unmanned ship motion model and motion equation;

[0096] S2-1, set the judgment procedure of the unmanned boat, determine its position in the environment through judgment, what kind of reward it will get, and decide whether to return to the starting point to explore again or continue the current exploration (the positions include normal waterways, danger zones and destinations, and the corresponding rewards are m, -M and M respectively. If it is in a danger zone or reaches the destination, it returns to the starting point to continue exploring and learning, otherwise it continues the current exploration)

[0097] S2-2, set the motion equation of the unmanned ship. When setting the equation, it is necessary to ensure that the ship can sail normally and will not turn the rudder too fast to cause the ship to capsize and cause losses;

[0098] S3, initialize the state and action of the unmanned ship, collect the exploration state and action of the unmanned ship

[0099] S4, determine whether the RRT* path has been introduced in the environment;

[0100] S4-1, if the RRT* path is not introduced in the environment, then add the code corresponding to RRT*, plan a path for the unmanned ship, draw it in the visualization path and store the points on the path, and then return to S3;

[0101] S4-2, if there is an RRT* path in the environment, add the corresponding code of the Sarsa algorithm and make a Q table to store the state and action of the unmanned ship and the corresponding Q-Value;

[0102] S5, add the reward function "N-distance decay reward equation" so that the unmanned ship calculates the horizontal distance Δx from the path at each step during exploration t , substitute into the reward function "N-distance decay reward equation", and get R t , with R t Rewards for unmanned ships;

[0103] The most important thing about this strategy is to introduce the "N-distance decay reward equation". The following are the specific implementation steps of this reward function:

[0104] S5-1, according to the initial state of the unmanned ship, obtain the initial state of the unmanned ship center coordinates (x0, y0); associate the horizontal and vertical coordinates of the path with the index [0][1] respectively, that is, path[0] and path[1] can be used to represent the horizontal and vertical coordinates of the path respectively;

[0105] S5-2, obtain the center coordinates (x1, y1) of the unmanned ship during each movement;

[0106] S5-3, when y1 = path[1];

[0107] S5-4, calculation of Δx t =x1-path[0], Δx t Substitute into the "N-distance decay reward equation" to obtain the reward R t , and R t Give unmanned ships;

[0108] S6, determining whether the learning is completed or the set number of iterations has been reached;

[0109] S6-1, if the determination is no, return to S4-2;

[0110] S6-2, if the judgment is yes, then end the loop and enter S7;

[0111] S7, output the optimal path obtained through reinforcement learning.

[0112] The present invention also proposes a path planning device based on RRT path intelligent guided unmanned ship reinforcement learning, comprising:

[0113] a software selection module for determining programming software and its associated libraries;

[0114] The parameter setting module is used to set the state space and action space of the unmanned ship, the visualization exploration environment of the unmanned ship and the motion model of the unmanned ship;

[0115] Initialization module, used to initialize the state and action of the unmanned ship and collect the exploration state and action of the unmanned ship;

[0116] The first judgment module is used to judge whether there is an RRT* path in the exploration environment. If there is no RRT* path, an RRT* path is obtained by exploring through the RRT* algorithm and recorded as the path path, and the exploration environment is added to return to step S3 for execution. If there is an RRT* path, the process goes to step S5;

[0117] An exploration module, used to explore by using a Sarsa algorithm, create and initialize a Q table, and the Q table is updated as the unmanned ship moves;

[0118] A reward value calculation module is used to calculate the horizontal distance between the current exploration position and the path path each time when exploring through the Sarsa algorithm, and bring the horizontal distance into the N-distance decay reward equation to calculate the reward value, and reward the unmanned ship with the reward value;

[0119] The second judgment module is used to judge whether the exploration is completed, if so, go to step S8, if not, go to step S5 for execution;

[0120] The result output module is used to output the optimal path obtained after the exploration is completed.

[0121] In addition, each functional module in each embodiment of the present invention may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated module may be implemented in the form of hardware or in the form of software functional modules.

[0122] If the integrated module is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present invention implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or system that can carry the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electrical carrier signal, telecommunication signal and software distribution medium, etc.

[0123] Although the description of the present invention has been quite detailed and specifically describes several described embodiments, it is not intended to be limited to any of these details or embodiments or any particular embodiment, but should be regarded as providing a broad possible interpretation of these claims in view of the prior art by reference to the appended claims, thereby effectively covering the intended scope of the present invention. In addition, the above description of the present invention is based on the embodiments foreseeable by the inventor, and its purpose is to provide a useful description, and those non-substantial changes to the present invention that have not yet been foreseen may still represent equivalent changes to the present invention.

[0124] The above is only a preferred embodiment of the present invention. The present invention is not limited to the above implementation. As long as the technical effect of the present invention is achieved by the same means, it should belong to the protection scope of the present invention. Within the protection scope of the present invention, its technical scheme and / or implementation method can have various modifications and changes.

Claims

1. The path planning method based on RRT path intelligent guided unmanned ship reinforcement learning is characterized by: Includes the following: Step S1, determining programming software and its associated libraries; Step S2, setting the state space and action space of the unmanned ship, setting the visualization exploration environment of the unmanned ship and the unmanned ship motion model; Step S3, initializing the state and action of the unmanned ship, and collecting the exploration state and action of the unmanned ship; Step S4, determine whether there is an RRT* path in the exploration environment. If there is no RRT* path, use the RRT* algorithm to explore and obtain an RRT* path, record it as the path, and add it to the exploration environment and return to step S3 for execution. If there is an RRT* path, go to step S5; Step S5: Explore through the Sarsa algorithm, create and initialize a Q table, and the Q table is updated as the unmanned ship moves; Step S6: When exploring by the Sarsa algorithm, the horizontal distance between the current exploration position and the path is calculated for each exploration, and the horizontal distance is brought into the N-distance decay reward equation to calculate a reward value, and the unmanned ship is rewarded with the reward value; Step S7, determine whether the exploration is completed, if yes, go to step S8, if no, go to step S5 for execution; Step S8: Output the optimal path obtained after the exploration is completed.

2. The path planning method based on RRT path intelligent guided unmanned ship reinforcement learning according to claim 1 is characterized in that: Specifically, the determined programming software is Pycharm software and its associated libraries include pandas, numpy, matolotlib, math, time, sys, and tkinter.

3. The path planning method based on RRT path intelligent guided unmanned ship reinforcement learning according to claim 1 is characterized in that: Specifically, the step S2 includes: S2-1, set the judgment procedure of the unmanned boat, determine the location of the unmanned boat in the environment, what kind of reward it will get, and decide whether to return to the starting point to explore again or continue the current exploration. The location includes the normal channel, the danger zone and the destination. The corresponding rewards are m, -M and M respectively. If it is in the danger zone or reaches the destination, it returns to the starting point to continue exploration and learning, otherwise it continues the current exploration; S2-2, set the motion equation of the unmanned ship. When setting the equation, it is necessary to ensure that the ship can sail normally and will not steer too quickly.

4. The path planning method based on RRT path intelligent guided unmanned ship reinforcement learning according to claim 1 is characterized in that: Specifically, an RRT* path is obtained through exploration using the RRT* algorithm, including: S11, initialize the RRT* algorithm, set the starting point start and the end point goal for it, and create an RRT search tree T containing only start, that is, let the search tree T grow from start; S12, RRT* starts random sampling to explore a random point P in the environment as a candidate expansion direction of the search tree T, where P is neither the starting point nor the end point; S13, node selection of the RRT* algorithm, find a point closest to point P in the search tree T constructed by RRT*, recorded as point Q; S14, node expansion of the RRT* algorithm, starting from point Q, extending a certain distance in the direction of point P to obtain a new node R, which is the extension of the search tree T toward the end point; S15, determining whether the extended path between Q and R collides with an obstacle, if so, returning to S11 and performing the next iteration; S16, cost calculation of the RRT* algorithm, calculating the path cost from the starting point to point R, which also includes the length of the path, the time required along the path, or other objective functions; S17, select the parent node, find a group of nodes Q' near point R, and try to use some of them as the parent nodes of Q, and select the node Q' that can minimize the total cost from start to Q as the parent node of Q; S18, path iteration of the RRT* algorithm. Whenever a new node is added to the search tree, the algorithm checks and optimizes the path from the starting point to that node. This means that the algorithm does not stop immediately when it finds the first path to the end point, but continues to search and optimize the path. S19, the iteration process of the RRT* algorithm, repeating the above steps S11 to S16 until an optimal path that meets the requirements is found, or the set maximum number of iterations is reached.

5. The path planning method based on RRT path intelligent guided unmanned ship reinforcement learning according to claim 1 is characterized in that: Specifically, the step S6 includes: S6-1, obtain the initial state of the unmanned ship, record the initial state of the unmanned ship center coordinates (x0, y0), and use path[0] and path[1] to represent the horizontal coordinate and vertical coordinate of any point on the path respectively; S6-2, obtaining the center coordinates (x1, y1) of the unmanned ship during each movement; S6-3, when y1 = path[1]; S6-4, calculation of Δx t =x1-path[0], Δx t Substitute into the N-distance decay reward equation to obtain the reward R t , and R t Give unmanned ships.

6. The path planning method based on RRT path intelligent guided unmanned ship reinforcement learning according to claim 5 is characterized in that: Specifically, the N-distance decay reward equation is as follows: Among them, R t is the reward value of the current exploration position, Δx t is the horizontal distance between the unmanned boat and the path at the current exploration position, and N is a preset constant.

7. The path planning device based on RRT path intelligent unmanned ship reinforcement learning is characterized by: include: a software selection module for determining programming software and its associated libraries; The parameter setting module is used to set the state space and action space of the unmanned ship, the visualization exploration environment of the unmanned ship and the motion model of the unmanned ship; Initialization module, used to initialize the state and action of the unmanned ship and collect the exploration state and action of the unmanned ship; The first judgment module is used to judge whether there is an RRT* path in the exploration environment. If there is no RRT* path, an RRT* path is obtained by exploring through the RRT* algorithm and recorded as the path path, and the exploration environment is added to return to step S3 for execution. If there is an RRT* path, the process goes to step S5; An exploration module, used to explore by using a Sarsa algorithm, create and initialize a Q table, and the Q table is updated as the unmanned ship moves; A reward value calculation module is used to calculate the horizontal distance between the current exploration position and the path path each time when exploring through the Sarsa algorithm, and bring the horizontal distance into the N-distance decay reward equation to calculate the reward value, and reward the unmanned ship with the reward value; The second judgment module is used to judge whether the exploration is completed, if so, go to step S8, if not, go to step S5 for execution; The result output module is used to output the optimal path obtained after the exploration is completed.

Images