Unmanned boat path planning method, device, equipment and storage medium

Abstract

The present disclosure provides a method, device, equipment and storage medium for unmanned boat path planning, the method comprising: randomly generating a roadblock map, setting a safety threshold based on the size of the unmanned boat, and establishing a reward function based on the end point coordinates, roadblock information and the safety threshold; constructing the initial state information of the unmanned boat based on the heading angle, starting point coordinates and end point coordinates of the unmanned boat, inputting the initial state information into a first neural network for model training, and obtaining the target state information at the next moment; judging the target state information based on the roadblock information, and when the position coordinates in the target state information are within the range coordinates in the roadblock information, retraining the initial state information based on the first neural network, and increasing the number of iterations; when the number of iterations reaches the set value, constructing the optimal path based on the training result of the first neural network.

Description

Technical Field

[0001] The present disclosure relates to the field of path planning, and in particular to an unmanned boat path planning method, device, equipment and storage medium. Background Art

[0002] An unmanned vehicle (USV) is a small surface motion platform that has the functions of autonomous planning, autonomous navigation, autonomous environmental perception and target detection. This technology is used in a variety of tasks such as water survey, marine resource detection, hydrological and geographical research, and marine operations. In order to adapt to different mission requirements, the USV must have reliable track tracking and control capabilities.

[0003] With the continuous advancement of artificial intelligence technology, the research and application of unmanned driving has made great progress. Among them, the research on unmanned boats is relatively mature, but due to the complexity and uncertainty of the marine environment, the current research on unmanned boats is relatively limited. Unmanned boats are widely used in marine resource exploration, maritime rescue, marine environment monitoring, coordinated operations and other fields. In practical applications, one of the key factors that determines whether an unmanned boat can achieve effective navigation is path planning, which reflects the intelligence level of the unmanned boat to a certain extent.

[0004] Path planning for unmanned boats refers to planning an obstacle-free path from the initial position to the target position in a marine environment with obstacles, based on certain constraints and target requirements. The Q-Learning algorithm is one of the classic reinforcement learning algorithms, which has strong robustness and adaptability to uncertain environments. The classic Q-Learning algorithm has problems such as long learning time, low exploration efficiency, and slow convergence speed. In addition, the classic Q-Learning algorithm cannot meet the maneuverability and motion characteristics of unmanned boats, and does not sufficiently consider the heading angle, motion characteristics, length, and safety of the unmanned boat. Summary of the invention

[0005] The present disclosure provides an unmanned boat path planning method, device, equipment and storage medium to at least solve the above technical problems existing in the prior art.

[0006] According to a first aspect of the present disclosure, a method for unmanned boat path planning is provided, the method comprising:

[0007] Randomly generate a roadblock map, wherein the roadblock map includes roadblock information, the starting point coordinates and the end point coordinates of the unmanned boat;

[0008] Setting a safety threshold based on the size of the unmanned boat, and establishing a reward function based on the end point coordinates, the roadblock information, and the safety threshold;

[0009] constructing initial state information of the unmanned boat based on the heading angle of the unmanned boat, the starting point coordinates and the end point coordinates, inputting the initial state information into a first neural network for model training, and obtaining target state information at the next moment;

[0010] The target state information is determined based on the roadblock information, and when the position coordinates in the target state information are within the range coordinates in the roadblock information, the initial state information is retrained based on the first neural network, and the number of iterations is increased;

[0011] When the number of iterations reaches a set value, constructing an optimal path based on the training result of the first neural network;

[0012] Wherein, the reward function includes a safety area function, a target point function and an obstacle area function, and the obstacle area function is constructed based on the roadblock information and the safety threshold;

[0013] wherein, an initial action of the unmanned boat is defined based on a first reinforcement learning algorithm, and the initial action corresponds to the heading angle;

[0014] Based on the diagonal search method, the number of the initial actions is increased to obtain the optimized actions;

[0015] Initial state information of the unmanned boat is constructed based on the optimization action, the starting point coordinates and the end point coordinates, and the number of the initial state information corresponds to the number of the optimization actions.

[0016] wherein, selecting an optimization action in the initial state information, selecting the safety area function or the target point function or the obstacle area function in the reward function based on the current coordinates of the unmanned boat, and calculating the state information of the unmanned boat at the next moment at the current coordinates based on the safety area function or the target point function or the obstacle area function;

[0017] The first neural network includes an input layer, a hidden layer and an output layer, and the state information at the next moment and the optimization action are input into the input layer for model training;

[0018] The hidden layer calculates the first neural network node corresponding to the state information at the next moment using a Gaussian function based on the state information at the next moment and the optimization action;

[0019] The output layer calculates a first weight based on the current coordinates of the unmanned boat and the end point coordinates, and obtains a training result of the first neural network based on the first weight and the first neural network node, wherein the training result of the first neural network is an action at a next moment corresponding to the optimization action, and the number of actions at the next moment corresponds to the number of the optimization actions;

[0020] The maximum value in the training result of the first neural network is selected as the target state information at the next moment.

[0021] In one possible implementation manner, constructing the optimal path based on the training result of the first neural network includes:

[0022] Calculating the training result of the first neural network based on the optimal path strategy until the training result converges;

[0023] The optimal path is constructed based on the converged training results.

[0024] In one embodiment, the method further comprises:

[0025] All position point information in the optimal path of the unmanned boat is obtained, and the position point information is calculated based on a third-order Bezier curve to obtain a smooth path of the unmanned boat.

[0026] According to a second aspect of the present disclosure, a path planning device for an unmanned boat is provided, the device comprising:

[0027] A map generating unit, used for randomly generating a roadblock map, wherein the roadblock map includes roadblock information, the starting point coordinates and the ending point coordinates of the unmanned boat;

[0028] A reward function setting unit, configured to set a safety threshold based on the size of the unmanned boat, and to establish a reward function based on the end point coordinates, the roadblock information and the safety threshold;

[0029] A path planning unit, configured to construct initial state information of the unmanned boat based on the heading angle of the unmanned boat, the starting point coordinates and the end point coordinates, input the initial state information into a first neural network for model training, and obtain target state information at the next moment;

[0030] a path iteration unit, configured to determine the target state information based on the roadblock information, and when the position coordinates in the target state information are within the range coordinates in the roadblock information, retrain the initial state information based on the first neural network, and increase the number of iterations by one;

[0031] A path generation unit, configured to construct an optimal path based on the training result of the first neural network when the number of iterations reaches a set value;

[0032] The reward function setting unit is further configured to: the reward function includes a safety area function, a target point function and an obstacle area function, and construct the obstacle area function based on the roadblock information and the safety threshold;

[0033] The path planning unit is also used to define the initial action of the unmanned boat based on a first reinforcement learning algorithm, the initial action corresponding to the heading angle; based on a diagonal search method, increase the number of the initial actions to obtain an optimized action; construct the initial state information of the unmanned boat based on the optimized action, the starting point coordinates and the end point coordinates, and the number of the initial state information corresponds to the number of the optimized actions; wherein, the optimized action in the initial state information is selected, the safety area function or the target point function or the obstacle area function in the reward function is selected based on the current coordinates of the unmanned boat, and the state information of the unmanned boat at the next moment at the current coordinates is calculated based on the safety area function or the target point function or the obstacle area function; the first A neural network includes an input layer, a hidden layer and an output layer, wherein the state information at the next moment and the optimization action are input into the input layer for model training; the hidden layer calculates the first neural network node corresponding to the state information at the next moment using a Gaussian function based on the state information at the next moment and the optimization action; the output layer calculates a first weight based on the current coordinates of the unmanned boat and the end point coordinates, and obtains the training result of the first neural network based on the first weight and the first neural network node, wherein the training result of the first neural network is the action at the next moment corresponding to the optimization action, and the number of actions at the next moment corresponds to the number of optimization actions; and the maximum value in the training result of the first neural network is selected as the target state information at the next moment.

[0034] In one possible implementation manner, the path generation unit is further used to calculate the training results of the first neural network based on the optimal path strategy until the training results converge; and construct the optimal path based on the converged training results.

[0035] In one embodiment, the device further comprises:

[0036] The smoothing processing unit is used to obtain all position point information in the optimal path of the unmanned boat, calculate the position point information based on the third-order Bezier curve, and obtain the smooth path of the unmanned boat.

[0037] According to a third aspect of the present disclosure, there is provided an electronic device, including:

[0038] at least one processor; and

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present disclosure.

[0041] According to a fourth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described in the present disclosure.

[0042] The unmanned boat path planning method, device, equipment and storage medium disclosed in the present invention optimize the initial action of the unmanned boat and set the safety threshold of obstacles, use the first neural network to perform model training, construct the state information of the unmanned boat, and iterate, and finally construct a complete navigation path. The heading angle, angular velocity, length and other indicators of the unmanned boat, as well as the maneuverability and motion characteristics of the unmanned boat are comprehensively considered to ensure the safety and stability of the unmanned boat during navigation.

[0043] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0045] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0046] Figure 1 The schematic diagram shows the implementation process of an unmanned boat path planning method according to an embodiment of the present disclosure. Figure 1 ;

[0047] Figure 2 The schematic diagram shows the implementation process of an unmanned boat path planning method according to an embodiment of the present disclosure. Figure 2 ;

[0048] Figure 3 The safety threshold is introduced in the embodiment of the present disclosure. Schematic diagram of the circular restricted area in front and behind;

[0049] Figure 4The safety threshold is introduced in the embodiment of the present disclosure. Schematic diagram of the front and rear quadrilateral restricted areas;

[0050] Figure 5 Schematic diagrams showing the action space before and after improvement in the embodiment of the present disclosure;

[0051] Figure 6 A schematic diagram of the network structure of the RBF-based Q-Learning algorithm in an embodiment of the present disclosure is shown;

[0052] Figure 7 A schematic diagram of a path planning device for an unmanned boat according to an embodiment of the present disclosure is shown;

[0053] Figure 8 A schematic diagram of the structure of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0054] In order to make the purpose, features, and advantages of the present disclosure more obvious and easy to understand, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0055] Figure 1 The schematic diagram shows the implementation process of an unmanned boat path planning method according to an embodiment of the present disclosure. Figure 1 ,like Figure 1 As shown, a method for unmanned boat path planning in an embodiment of the present disclosure includes the following steps:

[0056] Step 101, randomly generate a roadblock map, wherein the roadblock map includes roadblock information, the starting point coordinates and the end point coordinates of the unmanned boat.

[0057] In the disclosed embodiment, since the grid modeling is not suitable for path planning in a large-scale marine environment, the path planning effect of the unmanned boat adopts a randomly generated obstacle map and determines the starting and ending coordinates of the unmanned boat. Among them, the roadblock information in the obstacle map is determined by the randomly generated roadblocks in the map, including the roadblock size and roadblock coordinates.

[0058] Step 102: setting a safety threshold based on the size of the unmanned boat, and establishing a reward function based on the end point coordinates, the roadblock information and the safety threshold.

[0059] In the disclosed embodiment, a safety threshold is set according to the size of the unmanned boat to ensure the safety of the unmanned boat during driving so that it can avoid the above-mentioned roadblocks. Preferably, the safety threshold is twice the length of the unmanned boat. According to the coordinates of the roadblock, the size of the roadblock and the safety threshold, an envelope is set to block the restricted area of ​​the obstacle, that is, the area where the unmanned boat is prohibited from navigating. In reinforcement learning, the reward function plays an important role in the behavior decision-making and obstacle avoidance process of the USV. Therefore, a nonlinear segmentation function is established as a reward function, and the obstacle is taken as the target point. The reward value of the safety zone close to the target point is relatively large, while the reward value of the safety zone far from the target point is relatively small. Among them, the reward function includes a safety area function, a target point function and an obstacle area function, and the obstacle area function is constructed based on the roadblock information and the safety threshold.

[0060] Step 103, constructing the initial state information of the unmanned boat based on the heading angle of the unmanned boat, the starting point coordinates and the end point coordinates, inputting the initial state information into a first neural network for model training, and obtaining the target state information at the next moment.

[0061] In the disclosed embodiment, the initial action of the unmanned boat is defined based on the first reinforcement learning algorithm, and the initial action corresponds to the heading angle, wherein the first reinforcement learning algorithm is a Q-Learning algorithm (neural network smoothing and fast Q-Learning, NSFQ), the initial action includes an initial position coordinate and an initial heading angle, and based on a diagonal search method, the number of the initial actions is increased to obtain an optimized action, which increases the basic four heading angles to eight heading angles; the initial state information of the unmanned boat is constructed based on the optimized action, the starting point coordinates and the end point coordinates, and the number of the initial state information corresponds to the number of the optimized actions.

[0062] In the disclosed embodiment, an optimization action in the initial state information is selected, the safety area function or the target point function or the obstacle area function in the reward function is selected based on the current coordinates of the unmanned boat, and the state information of the unmanned boat at the next moment under the current coordinates is calculated based on the safety area function or the target point function or the obstacle area function; the first neural network includes an input layer, a hidden layer and an output layer, and the state information at the next moment and the optimization action are input into the input layer for model training; the hidden layer calculates the first neural network node corresponding to the state information at the next moment using a Gaussian function based on the state information at the next moment and the optimization action; the output layer calculates a first weight based on the current coordinates of the unmanned boat and the end point coordinates, and obtains a training result of the first neural network based on the first weight and the first neural network node, wherein the training result of the first neural network is the action at the next moment corresponding to the optimization action, and the number of actions at the next moment corresponds to the number of optimization actions; wherein the first neural network model is a radial basis function network (RBF), and the maximum value in the training result of the first neural network is selected as the target state information at the next moment, wherein the maximum value in the training result is the training result with the largest Q value corresponding to the above-mentioned Q-Learning algorithm.

[0063] Step 104, judging the target state information based on the roadblock information, when the position coordinates in the target state information are within the range coordinates in the roadblock information, retraining the initial state information based on the first neural network, and increasing the number of iterations by one.

[0064] In the disclosed embodiment, the position of the unmanned boat is determined by judging the target state information in real time. If it is in a feasible area, the action of the next state is selected for exploration. If the position coordinates are within the range coordinates in the roadblock information, the learning exploration is performed again according to the initial state, and the number of iterations is increased, and it is determined whether the maximum number of iterations has been reached. If not, it indicates that the iteration has not ended, and the learning exploration continues.

[0065] Step 105: When the number of iterations reaches a set value, an optimal path is constructed based on the training result of the first neural network.

[0066] In the disclosed embodiment, the training results of the first neural network are calculated based on the optimal path strategy until the training results converge; and the optimal path is constructed based on the converged training results. Due to the limitation of the action space, the planned initial path is not smooth, and there are many unnecessary turns and broken connections. Taking into account the heading angle of the unmanned boat and its continuity in actual navigation, the Bezier curve is used to smooth the initial path, and all position point information in the optimal path of the unmanned boat is obtained. The position point information is calculated based on the third-order Bezier curve to obtain the smooth path of the unmanned boat. Because the higher the order of the Bezier curve, the greater the degree of optimization and the smoother the path. But at the same time, it is also more likely to collide. Taking into account the safety of the unmanned boat and the smoothness requirements of the path, the third-order Bezier curve is used to smooth the initial path.

[0067] Figure 2 The schematic diagram shows the implementation process of an unmanned boat path planning method according to an embodiment of the present disclosure. Figure 2 ,like Figure 2 As shown, a method for unmanned boat path planning in an embodiment of the present disclosure includes the following steps:

[0068] Step 201, generating an obstacle map as a roadblock environment.

[0069] In the disclosed embodiment, since the grid modeling is not suitable for path planning in large-scale marine environments, the path planning effect of the unmanned boat adopts a randomly generated obstacle map. Determine the starting point and end point coordinates. The current position of the unmanned boat is , among which, The horizontal coordinate of the position of the unmanned boat at this time is , the vertical axis is , starting point position coordinates , target point position coordinates .

[0070] Step 202: Introduce the envelope of the safety threshold as a no-fly zone covering obstacles.

[0071] In order to ensure the safety of the unmanned vehicle (USV), a safety threshold is proposed in the embodiment of the present disclosure. The envelope with twice the length of the unmanned boat and increased safety threshold will be used as a restricted area to block obstacles. A restricted area refers to an area covered by obstacles, that is, an area where USVs are prohibited from navigating. In the marine environment, most obstacles are irregular in shape, which makes the environment modeling process very complicated. In order to simplify the processing, circular envelopes and quadrilateral envelopes are used to envelop these obstacles. Among them, the circular envelope can be used to simplify the modeling complexity of relatively small irregular obstacles. Figure 3 The safety threshold is introduced in the embodiment of the present disclosure. Schematic diagram of the circular restricted area before and after, without introducing a safety threshold The circular restricted area is represented by a solid circle, and a safety threshold is introduced The restricted area after is indicated by a dotted circle. No safety threshold is introduced The circular restricted area is expressed as follows:

[0072]

[0073] Introducing safety thresholds The circular restricted area is expressed as follows:

[0074]

[0075]

[0076]

[0077] Among them, r is the safety threshold not introduced The radius of the solid circle, To introduce a safety threshold The radius of the dotted circle, The center of the restricted area. is the length of USV.

[0078] In the disclosed embodiment, the quadrilateral envelope surface can simplify the modeling complexity of larger irregular obstacles. Figure 4 The safety threshold is introduced in the embodiment of the present disclosure. Schematic diagram of the quadrilateral restricted area before and after, without introducing a safety threshold The quadrilateral restricted area is represented by a solid quadrilateral, and a safety threshold is introduced. The restricted quadrilateral is represented by a dotted quadrilateral. The solid quadrilateral consists of four intersecting lines. The expression of the four intersecting lines can be expressed as:

[0079]

[0080] in, is the serial number of the quadrilateral lines, Straight lines The slope of Straight lines The intercept on the y-axis. No safety threshold is introduced The quadrilateral restricted area is as follows:

[0081]

[0082] Introducing safety thresholds The dotted quadrilateral consists of four intersecting lines Surrounded. The new four intersecting lines can be expressed as:

[0083]

[0084] in, Straight lines The intercept on the y-axis. Due to the safety threshold With the introduction of , the intercept of the straight line on the y-axis changes to:

[0085]

[0086] New Straight Line The intercept on the y-axis can be expressed as:

[0087]

[0088] Introducing safety thresholds The quadrilateral restricted area is expressed by the following formula:

[0089]

[0090] When the slope of an edge does not exist, the inequality constraint for that edge becomes constraints.

[0091] The above formula can be transformed into:

[0092]

[0093]

[0094] in, is the minimum function, For straight line The expression of the straight line There is no slope.

[0095] Step 203, design the action space that the unmanned boat can take during navigation, introduce the RBF neural network to approximate the true Q value, and design the reward function R according to the target point and obstacle information.

[0096] In the disclosed embodiment, the role of the USV is actually the size of its heading angle during obstacle avoidance or path planning. When the Q-Learning algorithm is used to plan the path of the USV, the state of the USV in real self-navigation is continuous. At this time, it is necessary to discretize the observed behavior of the USV to obtain a discrete action space. The present invention optimizes the action space of the unmanned boat by considering the heading angle. Under the current heading, the USV can avoid obstacles by turning the rudder at any angle in the action space. The action space is based on the current heading of the USV. Among them, the action space of the classic Q-Learning algorithm can be defined by the following formula: , where 1, 2, 3, and 4 represent the four discrete actions of the USV: right, up, left, and down. At this time, the heading angle of the USV can be obtained by the following formula: In order to shorten the path length of the unmanned boat while taking into account its heading angle control and navigation safety, the action space of the Q-Learning algorithm is improved. The improved action space adds the search behavior in the diagonal direction, which increases the action space of the USV to 8 discrete actions. The improved USV action space can be defined by the following formula: , where 1, 2, 3, 4, 5, 6, 7 and 8 represent the USV forward, turn left 45 degrees, turn left 90 degrees, turn left 135 degrees, backward, turn right 135 degrees, turn right 90 degrees, and turn right 45 degrees. At this time, the heading angle of the USV can be obtained by the following formula: , Figure 5 A schematic diagram of the action space before and after improvement in an embodiment of the present disclosure is shown, wherein the left figure is the action space of the Q-Learning algorithm, and the right figure is the improved action space.

[0097] In the disclosed embodiment, the reward function plays an important role in the USV's behavior decision-making and obstacle avoidance process. Since the environmental state set and action state of the USV in the Q-Learning algorithm are limited, and the USV is continuous and systematic in the actual navigation process, the reward function is generalized to a nonlinear segmentation function, wherein the reward function includes a safety area function, a target point function, and an obstacle area function, and the obstacle area function is constructed based on the roadblock information and the safety threshold as shown in the following formula.

[0098]

[0099] in, For the obstacle area, For safe areas, is the target point. is a very small positive number, whose size should be determined by the length of the planned path. is the distance from the current position to the target point. Based on the current coordinates of the unmanned boat, the safety area function, the target point function or the obstacle area function in the reward function is selected, and the state information of the unmanned boat at the next moment under the current coordinates is calculated based on the safety area function, the target point function or the obstacle area function. It can be seen from the above formula that the reward value of the safety zone close to the target point is relatively large, while the reward value of the safety zone far from the target point is relatively small.

[0100] Step 204, constructing the initial state of the unmanned boat and determining the next state of the unmanned boat.

[0101] In the disclosed embodiment, the classic Q-Learning algorithm uses grid-based modeling, so the size of its state space can be determined by the number of grids. However, in the present invention, the NSFQ algorithm does not use a grid-based modeling method. Therefore, the state space of the algorithm must be reconstructed, and the initial state of the unmanned boat consists of the position coordinates and heading angle of the unmanned boat. During the navigation of the USV, the longitudinal speed of the USV is u, which remains unchanged, and the lateral speed is , the yaw angular velocity is r. Therefore, if the coordinate position of the starting point is known , the end point coordinate position and the heading angle at a certain moment , the position of USV at any time can be obtained. At this time, the position of USV can be converted into the following formula:

[0102]

[0103] The above formula can be used to calculate the position coordinates of the USV at each moment. The state space of the Q-Learning algorithm consists of the position coordinates of the USV during navigation. The NSFQ algorithm takes into account the heading angle at each moment, so its state space consists of the position coordinates and heading angle of the USV at each moment. The initial state space is defined as: .in, is the position coordinate of USV, is the heading angle of the USV.

[0104] In the present disclosure, according to Strategy, select USV in state The specific action a under the action is obtained The corresponding immediate reward R, and get the next state , and then get the next state The position coordinates and heading of the

[0105] Step 205: Approximate the Q value using the RBF neural network and update the Q value.

[0106] In the embodiment of the present disclosure, the classic Q-learning algorithm uses a Q table to describe the state-action value function , and realize the learning and storage of Q table. However, the working environment of USV is relatively complex, and the number of learning parameters increases exponentially with the increase of state dimension, that is, Q table occupies a large amount of memory space, reduces the calculation efficiency, and causes "dimensionality disaster". The present invention adopts RBF neural network to replace Q table. It is used to approximate the action value function of Q-learning algorithm RBF neural network has a strong function approximation ability and can approximate the real Q function faster and more effectively. The results show that RBF neural network does not have the problem of catastrophic forgetting. Figure 6 The schematic diagram of the network structure of the RBF-based Q-Learning algorithm in the embodiment of the present disclosure is shown, wherein the RBF neural network is a three-layer static feedforward network, including an input layer, a hidden layer, and an output layer. The first layer is the input layer, which consists of signal source nodes. Its input consists of state variables and an action variable The input number is , the input vector is: , the second layer is the hidden layer, and the basis function is Generally, an m-dimensional Gaussian function is selected, and the expression of the k-th RBF node is derived from the following formula:

[0107]

[0108] in, is the center vector of the kth RBF node, is the width of the kth RBF node.

[0109] The third layer is the output layer, which responds to the influence of the input pattern. The action value function of the Q-learning algorithm Approximated by only one output node. Action-value function The expression of can be obtained by the following formula:

[0110]

[0111] in, is the weight value. Assume that the coordinates of the target point are , the current position coordinates of USV are ,but The size of can be expressed as:

[0112] ,

[0113] Step 206, determine the current state of the unmanned boat, further explore or iterate based on the determination result, and output the optimal path.

[0114] In the embodiment of the present disclosure, after the Q value update is completed, the unmanned boat performs a state transition. As a status Make a judgment, if the state of USV If it is in the obstacle area or target area, it will re-explore based on the initial state of the unmanned boat and increase the number of iterations N by 1. If it is in the feasible area, it will continue to learn and explore at the current position. At the same time, it will determine whether the maximum number of iterations has been reached. If not, it means that the iteration has not ended yet, and iterative learning will continue. If the maximum number of iterations has been reached, the optimal path strategy will be output. , until the Q value converges.

[0115] In the disclosed embodiment, the action space is enhanced by combining the heading information, and the number of discrete actions of the actual search action of the USV is increased to 8. Due to the limitation of the action space, the planned initial path is not smooth, and there are many unnecessary turns and broken connections. Considering the heading angle of the unmanned boat and its continuity in actual navigation, the present invention uses a Bezier curve to smooth the initial path. The equation of the n-order Bezier curve is defined as follows:

[0116]

[0117]

[0118] in, is the motion control point of the Bezier curve; is the location point, and are the starting point and the end point respectively. As the order of the Bezier curve is higher, the optimization degree is greater and the path is smoother. But at the same time, it is more likely to collide. Considering the safety of the unmanned boat and the smoothness requirements of the path, the third-order Bezier curve is used to smooth the initial path.

[0119] Figure 7 A schematic diagram of a path planning device for an unmanned boat according to an embodiment of the present disclosure is shown; Figure 7 As shown, an unmanned boat path planning device according to an embodiment of the present disclosure includes:

[0120] The map generation unit 701 is used to randomly generate a roadblock map, wherein the roadblock map includes roadblock information, the starting point coordinates and the end point coordinates of the unmanned boat.

[0121] The reward function setting unit 702 is used to set a safety threshold based on the size of the unmanned boat, and to establish a reward function based on the end point coordinates, the roadblock information and the safety threshold.

[0122] The reward function setting unit 702 is further configured to: the reward function includes a safety area function, a target point function and an obstacle area function; and the obstacle area function is constructed based on the roadblock information and the safety threshold.

[0123] The path planning unit 703 is used to construct the initial state information of the unmanned boat based on the heading angle of the unmanned boat, the starting point coordinates and the end point coordinates, input the initial state information into the first neural network for model training, and obtain the target state information at the next moment.

[0124] The path planning unit 703 is also used to define the initial action of the unmanned boat based on the first reinforcement learning algorithm, and the initial action corresponds to the heading angle; based on the diagonal search method, the number of the initial actions is increased to obtain the optimized action; based on the optimized action, the starting point coordinates and the end point coordinates, the initial state information of the unmanned boat is constructed, and the number of the initial state information corresponds to the number of the optimized actions; wherein, the optimized action in the initial state information is selected, and the safety area function or the target point function or the obstacle area function in the reward function is selected based on the current coordinates of the unmanned boat, and the state information of the unmanned boat at the next moment at the current coordinates is calculated based on the safety area function or the target point function or the obstacle area function; the first A neural network includes an input layer, a hidden layer and an output layer, wherein the state information at the next moment and the optimization action are input into the input layer for model training; the hidden layer calculates the first neural network node corresponding to the state information at the next moment using a Gaussian function based on the state information at the next moment and the optimization action; the output layer calculates a first weight based on the current coordinates of the unmanned boat and the end point coordinates, and obtains the training result of the first neural network based on the first weight and the first neural network node, wherein the training result of the first neural network is the action at the next moment corresponding to the optimization action, and the number of actions at the next moment corresponds to the number of optimization actions; and the maximum value in the training result of the first neural network is selected as the target state information at the next moment.

[0125] a path iteration unit 704, configured to determine the target state information based on the roadblock information, and when the position coordinates in the target state information are within the range coordinates in the roadblock information, retrain the initial state information based on the first neural network, and increase the number of iterations by one;

[0126] The path generation unit 705 is used to construct an optimal path based on the training result of the first neural network when the number of iterations reaches a set value.

[0127] The path generation unit 705 is further configured to calculate the training result of the first neural network based on the optimal path strategy until the training result converges; and construct the optimal path based on the converged training result.

[0128] The smoothing processing unit 706 is used to obtain all position point information in the optimal path of the unmanned boat, calculate the position point information based on the third-order Bezier curve, and obtain the smooth path of the unmanned boat.

[0129] In an exemplary embodiment, the map generation unit 701, the reward function setting unit 702, the path planning unit 703, the path iteration unit 704, the path generation unit 705, the smoothing processing unit 706, etc. can be implemented by one or more central processing units (CPU, Central Processing Unit), graphics processing units (GPU, Graphics Processing Unit), application specific integrated circuits (ASIC, Application Specific Integrated Circuit), DSP, programmable logic devices (PLD, Programmable Logic Device), complex programmable logic devices (CPLD, Complex Programmable Logic Device), field programmable gate arrays (FPGA, Field-Programmable Gate Array), general processors, controllers, microcontrollers (MCU, Micro Controller Unit), microprocessors (Microprocessors), or other electronic components.

[0130] Regarding the device in the above embodiment, the specific manner in which each module and unit performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0131] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0132] Figure 8A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0133] like Figure 8 As shown, the device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 to a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0134] A number of components in the device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0135] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as an unmanned boat path planning method. For example, in some embodiments, an unmanned boat path planning method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of an unmanned boat path planning method described above may be executed. Alternatively, in other embodiments, the computing unit 801 may be configured to execute an unmanned boat path planning method in any other appropriate manner (e.g., by means of firmware).

[0136] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0137] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0138] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0139] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0140] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0141] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0142] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0143] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0144] The above is only a specific embodiment of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any person skilled in the art who is familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present disclosure, which should be included in the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be based on the protection scope of the claims.

Claims

1. A method for unmanned boat path planning, characterized in that: The method comprises: Randomly generate a roadblock map, wherein the roadblock map includes roadblock information, the starting point coordinates and the end point coordinates of the unmanned boat; Setting a safety threshold based on the size of the unmanned boat, and establishing a reward function based on the end point coordinates, the roadblock information, and the safety threshold; constructing initial state information of the unmanned boat based on the heading angle of the unmanned boat, the starting point coordinates and the end point coordinates, inputting the initial state information into a first neural network for model training, and obtaining target state information at the next moment; The target state information is determined based on the roadblock information, and when the position coordinates in the target state information are within the range coordinates in the roadblock information, the initial state information is retrained based on the first neural network, and the number of iterations is increased; When the number of iterations reaches a set value, constructing an optimal path based on the training result of the first neural network; Wherein, the reward function includes a safety area function, a target point function and an obstacle area function, and the obstacle area function is constructed based on the roadblock information and the safety threshold; wherein, an initial action of the unmanned boat is defined based on a first reinforcement learning algorithm, and the initial action corresponds to the heading angle; Based on the diagonal search method, the number of the initial actions is increased to obtain the optimized actions; constructing initial state information of the unmanned boat based on the optimization action, the starting point coordinates and the end point coordinates, wherein the number of the initial state information corresponds to the number of the optimization actions; wherein, selecting an optimization action in the initial state information, selecting the safety area function or the target point function or the obstacle area function in the reward function based on the current coordinates of the unmanned boat, and calculating the state information of the unmanned boat at the next moment at the current coordinates based on the safety area function or the target point function or the obstacle area function; The first neural network includes an input layer, a hidden layer and an output layer, and the state information at the next moment and the optimization action are input into the input layer for model training; The hidden layer calculates the first neural network node corresponding to the state information at the next moment using a Gaussian function based on the state information at the next moment and the optimization action; The output layer determines a first weight based on the current coordinates of the unmanned boat and the end point coordinates according to the inverse of the Euclidean distance formula, and obtains a training result of the first neural network based on the first weight and the first neural network node, wherein the training result of the first neural network is an action value function, and the action value function corresponds to the actions at the next moment that are the same in number as the optimized actions; The maximum value of the action-value function is selected as the target state information at the next moment.

2. The method according to claim 1, characterized in that The constructing the optimal path based on the training result of the first neural network includes: Calculating the training result of the first neural network based on the optimal path strategy until the training result converges; The optimal path is constructed based on the converged training results.

3. The method according to claim 1, characterized in that The method further comprises: All position point information in the optimal path of the unmanned boat is obtained, and the position point information is calculated based on a third-order Bezier curve to obtain a smooth path of the unmanned boat.

4. An unmanned boat path planning device, characterized in that: The device comprises: A map generating unit, used for randomly generating a roadblock map, wherein the roadblock map includes roadblock information, the starting point coordinates and the ending point coordinates of the unmanned boat; A reward function setting unit, configured to set a safety threshold based on the size of the unmanned boat, and to establish a reward function based on the end point coordinates, the roadblock information and the safety threshold; A path planning unit, configured to construct initial state information of the unmanned boat based on the heading angle of the unmanned boat, the starting point coordinates and the end point coordinates, input the initial state information into a first neural network for model training, and obtain target state information at the next moment; a path iteration unit, configured to determine the target state information based on the roadblock information, and when the position coordinates in the target state information are within the range coordinates in the roadblock information, retrain the initial state information based on the first neural network, and increase the number of iterations by one; A path generation unit, configured to construct an optimal path based on the training result of the first neural network when the number of iterations reaches a set value; The reward function setting unit is further configured to: the reward function includes a safety area function, a target point function and an obstacle area function, and construct the obstacle area function based on the roadblock information and the safety threshold; The path planning unit is also used to define the initial action of the unmanned boat based on a first reinforcement learning algorithm, the initial action corresponding to the heading angle; based on a diagonal search method, increase the number of the initial actions to obtain an optimized action; construct the initial state information of the unmanned boat based on the optimized action, the starting point coordinates and the end point coordinates, and the number of the initial state information corresponds to the number of the optimized actions; wherein, the optimized action in the initial state information is selected, the safety area function or the target point function or the obstacle area function in the reward function is selected based on the current coordinates of the unmanned boat, and the state information of the unmanned boat at the next moment at the current coordinates is calculated based on the safety area function or the target point function or the obstacle area function; the first A neural network includes an input layer, a hidden layer and an output layer, wherein the state information at the next moment and the optimization action are input into the input layer for model training; the hidden layer calculates the first neural network node corresponding to the state information at the next moment using a Gaussian function based on the state information at the next moment and the optimization action; the output layer determines a first weight based on the current coordinates of the unmanned boat and the end point coordinates according to the inverse of the Euclidean distance formula, and obtains the training result of the first neural network based on the first weight and the first neural network node, wherein the training result of the first neural network is an action value function, and the action value function corresponds to the actions at the next moment that are the same in number as the optimization actions; and the maximum value of the action value function is selected as the target state information at the next moment.

5. The device according to claim 4, characterized in that The path generation unit is further used to calculate the training result of the first neural network based on the optimal path strategy until the training result converges; and construct the optimal path based on the converged training result.

6. The device according to claim 4, characterized in that The device also includes: The smoothing processing unit is used to obtain all position point information in the optimal path of the unmanned boat, calculate the position point information based on the third-order Bezier curve, and obtain the smooth path of the unmanned boat.

7. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 3.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to make a computer execute the method according to any one of claims 1-3.

Images