An unmanned ship path planning method, device and equipment

Abstract

This invention relates to the field of unmanned surface vessel (USV) technology, and discloses a USV path planning method, apparatus, and equipment. The method includes generating a prior reference path; acquiring the current position coordinates and current heading angle, and calculating a heading deviation reward value; calculating the minimum distance between the current position of the USV and the prior reference path; setting a layered region threshold based on the total distance between the starting point and the ending point, determining the specific interval within which the minimum distance falls under the layered region threshold, and selecting the corresponding gradient reward coefficient based on the interval to calculate the layered guidance reward value; calculating the total reward value at the current moment; updating the action policy of a preset reinforcement learning model using the total reward value, and outputting the USV's rudder angle control command based on the updated action policy. This invention effectively solves the problems of slow search convergence and poor path planning quality in complex environments in existing technologies, significantly improving the efficiency and feasibility of USV path planning technology.

Description

Technical Field

[0001] This invention relates to the field of unmanned surface vessel (USV) technology, and more specifically, to a USV path planning method, apparatus, and equipment. Background Technology

[0002] With increasing maritime traffic density, various maritime accidents are occurring frequently. Statistics show that the vast majority of these accidents are caused by human factors. Therefore, improving the level of autonomous navigation for ships has become a critical issue that urgently needs to be addressed. Unmanned surface vessels (USVs), as a type of vessel system that does not require human piloting, are better able to adapt to the complex and challenging marine environment compared to manned vessels, giving them unique advantages in mission execution. Path planning, as one of the core technologies for autonomous navigation of USVs, aims to plan an optimized navigation path to ensure that the USV can reach its target location quickly and safely. This is of great significance for improving its overall combat capabilities and mission completion efficiency.

[0003] Existing unmanned surface vessel (USV) path planning technologies mainly include traditional methods such as the A* algorithm and Dijkstra's algorithm, as well as the RRT (Rapid Expanding Random Tree) algorithm, which has been widely used in recent years based on sampling optimization. Although the RRT algorithm overcomes the shortcomings of traditional algorithms in getting trapped in local optima in complex environments to some extent and can better explore the state space, it still falls short in complex ocean dynamic environments when dealing with wind and wave interference and obstacle avoidance. Moreover, existing strategies often generate excessive rudder angles in autonomous driving control, increasing the risk of ship capsizing and resulting in low path planning efficiency and feasibility. Summary of the Invention

[0004] To overcome the shortcomings of low efficiency and feasibility in existing unmanned surface vessel (USV) path planning technologies, this invention proposes the following technical solution:

[0005] Firstly, this invention proposes an unmanned surface vessel (USV) path planning method, comprising:

[0006] S1: Generate a static, collision-free path connecting the start and end points as a priori reference path to guide the navigation of the unmanned surface vessel;

[0007] S2: Obtain the current position coordinates and current bow angle of the unmanned surface vessel;

[0008] S3: Calculate the heading deviation bonus value based on the current position coordinates and the current heading angle;

[0009] S4: Calculate the minimum distance between the current position of the unmanned surface vessel and the prior reference path;

[0010] S5: Set a tiered region threshold based on the total distance between the starting point and the ending point, determine the specific interval in which the minimum distance falls within the tiered region threshold, and select the corresponding gradient reward coefficient based on the interval to calculate the tiered guidance reward value.

[0011] S6: Calculate the total reward value at the current moment based on the heading deviation reward value and the hierarchical guidance reward value;

[0012] S7: Update the action policy of the preset reinforcement learning model using the total reward value, and output the rudder angle control command of the unmanned surface vessel according to the updated action policy;

[0013] S8: Control the unmanned surface vessel to navigate to the next moment state according to the rudder angle control command;

[0014] S9: Repeat S2~S8 until the unmanned surface vessel reaches the destination.

[0015] As a preferred technical solution, a static, collision-free path connecting the starting point and the ending point is generated as a priori reference path to guide the navigation of the unmanned surface vessel, including:

[0016] Set the starting point of the unmanned surface vessel as the root node of a fast-exploration random tree, and add the root node to the node set;

[0017] A state sampling point Xrand is randomly generated within the free space of the digital map in the virtual environment;

[0018] Traverse all existing nodes in the node set, calculate the distance between each node and the state sampling point Xrand, and select the node with the smallest distance value as the nearest node Xnear.

[0019] Starting from the nearest node Xnear, extend the growth step by a preset amount in the direction pointing to the state sampling point Xrand, and calculate the new node Xnew in the extension direction.

[0020] Detect whether the path segment connecting the nearest node Xnear and the new node Xnew intersects with an obstacle in the digital map of the virtual environment;

[0021] If no intersection exists, the new node Xnew is added to the node set and connected as a child node to the nearest node Xnear; if an intersection exists, the new node Xnew is discarded.

[0022] The steps are repeated until the distance between the newly added node and the endpoint is less than a preset threshold range. Then, the newly added node is connected to the endpoint, and the connection from the starting point to the endpoint is extracted as a priori reference path.

[0023] As a preferred technical solution, the heading deviation bonus value is calculated according to the following formula. :

[0024]

[0025] In the formula, This is the current bow angle. Let be the target heading angle, and k be the sensitivity coefficient.

[0026] As a preferred technical solution, calculating the minimum distance between the current position of the unmanned surface vessel and the prior reference path includes:

[0027] Calculate the Euclidean distance from the current position of the unmanned surface vessel to any path node on the prior reference path according to the following formula. :

[0028]

[0029] In the formula, and These are the x and y coordinates of the unmanned surface vessel's current position, respectively. and These are the x and y coordinates of a path node on the prior reference path, respectively.

[0030] Iterate through all path nodes on the prior reference path, repeat the above calculation steps to obtain the corresponding Euclidean distance set, and select the minimum value in the Euclidean distance set as the minimum distance between the current position of the unmanned surface vessel and the prior reference path. .

[0031] As a preferred technical solution, a tiered region threshold is set based on the total distance between the starting point and the ending point. The specific interval within which the minimum distance falls under the tiered region threshold is determined. A corresponding gradient reward coefficient is selected based on the interval to calculate the tiered guidance reward value, including:

[0032] The Euclidean distance d between the starting and ending coordinates of the unmanned surface vessel (USV) is calculated using the following formula:

[0033]

[0034] In the formula, and These are the x and y coordinates of the starting position of the unmanned surface vessel, respectively. and Let x and y be the x and y coordinates of the unmanned surface vessel's endpoint, respectively.

[0035] Based on the Euclidean distance Determine the layered region threshold H, and divide the range determined by the layered region threshold H into several intervals with different distance gradients;

[0036] The minimum distance from the current position of the unmanned surface vessel to the prior reference path. The minimum distance is determined by comparing it with the intervals of different distance gradients. The specific interval it is located in;

[0037] Calculate the hierarchical guidance reward value according to the following formula. :

[0038]

[0039] In the formula, As a preset constant, This represents the gradient reward coefficient.

[0040] As a preferred technical solution, the rules for determining the gradient reward coefficient are as follows:

[0041] .

[0042] As a preferred technical solution, the total reward value at the current moment is calculated based on the heading deviation reward value and the hierarchical guidance reward value, including:

[0043] The timing efficiency penalty is calculated according to the following formula. :

[0044]

[0045] in, The penalty / reward coefficient, t Number of steps used in the current training round The total number of steps allowed to reach the finish line. Let be the Euclidean distance between the current position and the destination of the unmanned surface vessel. The initial distance between the unmanned surface vessel and the destination;

[0046] The total reward value at the current moment is calculated according to the following formula. :

[0047]

[0048] in, , , These are respectively: ship deviation reward value, hierarchical guidance reward value, and timing efficiency penalty. , and These are the corresponding weighting coefficients.

[0049] As a preferred technical solution, the reinforcement learning model includes an action space and a state space; the action space includes several discrete rudder angle values, and the state space includes the unmanned surface vessel's position coordinates, heading angle, and velocity information;

[0050] The action policy of the preset reinforcement learning model is updated using the total reward value, and the rudder angle control command of the unmanned surface vessel is output according to the updated action policy, including:

[0051] Construct a Q-value table to map the correspondence between the state space and the action space;

[0052] Using the calculated total reward value at the current moment, the action value values ​​in the Q-value table corresponding to the current state and the currently executed rudder angle action are corrected to obtain a Q-value table containing the updated action values.

[0053] Obtain the real-time status information of the unmanned surface vessel at the next moment after performing the current action;

[0054] Based on the real-time status information of the next moment, retrieve the Q-value table containing the updated action value, and obtain the action value value corresponding to all candidate rudder angle actions in the action space under this state;

[0055] Compare the action value values ​​corresponding to all candidate rudder angle actions, select the rudder angle with the highest action value value as the rudder angle control command for the unmanned surface vessel at the next moment, and output it.

[0056] Secondly, the present invention also proposes an unmanned surface vessel (USV) path planning device, applied in the USV path planning method as described in any of the embodiments of the first aspect, comprising:

[0057] The generation module is used to generate a static, collision-free path connecting the start and end points as a priori reference path to guide the navigation of the unmanned surface vessel.

[0058] The acquisition module is used to acquire the current position coordinates and current bow angle of the unmanned surface vessel.

[0059] The first calculation module is used to calculate the heading deviation bonus value based on the current position coordinates and the current heading angle.

[0060] The second calculation module is used to calculate the minimum distance between the current position of the unmanned surface vessel and the prior reference path;

[0061] The third calculation module is used to set a stratified region threshold based on the total distance between the starting point and the ending point, determine the specific interval in which the minimum distance falls within the stratified region threshold, and select the corresponding gradient reward coefficient based on the interval to calculate the stratified guidance reward value.

[0062] The fourth calculation module is used to calculate the total reward value at the current moment based on the heading deviation reward value and the hierarchical guidance reward value;

[0063] The output module is used to update the action policy of the preset reinforcement learning model using the total reward value, and output the rudder angle control command of the unmanned surface vessel according to the updated action policy.

[0064] The control module is used to control the unmanned surface vessel to navigate to the next moment state according to the rudder angle control command;

[0065] The loop control module is used to determine whether the unmanned surface vessel has reached the destination, and if it has not reached the destination, it triggers the acquisition module, the first calculation module, the second calculation module, the third calculation module, the fourth calculation module, the output module and the control module to run repeatedly until the unmanned surface vessel reaches the destination.

[0066] Thirdly, the present invention also proposes an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the operations performed by the unmanned surface vessel path planning method as described in any of the embodiments of the first aspect.

[0067] The beneficial effects of the present invention include at least the following:

[0068] This invention provides global navigation guidance for unmanned surface vessels (USVs) by generating a static, collision-free path connecting the start and end points as a priori reference path, effectively overcoming the limitations of traditional reinforcement learning in its initial blind exploration. Furthermore, by introducing a heading deviation reward value, the heading of the USV can be constrained in real time, avoiding navigation instability caused by excessive rudder angle control or large heading fluctuations. Further, by combining a hierarchical region threshold set based on the total distance between the start and end points, gradient reward coefficients are selected differently according to the distance the USV deviates from the priori reference path to calculate hierarchical guidance reward values, constructing a hierarchical reward mechanism with clear gradient guidance. This mechanism can transform sparse rewards in the environment into highly guiding dense gradient rewards, prompting the USV to quickly converge to the priori reference path during training, thereby continuously correcting its action strategy in dynamic interactions. This effectively solves the problems of slow search convergence and poor path planning quality in complex environments in existing technologies, significantly improving the efficiency and feasibility of USV path planning technology. Attached Figure Description

[0069] Figure 1 This is a flowchart illustrating the unmanned surface vessel path planning method provided in an embodiment of the present invention.

[0070] Figure 2 This is a schematic diagram of the process for generating a priori reference paths provided in an embodiment of the present invention.

[0071] Figure 3 The calculation level guidance reward value provided in the embodiments of the present invention A flowchart.

[0072] Figure 4 This is an architectural diagram of the unmanned surface vessel path planning device provided in an embodiment of the present invention.

[0073] Figure 5 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0074] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred technical solutions. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred technical solutions are only for illustrating the present invention and are not intended to limit the scope of protection of the present invention.

[0075] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0076] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0077] Example 1

[0078] This embodiment proposes an unmanned surface vessel (USV) path planning method, such as... Figure 1 As shown, Figure 1 This is a flowchart illustrating an unmanned surface vessel (USV) path planning method provided in this embodiment. The method includes the following steps:

[0079] S1: Generate a static, collision-free path connecting the start and end points as a priori reference path to guide the navigation of the unmanned surface vessel;

[0080] S2: Obtain the current position coordinates and current bow angle of the unmanned surface vessel;

[0081] S3: Calculate the heading deviation bonus value based on the current position coordinates and the current heading angle;

[0082] S4: Calculate the minimum distance between the current position of the unmanned surface vessel and the prior reference path;

[0083] S5: Set a tiered region threshold based on the total distance between the starting point and the ending point, determine the specific interval in which the minimum distance falls within the tiered region threshold, and select the corresponding gradient reward coefficient based on the interval to calculate the tiered guidance reward value.

[0084] S6: Calculate the total reward value at the current moment based on the heading deviation reward value and the hierarchical guidance reward value;

[0085] S7: Update the action policy of the preset reinforcement learning model using the total reward value, and output the rudder angle control command of the unmanned surface vessel according to the updated action policy;

[0086] S8: Control the unmanned surface vessel to navigate to the next moment state according to the rudder angle control command;

[0087] S9: Repeat S2~S8 until the unmanned surface vessel reaches the destination.

[0088] Understandably, by generating a static, collision-free path connecting the start and end points as a priori reference path, global navigation guidance is provided for the unmanned surface vessel (USV), effectively overcoming the limitations of traditional reinforcement learning in its initial blind exploration. Furthermore, by introducing a heading deviation reward value, the USV's bow orientation can be constrained in real time, preventing navigation instability caused by excessive rudder angle control or large heading fluctuations. Moreover, by combining a hierarchical region threshold set based on the total distance between the start and end points, gradient reward coefficients are selectively chosen based on the distance the USV deviates from the priori reference path to calculate hierarchical guidance reward values, constructing a hierarchical reward mechanism with clear gradient guidance. This mechanism can transform sparse rewards in the environment into highly guiding dense gradient rewards, prompting the USV to quickly converge to the priori reference path during training, thereby continuously correcting its action strategy in dynamic interactions. This effectively solves the problems of slow search convergence and poor path planning quality in complex environments in existing technologies, significantly improving the efficiency and feasibility of USV path planning technology.

[0089] Example 2

[0090] This embodiment is an improvement on the unmanned surface vessel path planning method proposed in Embodiment 1.

[0091] In this embodiment, generating a static, collision-free path connecting the start and end points serves as a priori reference path to guide the navigation of the unmanned surface vessel, including:

[0092] Set the starting point of the unmanned surface vessel as the root node of a fast-exploration random tree, and add the root node to the node set;

[0093] A state sampling point Xrand is randomly generated within the free space of the digital map in the virtual environment;

[0094] Traverse all existing nodes in the node set, calculate the distance between each node and the state sampling point Xrand, and select the node with the smallest distance value as the nearest node Xnear.

[0095] Starting from the nearest node Xnear, extend the growth step by a preset amount in the direction pointing to the state sampling point Xrand, and calculate the new node Xnew in the extension direction.

[0096] Detect whether the path segment connecting the nearest node Xnear and the new node Xnew intersects with an obstacle in the digital map of the virtual environment;

[0097] If no intersection exists, the new node Xnew is added to the node set and connected as a child node to the nearest node Xnear; if an intersection exists, the new node Xnew is discarded.

[0098] The steps are repeated until the distance between the newly added node and the endpoint is less than a preset threshold range. Then, the newly added node is connected to the endpoint, and the connection from the starting point to the endpoint is extracted as a priori reference path.

[0099] As an example, in the specific implementation process, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the process for generating a priori reference paths provided in this embodiment of the invention. First, a virtual environment digitization platform is established. The size of the map, the size and coordinates of static obstacles, the waterway, and the coordinates of both banks of the waterway are initialized. The start and end points of the unmanned surface vessel (USV) are added, and its motion state is initialized. The start point of the USV is placed into the growth tree node set as the root node of the RRT algorithm. An initial growth step size is set according to the known environmental information. When the map is large, the step size is increased to avoid slow search speed; when the map area is small, the step size is decreased to prevent collisions with obstacles. Then, state points are randomly sampled from the environment, and the nearest node in the tree is found. A new node is generated by growing one step size in the direction connecting the two nodes. If the path does not pass through an obstacle, it is added to the tree. This process is repeated until the distance from the newly generated node to the target point is less than one step size. The new node is then directly connected to the target point, thus obtaining an RRT path planning curve connecting the start and end points.

[0100] Understandably, the RRT algorithm, as a sampling optimization technique, is better able to explore the state space and generate feasible and near-optimal paths in complex environments compared to traditional algorithms. This embodiment introduces the RRT algorithm for training, using relevant data on interaction with the environment as prior knowledge for path planning of the unmanned surface vessel (USV). Utilizing the generated paths as a reference improves the USV's initial tendency to blindly explore, guiding its training and effectively enhancing path planning efficiency and the vessel's ability to learn towards its destination.

[0101] In this embodiment, the heading deviation bonus value is calculated according to the following formula. :

[0102]

[0103] In the formula, This is the current bow angle. Let be the target heading angle, and k be the sensitivity coefficient. , The range of values ​​is generally as follows: The values ​​within this range are used. When the value obtained through cosine calculation is negative, it provides negative feedback to the unmanned surface vessel (USV), meaning that the current behavior of the USV is a constraint on its behavior; conversely, it encourages the USV's behavior.

[0104] In this embodiment, calculating the minimum distance between the current position of the unmanned surface vessel and the prior reference path includes:

[0105] The minimum distance between the current position of the unmanned surface vessel and the prior reference path is calculated according to the following formula, including:

[0106] Calculate the Euclidean distance from the current position of the unmanned surface vessel to any path node on the prior reference path according to the following formula. :

[0107]

[0108] In the formula, and These are the x and y coordinates of the unmanned surface vessel's current position, respectively. and These are the x and y coordinates of a path node on the prior reference path, respectively.

[0109] Iterate through all path nodes on the prior reference path, repeat the above calculation steps to obtain the corresponding Euclidean distance set, and select the minimum value in the Euclidean distance set as the minimum distance between the current position of the unmanned surface vessel and the prior reference path. .

[0110] As an example, in the specific implementation process, the RRT algorithm is first used as the basic algorithm for acquiring prior knowledge. Before training, an optimal path from the starting point to the destination under the given environmental factors is obtained as the target RRT path curve. Subsequently, the current actual coordinates of the unmanned surface vessel (USV) and the coordinates on the target RRT path curve are obtained. The Euclidean distances between the current position of the USV and each point on the target RRT path curve are calculated, and the minimum value is selected.

[0111] Understandably, by calculating the minimum distance from the current position of the unmanned surface vessel (USV) to the prior reference path, the actual navigation state of the USV can be closely linked to prior knowledge. This distance constraint mechanism provides a gradient-based quantitative basis for subsequent hierarchical rewards, enabling the USV to continuously converge using prior knowledge during training. This effectively solves the problems of slow convergence of the optimal path and easy deviation of the ship's trajectory from the destination caused by the diversification of reward values ​​in traditional methods, significantly improving the learning and update rates of reinforcement learning algorithms.

[0112] In this embodiment, a tiered region threshold is set based on the total distance between the starting point and the ending point. The specific interval within which the minimum distance falls under the tiered region threshold is determined. A corresponding gradient reward coefficient is selected based on the interval to calculate the tiered guidance reward value, including:

[0113] The Euclidean distance d between the starting and ending coordinates of the unmanned surface vessel (USV) is calculated using the following formula:

[0114]

[0115] In the formula, and These are the x and y coordinates of the starting position of the unmanned surface vessel, respectively. and Let x and y be the x and y coordinates of the unmanned surface vessel's endpoint, respectively.

[0116] Based on the Euclidean distance Determine the layered region threshold H, and divide the range determined by the layered region threshold H into several intervals with different distance gradients;

[0117] The minimum distance from the current position of the unmanned surface vessel to the prior reference path. The minimum distance is determined by comparing it with the intervals of different distance gradients. The specific interval in which it is located.

[0118] It should be noted that H is determined by the Euclidean distance d between the starting and ending points, the size of the environment, and prior knowledge. Prior knowledge refers to the experiential information acquired by the unmanned surface vessel (USV) through interaction with the environment via other learning methods before actual operation.

[0119] Understandably, this embodiment introduces the RRT algorithm as prior knowledge to perform regional gradient-based reward values, transforming the sparse reward values ​​obtained by the unmanned surface vessel (USV) into a hierarchical mutation reward mechanism. This design emphasizes the reward gradients brought by different regions, breaks the data correlation at edge positions, and gradually reduces the exploration range, thereby improving the USV's tendency to blindly explore. This provides guidance for the USV's training, enabling it to reach the endpoint more efficiently, while significantly improving the learning and update rates of the reinforcement learning algorithm.

[0120] like Figure 3 As shown, Figure 3 The calculation level guidance reward value provided in the embodiments of the present invention The flowchart illustrates the process. The hierarchical guidance reward value is calculated using the following formula. :

[0121]

[0122] In the formula, As a preset constant, This represents the gradient reward coefficient.

[0123] In this embodiment, the rules for determining the gradient reward coefficient are as follows:

[0124] .

[0125] In this embodiment, the total reward value at the current moment is calculated based on the heading deviation reward value and the hierarchical guidance reward value, including:

[0126] The timing efficiency penalty is calculated according to the following formula. :

[0127]

[0128] in, The penalty / reward coefficient, t Number of steps used in the current training round The total number of steps allowed to reach the finish line. Let be the Euclidean distance between the current position and the destination of the unmanned surface vessel. The initial distance between the unmanned surface vessel and the destination;

[0129] The total reward value at the current moment is calculated according to the following formula. :

[0130]

[0131] in, , , These are respectively: ship deviation reward value, hierarchical guidance reward value, and timing efficiency penalty. , and These are the corresponding weighting coefficients.

[0132] It should be noted that by introducing hyperparameters , , As a weighting coefficient, the weights of each reward item are dynamically adjusted, thereby enabling refined guidance of the unmanned surface vessel's behavioral learning process. Specifically, increasing... It will enhance the heading deviation angle bonus. The impact encourages ships to more strictly track the target course, improving the stability of path tracking; adjustment Controllable tiered rewards The intensity of the contribution affects the ship's hierarchical decision-making ability in a structured environment with multiple obstacles; while adjusting This can significantly change the timing efficiency penalty. The weight of each factor is used to weigh the ship's motion efficiency and safety, with higher factors being considered. This strengthens the time penalty, prompting ships to complete tasks faster, but may sacrifice some path smoothness. By dynamically optimizing these weighting coefficients, the system can adapt to the needs of different mission scenarios, balance the contradiction between exploration and utilization, and ultimately improve the learning efficiency and strategy performance of unmanned surface vessels in complex environments. The weight represents the contribution of the reward value to the total reward value and its effectiveness in regulating ship behavior.

[0133] In this embodiment, the reinforcement learning model includes an action space and a state space; the action space includes several discrete rudder angle values, and the state space includes the unmanned surface vessel's position coordinates, heading angle, and speed information.

[0134] The action policy of the preset reinforcement learning model is updated using the total reward value, and the rudder angle control command of the unmanned surface vessel is output according to the updated action policy, including:

[0135] Construct a Q-value table to map the correspondence between the state space and the action space;

[0136] Using the calculated total reward value at the current moment, the action value values ​​in the Q-value table corresponding to the current state and the currently executed rudder angle action are corrected to obtain a Q-value table containing the updated action values.

[0137] Obtain the real-time status information of the unmanned surface vessel at the next moment after performing the current action;

[0138] Based on the real-time status information of the next moment, retrieve the Q-value table containing the updated action value, and obtain the action value value corresponding to all candidate rudder angle actions in the action space under this state;

[0139] Compare the action value values ​​corresponding to all candidate rudder angle actions, select the rudder angle with the highest action value value as the rudder angle control command for the unmanned surface vessel at the next moment, and output it.

[0140] Example 3

[0141] like Figure 4 As shown, this embodiment proposes an unmanned surface vessel (USV) path planning device, which is applied to the USV path planning method described in the above embodiment, including: a generation module 100, an acquisition module 200, a first calculation module 300, a second calculation module 400, a third calculation module 500, a fourth calculation module 600, an output module 700, a control module 800, and a loop control module 900.

[0142] The generation module 100 generates a static, collision-free path connecting the start and end points as a priori reference path to guide the unmanned surface vessel's (USV) navigation; the acquisition module 200 acquires the USV's current position coordinates and current heading angle; the first calculation module 300 calculates a heading deviation reward value based on the current position coordinates and the current heading angle; the second calculation module 400 calculates the minimum distance between the USV's current position and the priori reference path; and the third calculation module 500 sets a layered region threshold based on the total distance between the start and end points, determines the specific interval within which the minimum distance falls under the layered region threshold, and selects the corresponding gradient reward coefficient based on the interval to calculate the layered guidance reward value. The four calculation modules 600 are used to calculate the total reward value at the current moment based on the heading deviation reward value and the hierarchical guidance reward value; the output module 700 is used to update the action policy of the preset reinforcement learning model using the total reward value, and output the rudder angle control command of the unmanned surface vessel according to the updated action policy; the control module 800 is used to control the unmanned surface vessel to navigate to the next moment state according to the rudder angle control command; the loop control module 900 is used to determine whether the unmanned surface vessel has reached the destination, and if it has not reached the destination, trigger the acquisition module, the first calculation module, the second calculation module, the third calculation module, the fourth calculation module, the output module and the control module to run repeatedly until the unmanned surface vessel reaches the destination.

[0143] It should be noted that the foregoing explanation of the unmanned surface vessel path planning method embodiment also applies to the unmanned surface vessel path planning device of this embodiment, and will not be repeated here.

[0144] Example 4

[0145] Figure 5This is a schematic diagram of the structure of the electronic device 1000 provided in this embodiment. The electronic device 1000 includes: a memory 1001, a processor 1002, and a computer program stored in the memory 1001 and executable on the processor 1002.

[0146] When the processor 1002 executes the program, it implements the unmanned surface vessel path planning method provided in the above embodiments.

[0147] Furthermore, the electronic device 1000 also includes a communication interface 1003 for communication between the memory 1001 and the processor 1002.

[0148] The memory 1001 may include high-speed RAM (Random Access Memory) and may also include non-volatile memory, such as at least one disk storage.

[0149] If the memory 1001, processor 1002, and communication interface 1003 are implemented independently, then the communication interface 1003, memory 1001, and processor 1002 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0150] Optionally, in a specific implementation, if the memory 1001, processor 1002, and communication interface 1003 are integrated on a single chip, then the memory 1001, processor 1002, and communication interface 1003 can communicate with each other through an internal interface.

[0151] The processor 1002 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention.

[0152] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described unmanned surface vessel path planning method.

[0153] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0154] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0155] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0156] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0157] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0158] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. An unmanned surface vehicle path planning method, characterized in that, The method comprises the following steps: S1: generating a static non-collision path connecting the starting point and the ending point as a prior reference path for guiding the navigation of the unmanned ship; S2: obtaining the current position coordinates and the current heading angle of the unmanned ship; S3: Calculate a heading deviation reward value based on the current position coordinates and the current heading angle : In the formula, is the current bow heading angle, is the target heading angle, k is the sensitivity coefficient; S4: calculating the minimum distance between the current position of the unmanned ship and the prior reference path; S5: setting a hierarchical region threshold value according to the total distance between the starting point and the ending point, judging the specific interval of the minimum distance falling into the hierarchical region threshold value, and selecting a corresponding gradient reward coefficient according to the interval to calculate a hierarchical guidance reward value, comprising: The Euclidean distance between the coordinates of the starting position and the coordinates of the end position of the unmanned surface vehicle is calculated according to the following formula : wherein, and are the horizontal and vertical coordinates of the starting position of the unmanned boat, and are the horizontal and vertical coordinates of the ending position of the unmanned boat; based on the euclidean distance determining a layered region threshold H, and dividing a range determined by the layered region threshold H into several intervals having different distance gradients; a minimum distance from a current position of the unmanned surface vehicle to a priori reference path comparing the different distance gradients to intervals, determining the minimum distance a specific interval in which the minimum distance is located The hierarchical guidance reward value is calculated according to the following formula : In the formula, is a preset constant, is a gradient reward coefficient; S6: calculating a total reward value at the current time based on the heading deviation reward value and the hierarchical guidance reward value; S7: updating the action strategy of the preset reinforcement learning model by using the total reward value, and outputting the rudder angle control instruction of the unmanned ship according to the updated action strategy; S8: controlling the unmanned ship to navigate to the next time state according to the rudder angle control instruction; S9: cyclically executing S2-S8 until the unmanned ship reaches the ending point.

2. The unmanned surface vehicle path planning method of claim 1, wherein, The method comprises the following steps: The starting point of the unmanned ship is set as the root node of the rapid exploration random tree, and the root node is added to the node set; A state sampling point Xrand is randomly generated in the free space of the virtual environment digital map; All existing nodes in the node set are traversed, the distances between the nodes and the state sampling point Xrand are calculated, and the node with the smallest distance value is selected as the nearest node Xnear; A new node Xnew is calculated in the extension direction by taking the nearest node Xnear as the starting point and extending a preset growth step length in the direction pointing to the state sampling point Xrand; It is detected whether the path segment connecting the nearest node Xnear and the new node Xnew intersects with the obstacles in the virtual environment digital map; If there is no intersection, the new node Xnew is added to the node set, and the new node Xnew is connected to the nearest node Xnear as a child node; if there is an intersection, the new node Xnew is discarded; The steps are cyclically executed until the distance between the newly added node and the ending point is less than a preset threshold range, the newly added node is connected to the ending point, and the connection line from the starting point to the ending point is extracted as the prior reference path.

3. The method of claim 1, wherein, The minimum distance between the current position of the unmanned ship and the prior reference path is calculated, comprising: The Euclidean distance from the current position of the unmanned surface vehicle to any one path node on the prior reference path is calculated according to the following formula : wherein, and are the horizontal and vertical coordinates of the current position of the unmanned boat, and are the horizontal and vertical coordinates of a path node on the priori reference path, respectively. Traverse all path nodes on the prior reference path, repeat the above calculation steps to obtain a corresponding set of Euclidean distances, and select the minimum value in the set of Euclidean distances as the minimum distance between the current position of the unmanned ship and the prior reference path .

4. The method of claim 3, wherein, The gradient reward coefficient has the following value rules: 。 5. The method of claim 4, wherein, The total reward value at the current time is calculated based on the heading deviation reward value and the hierarchical guidance reward value, comprising: The timing efficiency penalty is calculated according to the following formula : wherein, is a penalty reward coefficient, t is the number of steps used in the current training episode, is the total number of steps allowed to reach the goal, is the Euclidean distance between the current position of the USV and the goal, is the initial distance of the USV from the goal; The total reward value at the current time instant is given by : wherein, , , are a ship deviation reward value, a hierarchical guidance reward value and a timing efficiency penalty, respectively, , and are corresponding weight coefficients, respectively.

6. The method of claim 1-5, wherein, The reinforcement learning model comprises an action space and a state space; the action space comprises a plurality of discrete rudder angle values, and the state space comprises unmanned ship position coordinates, heading angle and speed information; The action strategy of the preset reinforcement learning model is updated by using the total reward value, and the rudder angle control instruction of the unmanned ship is output according to the updated action strategy, comprising: A Q value table for mapping the corresponding relationship between the state space and the action space is constructed; The total reward value at the current moment is calculated, and the action value corresponding to the current moment state and the current executed rudder angle action in the Q value table is corrected to obtain a Q value table containing updated action values; Real-time state information of the next moment after the unmanned ship shifts to the current action is obtained; According to the real-time state information of the next moment, the Q value table containing the updated action values is searched to obtain the action value corresponding to all candidate rudder angle actions in the action space under the state; The action values corresponding to all candidate rudder angle actions are compared, and the rudder angle with the maximum action value is selected as the rudder angle control instruction of the unmanned ship at the next moment and is output.

7. An unmanned surface vehicle path planning device, applied to the unmanned surface vehicle path planning method according to any one of claims 1-6, characterized in that, Comprise: The generating module is used for generating a static non-collision path connecting the starting point and the ending point as a priori reference path for guiding the unmanned ship to sail; The acquisition module is used for acquiring the current position coordinates and the current heading angle of the unmanned ship; The first calculation module is used for calculating a heading deviation reward value according to the current position coordinates and the current heading angle; The second calculation module is used for calculating the minimum distance between the current position of the unmanned ship and the a priori reference path; The third calculation module is used for setting a hierarchical region threshold value according to the total distance between the starting point and the ending point, judging the specific interval of the minimum distance falling into the hierarchical region threshold value, and selecting a corresponding gradient reward coefficient according to the interval to calculate a hierarchical guidance reward value; The fourth calculation module is used for calculating a total reward value at the current moment based on the heading deviation reward value and the hierarchical guidance reward value; The output module is used for updating the action strategy of a preset reinforcement learning model by using the total reward value, and outputting the rudder angle control instruction of the unmanned ship according to the updated action strategy; The control module is used for controlling the unmanned ship to sail to the next moment state according to the rudder angle control instruction; The loop control module is used for determining whether the unmanned ship reaches the ending point, and triggering the acquisition module, the first calculation module, the second calculation module, the third calculation module, the fourth calculation module, the output module and the control module to repeatedly run when the unmanned ship does not reach the ending point, until the unmanned ship reaches the ending point.

8. An electronic device, comprising: The electronic device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements the operations performed by the unmanned ship path planning method according to any one of claims 1 to 6 when executing the computer program.

Images