Unmanned ship path planning method, device and equipment

Abstract

The invention relates to the technical field of unmanned ships, and discloses an unmanned ship path planning method, device and equipment, and the method comprises the steps: generating a prior reference path; obtaining a current position coordinate and a current prow orientation angle, and calculating a course deviation award value; calculating the minimum distance between the current position of the unmanned ship and the prior reference path; setting a hierarchical region threshold value according to the total distance between the starting point and the ending point, judging a specific interval in which the minimum distance falls into the hierarchical region threshold value, and selecting a corresponding gradient reward coefficient according to the interval to calculate a hierarchical guide reward value; calculating a total reward value at the current moment; and updating an action strategy of a preset reinforcement learning model by using the total reward value, and outputting a rudder angle control instruction of the unmanned ship according to the updated action strategy. According to the method, the problems of slow search convergence and poor path planning quality in a complex environment in the prior art can be effectively solved, and the efficiency and feasibility of an unmanned ship path planning technology are remarkably improved.

Description

Technical Field

[0001] This invention relates to the field of unmanned surface vessel (USV) technology, and more specifically, to a USV path planning method, apparatus, and equipment. Background Technology

[0002] With increasing maritime traffic density, various maritime accidents are occurring frequently. Statistics show that the vast majority of these accidents are caused by human factors. Therefore, improving the level of autonomous navigation for ships has become a critical issue that urgently needs to be addressed. Unmanned surface vessels (USVs), as a type of vessel system that does not require human piloting, are better able to adapt to the complex and challenging marine environment compared to manned vessels, giving them unique advantages in mission execution. Path planning, as one of the core technologies for autonomous navigation of USVs, aims to plan an optimized navigation path to ensure that the USV can reach its target location quickly and safely. This is of great significance for improving its overall combat capabilities and mission completion efficiency.

[0003] Existing unmanned surface vessel (USV) path planning technologies mainly include traditional methods such as the A* algorithm and Dijkstra's algorithm, as well as the RRT (Rapid Expanding Random Tree) algorithm, which has been widely used in recent years based on sampling optimization. Although the RRT algorithm overcomes the shortcomings of traditional algorithms in getting trapped in local optima in complex environments to some extent and can better explore the state space, it still falls short in complex ocean dynamic environments when dealing with wind and wave interference and obstacle avoidance. Moreover, existing strategies often generate excessive rudder angles in autonomous driving control, increasing the risk of ship capsizing and resulting in low path planning efficiency and feasibility. Summary of the Invention

[0004] To overcome the shortcomings of low efficiency and feasibility in existing unmanned surface vessel (USV) path planning technologies, this invention proposes the following technical solution: In the first aspect, this invention proposes an unmanned surface vessel (USV) path planning method, comprising: S1: Generate a static, collision-free path connecting the start and end points as a priori reference path to guide the navigation of the unmanned surface vessel; S2: Obtain the current position coordinates and current bow angle of the unmanned surface vessel; S3: Calculate the heading deviation bonus value based on the current position coordinates and the current heading angle; S4: Calculate the minimum distance between the current position of the unmanned surface vessel and the prior reference path; S5: Set a tiered region threshold based on the total distance between the starting point and the ending point, determine the specific interval in which the minimum distance falls within the tiered region threshold, and select the corresponding gradient reward coefficient based on the interval to calculate the tiered guidance reward value. S6: Calculate the total reward value at the current moment based on the heading deviation reward value and the hierarchical guidance reward value; S7: Update the action policy of the preset reinforcement learning model using the total reward value, and output the rudder angle control command of the unmanned surface vessel according to the updated action policy; S8: Control the unmanned surface vessel to navigate to the next moment state according to the rudder angle control command; S9: Repeat S2~S8 until the unmanned surface vessel reaches the destination.

[0005] As a preferred technical solution, a static, collision-free path connecting the starting point and the ending point is generated as a priori reference path to guide the navigation of the unmanned surface vessel, including: Set the starting point of the unmanned surface vessel as the root node of a fast-exploration random tree, and add the root node to the node set; A state sampling point Xrand is randomly generated within the free space of the digital map in the virtual environment; Traverse all existing nodes in the node set, calculate the distance between each node and the state sampling point Xrand, and select the node with the smallest distance value as the nearest node Xnear. Starting from the nearest node Xnear, extend the growth step by a preset amount in the direction pointing to the state sampling point Xrand, and calculate the new node Xnew in the extension direction. Detect whether the path segment connecting the nearest node Xnear and the new node Xnew intersects with an obstacle in the digital map of the virtual environment; If no intersection exists, the new node Xnew is added to the node set and connected as a child node to the nearest node Xnear; if an intersection exists, the new node Xnew is discarded. The steps are repeated until the distance between the newly added node and the endpoint is less than a preset threshold range. Then, the newly added node is connected to the endpoint, and the connection from the starting point to the endpoint is extracted as a priori reference path.

[0006] As a preferred technical solution, the heading deviation bonus value is calculated according to the following formula. :

[0007] In the formula, This is the current bow angle. Let be the target heading angle, and k be the sensitivity coefficient.

[0008] As a preferred technical solution, calculating the minimum distance between the current position of the unmanned surface vessel and the prior reference path includes: Calculate the Euclidean distance from the current position of the unmanned surface vessel to any path node on the prior reference path according to the following formula. :

[0009] In the formula, and These are the x and y coordinates of the unmanned surface vessel's current position, respectively. and These are the x and y coordinates of a path node on the prior reference path, respectively. Iterate through all path nodes on the prior reference path, repeat the above calculation steps to obtain the corresponding Euclidean distance set, and select the minimum value in the Euclidean distance set as the minimum distance between the current position of the unmanned surface vessel and the prior reference path. .

[0010] As a preferred technical solution, a tiered region threshold is set based on the total distance between the starting point and the ending point. The specific interval within which the minimum distance falls under the tiered region threshold is determined. A corresponding gradient reward coefficient is selected based on the interval to calculate the tiered guidance reward value, including: The Euclidean distance d between the starting and ending coordinates of the unmanned surface vessel (USV) is calculated using the following formula:

[0011] In the formula, and These are the x and y coordinates of the starting position of the unmanned surface vessel, respectively. and Let x and y be the x and y coordinates of the unmanned surface vessel's endpoint, respectively. Based on the Euclidean distance Determine the layered region threshold H, and divide the range determined by the layered region threshold H into several intervals with different distance gradients; The minimum distance from the current position of the unmanned surface vessel to the prior reference path. The minimum distance is determined by comparing it with the intervals of different distance gradients. The specific interval it is located in; Calculate the tiered guidance reward value according to the following formula. :

[0012] In the formula, As a preset constant, This represents the gradient reward coefficient.

[0013] As a preferred technical solution, the rules for determining the gradient reward coefficient are as follows: .

[0014] As a preferred technical solution, the total reward value at the current moment is calculated based on the heading deviation reward value and the hierarchical guidance reward value, including: The timing efficiency penalty is calculated according to the following formula. :

[0015] in, The penalty / reward coefficient, t Number of steps used in the current training round The total number of steps allowed to reach the finish line. Let be the Euclidean distance between the current position and the destination of the unmanned surface vessel. The initial distance between the unmanned surface vessel and the destination; The total reward value at the current moment is calculated according to the following formula. :

[0016] in, , , These are respectively: ship deviation reward value, hierarchical guidance reward value, and timing efficiency penalty. , and These are the corresponding weighting coefficients.

[0017] As a preferred technical solution, the reinforcement learning model includes an action space and a state space; the action space includes several discrete rudder angle values, and the state space includes the unmanned surface vessel's position coordinates, heading angle, and velocity information; The action policy of the preset reinforcement learning model is updated using the total reward value, and the rudder angle control command of the unmanned surface vessel is output according to the updated action policy, including: Construct a Q-value table to map the correspondence between the state space and the action space; Using the calculated total reward value at the current moment, the action value values ​​in the Q-value table corresponding to the current state and the currently executed rudder angle action are corrected to obtain a Q-value table containing the updated action values. Obtain the real-time status information of the unmanned surface vessel at the next moment after performing the current action; Based on the real-time status information of the next moment, retrieve the Q-value table containing the updated action value, and obtain the action value value corresponding to all candidate rudder angle actions in the action space under this state; Compare the action value values ​​corresponding to all candidate rudder angle actions, select the rudder angle with the highest action value value as the rudder angle control command for the unmanned surface vessel at the next moment, and output it.

[0018] Secondly, the present invention also proposes an unmanned surface vessel (USV) path planning device, applied in the USV path planning method as described in any of the embodiments of the first aspect, comprising: The generation module is used to generate a static, collision-free path connecting the start and end points as a priori reference path to guide the navigation of the unmanned surface vessel. The acquisition module is used to acquire the current position coordinates and current bow angle of the unmanned surface vessel. The first calculation module is used to calculate the heading deviation bonus value based on the current position coordinates and the current heading angle. The second calculation module is used to calculate the minimum distance between the current position of the unmanned surface vessel and the prior reference path; The third calculation module is used to set a stratified region threshold based on the total distance between the starting point and the ending point, determine the specific interval in which the minimum distance falls within the stratified region threshold, and select the corresponding gradient reward coefficient based on the interval to calculate the stratified guidance reward value. The fourth calculation module is used to calculate the total reward value at the current moment based on the heading deviation reward value and the hierarchical guidance reward value; The output module is used to update the action policy of the preset reinforcement learning model using the total reward value, and output the rudder angle control command of the unmanned surface vessel according to the updated action policy. The control module is used to control the unmanned surface vessel to navigate to the next moment state according to the rudder angle control command; The loop control module is used to determine whether the unmanned surface vessel has reached the destination, and if it has not reached the destination, it triggers the acquisition module, the first calculation module, the second calculation module, the third calculation module, the fourth calculation module, the output module and the control module to run repeatedly until the unmanned surface vessel reaches the destination.

[0019] Thirdly, the present invention also proposes an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the operations performed by the unmanned surface vessel path planning method as described in any of the embodiments of the first aspect.

[0020] The beneficial effects of the present invention include at least the following: This invention provides global navigation guidance for unmanned surface vessels (USVs) by generating a static, collision-free path connecting the start and end points as a priori reference path, effectively overcoming the limitations of traditional reinforcement learning in its initial blind exploration. Furthermore, by introducing a heading deviation reward value, the heading of the USV can be constrained in real time, avoiding navigation instability caused by excessive rudder angle control or large heading fluctuations. Further, by combining a hierarchical region threshold set based on the total distance between the start and end points, gradient reward coefficients are selected differently according to the distance the USV deviates from the priori reference path to calculate hierarchical guidance reward values, constructing a hierarchical reward mechanism with clear gradient guidance. This mechanism can transform sparse rewards in the environment into highly guiding dense gradient rewards, prompting the USV to quickly converge to the priori reference path during training, thereby continuously correcting its action strategy in dynamic interactions. This effectively solves the problems of slow search convergence and poor path planning quality in complex environments in existing technologies, significantly improving the efficiency and feasibility of USV path planning technology. Attached Figure Description

[0021] Figure 1 This is a flowchart illustrating the unmanned surface vessel path planning method provided in an embodiment of the present invention.

[0022] Figure 2 This is a schematic diagram of the process for generating a priori reference paths provided in an embodiment of the present invention.

[0023] Figure 3 The calculation level guidance reward value provided in the embodiments of the present invention A flowchart.

[0024] Figure 4 This is an architectural diagram of the unmanned surface vessel path planning device provided in an embodiment of the present invention.

[0025] Figure 5 This is a schematic diagram of the structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0026] The embodiments of the present invention will be described below with reference to the accompanying drawings and preferred technical solutions. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be understood that the preferred technical solutions are only for illustrating the present invention and not for limiting the scope of protection of the present invention.

[0027] It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Therefore, the drawings only show the components related to the present invention and are not drawn according to the actual number, shape and size of the components in the actual implementation. In the actual implementation, the form, quantity and proportion of each component can be arbitrarily changed, and the layout of the components may also be more complex.

[0028] In the following description, numerous details are explored to provide a more thorough explanation of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details. In other embodiments, well-known structures and devices are shown in block diagram form rather than in detail to avoid obscuring embodiments of the invention.

[0029] Example 1 This embodiment proposes an unmanned surface vessel (USV) path planning method, such as... Figure 1 As shown, Figure 1 This is a flowchart illustrating an unmanned surface vessel (USV) path planning method provided in this embodiment. The method includes the following steps: S1: Generate a static, collision-free path connecting the start and end points as a priori reference path to guide the navigation of the unmanned surface vessel; S2: Obtain the current position coordinates and current bow angle of the unmanned surface vessel; S3: Calculate the heading deviation bonus value based on the current position coordinates and the current heading angle; S4: Calculate the minimum distance between the current position of the unmanned surface vessel and the prior reference path; S5: Set a tiered region threshold based on the total distance between the starting point and the ending point, determine the specific interval in which the minimum distance falls within the tiered region threshold, and select the corresponding gradient reward coefficient based on the interval to calculate the tiered guidance reward value. S6: Calculate the total reward value at the current moment based on the heading deviation reward value and the hierarchical guidance reward value; S7: Update the action policy of the preset reinforcement learning model using the total reward value, and output the rudder angle control command of the unmanned surface vessel according to the updated action policy; S8: Control the unmanned surface vessel to navigate to the next moment state according to the rudder angle control command; S9: Repeat S2~S8 until the unmanned surface vessel reaches the destination.

[0030] Understandably, by generating a static, collision-free path connecting the start and end points as a priori reference path, global navigation guidance is provided for the unmanned surface vessel (USV), effectively overcoming the limitations of traditional reinforcement learning in its initial blind exploration. Furthermore, by introducing a heading deviation reward value, the USV's bow orientation can be constrained in real time, preventing navigation instability caused by excessive rudder angle control or large heading fluctuations. Moreover, by combining a hierarchical region threshold set based on the total distance between the start and end points, gradient reward coefficients are selectively chosen based on the distance the USV deviates from the priori reference path to calculate hierarchical guidance reward values, constructing a hierarchical reward mechanism with clear gradient guidance. This mechanism can transform sparse rewards in the environment into highly guiding dense gradient rewards, prompting the USV to quickly converge to the priori reference path during training, thereby continuously correcting its action strategy in dynamic interactions. This effectively solves the problems of slow search convergence and poor path planning quality in complex environments in existing technologies, significantly improving the efficiency and feasibility of USV path planning technology.

[0031] Example 2 This embodiment is an improvement on the unmanned surface vessel path planning method proposed in Embodiment 1.

[0032] In this embodiment, generating a static, collision-free path connecting the start and end points serves as a priori reference path to guide the navigation of the unmanned surface vessel, including: Set the starting point of the unmanned surface vessel as the root node of a fast-exploration random tree, and add the root node to the node set; A state sampling point Xrand is randomly generated within the free space of the digital map in the virtual environment; Traverse all existing nodes in the node set, calculate the distance between each node and the state sampling point Xrand, and select the node with the smallest distance value as the nearest node Xnear. Starting from the nearest node Xnear, extend the growth step by a preset amount in the direction pointing to the state sampling point Xrand, and calculate the new node Xnew in the extension direction. Detect whether the path segment connecting the nearest node Xnear and the new node Xnew intersects with an obstacle in the digital map of the virtual environment; If no intersection exists, the new node Xnew is added to the node set and connected as a child node to the nearest node Xnear; if an intersection exists, the new node Xnew is discarded. The steps are repeated until the distance between the newly added node and the endpoint is less than a preset threshold range. Then, the newly added node is connected to the endpoint, and the connection from the starting point to the endpoint is extracted as a priori reference path.

[0033] As an example, in the specific implementation process, such as Figure 2 As shown, Figure 2 This is a schematic diagram of the process for generating a priori reference paths provided in this embodiment of the invention. First, a virtual environment digitization platform is established. The size of the map, the size and coordinates of static obstacles, the waterway, and the coordinates of both banks of the waterway are initialized. The start and end points of the unmanned surface vessel (USV) are added, and its motion state is initialized. The start point of the USV is placed into the growth tree node set as the root node of the RRT algorithm. An initial growth step size is set according to the known environmental information. When the map is large, the step size is increased to avoid slow search speed; when the map area is small, the step size is decreased to prevent collisions with obstacles. Then, state points are randomly sampled from the environment, and the nearest node in the tree is found. A new node is generated by growing one step size in the direction connecting the two nodes. If the path does not pass through an obstacle, it is added to the tree. This process is repeated until the distance from the newly generated node to the target point is less than one step size. The new node is then directly connected to the target point, thus obtaining an RRT path planning curve connecting the start and end points.

[0034] Understandably, the RRT algorithm, as a sampling optimization technique, is better able to explore the state space and generate feasible and near-optimal paths in complex environments compared to traditional algorithms. This embodiment introduces the RRT algorithm for training, using relevant data on interaction with the environment as prior knowledge for path planning of the unmanned surface vessel (USV). Utilizing the generated paths as a reference improves the USV's initial tendency to blindly explore, guiding its training and effectively enhancing path planning efficiency and the vessel's ability to learn towards its destination.

[0035] In this embodiment, the heading deviation bonus value is calculated according to the following formula. :

[0036] In the formula, This is the current bow angle. Let be the target heading angle, and k be the sensitivity coefficient. , The range of values ​​is generally as follows: The values ​​within this range are used. When the value obtained through cosine calculation is negative, it provides negative feedback to the unmanned surface vessel (USV), meaning that the current behavior of the USV is a constraint on its behavior; conversely, it encourages the USV's behavior.

[0037] In this embodiment, calculating the minimum distance between the current position of the unmanned surface vessel and the prior reference path includes: The minimum distance between the current position of the unmanned surface vessel and the prior reference path is calculated according to the following formula, including: Calculate the Euclidean distance from the current position of the unmanned surface vessel to any path node on the prior reference path according to the following formula. :

[0038] In the formula, and These are the x and y coordinates of the unmanned surface vessel's current position, respectively. and These are the x and y coordinates of a path node on the prior reference path, respectively. Iterate through all path nodes on the prior reference path, repeat the above calculation steps to obtain the corresponding Euclidean distance set, and select the minimum value in the Euclidean distance set as the minimum distance between the current position of the unmanned surface vessel and the prior reference path. .

[0039] As an example, in the specific implementation process, the RRT algorithm is first used as the basic algorithm for acquiring prior knowledge. Before training, an optimal path from the starting point to the destination under the given environmental factors is obtained as the target RRT path curve. Subsequently, the current actual coordinates of the unmanned surface vessel (USV) and the coordinates on the target RRT path curve are obtained. The Euclidean distances between the current position of the USV and each point on the target RRT path curve are calculated, and the minimum value is selected.

[0040] Understandably, by calculating the minimum distance from the current position of the unmanned surface vessel (USV) to the prior reference path, the actual navigation state of the USV can be closely linked to prior knowledge. This distance constraint mechanism provides a gradient-based quantitative basis for subsequent hierarchical rewards, enabling the USV to continuously converge using prior knowledge during training. This effectively solves the problems of slow convergence of the optimal path and easy deviation of the ship's trajectory from the destination caused by the diversification of reward values ​​in traditional methods, significantly improving the learning and update rates of reinforcement learning algorithms.

[0041] In this embodiment, a tiered region threshold is set based on the total distance between the starting point and the ending point. The specific interval within which the minimum distance falls under the tiered region threshold is determined. A corresponding gradient reward coefficient is selected based on the interval to calculate the tiered guidance reward value, including: The Euclidean distance d between the starting and ending coordinates of the unmanned surface vessel (USV) is calculated using the following formula:

[0042] In the formula, and These are the x and y coordinates of the starting position of the unmanned surface vessel, respectively. and Let x and y be the x and y coordinates of the unmanned surface vessel's endpoint, respectively. Based on the Euclidean distance Determine the layered region threshold H, and divide the range determined by the layered region threshold H into several intervals with different distance gradients; The minimum distance from the current position of the unmanned surface vessel to the prior reference path. The minimum distance is determined by comparing it with the intervals of different distance gradients. The specific interval in which it is located.

[0043] It should be noted that H is determined by the Euclidean distance d between the starting and ending points, the size of the environment, and prior knowledge. Prior knowledge refers to the experiential information acquired by the unmanned surface vessel (USV) through interaction with the environment via other learning methods before actual operation.

[0044] Understandably, this embodiment introduces the RRT algorithm as prior knowledge to perform regional gradient-based reward values, transforming the sparse reward values ​​obtained by the unmanned surface vessel (USV) into a hierarchical mutation reward mechanism. This design emphasizes the reward gradients brought by different regions, breaks the data correlation at edge positions, and gradually reduces the exploration range, thereby improving the USV's tendency to blindly explore. This provides guidance for the USV's training, enabling it to reach the endpoint more efficiently, while significantly improving the learning and update rates of the reinforcement learning algorithm.

[0045] like Figure 3 As shown, Figure 3 The calculation level guidance reward value provided in the embodiments of the present invention The flowchart illustrates the process. The hierarchical guidance reward value is calculated using the following formula. :

[0046] In the formula, As a preset constant, This represents the gradient reward coefficient.

[0047] In this embodiment, the rules for determining the gradient reward coefficient are as follows: .

[0048] In this embodiment, the total reward value at the current moment is calculated based on the heading deviation reward value and the hierarchical guidance reward value, including: The timing efficiency penalty is calculated according to the following formula. :

[0049] in, The penalty / reward coefficient, t Number of steps used in the current training round The total number of steps allowed to reach the finish line. Let be the Euclidean distance between the current position and the destination of the unmanned surface vessel. The initial distance between the unmanned surface vessel and the destination; The total reward value at the current moment is calculated according to the following formula. :

[0050] in, , , These are respectively: ship deviation reward value, hierarchical guidance reward value, and timing efficiency penalty. , and These are the corresponding weighting coefficients.

[0051] It should be noted that by introducing hyperparameters , , As a weighting coefficient, the weights of each reward item are dynamically adjusted, thereby enabling refined guidance of the unmanned surface vessel's behavioral learning process. Specifically, increasing... It will enhance the heading deviation angle bonus. The impact encourages ships to more strictly track the target course, improving the stability of path tracking; adjustment Controllable tiered rewards The intensity of the contribution affects the ship's hierarchical decision-making ability in a structured environment with multiple obstacles; while adjusting This can significantly change the timing efficiency penalty. The weight of each factor is used to weigh the ship's motion efficiency and safety, with higher factors being considered. This strengthens the time penalty, prompting ships to complete tasks faster, but may sacrifice some path smoothness. By dynamically optimizing these weighting coefficients, the system can adapt to the needs of different mission scenarios, balance the contradiction between exploration and utilization, and ultimately improve the learning efficiency and strategy performance of unmanned surface vessels in complex environments. The weight represents the contribution of the reward value to the total reward value and its effectiveness in regulating ship behavior.

[0052] In this embodiment, the reinforcement learning model includes an action space and a state space; the action space includes several discrete rudder angle values, and the state space includes the unmanned surface vessel's position coordinates, heading angle, and speed information.

[0053] The action policy of the preset reinforcement learning model is updated using the total reward value, and the rudder angle control command of the unmanned surface vessel is output according to the updated action policy, including: Construct a Q-value table to map the correspondence between the state space and the action space; Using the calculated total reward value at the current moment, the action value values ​​in the Q-value table corresponding to the current state and the currently executed rudder angle action are corrected to obtain a Q-value table containing the updated action values. Obtain the real-time status information of the unmanned surface vessel at the next moment after performing the current action; Based on the real-time status information of the next moment, retrieve the Q-value table containing the updated action value, and obtain the action value value corresponding to all candidate rudder angle actions in the action space under this state; Compare the action value values ​​corresponding to all candidate rudder angle actions, select the rudder angle with the highest action value value as the rudder angle control command for the unmanned surface vessel at the next moment, and output it.

[0054] Example 3 like Figure 4 As shown, this embodiment proposes an unmanned surface vessel (USV) path planning device, which is applied to the USV path planning method described in the above embodiment, including: a generation module 100, an acquisition module 200, a first calculation module 300, a second calculation module 400, a third calculation module 500, a fourth calculation module 600, an output module 700, a control module 800, and a loop control module 900.

[0055] The generation module 100 generates a static, collision-free path connecting the start and end points as a priori reference path to guide the unmanned surface vessel's (USV) navigation; the acquisition module 200 acquires the USV's current position coordinates and current heading angle; the first calculation module 300 calculates a heading deviation reward value based on the current position coordinates and the current heading angle; the second calculation module 400 calculates the minimum distance between the USV's current position and the priori reference path; and the third calculation module 500 sets a layered region threshold based on the total distance between the start and end points, determines the specific interval within which the minimum distance falls under the layered region threshold, and selects the corresponding gradient reward coefficient based on the interval to calculate the layered guidance reward value. The four calculation modules 600 are used to calculate the total reward value at the current moment based on the heading deviation reward value and the hierarchical guidance reward value; the output module 700 is used to update the action policy of the preset reinforcement learning model using the total reward value, and output the rudder angle control command of the unmanned surface vessel according to the updated action policy; the control module 800 is used to control the unmanned surface vessel to navigate to the next moment state according to the rudder angle control command; the loop control module 900 is used to determine whether the unmanned surface vessel has reached the destination, and if it has not reached the destination, trigger the acquisition module, the first calculation module, the second calculation module, the third calculation module, the fourth calculation module, the output module and the control module to run repeatedly until the unmanned surface vessel reaches the destination.

[0056] It should be noted that the foregoing explanation of the unmanned surface vessel path planning method embodiment also applies to the unmanned surface vessel path planning device of this embodiment, and will not be repeated here.

[0057] Example 4 Figure 5 This is a schematic diagram of the structure of the electronic device 1000 provided in this embodiment. The electronic device 1000 includes: a memory 1001, a processor 1002, and a computer program stored in the memory 1001 and executable on the processor 1002.

[0058] When the processor 1002 executes the program, it implements the unmanned surface vessel path planning method provided in the above embodiments.

[0059] Furthermore, the electronic device 1000 also includes a communication interface 1003 for communication between the memory 1001 and the processor 1002.

[0060] The memory 1001 may include high-speed RAM (Random Access Memory) and may also include non-volatile memory, such as at least one disk storage.

[0061] If the memory 1001, processor 1002, and communication interface 1003 are implemented independently, then the communication interface 1003, memory 1001, and processor 1002 can be interconnected via a bus to complete communication between them. The bus can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0062] Optionally, in a specific implementation, if the memory 1001, processor 1002, and communication interface 1003 are integrated on a single chip, then the memory 1001, processor 1002, and communication interface 1003 can communicate with each other through an internal interface.

[0063] The processor 1002 may be a CPU (Central Processing Unit), an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits configured to implement embodiments of the present invention.

[0064] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described unmanned surface vessel path planning method.

[0065] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0066] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0067] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or more N executable instructions for implementing custom logic functions or processes, and the scope of preferred embodiments of the invention includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of the invention pertain.

[0068] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it can be implemented using any of the following techniques known in the art, or a combination thereof: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (FPGAs), field-programmable gate arrays (FPGAs), etc.

[0069] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.

[0070] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A path planning method for unmanned surface vessels, characterized in that, include: S1: Generate a static, collision-free path connecting the start and end points as a priori reference path to guide the navigation of the unmanned surface vessel; S2: Obtain the current position coordinates and current bow angle of the unmanned surface vessel; S3: Calculate the heading deviation bonus value based on the current position coordinates and the current heading angle; S4: Calculate the minimum distance between the current position of the unmanned surface vessel and the prior reference path; S5: Set a tiered region threshold based on the total distance between the starting point and the ending point, determine the specific interval in which the minimum distance falls within the tiered region threshold, and select the corresponding gradient reward coefficient based on the interval to calculate the tiered guidance reward value. S6: Calculate the total reward value at the current moment based on the heading deviation reward value and the hierarchical guidance reward value; S7: Update the action policy of the preset reinforcement learning model using the total reward value, and output the rudder angle control command of the unmanned surface vessel according to the updated action policy; S8: Control the unmanned surface vessel to navigate to the next moment state according to the rudder angle control command; S9: Repeat S2~S8 until the unmanned surface vessel reaches the destination.

2. The unmanned surface vessel path planning method according to claim 1, characterized in that, Generate a static, collision-free path connecting the start and end points as a priori reference path to guide the navigation of the unmanned surface vessel, including: Set the starting point of the unmanned surface vessel as the root node of a fast-exploration random tree, and add the root node to the node set; A state sampling point Xrand is randomly generated within the free space of the digital map in the virtual environment; Traverse all existing nodes in the node set, calculate the distance between each node and the state sampling point Xrand, and select the node with the smallest distance value as the nearest node Xnear. Starting from the nearest node Xnear, extend the growth step by a preset amount in the direction pointing to the state sampling point Xrand, and calculate the new node Xnew in the extension direction. Detect whether the path segment connecting the nearest node Xnear and the new node Xnew intersects with an obstacle in the digital map of the virtual environment; If no intersection exists, the new node Xnew is added to the node set and connected as a child node to the nearest node Xnear; if an intersection exists, the new node Xnew is discarded. The steps are repeated until the distance between the newly added node and the endpoint is less than a preset threshold range. Then, the newly added node is connected to the endpoint, and the connection from the starting point to the endpoint is extracted as a priori reference path.

3. The unmanned surface vessel path planning method according to claim 1, characterized in that, The heading deviation bonus value is calculated according to the following formula. : In the formula, This is the current bow angle. Let be the target heading angle, and k be the sensitivity coefficient.

4. The unmanned surface vessel path planning method according to claim 1, characterized in that, Calculating the minimum distance between the current position of the unmanned surface vessel and the prior reference path includes: Calculate the Euclidean distance from the current position of the unmanned surface vessel to any path node on the prior reference path according to the following formula. : In the formula, and These are the x and y coordinates of the unmanned surface vessel's current position, respectively. and These are the x and y coordinates of a path node on the prior reference path, respectively. Iterate through all path nodes on the prior reference path, repeat the above calculation steps to obtain the corresponding Euclidean distance set, and select the minimum value in the Euclidean distance set as the minimum distance between the current position of the unmanned surface vessel and the prior reference path. .

5. The unmanned surface vessel path planning method according to claim 4, characterized in that, A tiered region threshold is set based on the total distance between the starting point and the ending point. The specific interval within which the minimum distance falls under the tiered region threshold is determined. Based on the interval, a corresponding gradient reward coefficient is selected to calculate the tiered guidance reward value, including: The Euclidean distance d between the starting and ending coordinates of the unmanned surface vessel (USV) is calculated using the following formula: In the formula, and These are the x and y coordinates of the starting position of the unmanned surface vessel, respectively. and Let x and y be the x and y coordinates of the unmanned surface vessel's endpoint, respectively. Based on the Euclidean distance Determine the layered region threshold H, and divide the range determined by the layered region threshold H into several intervals with different distance gradients; The minimum distance from the current position of the unmanned surface vessel to the prior reference path. The minimum distance is determined by comparing it with the intervals of different distance gradients. The specific interval it is located in; Calculate the tiered guidance reward value according to the following formula. : In the formula, As a preset constant, This represents the gradient reward coefficient.

6. The unmanned surface vessel path planning method according to claim 5, characterized in that, The rules for determining the gradient reward coefficient are as follows: 。 7. The unmanned surface vessel path planning method according to claim 5, characterized in that, Based on the heading deviation reward value and the hierarchical guidance reward value, calculate the total reward value at the current moment, including: The timing efficiency penalty is calculated according to the following formula. : in, The penalty / reward coefficient, t Number of steps used in the current training round The total number of steps allowed to reach the finish line. Let be the Euclidean distance between the current position and the destination of the unmanned surface vessel. The initial distance between the unmanned surface vessel and the destination; The total reward value at the current moment is calculated according to the following formula. : in, , , These are respectively: ship deviation reward value, hierarchical guidance reward value, and timing efficiency penalty. , and These are the corresponding weighting coefficients.

8. The unmanned surface vessel path planning method according to any one of claims 1 to 7, characterized in that, The reinforcement learning model includes an action space and a state space; the action space includes several discrete rudder angle values, and the state space includes the unmanned surface vessel's position coordinates, heading angle, and velocity information. The action policy of the preset reinforcement learning model is updated using the total reward value, and the rudder angle control command of the unmanned surface vessel is output according to the updated action policy, including: Construct a Q-value table to map the correspondence between the state space and the action space; Using the calculated total reward value at the current moment, the action value values ​​in the Q-value table corresponding to the current state and the currently executed rudder angle action are corrected to obtain a Q-value table containing the updated action values. Obtain the real-time status information of the unmanned surface vessel at the next moment after performing the current action; Based on the real-time status information of the next moment, retrieve the Q-value table containing the updated action value, and obtain the action value value corresponding to all candidate rudder angle actions in the action space under this state; Compare the action value values ​​corresponding to all candidate rudder angle actions, select the rudder angle with the highest action value value as the rudder angle control command for the unmanned surface vessel at the next moment, and output it.

9. An unmanned surface vessel (USV) path planning device, comprising a generation module, characterized in that, include: The generation module is used to generate a static, collision-free path connecting the start and end points as a priori reference path to guide the navigation of the unmanned surface vessel. The acquisition module is used to acquire the current position coordinates and current bow angle of the unmanned surface vessel. The first calculation module is used to calculate the heading deviation bonus value based on the current position coordinates and the current heading angle. The second calculation module is used to calculate the minimum distance between the current position of the unmanned surface vessel and the prior reference path; The third calculation module is used to set a stratified region threshold based on the total distance between the starting point and the ending point, determine the specific interval in which the minimum distance falls within the stratified region threshold, and select the corresponding gradient reward coefficient based on the interval to calculate the stratified guidance reward value. The fourth calculation module is used to calculate the total reward value at the current moment based on the heading deviation reward value and the hierarchical guidance reward value; The output module is used to update the action policy of the preset reinforcement learning model using the total reward value, and output the rudder angle control command of the unmanned surface vessel according to the updated action policy. The control module is used to control the unmanned surface vessel to navigate to the next moment state according to the rudder angle control command; The loop control module is used to determine whether the unmanned surface vessel has reached the destination, and if it has not reached the destination, it triggers the acquisition module, the first calculation module, the second calculation module, the third calculation module, the fourth calculation module, the output module and the control module to run repeatedly until the unmanned surface vessel reaches the destination.

10. An electronic device, characterized in that, The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the operations performed by the unmanned surface vessel path planning method as described in any one of claims 1 to 8.

Images