Unmanned ship autonomous navigation method, electronic device, storage medium and program product

Abstract

The application provides an unmanned ship autonomous navigation method, electronic equipment, readable storage medium and computer program product, and relates to the unmanned ship navigation technical field.The method of the application comprises the following steps: constructing short-term navigation state data according to state information and environment information in the process of unmanned ship navigation; acquiring at least two navigation trajectories under the same navigation scene and respectively calculating discount returns; based on the difference between each discount return and the average discount return, constructing a relative reward feedback advantage, and taking the relative reward feedback advantage as a strategy update signal for driving strategy network optimization; inputting the short-term navigation state data into a decision layer of proximal strategy optimization, and taking the strategy update signal as the driving of a strategy network in the decision layer, and outputting continuous control actions under the constraint of proximal strategy optimization pruning. The application can improve the training stability, reward robustness, and safety of autonomous navigation and collision avoidance of the unmanned ship in a complex dynamic water environment.

Description

Technical Field

[0001] This invention relates to the field of unmanned surface vessel (USV) navigation technology, specifically to an autonomous navigation method, electronic device, storage medium, and program product for USVs. Background Technology

[0002] With the rapid development of intelligent shipping, unmanned systems, and marine engineering equipment technologies, unmanned surface vehicles (USVs) have been widely used in tasks such as port inspection, waterway mapping, maritime search and rescue, environmental monitoring, and maritime transport support operations. In practical applications, USVs typically need to navigate autonomously in complex waters affected by environmental disturbances such as wind, waves, and currents, integrating multi-source information such as target points, channel boundaries, dynamic approaching vessels, and obstacles. They also need to collaboratively complete tasks such as target tracking, path keeping, and collision avoidance while ensuring navigational safety.

[0003] Traditional autonomous navigation and collision avoidance methods for unmanned surface vessels typically rely on artificial potential field methods, A... Algorithms, Rapid Expanding Random Tree (RRT) algorithms, model predictive control, genetic algorithms, or rule-based collision avoidance strategies are commonly used. These methods can achieve certain control effects when the environmental structure is relatively clear, the number of obstacles is limited, or the dynamic model of the unmanned surface vessel (USV) is relatively accurate. However, in complex scenarios such as multi-vehicle encounters, dense dynamic obstacles, strong perceived noise, or significant sea state disturbances, traditional methods generally suffer from insufficient environmental adaptability, strong parameter dependence, poor real-time decision robustness, and difficulty in effectively balancing trajectory smoothness and navigation safety.

[0004] In recent years, deep reinforcement learning technology has become an important technical approach for autonomous navigation and intelligent collision avoidance control of unmanned surface vessels (USVs) by enabling agents to automatically learn end-to-end mappings from environmental states to control actions through continuous interaction with simulated or real physical environments. Among these, the Proximal Policy Optimization (PPO) algorithm has been widely applied to tasks such as rudder angle control, propulsion adjustment, path tracking, and complex encounter and collision avoidance of USVs due to its advantages such as simple network structure, relatively stable training process, and good adaptability to continuous action spaces.

[0005] However, existing PPO algorithms and their conventional improvements typically rely heavily on absolute reward values ​​and value baselines when constructing the advantage function. In autonomous navigation missions for unmanned surface vessels (USVs), the reward function is often a weighted combination of multiple components such as target proximity, track deviation, collision risk, maneuver smoothness, and navigation rule compliance, exhibiting significant artificial design attributes and experience dependence. When the scale, translation, shaping method, or local penalty weights of the reward function change, the traditional absolute advantage signal is prone to drastic alteration, leading to policy gradient direction shifts, network update amplitude imbalances, and significant oscillations during training.

[0006] Especially in complex encounter navigation tasks for unmanned surface vessels (USVs), the relative performance of different navigation trajectories within the same scenario often reflects decision quality more objectively than absolute reward values. For example, under the same waterway risk baseline, the same channel constraints, or the same rule penalties, the absolute rewards of two collision avoidance trajectories may both be at a low level. However, the core of determining the direction of strategy optimization lies in identifying which trajectory has higher safety, smoothness, and target proximity. If existing algorithms rely excessively on absolute reward anchoring mechanisms, their policy updates are highly susceptible to fluctuations from manual parameter tuning and interference from common scenario offsets, resulting in a significant decrease in the algorithm's training stability and cross-scenario generalization ability.

[0007] Therefore, there is an urgent need in this field for an autonomous navigation method for unmanned surface vessels that can effectively reduce the dependence of the policy optimization process on the absolute reward scale and the common reward baseline while maintaining a stable update framework for the proximal policy optimization algorithm. Summary of the Invention

[0008] This invention provides an autonomous navigation method for unmanned surface vessels, an electronic device, a storage medium, and a program product.

[0009] According to one aspect of the present invention, an autonomous navigation method for an unmanned surface vessel (USV) is provided, comprising: acquiring state information and environmental information during the USV's navigation process; constructing short-term navigation state data based on the state information and environmental information; sampling at least two navigation trajectories using a current policy under the same navigation scenario, and calculating the discounted reward for each navigation trajectory at a corresponding time; calculating the average discounted reward of each navigation trajectory as the average discounted reward under the corresponding navigation scenario; constructing a relative reward feedback advantage corresponding to each navigation trajectory based on the difference between the discounted reward of each navigation trajectory and the average discounted reward, and using the relative reward feedback advantage as a policy update signal driving policy network optimization; and inputting the short-term navigation state data into the decision of near-end policy optimization. The decision layer consists of a policy network driven by the policy update signal, which outputs continuous control actions under near-end policy optimization pruning constraints. By executing these continuous control actions, the unmanned surface vessel's (USV) motion state is updated, generating short-term navigation state data for the next moment and calculating the immediate reward value for the current moment. The immediate reward value and the short-term navigation state data for the next moment are fed back to the decision layer. The decision layer substitutes the policy update signal into the objective function of the policy network for gradient calculation and, combined with the loss function of the value network, iteratively optimizes the USV's autonomous navigation strategy by adjusting the parameters of the policy network and the value network, thus obtaining the USV's autonomous navigation decision model. Navigation of the USV is then performed based on this autonomous navigation decision model.

[0010] According to at least one embodiment of the unmanned surface vessel (USV) autonomous navigation method of the present invention, the short-term navigation state data includes at least one of the following: USV navigation state, target point state, reference channel state, target vessel state, and obstacle state; wherein, the USV navigation state includes at least one of the following: global coordinates, heading angle, heading angle change rate, speed, rudder angle, and rudder angle change rate; the target point state includes the target point relative distance, relative bearing angle, and relative coordinates; the reference channel state includes the USV's lateral deviation from the nearest point on the reference channel, heading deviation, and progress along the channel; the target vessel state includes at least one of the target vessel's position, speed, heading, relative distance, relative bearing, DCPA, TCPA, and collision risk index.

[0011] An autonomous navigation method for unmanned surface vessels according to at least one embodiment of the present invention, wherein the objective function of the policy network is... Represented as:

[0012] in, Indicates the policy network parameters; The expression represents the desired operation; min represents the minimum value operation. This represents the importance sampling ratio of the k-th navigation trajectory at time t; k represents the sequence number of the navigation trajectory. This represents the relative reward feedback advantage of the k-th navigation trajectory at time t; Indicates a cropping operation; This represents the cutting factor.

[0013] An autonomous navigation method for unmanned surface vessels according to at least one embodiment of the present invention, with an importance sampling ratio. The calculation formula is expressed as:

[0014] in, This represents the state of the k-th navigation trajectory at time t; This represents the action of the k-th navigation trajectory at time t; Represents the old parameters of the policy network; Indicates the current policy network; Represents the old policy network; Indicates the current policy network In state Take action below The probability of; Represents the old policy network In state Take action below The probability of.

[0015] According to at least one embodiment of the unmanned surface vessel autonomous navigation method of the present invention, the loss function of the value network Represented as:

[0016] in, Indicates the parameters of the value network; Indicates the desired operation; Represents the value function; Navigation scenario Next The navigation trajectory at time Discounted return.

[0017] According to at least one embodiment of the unmanned surface vessel autonomous navigation method of the present invention, the total loss function for adjusting the policy network parameters and the value network parameters is expressed as:

[0018] in, Represents the total loss function; Indicates the policy network parameters; Representing value network parameters Represents the objective function of the policy network; The loss function representing the value network; This represents the policy entropy regularization term; The weight coefficients represent the loss function of the value network; This represents the weight coefficient of the policy entropy regularization term.

[0019] According to at least one embodiment of the unmanned surface vessel autonomous navigation method of the present invention, the instantaneous reward value is determined based on the target point proximity, track tracking error, collision risk index, navigation rule compliance and action smoothness after the continuous control action is performed.

[0020] According to another aspect of the present invention, an electronic device is provided, comprising: a memory storing execution instructions; and a processor executing the execution instructions stored in the memory, such that the processor performs an autonomous navigation method for an unmanned surface vessel according to any embodiment of the present invention.

[0021] According to another aspect of the present invention, a readable storage medium is provided, wherein execution instructions are stored therein, which, when executed by a processor, are used to implement the autonomous navigation method of an unmanned surface vessel according to any embodiment of the present invention.

[0022] According to another aspect of the present invention, a computer program product is provided, comprising a computer program that, when executed by a processor, implements an autonomous navigation method for unmanned surface vessels according to any embodiment of the present invention.

[0023] The present invention has the following beneficial effects: (1) Achieve natural offsetting of the common reward baseline and reconstruct the strategy optimization driving mechanism.

[0024] By calculating the average of discounted rewards within the same navigation scenario and constructing a relative reward feedback advantage, and utilizing a scenario-based reward centralization mechanism, the common reward shift term shared by all trajectories in the same navigation scenario is naturally canceled out during the advantage construction process. This mechanism ensures that the optimization signal of the policy network is no longer constrained by the magnitude of the absolute reward, but is purely driven by the relative performance differences between trajectories within the same scenario.

[0025] (2) Decouple the update signal from the reward calibration to improve the robustness of the algorithm.

[0026] Because the update signal of the policy network is provided by the relative reward feedback advantage, the update signal and policy gradient direction of the policy network can still maintain a high degree of consistency when the reward function experiences a common baseline shift, a change in the reward scale, or a change in the manual reward weight calibration method. This effectively overcomes the shortcomings of traditional absolute reward advantage, which is easily affected by manual parameter tuning, and avoids drastic oscillations during training.

[0027] (3) Ensure stable convergence and high safety of training in complex dynamic waters.

[0028] By leveraging the synergistic effect of the aforementioned relative reward robustness mechanism and the near-end policy optimization pruning constraints, the sensitivity of the unmanned surface vessel (USV) policy optimization process to absolute reward scales and manual parameter tuning can be significantly reduced. In complex dynamic waters (such as multi-ship encounters, strong environmental disturbances, and dense channel constraints), stable convergence during training can be guaranteed, significantly improving the reward robustness of the USV's autonomous navigation strategy and collision avoidance safety during actual deployment. Attached Figure Description

[0029] The accompanying drawings illustrate exemplary embodiments of the invention and, together with the description thereof, serve to explain the principles of the invention. These drawings are included to provide a further understanding of the invention and are incorporated in and constitute a part of this specification.

[0030] Figure 1 This is a flowchart illustrating an autonomous navigation method for an unmanned surface vessel according to an embodiment of the present invention.

[0031] Figure 2 This is a schematic structural block diagram of an electronic device employing a processor-based hardware implementation according to an embodiment of the present invention. Detailed Implementation

[0032] The present invention will now be described in further detail with reference to the accompanying drawings and examples. It should be understood that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of the invention. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present invention are shown in the accompanying drawings.

[0033] It should be noted that, unless otherwise specified, the embodiments and features described in this invention can be combined with each other. The technical solution of this invention will now be described in detail with reference to the accompanying drawings and embodiments.

[0034] Figure 1 A schematic diagram illustrating the overall flow of an unmanned surface vessel (USV) autonomous navigation method according to one embodiment of the present invention is shown. Figure 1 The method shown includes steps S1 to S8.

[0035] S1: Acquire the state and environmental information during the unmanned surface vessel's navigation process, and construct short-term navigation state data based on the state and environmental information.

[0036] The status information corresponds to the unmanned surface vessel's (USV) own status. For example, the status information may include: USV position, heading, speed, rate of change of attitude, rudder angle, rate of change of rudder angle, and propulsion status, etc.

[0037] Environmental information corresponds to information related to the external environment during the navigation of the unmanned surface vessel. For example, environmental information may include: target point, channel centerline, channel boundary, static obstacles, dynamic target vessels, and water boundary, etc.

[0038] As one possible implementation, state information and environmental information can be obtained through an environmental perception module. For example, the environmental perception module may include at least one of the following: a positioning unit, an inertial measurement unit, a speed measurement unit, radar, a visual sensor, an AIS (Automatic Identification System) information receiving unit, and a simulation environment interface.

[0039] Short-term navigation status data can be used to describe the comprehensive navigation situation of an unmanned surface vessel (USV) at a given moment. As one possible implementation, short-term navigation status data can be used to characterize the USV's motion state, relative relationship to a target point, course deviation, relative motion information of the target vessel, and collision risk indicators. For example, short-term navigation status data may include at least one of the following: USV navigation state, target point state, reference course state, target vessel state, and obstacle state. Specifically, the USV navigation state includes at least one of the following: global coordinates, heading angle, rate of change of heading angle, speed, rudder angle, and rate of change of rudder angle. The target point state includes the target point's relative distance, relative bearing, and relative coordinates. The reference course state includes the USV's lateral deviation from the nearest point on the reference course, heading deviation, and progress along the course. The target vessel state includes at least one of the target vessel's position, speed, heading, relative distance, relative bearing, DCPA, TCPA, and collision risk indicators.

[0040] For example, in the process of constructing short-term navigation state data, the navigation state of the unmanned surface vessel, the state of the target point, the state of the reference channel, the state of the target vessel, and the state of the obstacle can be normalized and then spliced ​​together to form short-term navigation state data (state vector).

[0041] S2: Under the same navigation scenario, at least two navigation trajectories are sampled using the current strategy, and the discounted reward for each navigation trajectory at the corresponding time is calculated.

[0042] During the sampling process, the state, action, immediate reward value, next state, and termination flag at each moment can be recorded so that they can be stored in the trajectory cache for network updates.

[0043] As one possible implementation, the process of obtaining at least two navigation trajectories by the current strategy under the same navigation scenario may include: defining the same navigation scenario c with the same initial task, the same target point, the same channel constraints, and the same configuration of incoming ships or obstacles; sampling K navigation trajectories under the current strategy in navigation scenario c, where K≥2 represents the total number of navigation trajectories.

[0044] For example, the formula for calculating the discount return can be expressed as:

[0045] in, Navigation scenarios Next The navigation trajectory at time Discount rewards; In navigation scenario c, the first... The navigation trajectory at time Instant reward value; t Indicates the current moment; Represents the time-time variable; Indicates the discount factor; This indicates the termination time of the k-th navigation trajectory.

[0046] S3: Calculate the average of the discount reward for each navigation trajectory, and use it as the average discount reward for the corresponding navigation scenario.

[0047] For example, the formula for calculating the average discounted return can be expressed as:

[0048] in, Navigation scenario At any moment The average discount return; k represents the navigation trajectory number.

[0049] S4: Based on the difference between the discounted reward of each navigation trajectory and the average discounted reward, construct the relative reward feedback advantage corresponding to each navigation trajectory, and use the relative reward feedback advantage as the policy update signal to drive the policy network optimization.

[0050] For example, the formula for calculating the relative reward feedback advantage can be expressed as:

[0051] in, Indicates the first The navigation trajectory at time The relative reward feedback advantage. When When, it indicates the first The navigation trajectory is better than average in the same navigation scenario; when When, it indicates the first A navigation trajectory is worse than average within the same navigation scenario. Through the above processing method, the common reward baseline shared by all navigation trajectories in the same navigation scenario can be offset, thus allowing policy updates to focus more on the relative merits of the navigation trajectories.

[0052] Relative reward feedback advantage can be used to characterize the degree of deviation of the current navigation trajectory from the average navigation trajectory in the same scene, rather than its absolute gain level.

[0053] S5: Input short-term navigation state data into the decision layer of near-end policy optimization. The policy network in the decision layer is driven by the policy update signal and outputs continuous control actions under the pruning constraints of near-end policy optimization.

[0054] In this embodiment, the relative reward feedback advantage is used to participate in the policy network update. This can enhance the action probability of navigation trajectories that are better than the average level of the navigation scenario, and suppress the action probability of navigation trajectories that are worse than the average level of the navigation scenario. At the same time, the relative reward feedback advantage replaces the traditional generalized advantage estimation (GAE) advantage term to drive the policy network update, so that the optimization signal no longer directly depends on the absolute reward value, but depends on the relative performance difference between trajectories within the same navigation scenario.

[0055] As one possible implementation, the objective function of the policy network It can be represented as:

[0056] in, Indicates the policy network parameters; The expression represents the desired operation; min represents the minimum value operation. This represents the importance sampling ratio of the k-th navigation trajectory at time t; Indicates a cropping operation; Indicates the clipping factor. When When the policy network tends to increase the probability of the corresponding action, when At this time, the policy network tends to suppress the probability of the corresponding action. The above objective function, while retaining the PPO near-terminal stable update mechanism (retaining the probability ratio pruning mechanism, limiting the difference between the old and new policies through pruning operations to avoid overly aggressive policy updates), replaces the driving signal for policy updates from absolute reward advantage to relative reward feedback advantage.

[0057] Importance sampling ratio is used to measure the difference in probability between the current policy and the old policy taking the same action in a specific state. The calculation formula can be expressed as:

[0058] in, This represents the state of the k-th navigation trajectory at time t; This represents the action of the k-th navigation trajectory at time t; Represents the old parameters of the policy network; Indicates the current policy network; Represents the old policy network; Indicates the current policy network In state Take action below The probability of; Represents the old policy network In state Take action below The probability of.

[0059] S6: Update the motion state of the unmanned surface vessel by executing continuous control actions, generate short-term navigation state data for the next moment, and calculate the instantaneous reward value for the current moment.

[0060] As one possible implementation, after executing continuous control actions, the motion state of the unmanned surface vessel (such as position, heading, speed, and attitude) can be updated using the kinematic model, dynamic model, or high-fidelity simulation environment of the unmanned surface vessel, and the target point distance, course deviation, DCPA, TCPA, and collision risk indicators can be recalculated to form short-term navigation state data for the next moment.

[0061] The immediate reward value forms the basis for discounted return calculation and value network learning, describing the contribution of the current action. As one possible implementation, the immediate reward value can be determined based on the target point proximity, trajectory tracking error, collision risk index, navigation rule compliance, and action smoothness after executing continuous control actions. Positive rewards are given when the unmanned surface vessel (USV) advances towards the target point or along a reference course. For example, the formula for calculating the immediate reward value can be expressed as:

[0062] in, Represents the instantaneous reward value at time t; Indicates the degree of proximity to the target point; Indicates the tracking error; This indicates a collision risk index based on DCPA and TCPA. Indicates the degree of compliance with navigation rules (such as COLREGs); It indicates the smoothness of the action and can correspond to the degree of change in rudder angle, thrust, or yaw moment; , , , and All of these represent weighting coefficients.

[0063] For example, the formula for calculating the collision risk index can be expressed as:

[0064] in, Indicates a collision risk indicator; , , and All represent risk adjustment parameters; This indicates that you will encounter a distance in the near future; This indicates that the time will come soon. When the preset risk threshold is exceeded, a collision risk indicator is applied to the reward function to guide the unmanned surface vessel to take deceleration, turning or avoidance actions in advance.

[0065] S7: Feed back the instant reward value at the current moment and the short-term navigation state data at the next moment to the decision layer. The decision layer substitutes the policy update signal into the objective function of the policy network to calculate the gradient, and combines it with the loss function of the value network. By adjusting the parameters of the policy network and the value network, the autonomous navigation policy of the unmanned surface vessel is iteratively optimized, and the autonomous navigation decision model of the unmanned surface vessel is obtained.

[0066] As one possible implementation, the instantaneous reward value, the current short-term navigation state data, the next short-term navigation state data, and continuous control actions can be stored in the trajectory cache. After completing trajectory sampling in the same navigation scenario, the policy network parameters and value network parameters are updated based on the total loss function until a deployable autonomous navigation strategy model for unmanned surface vessels is obtained.

[0067] The value network continues to learn using the discounted return calculated in step S2 as a supervision signal to maintain a stable estimate of the long-term cumulative return of the value function. Thus, the policy network is used to learn a relative performance-driven navigation policy, while the value network provides a stable long-term value assessment. This clear division of labor avoids the problem in traditional PPO where value estimation errors are directly transmitted to policy updates.

[0068] As one possible implementation method, the loss function of the value network Represented as:

[0069] in, Indicates the parameters of the value network; Indicates the desired operation; Represents the value function; Navigation scenario Next The navigation trajectory at time Discounted return.

[0070] As one possible implementation, the total loss function used to adjust the policy network parameters and the value network parameters can be expressed as:

[0071] in, Represents the total loss function; Indicates the policy network parameters; Representing value network parameters Represents the objective function of the policy network; The loss function representing the value network; This represents the policy entropy regularization term; The weight coefficients represent the loss function of the value network; This represents the weight coefficient of the policy entropy regularization term.

[0072] The iteration termination conditions for the iterative optimization (training) process can be: reaching the maximum number of training rounds, the average reward reaching a preset threshold, the collision rate being lower than a preset threshold, the trajectory error being lower than a preset threshold, or the strategy converging, etc.

[0073] S8: Navigation of unmanned surface vessels based on an autonomous navigation decision-making model.

[0074] After iterative optimization, the policy network parameters can be solidified into an autonomous navigation decision model for unmanned surface vessels (USVs) and deployed to a simulation platform or USV controller for online inference of USV navigation.

[0075] The present invention has the following beneficial effects: (1) Achieve natural offsetting of the common reward baseline and reconstruct the strategy optimization driving mechanism.

[0076] By calculating the average of discounted rewards within the same navigation scenario and constructing a relative reward feedback advantage, and utilizing a scenario-based reward centralization mechanism, the common reward shift term shared by all trajectories in the same navigation scenario is naturally canceled out during the advantage construction process. This mechanism ensures that the optimization signal of the policy network is no longer constrained by the magnitude of the absolute reward, but is purely driven by the relative performance differences between trajectories within the same scenario.

[0077] (2) Decouple the update signal from the reward calibration to improve the robustness of the algorithm.

[0078] Because the update signal of the policy network is provided by the relative reward feedback advantage, the update signal and policy gradient direction of the policy network can still maintain a high degree of consistency when the reward function experiences a common baseline shift, a change in the reward scale, or a change in the manual reward weight calibration method. This effectively overcomes the shortcomings of traditional absolute reward advantage, which is easily affected by manual parameter tuning, and avoids drastic oscillations during training.

[0079] (3) Ensure stable convergence and high safety of training in complex dynamic waters.

[0080] By leveraging the synergistic effect of the aforementioned relative reward robustness mechanism and the near-end policy optimization pruning constraints, the sensitivity of the unmanned surface vessel (USV) policy optimization process to absolute reward scales and manual parameter tuning can be significantly reduced. In complex dynamic waters (such as multi-ship encounters, strong environmental disturbances, and dense channel constraints), stable convergence during training can be guaranteed, significantly improving the reward robustness of the USV's autonomous navigation strategy and collision avoidance safety during actual deployment.

[0081] According to a further embodiment of the present invention, an electronic device is also provided. Figure 2This diagram illustrates a schematic block diagram of an electronic device employing a processor-based hardware implementation according to an embodiment of the present invention. The hardware structure of the electronic device of the present invention can be implemented using a bus architecture. The bus architecture can include any number of interconnect buses and bridges, depending on the specific application and overall design constraints of the hardware. Bus 1100 connects various circuits including one or more processors 1200, memory 1300, and / or hardware modules. Bus 1100 can also connect various other circuits 1400 such as peripheral devices, voltage regulators, power management circuits, external antennas, etc. Bus 1100 can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Component (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, only one connecting line is used in this diagram, but this does not imply that there is only one bus or one type of bus. Memory 1300 stores a computer program, and when processor 1200 executes the computer program, processor 1200 is able to perform the following processes. Acquire state and environmental information during the unmanned surface vessel's (USV) navigation process, and construct short-term navigation state data based on this information. For each USV, at least two navigation trajectories are sampled using the current policy within the same navigation scenario, and the discounted reward for each trajectory at the corresponding time step is calculated. The average discounted reward of each trajectory is calculated as the average discounted reward for the corresponding navigation scenario. Based on the difference between the discounted reward of each trajectory and the average discounted reward, a relative reward feedback advantage is constructed for each trajectory, and this relative reward feedback advantage is used as the policy update signal driving the policy network optimization. The short-term navigation state data is input into the decision layer of the proximal policy optimization. The policy network in the decision layer is driven by the policy update signal and outputs continuous control actions under the pruning constraints of the proximal policy optimization. By executing continuous control actions, the USV's motion state is updated, generating the short-term navigation state data for the next time step, and the instantaneous reward value for the current time step is calculated. The immediate reward value at the current moment and the short-term navigation state data at the next moment are fed back to the decision layer. The decision layer substitutes the policy update signal into the objective function of the policy network for gradient calculation, and combines it with the loss function of the value network. By adjusting the parameters of the policy network and the value network, the autonomous navigation strategy of the unmanned surface vessel (USV) is iteratively optimized, resulting in the USV autonomous navigation decision model. USV navigation is then performed based on this decision model.

[0082] In one possible implementation, the electronic device may further include a communication interface, a sensor interface, and a control output interface. The communication interface is used to receive target point, mission route, channel boundary, target vessel AIS information, and remote monitoring commands. The sensor interface is used to receive positioning, heading, speed, radar, visual, or inertial measurement data. The control output interface is used to send continuous control commands to the propulsion system, servo motor, or motion control module.

[0083] As one possible implementation, the electronic device can be a training server, an edge computing device, an unmanned surface vessel (USV) onboard computer, or a simulation training platform. When the electronic device is a training server, it can generate multiple trajectories in the same navigation scenario through parallel simulation and perform iterative optimization to obtain an autonomous navigation decision model for the USV. When the electronic device is an USV onboard computer, it can load the trained USV autonomous navigation decision model and output control actions based on the real-time short-term navigation status to achieve online autonomous navigation and collision avoidance.

[0084] The present invention also provides a readable storage medium storing a computer program, which, when executed by a processor, is used to implement the methods described above. A "readable storage medium" can be any means capable of containing, storing, communicating, propagating, or transmitting a program for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples of a readable storage medium include: an electrical connection (electronic device) having one or more wires, a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and programmable read-only memory (EPROM or flash memory), fiber optic devices, and portable read-only memory (CDROM), etc.

[0085] This invention also provides a computer program product. The method of this invention can be implemented wholly or partially through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented wholly or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed, the processes or functions of this invention are performed wholly or partially.

[0086] Computer programs or instructions can be stored in a readable storage medium or transferred from one readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The readable storage medium can be any available medium capable of access, or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; an optical medium, such as a digital video optical disc; or a semiconductor medium, such as a solid-state drive. The computer-readable storage medium can be a volatile or non-volatile storage medium, or it can include both volatile and non-volatile types of storage media.

[0087] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0088] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus, and computer program products according to the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0089] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0090] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0091] In the description of this specification, the references to terms such as "one embodiment / mode," "some embodiments / modes," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, or characteristic described in connection with that embodiment / mode or example is included in at least one embodiment / mode or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment / mode or example. Moreover, the specific features, structures, or characteristics described may be combined in any suitable manner in one or more embodiments / modes or examples. Furthermore, without contradiction, those skilled in the art can combine and integrate the different embodiments / modes or examples described in this specification, as well as the features of different embodiments / modes or examples.

[0092] Those skilled in the art should understand that the above embodiments are merely for illustrating the present invention and are not intended to limit the scope of the invention. Those skilled in the art can make other changes or modifications based on the above disclosure, and these changes or modifications still fall within the scope of the present invention.

Claims

1. An autonomous navigation method for unmanned surface vessels, characterized in that, include: Acquire state and environmental information during the navigation of the unmanned surface vessel, and construct short-term navigation state data based on the state and environmental information; Under the same navigation scenario, at least two navigation trajectories are sampled using the current strategy, and the discounted reward for each navigation trajectory at the corresponding time is calculated. Calculate the average of the discount reward for each navigation trajectory, and use it as the average discount reward for the corresponding navigation scenario; Based on the difference between the discounted reward of each navigation trajectory and the average discounted reward, a relative reward feedback advantage corresponding to each navigation trajectory is constructed, and the relative reward feedback advantage is used as a policy update signal to drive the policy network optimization. The short-term navigation state data is input into the decision layer of the near-end policy optimization. The policy network in the decision layer is driven by the policy update signal and outputs continuous control actions under the pruning constraints of the near-end policy optimization. By executing the continuous control actions, the motion state of the unmanned surface vessel is updated, short-term navigation state data for the next moment is generated, and the instantaneous reward value for the current moment is calculated. The instant reward value at the current moment and the short-term navigation status data at the next moment are fed back to the decision layer. The decision layer substitutes the policy update signal into the objective function of the policy network to perform gradient calculation, and combines it with the loss function of the value network. By adjusting the parameters of the policy network and the value network, the autonomous navigation policy of the unmanned surface vessel is iteratively optimized to obtain the autonomous navigation decision model of the unmanned surface vessel. Unmanned surface vessel (USV) navigation is performed based on the aforementioned autonomous navigation decision-making model.

2. The autonomous navigation method for unmanned surface vessels as described in claim 1, characterized in that, The short-term navigation status data includes at least one of the following: unmanned surface vessel (USV) navigation status, target point status, reference channel status, target vessel status, and obstacle status; wherein, the USV navigation status includes at least one of the following: global coordinates, heading angle, rate of change of heading angle, speed, rudder angle, and rate of change of rudder angle; the target point status includes the target point's relative distance, relative bearing angle, and relative coordinates; the reference channel status includes the USV's lateral deviation from the nearest point on the reference channel, heading deviation, and progress along the channel; the target vessel status includes at least one of the target vessel's position, speed, heading, relative distance, relative bearing, DCPA, TCPA, and collision risk indicators.

3. The autonomous navigation method for unmanned surface vessels as described in claim 1, characterized in that, Objective function of policy network Represented as: in, Indicates the policy network parameters; The expression represents the desired operation; min represents the minimum value operation. This represents the importance sampling ratio of the k-th navigation trajectory at time t; k represents the sequence number of the navigation trajectory. This represents the relative reward feedback advantage of the k-th navigation trajectory at time t; Indicates a cropping operation; This represents the cutting factor.

4. The autonomous navigation method for unmanned surface vessels as described in claim 3, characterized in that, Importance sampling ratio The calculation formula is expressed as: in, This represents the state of the k-th navigation trajectory at time t; This represents the action of the k-th navigation trajectory at time t; Represents the old parameters of the policy network; Indicates the current policy network; Represents the old policy network; Indicates the current policy network In state Take action below The probability of; Represents the old policy network In state Take action below The probability of.

5. The autonomous navigation method for unmanned surface vessels as described in claim 1, characterized in that, Loss function of value network Represented as: in, Indicates the parameters of the value network; Indicates the desired operation; Represents the value function; Navigation scenario Next The navigation trajectory at time Discounted returns.

6. The autonomous navigation method for unmanned surface vessels as described in claim 1, characterized in that, The total loss function used to adjust the parameters of the policy network and the value network is expressed as: in, Represents the total loss function; Indicates the policy network parameters; Representing value network parameters Represents the objective function of the policy network; The loss function representing the value network; This represents the policy entropy regularization term; The weight coefficients represent the loss function of the value network; This represents the weight coefficient of the policy entropy regularization term.

7. The autonomous navigation method for unmanned surface vessels as described in claim 1, characterized in that, The instant reward value is determined based on the target point proximity, track tracking error, collision risk index, navigation rule compliance, and action smoothness after the continuous control actions are performed.

8. An electronic device, characterized in that, include: The memory stores execution instructions; as well as A processor that executes the execution instructions stored in the memory, causing the processor to perform the autonomous navigation method of any one of claims 1 to 7.

9. A readable storage medium, characterized in that, The readable storage medium stores execution instructions, which, when executed by a processor, are used to implement the autonomous navigation method of any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the autonomous navigation method for unmanned surface vessels as described in any one of claims 1 to 7.

Images