A method for large-scale dynamic obstacle avoidance and path tracking of unmanned surface vessels based on the fusion of deep reinforcement learning and line-of-sight navigation.

Abstract

This invention belongs to the field of unmanned surface vessel (USV) technology and discloses a large-scale dynamic obstacle avoidance and path tracking method for USVs based on the fusion of deep reinforcement learning and line-of-sight navigation. This invention boasts strong robustness and generalization ability: it employs a FEDT (Feature-Based Assisted Learning) strategy to train the DRL agent in a local dynamic environment, avoiding the complexity of training in a large-scale environment, while enabling the trained policy to effectively transfer and cope with unknown complex obstacles in a large-scale environment. It balances global tracking and local obstacle avoidance: by creatively combining LOS (Low-Installation-Side) navigation and DRL control, and utilizing an "obstacle-aware domain" for mode switching, it solves the problem that a single method cannot simultaneously achieve both long-distance tracking efficiency and short-distance obstacle avoidance flexibility. Smooth policy fusion: in obstacle avoidance mode, the LOS point is used as a temporary target for DRL, acting as a "bridge" to ensure that the USV can naturally and quickly converge back to the preset path after obstacle avoidance.

Description

Technical Field

[0001] This invention belongs to the field of unmanned surface vessel technology, and in particular relates to a method for large-scale dynamic obstacle avoidance and path tracking of unmanned surface vessels based on the fusion of deep reinforcement learning and line-of-sight navigation. Background Technology

[0002] With the continuous development of unmanned surface vehicles (USVs) technology, their autonomous navigation, obstacle avoidance, and path tracking capabilities have become key technologies for the safe and efficient operation of intelligent surface platforms. In applications such as marine mapping, environmental monitoring, search and rescue, and logistics transportation, USVs need to make autonomous decisions in complex and dynamic marine environments to safely avoid obstacles and achieve high-precision tracking of pre-set global trajectories. To adapt to the randomness and uncertainty of the aquatic environment, researchers both domestically and internationally have proposed various technical solutions based on traditional navigation, planning, and machine learning methods, attempting to improve the decision-making capabilities of USVs in dynamic and complex environments.

[0003] Traditional obstacle avoidance and path tracking methods primarily rely on rule- or model-driven strategies, such as decision planning directly based on artificial potential fields, geometric rules, or velocity obstacles. These methods typically construct mathematical models and rule bases, achieving obstacle avoidance and path adjustment through on-site perception and model prediction. However, these traditional strategies often suffer from insufficient robustness and weak adaptability to unknown environments in real-world, dynamically complex marine environments, making it difficult to maintain stable and safe trajectory control when the number of obstacles, their motion states, and external disturbances surge. Furthermore, for global path tracking tasks, purely rule-driven methods often struggle to maintain long-term high efficiency and system generalization capabilities in large-scale environments (see relevant research reviews and comparative analyses).

[0004] In recent years, Deep Reinforcement Learning (DRL), as an intelligent algorithm combining deep perception and sequential decision-making capabilities, has gradually become a research hotspot in the field of dynamic obstacle avoidance and autonomous navigation decision-making for unmanned surface vessels (USVs). Related research shows that by formulating the obstacle avoidance problem as a Markov Decision Process (MDP) and combining it with deep neural networks to enhance state representation and policy learning capabilities, DRL demonstrates good adaptability and robustness in handling complex obstacle avoidance scenarios. Existing technologies have disclosed methods for training path-tracking control strategies using algorithms such as PPO and DDPG, and have implemented policy learning for path tracking or single obstacle avoidance behavior of USVs under disturbed conditions in simulation environments. For example, a path-tracking method for USVs based on reinforcement learning uses MDP modeling and PPO policy optimization to improve the convergence of tracking control. ([Google Patents][2]) Meanwhile, some research has also attempted to directly use the deep reinforcement learning framework to implement dynamic collision avoidance decision-making for USVs, improving the training performance of the RL algorithm itself by combining methods such as policy gradients.

[0005] However, the aforementioned existing technologies still have significant shortcomings and deficiencies. On the one hand, when applying deep reinforcement learning directly to large-scale navigation tasks of unmanned vessels, the large dimensionality of the marine environment's state space and the variability in obstacle types and motion states often lead to problems such as state space explosion, slow convergence speed, or even non-convergence during model training. Furthermore, it is difficult to guarantee the stability of the trained strategy in complex environments and its efficiency in obstacle avoidance and global path tracking. On the other hand, most existing deep reinforcement learning methods implement obstacle avoidance or tracking in the form of end-to-end policies, lacking a control framework that effectively integrates local obstacle avoidance behavior with traditional global navigation methods. This results in insufficient practical performance when handling real-world, large-scale scenarios with multiple dynamic obstacles. Therefore, how to combine the advantages of deep reinforcement learning in handling complex local obstacle avoidance with the integration of traditional navigation methods to achieve efficient global path tracking, in order to solve the problem of safe navigation of unmanned vessels in large-scale, uncertain environments, remains a key technical problem that has not yet been effectively solved by existing technologies. Summary of the Invention

[0006] To address the problems existing in the prior art, this invention provides a method for large-scale dynamic obstacle avoidance and path tracking of unmanned surface vessels based on the fusion of deep reinforcement learning and line-of-sight navigation.

[0007] This invention is implemented as follows: a method for large-scale dynamic obstacle avoidance and path tracking of unmanned surface vessels based on the fusion of deep reinforcement learning and line-of-sight navigation includes:

[0008] Step 1: Construct a training environment limited to a local spatial range;

[0009] Step 2: In each training round, only one dynamic obstacle is set up, and the initial position and geometric dimensions of the obstacle are randomly changed in different training rounds;

[0010] Step 3: Through the above-mentioned high-dynamic but low-complexity training method, the reinforcement learning control strategy focuses on learning the single obstacle avoidance maneuverability during the training process, thereby forming a robust obstacle avoidance control strategy that can be transferred to complex scenarios.

[0011] Furthermore, the scale of the local spatial range is consistent with the obstacle perception range of the unmanned vessel in subsequent practical applications, so as to ensure the effectiveness of the trained control strategy in the application stage.

[0012] Furthermore, by randomly varying obstacle parameters in multiple training rounds, the control strategy enables a unified obstacle avoidance response capability for obstacles of different sizes and relative positions.

[0013] Another objective of this invention is to provide a target generation method for unmanned surface vessel path tracking and dynamic obstacle avoidance, comprising:

[0014] An obstacle perception domain with a fixed radius is constructed centered on the current position of the unmanned vessel;

[0015] When the unmanned vessel travels along a preset path, a line-of-sight target point is generated in front of the preset path based on the path deviation and forward sight distance parameters.

[0016] The line-of-sight target point serves as a heading control target in path tracking mode and as a temporary navigation target for local obstacle avoidance control in obstacle avoidance mode.

[0017] Furthermore, the radius of the obstacle perception domain is matched with the scale of the local training environment used in the training phase to ensure the perceptual consistency of the control strategy.

[0018] Furthermore, the line-of-sight target point is located in front of the current preset path, and is used to characterize the entry position of the unmanned vessel on the return path after obstacle avoidance.

[0019] Another objective of this invention is to provide a cooperative control method for unmanned surface vessel path tracking and dynamic obstacle avoidance, comprising:

[0020] When no obstacle is detected within the obstacle perception domain of the unmanned vessel, a path tracking control method based on line-of-sight target points is adopted;

[0021] When an obstacle is detected within the obstacle perception domain of the unmanned vessel, it switches to an obstacle avoidance control method based on reinforcement learning.

[0022] In obstacle avoidance control mode, the currently calculated line-of-sight target point is set as the short-term navigation target for obstacle avoidance control, so that the unmanned vessel can actively return to the preset path after completing local obstacle avoidance.

[0023] Furthermore, the obstacle avoidance control method generates control commands based on the unmanned vessel's own state, the obstacle's state, and the relative relationship between the unmanned vessel and the short-term navigation target.

[0024] Furthermore, by limiting the short-term navigation target to the line-of-sight target point, the obstacle avoidance process is kept consistent with the path tracking target, thus preventing the obstacle avoidance behavior from deviating from the global path.

[0025] Furthermore, by adjusting the forward-looking distance parameter, a performance trade-off between the path tracking accuracy and obstacle avoidance safety distance of the unmanned vessel can be achieved.

[0026] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0027] The purpose of this invention is to overcome the shortcomings of existing rule-based or model-based obstacle avoidance methods in complex dynamic environments, as well as the drawbacks of end-to-end deep reinforcement learning (DRL) methods in large-scale scenarios, such as state space explosion, convergence difficulties, and inefficient global path tracking. This invention constructs a control framework that leverages the advantages of deep reinforcement learning in handling complex local obstacle avoidance while combining traditional navigation methods to achieve efficient global path tracking, thereby solving the problem of safe navigation of unmanned vessels in large-scale uncertain environments.

[0028] This system effectively solves the problem of complex and difficult-to-converge training in large-scale environments by ① employing a unique feature-enhanced dynamic training strategy to train the DRL agent in local dynamic environments. The trained model has strong generalization ability and can effectively transfer to cope with unknown and complex obstacles in a wide range of environments. ② It combines the stability of classic LOS navigation with the flexibility of DRL control. By constructing an "obstacle perception domain" to achieve intelligent switching between dual modes, it successfully solves the industry pain point that a single method cannot simultaneously meet the needs of long-distance efficient tracking and short-distance agile obstacle avoidance. ③ In obstacle avoidance mode, the LOS point is used as a temporary guiding target for the DRL agent, acting as a "bridge". This mechanism ensures that after completing the obstacle avoidance action, the unmanned vessel can naturally, quickly and smoothly converge back to the preset path, avoiding control oscillations. ④ By opening the adjustment interface of the forward look-ahead distance parameter Δ, a simple and effective method is provided to balance the system's path tracking accuracy and obstacle avoidance safety margin, greatly improving the system's adaptability to different navigation tasks and safety level requirements.

[0029] ① Strong robustness and generalization ability: The FEDT strategy is used to train the DRL agent in a local dynamic environment, which avoids the complexity of training in a large-scale environment. At the same time, the trained policy can be effectively transferred and cope with unknown complex obstacles in a wide range of environments.

[0030] ② Balancing global tracking and local obstacle avoidance: By creatively combining LOS navigation and DRL control, and utilizing the "obstacle perception domain" for mode switching, the problem of a single method being unable to balance long-distance tracking efficiency and short-distance obstacle avoidance flexibility is solved.

[0031] ③ Smooth strategy fusion: In obstacle avoidance mode, the LOS point is used as a temporary target of DRL, which acts as a "bridge" to ensure that the unmanned ship can naturally and quickly converge back to the preset path after obstacle avoidance.

[0032] Adjustable performance balance: By adjusting the forward look-ahead distance parameter Δ, a simple and effective method is provided to balance the path tracking accuracy and obstacle avoidance safety margin of the system, thereby improving the system's adaptability to different task requirements. Attached Figure Description

[0033] Figure 1 This is a flowchart of a method for large-scale dynamic obstacle avoidance and path tracking of unmanned vessels based on the fusion of deep reinforcement learning and line-of-sight navigation, provided in an embodiment of the present invention.

[0034] Figure 2 This is a detailed flowchart of the large-scale dynamic obstacle avoidance and path tracking method for unmanned vessels based on the fusion of deep reinforcement learning and line-of-sight navigation provided in the embodiments of the present invention.

[0035] Figure 3 This is a block diagram of a large-scale dynamic obstacle avoidance and path tracking system for unmanned vessels based on the fusion of deep reinforcement learning and line-of-sight navigation, provided in an embodiment of the present invention.

[0036] Figure 4 This is a flowchart of the offline training subsystem provided in an embodiment of the present invention.

[0037] Figure 5 This is a flowchart of online operation and subsystem control provided in an embodiment of the present invention.

[0038] Figure 6 This is a graph showing the reward curve and the average reward curve obtained in every 100 training rounds of the robust obstacle avoidance algorithm based on the DDPG reinforcement learning model provided in this embodiment of the invention.

[0039] Figure 7 This is an obstacle avoidance test diagram of an unmanned surface vessel based on a robust obstacle avoidance strategy provided in an embodiment of the present invention, in different obstacle environments.

[0040] Figure 8This is a comparative experiment between the robust obstacle avoidance algorithm based on the DDPG reinforcement learning model provided in this embodiment of the invention and the traditional artificial potential field method. (a)-(c) show the trajectory, rudder angle δ, and heading φ of the unmanned vessel in a randomly generated obstacle environment under two different obstacle avoidance strategies.

[0041] Figure 9 This is a test diagram of the obstacle avoidance performance of a wave glider based on obstacle perception domain and LOS target tracking method in a large-scale uncertain obstacle environment, provided by an embodiment of the present invention.

[0042] Figure 10 This is a robustness test diagram of the obstacle avoidance control strategy under different target positions provided in the embodiments of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0044] like Figure 1 As shown in the figure, the method for large-scale dynamic obstacle avoidance and path tracking of unmanned surface vessels based on the fusion of deep reinforcement learning and line-of-sight navigation provided by the embodiments of the present invention includes the following steps in detail:

[0045] S101, constructing a training environment limited to a local spatial range;

[0046] S102, in each training round, only one dynamic obstacle is set, and the initial position and geometric dimensions of the obstacle are randomly changed in different training rounds;

[0047] S103, through the above-mentioned high-dynamic but low-complexity training method, the reinforcement learning control strategy focuses on learning the single obstacle avoidance maneuverability during the training process, thereby forming a robust obstacle avoidance control strategy that can be transferred to complex scenarios.

[0048] In this embodiment, the large-scale dynamic obstacle avoidance and path tracking method for unmanned surface vessels based on the fusion of deep reinforcement learning and line-of-sight navigation first constructs a training environment limited to a local spatial range during the training phase. This training environment reduces the state space dimension by spatially cropping the unmanned surface vessel's motion area, retaining only the local perception region highly relevant to obstacle avoidance decisions. This avoids training instability issues caused by large-scale environment modeling, thereby improving the convergence efficiency of the reinforcement learning strategy.

[0049] Based on this, in each training round, only one dynamic obstacle is set within the local spatial range, and the initial position, direction of motion, and geometric dimensions of the dynamic obstacle are randomly varied in different training rounds. By limiting the number of obstacles in each training round, the reinforcement learning model can focus on the relative motion relationship and avoidance maneuvering process between the unmanned vessel and a single dynamic obstacle during training, avoiding the amplification of the complexity of policy learning caused by the coupling interference of multiple obstacles.

[0050] Meanwhile, by randomizing obstacle parameters in different training rounds, the reinforcement learning model repeatedly learns obstacle avoidance control behavior under diverse relative motion scenarios, thereby enhancing the adaptability of the learned control strategy to changes in obstacle scale, relative velocity, and initial relative position. This training method effectively controls environmental complexity while maintaining the high dynamic characteristics of the training scenario, enabling the model to stably extract key features that influence obstacle avoidance decisions.

[0051] Through the aforementioned highly dynamic yet low-complexity training mechanism, the acquired reinforcement learning control strategy is independent of a specific number of obstacles or a fixed scene structure, instead forming an obstacle avoidance decision-making capability centered on relative motion relationships. When this strategy is applied to complex aquatic environments containing multiple dynamic obstacles, the system can achieve robust dynamic obstacle avoidance control in complex scenes by equivalently processing the perception results of multiple obstacles, thereby supporting the safe path tracking and stable operation of unmanned vessels during large-scale navigation missions.

[0052] The scale of the local spatial range provided in this embodiment of the invention is consistent with the obstacle perception range of the unmanned vessel in subsequent practical applications, so as to ensure the effectiveness of the trained control strategy in the application stage.

[0053] The present invention provides a control strategy that enables a uniform obstacle avoidance response capability for obstacles of different sizes and relative positions by randomly varying obstacle parameters in multiple training rounds.

[0054] The target generation method for unmanned surface vessel path tracking and dynamic obstacle avoidance provided in this embodiment of the invention includes:

[0055] An obstacle perception domain with a fixed radius is constructed centered on the current position of the unmanned vessel;

[0056] When the unmanned vessel travels along a preset path, a line-of-sight target point is generated in front of the preset path based on the path deviation and forward sight distance parameters.

[0057] The line-of-sight target point serves as a heading control target in path tracking mode and as a temporary navigation target for local obstacle avoidance control in obstacle avoidance mode.

[0058] The radius of the obstacle perception domain provided in this embodiment of the invention matches the scale of the local training environment used in the training phase, so as to ensure the perceptual consistency of the control strategy.

[0059] The line-of-sight target point provided in this embodiment of the invention is located in front of the current preset path and is used to characterize the entry position of the unmanned vessel on the return path after obstacle avoidance.

[0060] This invention provides a cooperative control method for unmanned surface vessel path tracking and dynamic obstacle avoidance, comprising:

[0061] When no obstacle is detected within the obstacle perception domain of the unmanned vessel, a path tracking control method based on line-of-sight target points is adopted;

[0062] When an obstacle is detected within the obstacle perception domain of the unmanned vessel, it switches to an obstacle avoidance control method based on reinforcement learning.

[0063] In obstacle avoidance control mode, the currently calculated line-of-sight target point is set as the short-term navigation target for obstacle avoidance control, so that the unmanned vessel can actively return to the preset path after completing local obstacle avoidance.

[0064] The obstacle avoidance control method provided in this embodiment of the invention generates control commands based on the unmanned vessel's own state, the obstacle's state, and the relative relationship between the unmanned vessel and the short-term navigation target.

[0065] The present invention provides a method that limits the short-term navigation target to the line-of-sight target point, so that the obstacle avoidance process is consistent with the path tracking target, and avoids the obstacle avoidance behavior from deviating from the global path.

[0066] The present invention provides a performance trade-off between path tracking accuracy and obstacle avoidance safety distance of unmanned surface vessels by adjusting the forward-looking distance parameter.

[0067] like Figure 2 As shown in the figure, the method for large-scale dynamic obstacle avoidance and path tracking of unmanned surface vessels based on the fusion of deep reinforcement learning and line-of-sight navigation provided by the embodiments of the present invention includes the following steps in detail:

[0068] S1: Feature Enhancement Dynamic Training (FEDT) strategy;

[0069] S2: Obstacle perception domain and LOS target point calculation;

[0070] S3: A perception-based dual-mode switching control mechanism;

[0071] S4: Performance trade-off parameter adjustment.

[0072] In this embodiment, the large-scale dynamic obstacle avoidance and path tracking method for unmanned surface vessels (USVs) based on the fusion of deep reinforcement learning and line-of-sight navigation first executes a Feature Enhancement Dynamic Training (FEDT) strategy during operation. This strategy constructs a state feature set including heading error, relative distance, obstacle motion state, and control action feedback to jointly model the obstacle avoidance and tracking behavior of the USV in a dynamic environment. During the training phase, key features are dynamically enhanced, enabling the deep reinforcement learning model to focus on state changes closely related to safe obstacle avoidance and path deviation during the decision-making process, thereby improving the strategy's perception sensitivity and decision-making stability in complex dynamic environments.

[0073] Subsequently, the obstacle perception domain and line-of-sight (LOS) target point calculation steps are performed. This step constructs a local obstacle perception domain around the UAV based on obstacle location information acquired by sensors and the UAV's own navigation status, characterizing potential collision risk areas. Simultaneously, combined with pre-set global trajectory information, target tracking points are dynamically selected on the trajectory using the LOS method, providing the UAV with a desired heading reference and ensuring good smoothness and directional guidance during path tracking.

[0074] Based on this, a perception-based dual-mode switching control mechanism is implemented. This mechanism switches control strategies in real time according to the risk assessment results within the obstacle perception domain. When no high-risk obstacles exist within the perception domain, the system adopts a path-tracking control mode based on LOS to reduce control jitter and ensure trajectory following accuracy. When a dynamic obstacle is detected entering a preset safety threshold range, the system automatically switches to an obstacle avoidance control mode based on deep reinforcement learning to achieve rapid avoidance of complex dynamic obstacles.

[0075] Finally, the performance trade-off parameter adjustment step is executed. This step dynamically adjusts the obstacle avoidance safety weight, path tracking weight, and control gain parameters based on the current navigation mission objectives and environmental changes, to achieve comprehensive optimization between obstacle avoidance safety, trajectory tracking accuracy, and energy consumption. Through the coordinated operation of the above steps, the unmanned surface vessel can achieve stable and efficient dynamic obstacle avoidance and path tracking control in a wide range of dynamic and complex aquatic environments.

[0076] The Feature Enhancement Dynamic Training (FEDT) strategy steps in S1 provided in this embodiment of the invention. To obtain a robust local obstacle avoidance strategy, this invention first trains an offline reinforcement learning agent based on the Deep Deterministic Policy Gradient (DDPG) algorithm; the training employs the "Feature Enhancement Dynamic Training (FEDT)" strategy:

[0077] S1.1: Construct a local training environment: Limit the training scene to a small local area (e.g., a square body of water of 100m×100m).

[0078] S1.2: Setting a Dynamic Single Obstacle: In each training episode, only one obstacle is set in the environment; however, the initial position and geometry of the obstacle are dynamically generated randomly at the start of each new episode.

[0079] S1.3: Training Objective: Through this highly dynamic but single-scene-element reinforcement training, the DDPG agent is forced to focus on mastering the core obstacle avoidance maneuverability to deal with various single obstacles, thereby obtaining a highly robust obstacle avoidance strategy model.

[0080] The obstacle perception domain and LOS target point calculation provided in embodiment S2 of this invention:

[0081] In practical navigation applications, this invention defines the perception range of the unmanned surface vessel and the target tracking method based on line of sight:

[0082] S2.1: Constructing the obstacle perception domain: Centered on the current position of the unmanned vessel, set a circular area with a fixed radius (e.g., R=100m) as the "obstacle perception domain"; the value of this radius R is consistent with the scale of the FEDT training environment to ensure the effective transfer of the training strategy in the new environment;

[0083] S2.2: Calculate the LOS line-of-sight point P LOS Based on the traditional line-of-sight (LOS) navigation method, a virtual line-of-sight target point P is calculated in front of the desired global path trajectory based on the current path tracking error e of the unmanned vessel and the preset lookahead distance parameter (Δ). LOS .

[0084] The perception-based dual-mode switching control mechanism in S3 provided in this embodiment of the invention:

[0085] This invention designs an automatic switching mechanism based on the state of the obstacle perception domain, dynamically switching between two control modes:

[0086] S3.1: Path tracking mode (no obstacles within the perception domain)

[0087] Triggering condition: This mode is activated when the obstacle perception range of the unmanned vessel (e.g., within a 100m radius) does not detect any obstacles;

[0088] Control strategy: The traditional LOS path tracking algorithm is adopted; the system directly points the desired heading of the unmanned vessel to the calculated LOS line-of-sight point P. LOS Its main task is to accurately and smoothly track the preset path;

[0089] S3.2: Robust obstacle avoidance mode (obstacles exist within the perception domain)

[0090] Triggering condition: When an obstacle is detected within the obstacle perception range of the unmanned vessel, it immediately switches to this mode;

[0091] Control strategy: Activate the DDPG reinforcement learning agent trained with the FEDT policy to take over control;

[0092] Key fusion mechanism: At this point, the system will merge the current LOS line of sight point P. LOS The temporary target point P is set as the "navigation target point" for the reinforcement learning agent; the agent adjusts its current observations (including its own state, obstacle information, and the distance from the temporary target point P). LOS (relative position), output control commands (such as rudder angle and speed);

[0093] Function: This mechanism ensures that, in obstacle avoidance mode, the short-term goal of the unmanned surface vessel is not a distant global endpoint, but rather "to bypass the current local obstacles and return to the current entry point on the preset path as quickly as possible (i.e., P)." LOS This cleverly balances obstacle avoidance safety and path tracking requirements.

[0094] Performance trade-off parameter adjustment in S4 provided in this embodiment of the invention

[0095] In this invention, the forward look-ahead distance parameter Δ is a key adjustable parameter used to balance tracking accuracy and obstacle avoidance safety:

[0096] Choosing a smaller Δ value can improve path tracking accuracy when there are no obstacles, but it will result in a shorter obstacle avoidance path and a reduced safety margin when avoiding obstacles.

[0097] Choosing a larger Δ value will appropriately reduce the path tracking accuracy, but it will enable the unmanned vessel to initiate obstacle avoidance actions earlier, increase the safe distance from obstacles, and improve safety.

[0098] Users can adjust the Δ value according to the actual task requirements.

[0099] like Figure 3 As shown in the figure, an embodiment of the present invention provides a large-scale dynamic obstacle avoidance and path tracking system for unmanned surface vessels based on the fusion of deep reinforcement learning and line-of-sight navigation, comprising:

[0100] The reinforcement module is used for the Feature Enhancement Dynamic Training (FEDT) strategy;

[0101] The target point calculation module is used for calculating the obstacle perception domain and the LOS target point.

[0102] A switching control module is used for a perception-based dual-mode switching control mechanism;

[0103] The parameter adjustment module is used for adjusting parameters to balance performance.

[0104] In this embodiment, when the unmanned surface vessel (USV) large-scale dynamic obstacle avoidance and path tracking system based on deep reinforcement learning and line-of-sight (LOS) navigation is running, the target point calculation module first performs perception modeling of the environment surrounding the USV. Based on obstacle position, velocity, and heading information collected by sensors, this module constructs a local obstacle perception domain and, combined with a preset global trajectory, calculates the desired tracking target point in real time using the line-of-sight (LOS) navigation method, thereby obtaining reference heading information that satisfies the trajectory constraints.

[0105] Subsequently, the reinforcement module receives the target point information and environmental state parameters output by the target point calculation module, and uses the Feature Reinforcement Dynamic Training (FEDT) strategy to perform online decision-making and reasoning on the deep reinforcement learning network. The reinforcement module constructs a multi-dimensional state space that includes distance error, heading deviation, obstacle risk, and control energy consumption, and imposes joint constraints on obstacle avoidance safety and path tracking accuracy based on the reward function, thereby outputting the optimal control action of the unmanned vessel at the current moment.

[0106] Based on this, the switching control module adaptively switches control modes according to the risk assessment results within the obstacle perception domain. When the unmanned vessel is in a low-risk navigation state, the system prioritizes the path tracking control mode based on LOS to ensure smooth trajectory and control stability; when a dynamic obstacle is detected entering the preset safety threshold range, the switching control module automatically activates the obstacle avoidance control mode based on deep reinforcement learning to achieve rapid response and safe avoidance in complex dynamic environments.

[0107] Meanwhile, the parameter adjustment module dynamically adjusts the path tracking weight, obstacle avoidance risk weight, and control gain parameters according to the navigation mission requirements and environmental changes, in order to optimize the performance trade-off between obstacle avoidance safety, tracking accuracy, and energy consumption. Through the coordinated work of the above modules, the unmanned surface vessel can achieve stable and efficient dynamic obstacle avoidance and path tracking in a wide range of dynamic and complex water environments.

[0108] Figure 4 and Figure 5 Description: A schematic diagram illustrating the working principle of an unmanned surface vessel control method based on the fusion of deep reinforcement learning and line-of-sight navigation.

[0109] The figures show the flowcharts for the offline training subsystem and the online operation and subsystem control, respectively.

[0110] A1: Create a small, localized virtual water training environment with a limited scale (e.g., 100m × 100m), which is consistent with the perceptual domain during actual operation.

[0111] A2: At the start of each training round, the initial position and geometry of a single obstacle can be dynamically generated randomly in the environment to ensure the high dynamism of the training samples.

[0112] A3: Through repeated iterations in the above-mentioned highly dynamic, single-element scenarios, the DDPG agent is forced to focus on mastering the core obstacle avoidance maneuverability to cope with various obstacles, and finally outputs an obstacle avoidance strategy model file with high generalization ability and robustness for the online control module to call.

[0113] B2: Based on the traditional line-of-sight (LOS) navigation principle, the current path tracking error e of the unmanned vessel is collected in real time, and combined with the preset lookahead distance (Δ), a virtual line-of-sight target point PLOS is dynamically calculated in front of the expected global path trajectory as the current navigation guide.

[0114] B3: In real time, a circular electronic fence with a fixed radius (R, for example, 100m) is drawn as the center of the current position of the unmanned vessel to serve as an "obstacle perception domain" for monitoring potential threats.

[0115] B4: Real-time monitoring of the status within the "obstacle perception domain". If no obstacle is detected within the perception domain, it is determined to be a safe state; if any obstacle is detected, it is immediately determined to be an obstacle avoidance state.

[0116] B5: Activate the traditional LOS path tracking algorithm. Direct the UAV's desired heading directly to the LOS line-of-sight point PLOS calculated from the forward-looking distance parameter Δ and the path tracking error e. Output control commands to guide the UAV to accurately and smoothly follow the preset global path.

[0117] B6: Activate the DDPG reinforcement learning agent trained in Module 1 to take over control. Key fusion mechanism execution: Dynamically change the current LOS line-of-sight point P. LOS It is temporarily set as the "short-term navigation target point" for the reinforcement learning agent.

[0118] B7: The agent, based on its current observations (its own state, obstacle information, and position relative to the temporary target point PLOS), outputs rudder angle and speed commands. This mechanism ensures that the unmanned vessel, while navigating around local obstacles, always aims to return to its current entry point on the preset path as quickly as possible.

[0119] Example 1: Construction of obstacle avoidance strategy under high randomness training with single obstacle

[0120] In this embodiment, a localized water training environment of a limited scale is first constructed, within which the unmanned surface vessel (USV) performs autonomous navigation tasks. In each training round, only one dynamic obstacle is placed within the environment. This obstacle has different initial positions and geometric dimensions in different rounds, so that the USV continuously faces changing obstacle avoidance situations during the training process.

[0121] Through multiple rounds of training, the unmanned surface vessel (USV) gradually develops a unified maneuvering response strategy for different single obstacle scenarios. This training method avoids interference from complex multi-obstacle scenarios in the learning process, enabling the USV to acquire stable obstacle avoidance capabilities under limited training conditions, providing a reliable foundation for subsequent practical applications in complex environments.

[0122] Example 2: Co-generation of obstacle perception domain and path forward-looking target

[0123] In this embodiment, during actual navigation, the unmanned surface vessel (USV) constructs an obstacle perception domain with a fixed radius centered on its current location to determine in real time whether obstacles exist in the surrounding waters. Simultaneously, based on the preset navigation path and the USV's path deviation, a line-of-sight target point is generated ahead of the path to guide its course.

[0124] This line-of-sight target point reflects both the path the unmanned vessel should return to and provides a clear reference position for the subsequent obstacle avoidance process. By keeping the scale of the perception domain consistent with that of the training phase, the obstacle avoidance strategy obtained during training can be directly adapted to the perception results during actual operation, thereby avoiding control failure or strategy mismatch issues.

[0125] Example 3: Path tracking control process in barrier-free conditions

[0126] In this embodiment, when no obstacle is detected within the obstacle perception domain of the unmanned surface vessel (USV), the system automatically enters path tracking mode. At this time, the USV uses the real-time calculated line-of-sight target point as the basis for heading control, and by continuously adjusting its heading and speed, it gradually approaches and stably tracks the preset path.

[0127] In this state, the control process prioritizes smoothness and accuracy, and the unmanned surface vessel (USV) does not trigger obstacle avoidance calculations, thereby reducing the system's computational burden. This implementation ensures that the USV can efficiently complete long-distance navigation missions within safe waters.

[0128] Example 4: Collaborative Control of Obstacle Avoidance and Path Regression When an Obstacle is Detected

[0129] In this embodiment, when an obstacle is detected within the obstacle perception domain of the unmanned vessel, the system immediately switches from path tracking to obstacle avoidance control. At this time, the control system no longer uses the global path endpoint as the navigation target, but instead sets the currently calculated line-of-sight target point as the short-term navigation target.

[0130] The unmanned surface vessel (USV) generates control commands based on its own state, the state of the obstacle, and its relative relationship with the short-term navigation target, completing partial detours while maintaining a safe distance. Once the obstacle leaves the perception field, the USV naturally sails towards the line-of-sight target point, thus smoothly returning to the preset path, achieving a balance between obstacle avoidance safety and path continuity.

[0131] Example 5: Performance Trade-off Application under Forward Sight Distance Parameter Adjustment

[0132] In this embodiment, the forward look-ahead distance parameter is adjusted according to specific task requirements. When performing high-precision path tracking tasks, a smaller forward look-ahead distance is selected to allow the unmanned surface vessel to more closely follow the preset path; when performing tasks with dense obstacles or high safety requirements, a larger forward look-ahead distance is selected to allow the unmanned surface vessel to initiate obstacle avoidance actions in advance.

[0133] In this way, the same control framework can be adapted to different application scenarios. It can achieve a flexible trade-off between path accuracy and obstacle avoidance safety without changing the overall control structure, demonstrating the adjustability and practical value of this method in actual engineering applications.

[0134] Evidence related to the technical effects obtained by the embodiments of the present invention.

[0135] A. Feature-enhanced dynamic training and algorithm robustness verification of unmanned surface vessels

[0136] To verify the robustness of the algorithm trained based on the DDPG reinforcement learning model, 100 different environments with 5 random obstacles were designed. In the 6000th training round, we extracted the obstacle avoidance control policy learned by the DDPG-based ROA algorithm. To obtain a reward function with better training performance, we first set different initial values ​​for Ki (i = 1,2,3,4). As the absolute value of K1 increases, it means that the action of moving towards the target will receive more reward, and the obstacle avoidance success rate also increases. However, when K1 exceeds 12, the obstacle avoidance success rate decreases significantly, meaning that the hull ignores the penalty for approaching the obstacle because the reward for moving towards the target is too high. By continuously adjusting the weights of the reward function, we finally obtained an obstacle avoidance policy based on reward function #6, achieving an obstacle avoidance success rate of 98%, as shown in Table 1. Figure 6 The diagram shows the reward feedback of the unmanned surface vessel (USV) during training, based on reward function #6. It can be observed that initially, the Actor neural network has no prior knowledge and therefore explores randomly. At this point, the USV lingers near the starting point or collides with obstacles, incurring a large penalty calculated based on the reward function. As training continues, the vessel approaches and eventually reaches the endpoint. The average reward per training epoch increases. After 5000 training epochs, the average reward remains at a high level.

[0137] Table 1 Obstacle Avoidance Success Rate under Different Reward Functions

[0138]

[0139] Four environments were selected from 100 randomly generated environments to demonstrate the robustness of the obstacle avoidance strategy, such as... Figure 7 As shown in the figure, gray circles represent obstacles, and red pentagrams represent the target location. The results demonstrate that the DDPG-based ROA algorithm can successfully avoid obstacles and reach the destination via a relatively short path, showcasing the algorithm's robustness.

[0140] B. Comparison of robust obstacle avoidance algorithm based on DDPG with traditional artificial potential field method

[0141] To demonstrate the advantages of the DDPG-based ROA algorithm, comparative simulation experiments were conducted with the traditional APF method. Figure 8 As can be seen, using the proposed DDPG-based ROA algorithm, the unmanned surface vessel (USV) can correctly handle the relationship between obstacle avoidance and target arrival. It can adjust its course in a timely manner based on its relationship with obstacles and targets, while also considering its own motion characteristics. In contrast, the APF method, based on simple distance calculations, suffers from poor adaptability. The APF method can produce sharp turns, which is unsuitable for USVs with limited maneuverability. Furthermore, if the obstacle layout is complex, it can easily get trapped in local optima, ultimately failing to avoid obstacles, such as... Figure 8 As shown in (a).

[0142] C. Obstacle avoidance testing in environments with a wide range of uncertain obstacles

[0143] To test the autonomous obstacle avoidance and path tracking performance of the unmanned surface vessel in a large-scale, uncertain obstacle environment, a 1000×1000 m obstacle environment was randomly generated, such as... Figure 9 As shown. When obstacles exist within the obstacle perception domain, a smaller Δ means a shorter obstacle avoidance path and a higher risk; while a larger Δ allows the unmanned surface vessel more space to take obstacle avoidance measures, resulting in safer obstacle avoidance. When there are no obstacles within the obstacle perception domain, Δ can be used to adjust the accuracy of path tracking. Sensitivity tests were conducted for different look-ahead distances Δ during simulation testing. Figure 9This paper demonstrates the obstacle avoidance results and performance of an unmanned surface vessel (USV) using an obstacle perception domain (APD) and a low-order object (LOS) target tracking method in a large-scale uncertain obstacle environment. Figure (a) shows the obstacle avoidance and path tracking trajectories with different look-ahead distances (Δ). Figure (b) shows the tracking errors of different trajectories. Figure (c) shows the distance between the USV and the nearest obstacle at different look-ahead distances (Δ). It can be seen that, under different look-ahead distance (Δ) settings, the USV can successfully achieve autonomous obstacle avoidance and path tracking in a large-scale uncertain obstacle environment using the APD and LOS target tracking method. Furthermore, the look-ahead distance (Δ) significantly affects the accuracy of path tracking during obstacle avoidance. When Δ is set smaller, the path tracking accuracy is relatively high. However, this also reduces the minimum distance between the USV and the obstacle, as shown by the red arrow in Figure (c), increasing the obstacle avoidance risk. Therefore, while ensuring path tracking accuracy, a larger Δ can effectively improve obstacle avoidance safety performance.

[0144] Furthermore, to further verify the robustness of the obstacle avoidance control strategy when facing different target locations, an obstacle environment was randomly generated. The unmanned surface vessel started from the center position and traveled in four different directions to reach the target point, such as... Figure 10 As shown in the figure, the blue circles represent the starting position, the red pentagrams represent the target positions in different directions, and the solid lines of different colors represent the obstacle avoidance trajectories of the unmanned surface vessel (USV) as it travels towards the target positions in different directions. The figure demonstrates that, facing different target positions, the USV can successfully avoid obstacles and reach the target positions using the DDPG-based ROA control strategy and the obstacle perception domain and LOS-based target tracking method, exhibiting good robustness.

[0145] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0146] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for large-scale dynamic obstacle avoidance and path tracking of unmanned surface vessels based on the fusion of deep reinforcement learning and line-of-sight navigation, characterized in that, Includes the following steps: Step 1: Construct a training environment limited to a local spatial range; Step 2: In each training round, only one dynamic obstacle is set up, and the initial position and geometric dimensions of the obstacle are randomly changed in different training rounds; Step 3: Through the above-mentioned high-dynamic but low-complexity training method, the reinforcement learning control strategy focuses on learning the single obstacle avoidance maneuverability during the training process, thereby forming a robust obstacle avoidance control strategy that can be transferred to complex scenarios.

2. The method as described in claim 1, characterized in that, The scale of the local spatial range is consistent with the obstacle perception range of the unmanned vessel in subsequent practical applications, so as to ensure the effectiveness of the trained control strategy in the application stage.

3. The method as described in claim 1, characterized in that, By randomly varying obstacle parameters over multiple training rounds, the control strategy achieves a unified obstacle avoidance response capability for obstacles of different sizes and relative positions.

4. A target generation method for unmanned surface vessel (USV) path tracking and dynamic obstacle avoidance, implementing the large-scale dynamic obstacle avoidance and path tracking method based on deep reinforcement learning and line-of-sight navigation fusion as described in any one of claims 1-3, characterized in that, include: An obstacle perception domain with a fixed radius is constructed centered on the current position of the unmanned vessel; When the unmanned vessel travels along a preset path, a line-of-sight target point is generated in front of the preset path based on the path deviation and forward sight distance parameters. The line-of-sight target point serves as a heading control target in path tracking mode and as a temporary navigation target for local obstacle avoidance control in obstacle avoidance mode.

5. The method as described in claim 4, characterized in that, The radius of the obstacle perception domain is matched with the scale of the local training environment used in the training phase to ensure the perceptual consistency of the control strategy.

6. The method as described in claim 4, characterized in that, The line-of-sight target point is located in front of the current preset path and is used to characterize the entry point of the unmanned vessel on the return path after obstacle avoidance.

7. A collaborative control method for unmanned surface vessel (USV) path tracking and dynamic obstacle avoidance, implementing the large-scale dynamic obstacle avoidance and path tracking method based on deep reinforcement learning and line-of-sight navigation fusion as described in any one of claims 1-3, characterized in that, include: When no obstacle is detected within the obstacle perception domain of the unmanned vessel, a path tracking control method based on line-of-sight target points is adopted; When an obstacle is detected within the obstacle perception domain of the unmanned vessel, it switches to an obstacle avoidance control method based on reinforcement learning. In obstacle avoidance control mode, the currently calculated line-of-sight target point is set as the short-term navigation target for obstacle avoidance control, so that the unmanned vessel can actively return to the preset path after completing local obstacle avoidance.

8. The method as described in claim 7, characterized in that, The obstacle avoidance control method generates control commands based on the unmanned vessel's own state, the obstacle's state, and the relative relationship between the unmanned vessel and the short-term navigation target.

9. The method as described in claim 7, characterized in that, By limiting the short-term navigation target to the line-of-sight target point, the obstacle avoidance process is kept consistent with the path tracking target, thus preventing the obstacle avoidance behavior from deviating from the global path.

10. The method as described in claim 7, characterized in that, By adjusting the forward-looking distance parameter, a performance trade-off between the path tracking accuracy and obstacle avoidance safety distance of the unmanned surface vessel can be achieved.

Images