An unmanned ship collision avoidance decision-making method based on navigation intention perception

Abstract

The application provides an unmanned ship collision avoidance decision-making method based on navigation intention perception, motion state data of a ship and a target ship is collected in real time through a multi-source sensor, a collision risk assessment model is constructed to screen threat targets; based on recursive Bayesian inference of a historical trajectory of the target ship, a posterior probability distribution of a navigation intention of the target ship is dynamically updated, the navigation intention is discretized into multiple behavior patterns; a state vector is constructed by taking the state of the ship, motion characteristics and intention distribution of the target ship as input into a strategy network based on reinforcement learning, and a collision avoidance control action is output; a compound reward function is used to guide optimization of the strategy network, and the reward function integrates distance reward, rule penalty and collision risk penalty.

Description

Technical Field

[0001] The present invention belongs to the technical field of unmanned vessel control (unmanned vessel collision avoidance, reinforcement learning), and specifically relates to an unmanned vessel collision avoidance decision-making method based on navigation intention perception. Background Art

[0002] In recent years, autonomous collision avoidance methods based on deep reinforcement learning (DRL) have become a research hotspot in the field of autonomous navigation for unmanned underwater vehicles. These methods, through interaction with the environment and leveraging reward signals to guide policy learning, enable end-to-end collision avoidance decision-making, demonstrating excellent environmental adaptability and policy optimization capabilities. Compared to traditional rule-based or optimization-based methods, DRL not only handles complex nonlinear decision-making problems but also achieves superior collision avoidance in dynamic scenarios with multiple objectives and constraints, offering broad application prospects.

[0003] In the DRL framework, the construction of the state space plays a decisive role in the performance of the strategy. To enable unmanned vehicles to accurately perceive the environment and make reasonable decisions, researchers have proposed a variety of state modeling methods. Common practices include: 1) a risk-ranking-based approach, which selects only the target vessels with the highest threat level at the current moment to construct the state vector; 2) a spatial partitioning-based approach, which divides the perception area into multiple sectors and combines a risk-ranking-based approach to select the most representative targets in each sector; 3) a grid-map-based representation method, which converts perceived obstacle information into grid-map data, thereby improving the strategy's ability to model multi-target interaction scenarios.

[0004] However, existing methods generally only utilize the target vessel's current geometric and kinematic state (such as position, speed, and heading) as input features. These methods lack modeling of the target vessel's changing behavior trends and overlook the navigational intention information embedded in its historical trajectory. In real-world maritime environments, target vessels often exhibit significant behavioral continuity, and their future actions can be largely inferred from past dynamics. Relying solely on single-moment states, reinforcement learning strategies struggle to accurately determine target behavior patterns. This can lead to response lags, strategy failure, and increased collision risk, especially when faced with sudden changes in direction or non-cooperative vessels. Therefore, it is imperative to introduce a dynamic inference mechanism for navigational intention at the state modeling level to characterize target behavior at a higher level and enhance the stability of the unmanned vehicle's collision avoidance strategy. To address this issue, this paper proposes a collision avoidance decision-making method for unmanned vehicles based on navigational intention perception. By designing a navigational intention perception module and explicitly integrating this information into the state space, the method effectively enhances the unmanned vehicle's collision avoidance decision-making capabilities in environments with uncertain target vessel behavior, ensuring safe navigation.

[0005] The invention with application number CN202510296976.6 is “Ship collision avoidance decision-making method and device in restricted waters based on uncertainty modeling”.

[0006] The invention with application number CN202210962382.0 is "An intelligent ship collision avoidance path planning method based on uncertain speed obstacles".

[0007] The invention with application number CN202411845856.9 is "A full-process path planning method for autonomous collision avoidance of intelligent ships."

[0008] The invention with application number CN202411439576.8 is “A method for constructing a high-reliability ship autonomous collision avoidance model”.

[0009] The above solutions are related existing technologies, but none of them fully consider the impact of the uncertainty of the target ship's navigation intentions during the unmanned boat collision avoidance process on the stability and safety of the decision-making strategy. They generally only construct the state space based on the geometry and motion state of the target ship at the current moment, ignoring the behavioral trend information contained in its historical trajectory. In actual navigation, the target ship will exhibit nonlinear and variable behavioral characteristics, such as sudden acceleration and deceleration, temporary turning, etc., which is especially significant when facing non-cooperative or small ships. Existing methods fail to effectively model the uncertainty of such behaviors, resulting in the strategy's lack of ability to predict the target's future actions, prone to response lags, increased risk of misjudgment, and other problems, reducing the adaptability and robustness of the collision avoidance strategy in complex dynamic environments. Summary of the Invention

[0010] In view of the defects and shortcomings of the existing technology, the present invention provides an unmanned boat collision avoidance decision-making method based on the deep integration of recursive Bayesian intention modeling and reinforcement learning. Its innovative design points include: dynamically analyzing the historical trajectory of the target ship through recursive Bayesian estimation, discretizing the navigation intention into 9 types of combination modes of heading (turn left / straight / turn right) and speed (acceleration / constant speed / deceleration), and updating the posterior probability distribution of intention in real time; constructing a multi-dimensional state vector including the state of the own ship, the motion characteristics of the target ship and the intention distribution, driving the Soft The Actor-Critic (SAC) strategy network makes forward-looking decisions. A five-factor collision risk assessment model (DCPA / TCPA / relative heading / distance / speed) is combined to screen threat targets, and a composite reward function is designed that integrates distance reward, heading reward, international rule penalty, and speed obstacle penalty to enforce the strategy to comply with COLREGs specifications. A dynamic simulation environment is used to simulate the target ship's random behavior of "maintaining speed and heading, avoiding collisions, and intentional collisions" (probability distribution [0.3, 0.3, 0.4]), enhancing the strategy's robustness in non-cooperative scenarios. This system systematically integrates multi-source perception (GNSS / radar / AIS) data with a Bayesian reasoning-reinforcement learning framework, effectively addressing the response lag and misjudgment problems caused by traditional methods that ignore the target ship's behavioral trends.

[0011] The solutions adopted by the present invention to solve the technical problems specifically include:

[0012] A collision avoidance decision-making method for unmanned boats based on navigation intention perception:

[0013] Multi-source sensors collect real-time motion data of the own ship and target ship, and build a collision risk assessment model to screen threat targets;

[0014] Based on recursive Bayesian reasoning of the target ship's historical trajectory, the posterior probability distribution of the target ship's navigation intention is dynamically updated, and the navigation intention is discretized into multiple behavior patterns;

[0015] The state of the own ship, the motion characteristics of the target ship, and the intention distribution are constructed into a state vector, which is input into the policy network based on reinforcement learning and outputs the collision avoidance control action;

[0016] The policy network optimization is guided by a composite reward function that combines distance reward, rule penalty, and collision risk penalty.

[0017] Furthermore, the multiple behavior modes of the navigation intention include a combination of heading left turn, straight ahead, right turn and speed acceleration, constant speed, and deceleration, for a total of 9 modes.

[0018] Furthermore, the recursive Bayesian reasoning includes:

[0019] The dynamic model predicts the state based on the target ship's surge acceleration and bow acceleration;

[0020] Modeling observation noise through Gaussian distribution and calculating the likelihood probability of predicted state and actual observation;

[0021] Recursively update the posterior probability distribution of intent.

[0022] Furthermore, the construction of the state vector gives priority to target ship data with the highest collision risk index, and fills in the insufficiency with zero vectors.

[0023] Furthermore, the composite reward function includes an International Collision Avoidance Regulations penalty term and a speed barrier penalty term, and a linear penalty is triggered when an action violates the rules or enters a speed barrier area.

[0024] Furthermore, the reinforcement learning-based policy network is a SAC architecture, and the output action is sampled from a Gaussian distribution through a reparameterization technique and mapped to a physical control range through a tanh function.

[0025] Furthermore, the method is trained in a dynamic simulation environment, and the behavior pattern of the target ship is randomly selected according to a preset probability, including three strategies: maintaining speed and direction, collision avoidance, and intentional collision.

[0026] Furthermore, the multi-source sensors include a global navigation satellite system, a radar and an automatic identification system, and the collected data are input into the model after being pre-processed by anomaly elimination, time synchronization and coordinate unification.

[0027] Furthermore, the collision risk assessment model calculates the risk index by linearly weighting five factors, including relative distance, relative speed, relative heading, minimum collision distance and time to closest point.

[0028] And, an unmanned boat collision avoidance decision system based on navigation intention perception, comprising:

[0029] Multi-source perception module, used to collect motion status data of own ship and target ship in real time;

[0030] Threat target screening module, used to build a collision risk assessment model to screen threat targets;

[0031] The intention reasoning module is used to dynamically update the posterior probability distribution of the target ship's navigation intention based on recursive Bayesian reasoning based on the target ship's historical trajectory;

[0032] Strategy network module, used to generate collision avoidance control actions based on own ship status, target ship motion characteristics and intention distribution;

[0033] The control execution module is used to convert action instructions into ship heading and speed control signals.

[0034] And, an electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the program.

[0035] A non-transitory computer-readable storage medium stores a computer program, which implements the steps of the method described above when executed by a processor.

[0036] Compared with the prior art, the present invention and its preferred embodiments have at least the following beneficial effects:

[0037] Through the deep integration of recursive Bayesian and reinforcement learning, the system breaks through the limitations of traditional collision avoidance methods in perceiving target vessel behavior trends. Dynamically modeling the distribution of intentions based on historical trajectories significantly improves the ability to predict and make real-time decisions in complex scenarios such as sudden changes in direction and non-cooperative targets.

[0038] A composite reward function design integrates international collision avoidance regulations, speed barrier constraints, and a multi-dimensional risk indicator system to drive the policy network to achieve a dynamic balance between safety, compliance, and economy, avoiding suboptimal decisions caused by rule violations or misjudgment of risks.

[0039] The dynamic simulation environment builds logic, simulating the target ship's random behavior patterns of "maintaining speed and direction, avoiding collisions, and intentionally colliding." This strengthens the strategy's generalization capabilities in uncertain interaction scenarios and addresses the strategy vulnerability caused by the single nature of traditional training scenarios.

[0040] Multi-source perception and a five-factor risk assessment model, combining heterogeneous GNSS, radar, and AIS data fusion with anomaly rejection mechanisms, improves environmental perception reliability. It also quantifies collision risk through linear weighting, enabling efficient screening and prioritization of threat targets.

[0041] The SAC algorithm framework is optimized based on a re-parameterized action sampling and physical constraint mapping mechanism, taking into account both strategy exploration efficiency and the engineering feasibility of control instructions, ensuring the smoothness and execution stability of collision avoidance actions. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0043] Figure 1 This is a flow chart of the collision avoidance decision-making process of an unmanned boat according to an embodiment of the present invention.

[0044] Figure 2 This is a schematic diagram of the target ship's navigation intention according to an embodiment of the present invention.

[0045] Figure 3 This is a schematic diagram of the target ship generation process according to an embodiment of the present invention.

[0046] Figure 4 Schematic diagram of the algorithm structure of an embodiment of the present invention. DETAILED DESCRIPTION

[0047] In order to make the features and advantages of the present invention more clearly understood, the following embodiments are given for detailed description:

[0048] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used in this specification have the same meanings as those commonly understood by those skilled in the art to which this application belongs.

[0049] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0050] To address the problem that existing solutions fail to adequately model the uncertainty of target vessel navigation intentions, resulting in delayed and insufficiently robust collision avoidance strategies when facing non-cooperative targets or dynamic behavior changes, making it difficult for unmanned vehicles (UAVs) to effectively cope with highly dynamic and variable navigation environments, this paper proposes an intelligent collision avoidance method for UAVs based on the fusion of Bayesian navigation intention reasoning and deep reinforcement learning. By introducing a recursive Bayesian estimation method to dynamically analyze the historical trajectory of the target vessel, probabilistic modeling of navigation intentions is achieved. This is then explicitly embedded into the state space as a navigation feature. Combined with deep reinforcement learning strategy training, this method effectively improves the collision avoidance strategy's ability to perceive and predict target behavior trends, thereby enhancing the UAV's decision-making stability, safety, and generalization capabilities in complex and uncertain sea conditions.

[0051] To this end, an embodiment of the present invention proposes an intelligent collision avoidance method for an unmanned boat based on Bayesian intention reasoning and the SAC algorithm. By introducing a recursive Bayesian method to dynamically infer the navigation intention of the target ship, and embedding the intention distribution as the navigation feature of the target ship into the state space, the reinforcement learning strategy is guided to make forward-looking decisions, thereby improving the collision avoidance capability and strategy stability of the unmanned boat in complex, dynamic and uncertain environments.

[0052] The collision avoidance decision flow chart proposed in the embodiment of the present invention is as follows: Figure 1 As shown, it mainly includes the following steps:

[0053] Step 1: Environmental perception to obtain navigation information;

[0054] Step 2: Determine whether there are any obstacle ships around the unmanned boat based on the data from step 1;

[0055] Step 3: Determine whether the obstructing vessel poses a collision risk to the own vessel;

[0056] Step 4: If there is no collision risk, maintain the current navigation state; if there is a collision risk, proceed to step 5;

[0057] Step 5: Infer the navigation intention of the obstacle ship;

[0058] Step 6: Calculate model input;

[0059] Step 7: Make collision avoidance decisions;

[0060] Step 8: Execute collision avoidance decision and determine whether to navigate to the target point;

[0061] Step 9: If the target point has not been reached, repeat steps 1-7 until the target point is reached.

[0062] For step 1: By integrating multi-source sensing equipment such as the Global Navigation Satellite System (GNSS), radar sensors, and the Automatic Identification System (AIS), the system collects real-time information on the vessel's navigation status and surrounding environmental targets. This data includes the vessel's position, speed, heading, and angular velocity, as well as the relative position, relative speed, and heading angle of surrounding targets. After undergoing anomaly elimination, time synchronization, and coordinate system preprocessing, this data becomes the fundamental input for obstacle identification and collision avoidance decision-making, ensuring the system's stable and reliable environmental perception capabilities in dynamic and complex environments.

[0063] For step 2: Based on the multi-source perception information obtained in step 1, effectively identify the target objects in the environment to achieve preliminary screening and classification of obstacles. The specific steps are as follows:

[0064] Step 2.1: The raw perception data acquired by multi-source sensor systems such as GNSS, radar, and AIS are pre-processed to extract the basic navigation characteristics of each target object;

[0065] Step 2.2: Eliminate noise data and retain only the suspected obstacle data with physical consistency and navigation characteristics;

[0066] Step 2.3: Extract target ship data.

[0067] For step 3: After extracting the target ship data, the system performs a collision risk assessment on all target ships to determine whether they pose a potential threat to the host ship. The specific assessment method uses a typical five-factor collision risk assessment model, which comprehensively considers factors such as the relative distance between the two ships, relative speed, relative heading, minimum collision distance (DCPA), and closest point time (TCPA) to calculate the target ship's collision risk index. The overall risk index is calculated using the following linear weighted model:

[0068] (1)

[0069] Where J is the weight vector of each factor, reflecting its relative contribution to the collision risk, and M is the fuzzy membership function of each risk factor, with a value range of [0,1].

[0070] For step 4: Based on the collision risk index calculated in step 3, determine whether there are any target ships that pose a substantial threat to the own vessel. If the risk assessment results for all target ships are equal to 0, the system determines that the current navigation environment is safe and no collision avoidance decision is required. The unmanned vehicle controller maintains the current navigation state. Conversely, if there is at least one target ship with a collision risk index greater than 0, proceed to step 5.

[0071] For step 5: When a target ship poses a potential collision threat to the host ship, a recursive Bayesian approach is used to infer the navigation intention of this target ship. This approach dynamically models the deviation between the target ship's historical motion state and the current observation to estimate the posterior probability distribution of its behavioral intention. This is then used as a partial feature and combined with the current navigation data to construct the collision avoidance decision model input. The specific steps are as follows:

[0072] Step 5.1: Initialize the prior intent distribution.

[0073] The present invention uses the navigation intention of the target ship It is discretized into nine typical modes, representing the combined action combination of heading (turn left / go straight / turn right) and speed (acceleration / constant speed / deceleration), such as Figure 2 As shown. At the initial moment, the system assigns a uniform prior distribution to each target ship:

[0074] (2)

[0075] Step 5.2: Calculate the predicted state.

[0076] The observed state of the target ship at time t-1 is , where (x, y) represents the position of the target ship, ψ is the target ship heading angle, and (u, v, r) are the target ship's surge, sway, and bow angular velocity, respectively. Calculate the predicted state at time t according to the prediction model The prediction model is as follows:

[0077] (3)

[0078] where a u 、a r are the surge and bow acceleration of the target ship respectively.

[0079] Step 5.3: Calculate the likelihood estimate.

[0080] The predicted state The actual observation state of the target ship at the current moment O t Compare and calculate the intention i The likelihood probability of the current observation occurring under . Assuming that the observation noise follows a zero-mean Gaussian distribution, the likelihood function is expressed as:

[0081] (4)

[0082] Where ∑ is the observation error covariance matrix and n=6 is the state dimension.

[0083] Step 5.4: Intent posterior update.

[0084] The Bayesian formula is used to update the posterior probability of each candidate intent. The calculation formula is as follows:

[0085] (5)

[0086] The updating process is recursively performed in each control cycle to form the posterior distribution vector of the target ship's behavior intention at the current moment .

[0087] In step 6, after inferring the target ship's navigation intention, the system constructs the input vector for the collision avoidance strategy network based on the current state information of the own ship and the target environment. This input vector consists of four parts: the own ship's navigation state, navigation target information, the target ship's motion state, and the target ship's navigation intention distribution.

[0088] (6)

[0089] The navigation status of the own ship is mainly determined by the longitudinal and longitudinal motion of the ship. os 、Hangdangv os and the bow angular velocity r os , as shown below:

[0090] (7)

[0091] Navigation target information mainly consists of the relative azimuth between the own ship and the target point and relative heading angle Composition is as follows:

[0092] (8)

[0093] The target ship's motion state is determined by the relative distance between the host ship and the target ship. , relative azimuth , relative heading angle , collision risk index C RI 、You will encounter situation C OL and the target ship's bow angular velocity r TS and speed V, as shown below:

[0094] (9)

[0095] The target ship intention distribution is obtained by step 5, as shown below:

[0096] (10)

[0097] When constructing the input vector, the navigation information and intention distribution of the target ship with the highest collision risk index are prioritized. If the number is insufficient, it is padded with zero vectors to ensure consistent input dimensions. All feature variables are normalized before input to improve training stability and strategy convergence efficiency.

[0098] Regarding step 7: After constructing the state vector, the system feeds it as input to the deep reinforcement learning policy network to calculate the collision avoidance action to be performed at the current moment. Preferably, this embodiment uses the policy network in the Soft Actor-Critic (SAC) architecture to generate actions. The policy network takes the state s as input and outputs a Gaussian distribution with mean μ(s) and standard deviation σ(s), representing the probability distribution of actions that can be taken in the current state. The system then samples actions from this distribution using a reparameterization technique:

[0099] (11)

[0100] Action a represents the current collision avoidance control command, typically the rudder angle used to control the ship's heading. Before execution, the action is mapped using a tanh function to ensure its range complies with the physical constraints of the ship's control.

[0101] In step 8, the collision avoidance control actions output by the policy network are analyzed, converted into corresponding control commands, and transmitted to the UAV's underlying control system. The control system adjusts the vessel's heading and speed accordingly, implementing the current collision avoidance decision.

[0102] For step 9: After each control cycle, the system evaluates the distance between the own ship and the target point in real time based on the latest own ship position data. If the distance is less than a preset threshold, the own ship is deemed to have successfully reached the target point, and the collision avoidance process terminates. Otherwise, the system continues to collect environmental state data, reconstructs the state vector, and enters the next collision avoidance decision cycle.

[0103] The unmanned underwater vehicle collision avoidance method based on Bayesian intention inference, as proposed in the embodiments of the present invention, addresses the shortcomings of existing collision avoidance strategies, which rely on a single state-space model and struggle to represent the uncertainty of target vessel behavior. Instead, it proposes a modeling mechanism that dynamically incorporates the target vessel's navigation intention distribution into the state space. By performing recursive Bayesian inference on the target vessel's historical motion trajectory, its potential navigation intention is inferred in real time. The inference results are embedded as navigation features in the state input, effectively enhancing the state space's ability to represent target behavior trends.

[0104] By introducing intent modeling, this invention enables the unmanned watercraft collision avoidance decision-making system to not only judge environmental conditions based on current geometric information but also make forward-looking decisions based on target behavior trends, thereby improving the strategy's response speed and decision-making accuracy. Compared with traditional methods that rely on modeling the current instantaneous state, this technology significantly enhances the adaptability of deep reinforcement learning strategies in dynamic and changing environments, improves convergence stability during training, and effectively improves the collision avoidance success rate and overall navigation safety during actual navigation.

[0105] To improve the UAV's collision avoidance capabilities under uncertain target vessel behavior, the above embodiment proposes a collision avoidance decision-making method based on the SAC algorithm and integrated with a recursive Bayesian intention inference mechanism. This method addresses the lack of adaptability of traditional collision avoidance strategies when faced with target vessel behavior uncertainty. By dynamically estimating the target vessel's navigation intention through recursive Bayesian inference and integrating the posterior distribution of the intention into the state space as an explicit feature, the collision avoidance strategy's ability to perceive the target's behavior patterns improves, achieving more robust collision avoidance decisions.

[0106] To train and validate the algorithm, the collision avoidance strategy proposed in this embodiment was implemented during the training phase using a simulation platform built using the Python language and the OpenAI Gym framework. This platform simulates a two-dimensional ocean environment with a 200 x 200 meter area and a 1-second simulation step. It supports dynamic encounters with multiple target vessels simultaneously. The unmanned vehicle is modeled using a three-degree-of-freedom (3-DOF) nonlinear dynamics model. State variables include position, surge velocity, heading, sway velocity, and bow angular velocity. Control inputs consist of a fixed thrust and an adjustable yaw moment. The target vessels use a consistent dynamics model to ensure interaction consistency.

[0107] The unmanned boat in the simulation environment adopts a three-degree-of-freedom kinematic model, and the model form is as follows:

[0108] (12)

[0109] Where T and τ are thrust and torque input respectively; (m1, m2) is the additional mass, is the damping coefficient, and L is the moment of inertia.

[0110] To ensure numerical stability and simulation accuracy, the platform uses the fourth-order Runge-Kutta method to perform numerical integration calculations on the above differential equations.

[0111] In order to generate training scenarios with collision risks, a set of dynamic target ship (TS) generation logic is designed, such as Figure 3 The specific generation process is as follows:

[0112] Step A1: First, randomly determine the number of target ships according to the preset probability distribution and the type of encounter with own ship (OS);

[0113] Step A2: For each target ship, calculate its possible heading range according to the selected encounter type and randomly generate a heading ;

[0114] Step A3: Randomly generate target ship speed and time of collision with own ship ;

[0115] Step A4: Randomly generate collision time , calculate the initial coordinates of the target ship according to the following equations :

[0116] (13)

[0117] Step A5: If it is the first target ship, it is directly retained; otherwise, it is necessary to determine whether the distance between it and the existing target ships meets the safety threshold to avoid generating target ships with too dense positions;

[0118] Step A6: Repeat steps A2 to A5 until the target number of ships is reached.

[0119] The TS generation mechanism is dynamically executed at the beginning of each training round to generate scenarios containing multiple target ships, high-risk encounters, behavioral diversity and uncertainty for training the collision avoidance model.

[0120] Furthermore, to simulate the uncertainty of TS navigation in real-world ocean environments, the simulation platform divides TS navigation behaviors into two types: collision avoidance and random behavior. When encountering an obstacle, the TS with collision avoidance behavior uses the velocity obstacle (VO) method to calculate and execute collision avoidance decisions. The TS with random behavior selects a navigation strategy in real time from three options: "maintaining speed and direction," "collision avoidance," and "intentional collision," with probabilities of [0.3, 0.3, 0.4], forming an uncertain strategy.

[0121] In order to realize efficient collision avoidance strategy training for unmanned vehicles in complex dynamic environments, this paper designs a collision avoidance decision-making method integrating navigation intention perception based on the Soft Actor-Critic (SAC) reinforcement learning framework. Its overall structure is as follows: Figure 4 The algorithm consists of a policy network and a value network. The policy network consists of a neural network that generates control actions. It also consists of two value evaluation networks and two corresponding target networks that evaluate the value of actions. In the policy network design, state space inputs include the own ship's navigation state, target point information, motion characteristics of surrounding target ships, and their posterior distribution of intentions. These inputs are preprocessed and uniformly encoded into fixed-length vectors, which are then fed into the policy and value networks.

[0122] The policy network uses a four-layer fully connected structure, with each layer containing 256 neurons and a Reluctant Unit (ReLU) activation function. The output layer generates the mean and standard deviation of a Gaussian distribution and samples the action vector using a differentiable random reparameterization technique. Taking into account actual control requirements, the action is mapped to the corresponding steering thrust to achieve steering control of the unmanned vehicle. All neural network structures in the value network are similar to the policy, consisting of four fully connected layers. The input is a concatenated vector of state and action, and the output is a Q-value estimate for the state-action pair. To alleviate the problem of Q-value overestimation, a dual-Q structure is adopted with a target network. Parameters are updated using an exponential sliding average to improve the stability of policy training.

[0123] During the training phase, the Adam optimizer was used, with an experience replay pool capacity of 200,000, a batch size of 256, a discount factor of 0.99, and a target network soft update coefficient of 0.005. The remaining hyperparameters remained consistent with the default values ​​in the SAC algorithm.

[0124] To effectively train autonomous collision avoidance strategies, a composite reward function is constructed by comprehensively considering target distance rewards, heading rewards, collision penalties, collision risk penalties, speed obstacle penalties, and international collision avoidance regulations penalties. The total reward function is calculated as follows:

[0125] (14)

[0126] in They represent the weighting coefficients of distance reward, heading reward, collision risk penalty, collision penalty, violation of collision avoidance rules penalty and speed obstacle penalty respectively. The specific values ​​of each reward and the specific meanings of each sub-item are as follows:

[0127] 1. Distance bonus (r d )

[0128] Encourage the unmanned boat to continue moving towards the target point. The specific calculation is:

[0129] (15)

[0130] in 、 They represent the Euclidean distance from the unmanned boat to the target point at the previous moment and the current moment respectively. represents the speed of the unmanned boat at the last moment. j represents the simulation step size.

[0131] 2. Heading reward item (r h )

[0132] The unmanned boat is encouraged to adjust its course towards the target point. The calculation formula is:

[0133] (16)

[0134] 3. Collision risk penalty (r cri )

[0135] This item is used to penalize the unmanned boat for taking collision avoidance actions that increase navigation risks. The calculation is as follows:

[0136] (17)

[0137] in Indicates the collision risk between own ship and the most dangerous target ship.

[0138] 4. Collision penalty term (r coll )

[0139] Triggered when the agent collides or successfully reaches the target point, the calculation is as follows:

[0140] (18)

[0141] in 、 、 and They respectively represent the distance between the unmanned boat and the target point, the distance between the unmanned boat and the i-th opponent ship, the distance threshold for judging whether the target point has been reached, and the safety area radius of the i-th opponent ship.

[0142] 5. Penalty for violation of collision avoidance rules (r C )

[0143] In order to improve the model's ability to comply with the International Regulations for Preventing Collisions at Sea, this penalty term is added in the single-vessel intersection scenario. This term is used to guide the unmanned boat to take avoidance measures that comply with the collision avoidance regulations. Its calculation is shown in the following formula:

[0144] (19)

[0145] 6.VO penalty term (r VO )

[0146] In ship collision avoidance, VO is a collision avoidance modeling method based on velocity space. It defines that within a given time window, if the speed of the OS falls within the speed barrier zone formed by the TS, the two ships may collide. To this end, this paper introduces a VO penalty term into the reward function to penalize action choices that potentially lead to collision trajectories during reinforcement learning training, thereby driving the strategy to automatically avoid unsafe speed zones. Its calculation is shown in the following formula:

[0147] (20)

[0148] Through the proposed technical solution, the present invention organically integrates recursive Bayesian and deep reinforcement learning frameworks to enhance the autonomous collision avoidance capabilities of unmanned aerial vehicles (UAVs) under conditions of target vessel behavior uncertainty. In its implementation, the target vessel's navigation intention is dynamically estimated recursively, and its posterior probability distribution is explicitly introduced into the state space, effectively enhancing the modeling capabilities of the policy network and value network for the target vessel's behavior patterns. Furthermore, a multi-factor composite reward function system is designed to guide the agent in learning action strategies that better comply with maritime collision avoidance regulations and provide increased safety during training. To ensure the adaptability and generalization of the training strategy, the present invention constructs a highly dynamic and uncertain simulation environment. The environment uses randomly generated parameters such as the number of target vessels, heading, speed, and position, encompassing a variety of typical behavior patterns, including "speed and direction maintenance," "collision avoidance," and "intentional collision." Combining these technical approaches, the present invention achieves high-precision dynamic modeling and strategy optimization of target behavior intentions, effectively improving the robustness and efficiency of the UAV's collision avoidance strategy under conditions of multi-target intersections, behavioral uncertainty, and rule constraints. This approach demonstrates excellent engineering practicality and is suitable for a variety of autonomous navigation application scenarios, including intelligent shipping, maritime monitoring, and rescue and search.

[0149] Based on the same inventive concept, the present invention also provides a computer device, which includes: one or more processors and a memory for storing one or more computer programs; the program includes program instructions, and the processor is used to execute the program instructions stored in the memory. The processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing core and control core of the terminal, which is used to implement one or more instructions, specifically for loading and executing one or more instructions in a computer storage medium to implement the above method.

[0150] It should be further explained that, based on the same inventive concept, the present invention also provides a computer storage medium having a computer program stored thereon, which, when executed by a processor, performs the above-described method. The storage medium may be any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more conductors, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0151] It should be noted that, unless otherwise defined, the technical or scientific terms used in the present invention should have the usual meanings understood by people with ordinary skills in the field to which the present invention belongs. The "first", "second" and similar words used in the present invention do not indicate any order, quantity or importance, but are only used to distinguish different components. "Include" or "comprise" and similar words mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. "Connect" or "connected" and similar words are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0152] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any other manner. Any person skilled in the art may utilize the above-disclosed technical content to modify or modify the present invention into equivalent embodiments. However, any simple modifications, equivalent variations, and modifications to the above embodiments that do not depart from the technical content of the present invention and are based on the technical essence of the present invention remain within the scope of protection of the present invention.

[0153] The present invention is not limited to the above-mentioned optimal implementation mode. Anyone can derive various other forms of unmanned boat collision avoidance decision-making methods based on navigation intention perception under the inspiration of the present invention. All equal changes and modifications made within the scope of application of the present invention should fall within the scope of the present invention.

Claims

1. A collision avoidance decision-making method for an unmanned vehicle based on navigation intention perception, characterized by: Multi-source sensors collect real-time motion data of the own ship and target ship, and build a collision risk assessment model to screen threat targets; Based on recursive Bayesian reasoning of the target ship's historical trajectory, the posterior probability distribution of the target ship's navigation intention is dynamically updated. The navigation intention is discretized into multiple behavior patterns; the multiple behavior patterns of the navigation intention include a combination of left turn, straight ahead, right turn, and acceleration, constant speed, and deceleration, for a total of 9 patterns; The recursive Bayesian inference includes: The dynamic model predicts the state based on the target ship's surge acceleration and bow acceleration; Modeling observation noise through Gaussian distribution and calculating the likelihood probability of predicted state and actual observation; Recursively update the posterior probability distribution of intention; The state vector of the own ship, the motion characteristics of the target ship, and the intention distribution are constructed and input into the strategy network of the SAC architecture to output the collision avoidance control action. The state vector is constructed by giving priority to the target ship data with the highest collision risk index, and any insufficient data is padded with zero vectors. The policy network is optimized using a composite reward function that combines distance rewards, rule penalties, and collision risk penalties. The composite reward function includes an International Collision Avoidance Regulations penalty and a speed barrier penalty, triggering a linear penalty when an action violates the rules or enters a speed barrier area. The method is trained in a dynamic simulation environment. The simulation platform divides the navigation behavior of a dynamic target ship into two types: collision avoidance behavior and random behavior. When encountering an obstacle, the dynamic target ship with collision avoidance behavior uses the speed obstacle method to calculate and execute collision avoidance decisions. The dynamic target ship with random behavior selects a navigation strategy in real time from the three behaviors of "maintaining speed and direction," "collision avoidance," and "intentional collision" with a probability of [0.3, 0.3, 0.4] during navigation, forming an uncertain strategy. The output action is sampled from a Gaussian distribution via a reparameterization technique and mapped to the physical control range via a tanh function.

2. The unmanned vehicle collision avoidance decision-making method based on navigation intention perception according to claim 1 is characterized in that: The multi-source sensors include a global navigation satellite system, a radar, and an automatic identification system. The collected data are input into the model after being pre-processed with anomaly elimination, time synchronization, and coordinate unification.

3. The unmanned vehicle collision avoidance decision-making method based on navigation intention perception according to claim 1 is characterized in that: The collision risk assessment model calculates the risk index by linearly weighting five factors, including relative distance, relative speed, relative heading, minimum collision distance and time to closest point.

4. An unmanned boat collision avoidance decision system based on navigation intention perception, characterized in that: For executing the method according to claim 1, comprising: Multi-source perception module, used to collect motion status data of own ship and target ship in real time; Threat target screening module, used to build a collision risk assessment model to screen threat targets; The intention reasoning module is used to dynamically update the posterior probability distribution of the target ship's navigation intention based on recursive Bayesian reasoning based on the target ship's historical trajectory; Strategy network module, used to generate collision avoidance control actions based on own ship status, target ship motion characteristics and intention distribution; The control execution module is used to convert action instructions into ship heading and speed control signals.

Images