Unmanned ship smooth collision avoidance method considering marine environment disturbance

Abstract

The invention relates to an unmanned ship smooth collision avoidance method considering marine environment disturbance, and belongs to the field of unmanned ship collision avoidance. The method comprises the following steps: acquiring observation data of a current ship and surrounding target ships at the current moment, preprocessing the observation data, and storing the preprocessed observation data into a historical state cache; traversing historical cache data of each moment in the historical state cache, performing geometric and kinematic calculation to generate feature vectors of corresponding moments, and stacking the feature vectors of all the moments in sequence according to a time sequence; and inputting a stacking result of the feature vectors at each moment into a strategy network to obtain a collision avoidance control instruction at the current moment, and sending the collision avoidance control instruction to an unmanned ship bottom layer control system for execution. According to the method, a Mama sequence encoder is introduced, ship states at multiple moments are modeled, and time sequence characteristics reflecting historical dynamic change trends are extracted; and an action change rate penalty term is added in the reward function to guide the strategy to output smooth and executable collision avoidance actions, so that decision optimization considering both security and execution stability is realized.

Description

Technical Field

[0001] This invention belongs to the field of unmanned surface vessel (USV) collision avoidance, specifically relating to a smooth collision avoidance method for USVs that takes into account marine environmental disturbances. Background Technology

[0002] With the widespread application of unmanned surface vehicles (USVs) in fields such as ocean patrol, hydrographic surveying, environmental monitoring, and military reconnaissance, the safety and stability issues during collision avoidance have received increasing attention. In actual collision avoidance processes, the frequent and drastic adjustments of rudder angles or torques by USVs can cause severe wear on the actuators and even lead to system failures. Therefore, there is an urgent need to design intelligent collision avoidance strategies that balance safety and control stability.

[0003] In recent years, autonomous collision avoidance methods based on deep reinforcement learning (DRL) have gradually become a research hotspot. These methods, through continuous interaction with the environment and using reward signals to guide policy learning, can achieve end-to-end collision avoidance control from perception to decision-making, exhibiting good environmental adaptability and policy optimization capabilities. In complex and ever-changing maritime traffic scenarios, DRL demonstrates stronger generalization and dynamic response capabilities compared to traditional methods. However, existing research mostly focuses on collision avoidance success rate and rule compliance under calm water conditions, while paying insufficient attention to the safety of the policy and the stability of control execution under marine environmental disturbances. Especially under complex disturbances such as wind, waves, and currents, if the learned policy exhibits significant fluctuations in action output, it may not only cause controller oscillations and system fatigue but also exacerbate the risk of structural damage, thus seriously threatening the navigation safety and equipment lifespan of unmanned vessels.

[0004] To improve the practical usability of DRL strategies, some studies have attempted to introduce control smoothing terms into the reward function to penalize excessive action changes, thereby prompting the strategy to generate gentler control commands. However, experiments show that relying solely on reward terms to balance strategy performance and control stability has significant limitations: on the one hand, excessive smoothing penalties may inhibit the strategy's responsiveness and reduce collision avoidance success rate; on the other hand, in dynamic multi-objective scenarios, there is a significant conflict between smoothness and safety, making it difficult to achieve a balance through static weight design. Therefore, how to effectively suppress drastic fluctuations in strategy output while improving collision avoidance safety remains a key problem that urgently needs to be solved in current DRL unmanned surface vessel collision avoidance control.

[0005] Existing technologies fail to adequately consider the impact of the smoothness of unmanned surface vessels' (USVs) action outputs during collision avoidance on system stability and operational safety. They typically prioritize collision avoidance success rate and rule compliance as optimization objectives, neglecting the dynamic smoothness requirements of collision avoidance strategies during continuous navigation. In practical applications, USVs operate under long-term conditions of multi-source disturbances such as wind, waves, and currents. If the strategy output exhibits drastic fluctuations, it can easily lead to frequent and significant adjustments in servo motors, abnormal propeller loads, and in severe cases, even control system oscillations, mechanical fatigue, or hardware failures. While some methods have attempted to constrain the magnitude of strategy changes by adding action smoothing terms to the reward function, this approach alone fails to balance safety and control smoothness in collision avoidance tasks, resulting in performance degradation when facing high-risk interactions or complex disturbances. Summary of the Invention

[0006] The purpose of this invention is to address the problems of existing solutions neglecting motion smoothness modeling during unmanned surface vessel (USV) collision avoidance, leading to drastic fluctuations in control commands, accelerated actuator wear, and insufficient strategy stability and reliability under complex ocean disturbance conditions. This invention provides a smooth collision avoidance method for USVs that considers ocean environmental disturbances. This method introduces a motion smoothness reward function into a deep reinforcement learning framework, guiding the strategy to output smooth control commands while achieving effective collision avoidance. Simultaneously, it improves the Actor-Critic network by incorporating the Mamba module, which has advantages in sequence modeling, to perform temporal modeling of historical dynamic states during navigation. This allows the strategy to perceive the evolution trend of states over time, enhancing its understanding and adaptability to target behavior patterns and disturbed environments, thereby significantly improving the success rate and robustness of the collision avoidance strategy. Through this method, while ensuring collision avoidance success rate and rule compliance, drastic fluctuations in motion output can be effectively reduced, improving the safe operation and control smoothness of USVs under real and complex sea conditions such as wind, waves, and currents.

[0007] To achieve the above objectives, the technical solution of the present invention is: a smooth collision avoidance method for unmanned surface vessels considering marine environmental disturbances, comprising:

[0008] Acquire the observation data of the current ship and surrounding target ships, perform preprocessing operations, and store the data in the historical status cache;

[0009] Iterate through the historical cache data for each moment in the historical state cache, perform geometric and kinematic calculations, generate the feature vector for the corresponding moment, and stack the feature vectors of each moment in chronological order.

[0010] The stacked result of the feature vectors at each time step is input into the SAC algorithm based on Mamba to obtain the collision avoidance control command at the current time step, and then sent to the unmanned ship's underlying control system for execution.

[0011] Furthermore, acquire the current observation data of the ship and surrounding target ships, and perform preprocessing operations, specifically including:

[0012] The ship's position in the geographic coordinate system was obtained using a dual-antenna GPS system. and heading ; Combine IMU to obtain the ship's pitch speed sway speed and bow angular velocity ;

[0013] Millimeter-wave radar is used to detect surrounding target ships in real time and obtain their relative distances. Relative azimuth and radial velocity ;

[0014] The data output from the dual-antenna GPS, IMU, and millimeter-wave radar are unified to the system clock reference, and data time alignment is achieved through interpolation or extrapolation; the relative coordinates measured by the millimeter-wave radar are converted into absolute position and velocity information in a geodetic coordinate system consistent with the GPS position;

[0015] Physical constraint verification and statistical filtering were performed on the acquired observation data of the ship and surrounding target ships to remove outliers and noise; for data lost in a short period of time, a constant speed or constant rotation rate model was used for prediction and compensation.

[0016] Furthermore, before storing the preprocessed observation data of the current ship and surrounding target ships in the historical state cache, the most dangerous target ship needs to be identified. The identification method is as follows:

[0017] The collision risk between the two ships is assessed based on the current observation data of both the ship and the target ship. Based on the assessment results, all detected target ships are ranked, and the target ship with the highest collision risk index is selected as the most dangerous target ship. A five-factor collision risk assessment model is used to calculate the collision risk index of the target ship based on the relative distance, relative speed, relative heading, minimum collision distance, and time to closest point. The calculation formula is as follows:

[0018]

[0019] In the formula This is the weight vector for each factor, reflecting its relative contribution to collision risk. Let be the fuzzy membership function for each risk factor, with a value range of . , , , , , These represent the weights of nearest encounter distance (DCPA), nearest encounter time (TCPA), relative bearing, relative distance, and relative heading in the collision risk, respectively. , , , and These represent the risk membership degrees corresponding to the nearest encounter distance (DCPA), nearest encounter time (TCPA), relative bearing, relative distance, and relative heading, respectively. Indicates transpose;

[0020] After identifying the most dangerous target vessel, the pre-processed observation data of both the current vessel and the most dangerous target vessel are stored in a file with a maximum length of [length missing]. Historical state cache Among them Indicates the first The observation data is spliced ​​together from the observation data of the ship itself and the most dangerous target ship. This indicates the ship's position in the x-direction, y-direction, heading angle, pitch speed, sway speed, and bow turning angular velocity in the global coordinate system. This indicates the position of the most dangerous target ship in the x-direction, y-direction, heading angle, pitch speed, sway speed, and bow turning speed in the global coordinate system.

[0021] Furthermore, before traversing the historical cache data at each moment in the historical state cache, it is necessary to determine whether the sequence length of the historical state cache is less than L. If so, it is padded to the length L by copying adjacent frames.

[0022] Furthermore, the historical cache data for each time step is traversed in the historical state cache, and geometric and kinematic calculations are performed to generate the feature vector for the corresponding time step. The feature vectors for each time step are then stacked sequentially in chronological order, specifically as follows:

[0023] Traverse the historical state cache data for each time step in the historical state cache of sequence length L, and perform geometric and kinematic calculations to generate the feature vector for the corresponding time step. The calculation formula is as follows:

[0024]

[0025] in , , These are the ship's pitch speed, sway speed, and bow turning angular velocity, respectively. , These are the relative bearing and relative heading angles between the ship and the target point, respectively. , , , , These include the relative distance, relative bearing, relative heading, collision risk index, and the resulting encounter situation between the vessel and the target vessel. The values ​​1, 2, 3, 4, and 5 represent the situations in which the ship encounters the other ship, crosses to the left, crosses to the right, overtakes, and is overtaken, respectively. For speed;

[0026] After calculating the feature vector for each time step in the historical state cache of sequence length L, the feature vectors from each time step are stacked sequentially in chronological order, using the following calculation formula:

[0027]

[0028] in Represents the state sequence matrix, Indicates the first The feature vector at time step, Indicates the sequence length. Represents the state vector of a single frame Dimensions.

[0029] Furthermore, the stacked result of the feature vectors at each time step is input into the SAC algorithm based on Mamba to obtain the collision avoidance control command at the current time step, specifically including:

[0030] The stacked state sequence matrix is ​​processed through a fully connected layer. Projected latent space dimension of Mamba module :

[0031]

[0032] in , These are the weight matrix and bias vector of the input linear mapping layer, respectively. is the sequence feature representation after projection, and L is the sequence length of the historical state cache;

[0033] Projected sequence features Input the Mamba module to perform temporal modeling and information fusion of sequence features:

[0034]

[0035] in, This represents time series modeling based on a selective state-space model. This represents the output sequence of the Mamba module.

[0036] Through the linear mapping layer Mapping to the policy latent space dimension :

[0037]

[0038] in , These are the weight matrix and bias vector of the linear mapping layer, respectively. This represents the output of the linear mapping layer.

[0039] In the policy latent space, the mean of the action distribution is output through two independent fully connected layers. With log standard deviation :

[0040]

[0041] in , These are the weight matrix and bias vector of the mean head, respectively. , These are the weight matrix and bias vector of the standard deviation correlation branch, respectively. This represents a truncation function used to truncate... The value is limited to the range [-10, 2].

[0042] logarithmic standard deviation Indexing yields the standard deviation :

[0043]

[0044] The action is obtained by sampling from a Gaussian distribution using the reparameterization method in the SAC algorithm. :

[0045]

[0046] in This represents element-wise multiplication. To obtain from the standard multivariate normal distribution The noise vector obtained from sampling, Represents the identity matrix.

[0047] Then, the tanh activation function is used to... Mapping to the (-1,1) interval yields the action to be executed. :

[0048]

[0049] Based on the physical constraints of the unmanned vessel actuators, Linear mapping to actual control range :

[0050]

[0051] in This is the collision avoidance control command for the current moment, which is then sent to the unmanned vessel's underlying control system for execution. They represent The maximum and minimum values.

[0052] Furthermore, after the unmanned vessel's underlying control system executes the collision avoidance control command for the current moment, the system continues to collect observation data of the vessel and surrounding target vessels in real time through multi-source sensors and update the historical state cache to update the input sequence for the next control cycle, thereby realizing a closed-loop control process of perception-decision-execution.

[0053] Furthermore, the SAC algorithm based on Mamba is improved by introducing a smoothness constraint reward to guide the policy network of the Mamba-improved SAC algorithm to output smooth and executable collision avoidance control commands.

[0054] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method described above.

[0055] The present invention also provides a computer program product, including a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of the method described above.

[0056] Compared with the prior art, the present invention has the following beneficial effects:

[0057] This invention proposes a smooth collision avoidance decision-making method for unmanned surface vessels (USVs) based on smooth rewards and Mamba temporal modeling. Addressing the problems of large control command fluctuations, rapid actuator wear, and difficulty in fully utilizing historical dynamic information in existing collision avoidance strategies, this method designs a smoothness-constrained reward function and introduces the Mamba structure, which possesses long-term dependency modeling capabilities, to improve the actor and critic networks in the existing SAC algorithm, giving them temporal awareness. This method effectively suppresses drastic changes in collision avoidance actions while maintaining a high collision avoidance success rate.

[0058] By employing smoothness-based reward constraints, the strategy tends to generate control commands with smaller fluctuations and better continuity during training, thereby improving the smoothness of collision avoidance maneuvers. Through Mamba time-series modeling, the strategy can perceive the evolution of the relative situation between the ship and the target ship over time, forming more proactive collision avoidance maneuvers and improving the collision avoidance success rate. Compared to traditional DRL methods, this invention significantly improves the smoothness of collision avoidance maneuvers and the overall operational safety. Attached Figure Description

[0059] Figure 1 A schematic diagram of the collision avoidance decision-making process for unmanned vessels.

[0060] Figure 2 This is a schematic diagram of the reasoning process for a smooth collision avoidance strategy.

[0061] Figure 3 This is a schematic diagram of the algorithm training. Detailed Implementation

[0062] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0063] This invention provides a smooth collision avoidance method for unmanned surface vessels that takes into account disturbances in the marine environment, including:

[0064] Acquire the observation data of the current ship and surrounding target ships, perform preprocessing operations, and store the data in the historical status cache;

[0065] Iterate through the historical cache data for each moment in the historical state cache, perform geometric and kinematic calculations, generate the feature vector for the corresponding moment, and stack the feature vectors of each moment in chronological order.

[0066] The stacked result of the feature vectors at each time step is input into the SAC algorithm based on Mamba to obtain the collision avoidance control command at the current time step, and then sent to the unmanned ship's underlying control system for execution.

[0067] The following is a detailed implementation process of the present invention.

[0068] This invention discloses a smooth collision avoidance method for unmanned surface vessels that takes into account marine environmental disturbances. The collision avoidance process is as follows: Figure 1 As shown. It mainly includes the following steps:

[0069] Step 1: Environmental perception and data preprocessing;

[0070] Step 2: Organizing the historical sequence;

[0071] Step 3: Input the calculation model;

[0072] Step 4: Strategic Reasoning;

[0073] Step 5: Instruction issuance and closed-loop update.

[0074] For step 1: The system first uses multi-source sensors to comprehensively perceive the navigation environment, acquiring the navigation status of the ship and surrounding target ships, as well as marine environmental disturbance factors. The perceived data mainly includes the ship's position, speed, heading, sway and roll velocities, and turning angular velocity, as well as the relative position, relative speed, and heading angle of the target ships. To ensure the consistency and reliability of the data in subsequent processing, the system preprocesses the collected information to form a structured, non-redundant, and dimensionlessly consistent observation dataset. This preprocessed environmental perception data will serve as the input basis for subsequent strategy reasoning. The specific steps are as follows:

[0075] Step 1.1: Obtain the status of this ship.

[0076] The ship's position in the geographic coordinate system was obtained using a dual-antenna GPS system. ,course Combined with IMU to obtain oscillation speed sway speed Angular velocity of the bow These motion state parameters are used to improve the accuracy of positioning and attitude calculation through sensor fusion algorithms.

[0077] Step 1.2: Obtain the target ship status.

[0078] Using millimeter-wave radar to detect surrounding target ships in real time and obtain their relative distances Relative azimuth radial velocity Information such as radar points are correlated and tracked using a multi-target tracking algorithm to obtain an estimate of the target ship's continuous motion state.

[0079] Step 1.3: Time Synchronization and Coordinate Transformation

[0080] The data output from the dual-antenna GPS, IMU, and millimeter-wave radar are unified to the system clock reference, and data time alignment is achieved through interpolation or extrapolation; the relative coordinates measured by the radar are converted into absolute position and velocity information in a geodetic coordinate system consistent with the GPS position.

[0081] Step 1.4: Data Cleaning and Anomaly Removal

[0082] Physical constraints and statistical filtering are performed on parameters such as position, speed, and heading to remove outliers and noise. For data lost in a short period of time, a constant speed or constant rotation rate model is used for prediction and compensation to ensure data continuity and stability.

[0083] For step 2: After processing the observation data of the current ship and the target ship, the system stores it in the historical state cache to extract a certain length of time-series information, providing a data foundation for subsequent calculation model input. The specific steps are as follows:

[0084] Step 2.1: Identify the most dangerous target vessel

[0085] The collision risk between the two ships is assessed based on the current observation data. Based on the assessment results, all detected target ships are ranked, and the target ship with the highest collision risk index is selected as the primary tracking target for this period. When the number of detected target ships is insufficient, empty slots are recorded and occupied by a zero vector. The specific assessment method is a five-factor collision risk assessment model, which comprehensively considers factors such as the relative distance, relative speed, relative heading, minimum collision distance (DCPA), and time to closest point (TCPA) between the two ships to calculate the collision risk index of the target ship. The overall risk index is calculated using the following linear weighted model:

[0086]

[0087] In the formula This is the weight vector for each factor, reflecting its relative contribution to collision risk. Let be the fuzzy membership function for each risk factor, with a value range of . , , , , , These represent the weights of nearest encounter distance (DCPA), nearest encounter time (TCPA), relative bearing, relative distance, and relative heading in the collision risk, respectively. , , , and These represent the risk membership degrees corresponding to the five factors mentioned above. Indicates transpose;

[0088] Step 2.2: Historical Cache Maintenance

[0089] The current moment The maximum length of the observation data of the selected target ship and the observation data of this ship to be stored is State sequence cache ,in Indicates the first The observation data is spliced ​​together from the observation data of the current ship and the target ship at any given moment. This indicates the ship's position in the x-direction, y-direction, heading angle, pitch speed, sway speed, and bow turning angular velocity in the global coordinate system. This represents the other ship's position in the x-direction, y-direction, heading angle, pitch speed, sway speed, and bow turning speed in the global coordinate system. The cache uses a first-in, first-out (FIFO) strategy; when new data arrives, the oldest historical state is removed to ensure a constant cache length.

[0090] For step 3: After completing steps 1 and 2, the system at the current time... The observation data from both the current ship and the target ship are used to generate a fixed-length input for policy network inference. The specific steps are as follows:

[0091] Step 3.1: Historical sequence length check and padding

[0092] From the historical cache of step 2 Read the most recent observation sequence. If the length of the current time-in-history cache sequence is less than the minimum sequence length required by the Mamba time-series coding module... Then, it is completed by copying adjacent frames. Specifically: the current time... The adjacent time, i.e. Available data can be repeatedly inserted into the end of the sequence until the length requirement is met. This operation ensures that even when data is insufficient at the beginning of system startup, the improved Mamba policy module can still receive input sequences with consistent dimensions and fixed lengths.

[0093] Step 3.2: Input the calculation model

[0094] After completing the historical sequence completion in step 3.1, the system sequentially iterates through the historical cache data for each completed moment, performing geometric and kinematic calculations on the observation data of the local ship and the target ship to generate the feature vector for the corresponding moment. The specific calculation formula for the feature vector at each moment is as follows:

[0095]

[0096] in , , These are the ship's pitch speed, sway speed, and bow turning angular velocity, respectively. , These are the relative bearing and relative heading angles between the ship and the target point, respectively. , , , , These include the relative distance, relative bearing, relative heading, collision risk index, and the resulting encounter situation between the vessel and the target vessel. The values ​​1, 2, 3, 4, and 5 represent the situations in which the ship encounters the other ship, crosses to the left, crosses to the right, overtakes, and is overtaken, respectively. The target ship's speed;

[0097] After calculating the feature vector for each time step in the historical state cache of sequence length L, the feature vectors from each time step are stacked sequentially in chronological order to form the model input. The calculation formula is as follows:

[0098]

[0099] in Represents the state sequence matrix, Indicates the first Momentary, Indicates the sequence length. Represents the state vector of a single frame Dimensions.

[0100] For step 4: After completing the model input calculation in step 3, the result is input into the policy network to obtain the real-time collision avoidance action. The specific calculation process is as follows: Figure 2 As shown. The specific steps are as follows:

[0101] Step 4.1: Linear Projection

[0102] The stacked state sequence matrix is ​​processed through a fully connected layer. Projected latent space dimension of Mamba module :

[0103]

[0104] in , These are the weight matrix and bias vector of the input linear mapping layer, respectively. Let L be the sequence feature representation after projection, and let H be the sequence length of the historical state cache.

[0105] Step 4.2: Mamba Timing Coding

[0106] Projected sequence features Input the Mamba module to perform temporal modeling and information fusion of sequence features:

[0107]

[0108] in, This represents time series modeling based on a selective state-space model. This represents the output sequence of Mamba.

[0109] Step 4.3: Mamba encoder output mapping

[0110] Through the linear mapping layer Mapping to the policy latent space dimension :

[0111]

[0112] in , These are the weight matrix and bias vector of the linear mapping layer, respectively. This represents the output of the linear mapping layer.

[0113] Step 4.4: Generation of Gaussian Policy Distribution

[0114] In the policy latent space, the mean of the action distribution is output through two independent fully connected layers. With log standard deviation :

[0115]

[0116] in , These are the weight matrix and bias vector of the mean head, respectively. , These are the weight matrix and bias vector of the standard deviation correlation branch, respectively. This represents a truncation function used to truncate... The value is limited to the range [-10, 2].

[0117] logarithmic standard deviation Indexing yields the standard deviation :

[0118]

[0119] Step 4.5: Reparameterized Sampling and Motion Squeezing

[0120] The action is obtained by sampling from a Gaussian distribution using the reparameterization method in the SAC algorithm. :

[0121]

[0122] in This represents element-wise multiplication. To obtain from the standard multivariate normal distribution The noise vector obtained from sampling, Represents the identity matrix.

[0123] Then, the tanh activation function is used to... Mapping to the (-1,1) interval yields the action to be executed. :

[0124]

[0125] Step 4.6: Mapping Actions to Control Commands

[0126] Based on the physical constraints of the unmanned vessel actuators, Linear mapping to actual control range :

[0127]

[0128] in This is the collision avoidance control command for the current moment, which is then sent to the unmanned vessel's underlying control system for execution. They represent The maximum and minimum values.

[0129] For step 5: After completing the strategy reasoning in step 4 and obtaining the control action vector at the current moment, the control command is sent to the unmanned surface vessel's underlying execution unit in real time through the communication interface. The underlying control system adjusts the vessel's heading, propulsion speed, and other motion parameters according to the received command, thereby implementing collision avoidance actions at the physical level.

[0130] After the command is executed, the system continues to collect real-time information on the ship's status and the environment through multi-source sensors and updates historical cached observation data. This updated data is used to update the input sequence for the next control cycle, realizing a closed-loop control process of perception-decision-execution. Through this closed-loop mechanism, the system can continuously adjust its collision avoidance strategy in the presence of dynamic factors such as changes in sea state and target ship maneuvers, ensuring the navigation safety and path smoothness of the unmanned vessel.

[0131] To improve the collision avoidance smoothness of unmanned surface vessels (USVs) under dynamic multi-target encounters and real-world sea state disturbances, this invention proposes a collision avoidance decision-making method for USVs that integrates a Mamba temporal modeling structure and action smoothness constraints, based on the Soft Actor-Critic (SAC) algorithm. This method addresses the performance degradation of existing collision avoidance strategies when facing multi-target interactions and control command stability requirements. It introduces a Mamba sequence encoder into the Actor-Critic network to model the ship's state at multiple time points, thereby extracting temporal features reflecting historical dynamic trends. Furthermore, it adds a penalty term for the rate of change of actions to the reward function, guiding the strategy to output smooth and executable collision avoidance actions, achieving decision optimization that balances safety and execution stability. The specific training steps are as follows:

[0132] Step 1: Dynamic Modeling and Environmental Disturbance Design

[0133] Step 2: State and Action Space Design

[0134] Step 3: Construct the reward function

[0135] Step 4: Actor-Critic Network Design

[0136] Step 5: Loss Function Design

[0137] Step 6: Design of the timing buffer pool

[0138] Step 7: Training Process Design

[0139] For step 1: Fossen's three-degree-of-freedom ship motion mathematical model is used to characterize the motion of the unmanned ship during the training process; in order to simulate external disturbances such as wind, waves and currents in real sea conditions, Ornstein-Uhlenbeck (OU) noise is used to simulate the interference of the marine environment on the motion of the unmanned ship.

[0140] The mathematical model of the three-degree-of-freedom ship motion used in the training is as follows:

[0141]

[0142] in This is the velocity vector in the ship's coordinate system. express The derivative, It is the sum of the rigid body and its added mass. For the Coriolis and centripetal force matrices, For the viscous damping matrix, To control the inputs (thrust and yaw moment). Environmental disturbance forces / moments. Hull attitude and position vectors. The relationship with speed is:

[0143]

[0144] in This is the hull-to-ground coordinate system transformation matrix.

[0145] To efficiently generate time-varying perturbation signals during training, OU noise is used. Modeling:

[0146]

[0147] in Let be the OU state vector at time t. The mean, For the regression rate, The fluctuation amplitude, This is a standard Wiener process. After discretization, it can be written as:

[0148]

[0149] in For discrete time steps, Let be independent and identically distributed standard Gaussian random variables, satisfying , Represents the identity matrix.

[0150] Environmental disturbance at time t Through the proportionality coefficient The matrix is ​​mapped from the OU state.

[0151]

[0152] The generated disturbance signal can maintain a certain smoothness while adjusting the amplitude and correlation time, making it convenient to simulate different sea conditions.

[0153] Regarding step 2: During training, the agent's state vector is composed of information about the current vessel, the most dangerous target vessel, and the target point. Features are calculated and combined using the specific formula for calculating the feature vector at each moment, as described above, ensuring a fixed input dimension and collision risk perception capability. A continuous action space is employed, with the unmanned vessel's yaw moment as the sole control input, controlling heading changes to achieve collision avoidance and navigation tasks.

[0154] For step 3: To achieve effective training of the autonomous collision avoidance strategy, a composite reward function is constructed, comprehensively considering target distance reward, heading reward, collision penalty, collision risk penalty, speed obstacle penalty, international collision avoidance rule penalty, and smoothing reward. The overall reward function is calculated as follows:

[0155]

[0156] in These represent the weighted coefficients for distance bonus, heading bonus, collision risk penalty, collision penalty, violation of collision avoidance rules penalty, speed obstacle penalty, and smoothness bonus, respectively. The specific numerical values ​​for each reward component, and the detailed meanings of each sub-item, are as follows:

[0157] (1) Distance Reward Items ( )

[0158] The unmanned surface vessel is encouraged to continue moving towards the target point, specifically calculated as follows:

[0159]

[0160] in , These represent the Euclidean distances from the unmanned surface vessel to the target point at the previous and current moments, respectively. This indicates the speed of the unmanned surface vessel at the previous moment. This represents the simulation step size, set to 1.

[0161] (2) Heading bonus items ( )

[0162] The unmanned surface vessel is encouraged to adjust its course toward the target point, and the calculation formula is as follows:

[0163]

[0164] (3) Collision risk penalty items ( )

[0165] This term is used to penalize unmanned surface vessels for collision avoidance behaviors that increase navigational risks, and its calculation is shown in the following formula:

[0166]

[0167] in This indicates the collision risk between this vessel and the most dangerous target vessel.

[0168] (4) Collision penalty items ( )

[0169] Triggered when the agent collides with another object or successfully reaches the target point, the calculation is as follows:

[0170]

[0171] in , , and Represent the distance between the unmanned surface vessel (USV) and the target point, respectively, and the distance between the USV and the target point. Distance between the two opposing vessels, distance threshold for determining whether the target point has been reached, and the first The radius of the safety zone of the other vessel.

[0172] (5) Penalties for violating collision avoidance rules ( )

[0173] To improve the model's compliance with the International Regulations for Preventing Collisions at Sea (COCR), this penalty term is added to the single-vessel encounter scenario. This term guides the unmanned surface vessel (USV) to take avoidance measures that comply with the COCR. Its calculation is shown in the following formula:

[0174]

[0175] (6) Speed ​​obstacle penalty item ( )

[0176] In ship collision avoidance, VO (Velocity-Space) is a collision avoidance modeling method based on velocity space. It defines that within a given time window, if the speed of the ship (OS) falls within the speed barrier area formed by the target ship or the opposing ship (TS), a collision is likely to occur. Therefore, this invention introduces a VO penalty term into the reward function to penalize action choices that potentially lead to a collision trajectory during reinforcement learning training, thereby driving the strategy to automatically avoid unsafe speed areas. Its calculation is shown in the following formula:

[0177]

[0178] in This represents the velocity vector of the ship. This indicates that there is no risk of predicted collision at the current speed selection, and the reward value is 0 in this case; This indicates that the ship's speed has fallen into the speed barrier area, and a collision may occur in the future, therefore a negative reward is given.

[0179] (7) Smooth reward ( )

[0180] Based on the second-order difference of the action, a smooth reward is designed to suppress abrupt changes in the action:

[0181]

[0182] in , and These represent the Actor network in... , and Actions that are calculated in real time.

[0183] For step 4: In the SAC, the Actor network provides a continuous action distribution with state inputs and outputs to generate yaw moment control commands; the Critic receives state-action pairs and outputs a Q-value estimate to evaluate the strategy's effectiveness. An automatic temperature parameter adjustment mechanism balances exploratory and optimal approaches.

[0184] To enhance the ability to model the temporal characteristics of target ship interactions, this invention introduces the Mamba sequence modeling module into both the Actor and Critic networks, enabling them to process state sequence inputs and extract the dynamic evolution of the state space during navigation, thereby improving the safety and smoothness of decision-making.

[0185] Actor Network: The Mamba module first dynamically models the input state sequence, extracts the temporal evolution pattern during the navigation process, and uses the last frame of the output as the temporal feature representation of the current decision. After mapping it to the policy latent space, it generates the control distribution, which is used to output the yaw moment action.

[0186] The Critic network also inputs the state sequence into the Mamba module for temporal feature extraction, concatenates it with the current candidate action at the feature layer, and then calculates the corresponding Q-value estimate through a multilayer perceptron. This structure ensures that the Critic's evaluation of the quality of actions not only depends on the instantaneous state, but also incorporates the dynamic context of historical interactions, improving its ability to assess the value of complex ship interaction scenarios.

[0187] The network structure is consistent with the reasoning process of the "policy reasoning" part in the invention: during the training phase, the Mamba module is also responsible for extracting temporal features from the state sequence and using the last frame as the input for the current decision. The Actor outputs the action, and the Critic calculates the Q value based on the same temporal features and actions, thus achieving functional unification with the actual reasoning phase, thereby ensuring the consistency between training and reasoning and the transferability of policy effects.

[0188] Regarding step 5: In this invention, the Actor loss function remains consistent with the standard SAC, still primarily aiming to maximize the entropy regularization strategy, without any additional modifications.

[0189] To improve the evaluation capability of long-term actions in complex interactive scenarios, this invention introduces a weighted combination of n-step cumulative rewards, in addition to the traditional single-step reward. The target value of the Critic network is given by the following formula:

[0190]

[0191] in These are weighting coefficients used to balance the stability of short-term estimates with the modeling of long-term effects; Discount factor; , They represent in and Momentary rewards; , They represent in and The output of the Mamba module at any given time; , They represent in and The output of the Actor network at each time step; Indicates that the Critic network is in and Output under the given conditions; Indicates that the Critic network is in and Output under certain conditions.

[0192] This design allows Critic's value assessment to not only rely on single-step rewards but also combine multi-step cumulative rewards, thereby improving learning efficiency and stability under dynamic interactions and complex environmental disturbances.

[0193] Regarding step 6: To meet the Mamba temporal encoder's requirement for continuous historical frames as input and to simultaneously utilize n-step temporal information in the Critic target, this invention designs a temporal experience replay pool that supports fixed-length sequence sampling, n-step reward calculation, and GPU vectorization, based on the conventional experience pool. This buffer pool not only stores single-step transitions but also ensures that the batch-sampled samples are temporally continuous, facilitating direct input into the Actor-Critic sequence network.

[0194] First, during the interaction, the buffer pool sequentially stores the unmanned vessel's state, control actions, reward value, completion marker, and next-moment state, forming a continuous trajectory segment. Unlike traditional single-step storage, this buffer pool automatically maintains a fixed-length state sequence during storage, ensuring that each sampled data contains historical information from the navigation process, thus meeting the input requirements of the Mamba module.

[0195] Secondly, to improve training efficiency, an n-step reward calculation mechanism is directly introduced within the buffer pool. In the sampled state sequence, the buffer pool pre-calculates the multi-step cumulative reward in the last frame of the sequence and obtains this reward directly through sampling, thereby avoiding additional computational overhead during training.

[0196] Finally, the buffer pool fully utilizes parallel processing and vectorized operations in its implementation, enabling it to maintain high throughput even during large-scale training. Through this design, the Actor-Critic network can obtain single-step feedback for short-term fine-tuning during training, and also utilize multi-step rewards to capture global behavioral trends, thereby improving the stability of decision-making and the success rate of collision avoidance.

[0197] Regarding step 7: The training process mainly includes five stages: environment interaction, data storage, sequence sampling, network update, and parameter optimization. The overall process is as follows: Figure 3 As shown, the specific steps are as follows:

[0198] Step 7.1: Environmental Interaction and Data Acquisition

[0199] The unmanned surface vessel (USV) agent selects actions according to its current policy in a simulated environment. The state space includes the USV's position, heading, velocity, target point information, and the kinematic characteristics of surrounding target vessels. Actions are generated by a policy network and mapped to control torque signals, driving the USV to complete its navigation. The environment provides a reward value based on the action feedback and the state at the next moment, and indicates whether to terminate the operation.

[0200] Step 7.2: Experience Storage and Pre-calculation of n-Step Cumulative Returns

[0201] The obtained state, action, reward, next state, and termination flag are stored in the experience buffer pool. When storing sequence data, the buffer pool directly calculates the multi-step cumulative reward at the last frame of the sequence.

[0202] Step 7.3: Sequence Sampling and Temporal Modeling

[0203] During the parameter update phase, the buffer pool samples fixed-length sequences (length 3) at a batch size of 256. These sampled sequences are fed into the policy network and the value network, where the sequence portion undergoes temporal feature extraction by the Mamba module. This module comprises an input projection layer (dimension 256), a state convolution kernel of size 4, hidden state dimensions of 64, and gating units. The Mamba module dynamically models historical features, effectively capturing the temporal correlation of target ship behavior patterns.

[0204] Step 7.4: Policy Network and Value Network Update

[0205] The policy network (Actor) consists of two fully connected layers, each with 256 neurons, using ReLU as the activation function. Its output is the mean and standard deviation of the action distribution. It samples and calculates the corresponding policy loss using a reparameterization technique. The value network (Critic) employs a double-Q structure, with each Q-network being a two-layer fully connected network (256 neurons per layer). It estimates the value of state-action pairs and updates them in conjunction with the target network to reduce overestimation bias.

[0206] Step 7.5: Parameter Optimization and Target Network Soft Update

[0207] Gradient descent is performed using the Adam optimizer, with a learning rate set to... The target network replaces parameters through a soft update mechanism.

[0208] Step 7.6: Automatic Temperature Regulation Mechanism

[0209] During training, temperature parameters are automatically adjusted to balance exploration and utilization, ensuring that the strategy can both be fully explored and converge to a stable control strategy.

[0210] Step 7.7: Iteration and Policy Convergence

[0211] The above process continues to iterate until the agent achieves a stable high success rate and smooth control performance in collision avoidance tasks, enabling the unmanned surface vessel to navigate safely in complex dynamic environments.

[0212] By introducing the Mamba temporal building module and motion smoothness reward design into the SAC framework, the present invention can significantly improve the collision avoidance performance and control stability of unmanned surface vessels in complex dynamic environments.

[0213] Specifically, the Mamba module, as a highly efficient sequence modeling unit, is embedded in the policy network and value network structure, overcoming the shortcomings of traditional fully connected layers in terms of temporal dependencies. This module can effectively capture the temporal evolution patterns of target ship motion characteristics and historical interaction states, enabling the network to have stronger dynamic modeling and generalization capabilities when dealing with multi-target ship scenarios and uncertain behavioral patterns.

[0214] In designing the reward function, this invention incorporates a motion smoothness reward term. By constraining the rate of change of control commands in adjacent time steps, it guides the strategy output to remain stable during collision avoidance, preventing excessive actuator wear and ship maneuvering risks caused by motion oscillations. This reward term, together with safety and compliance rewards, constitutes a composite incentive system, ensuring that the unmanned surface vessel can achieve smooth and compliant maneuvers while avoiding collisions during training.

[0215] By combining the above technical solutions, this invention not only enhances temporal modeling capabilities through Mamba in the network structure, but also optimizes motion stability through smoothness rewards in the training objective, achieving high success rates and low oscillations for unmanned surface vessels (USVs) under conditions of multi-target encounters and uncertain disturbances. This method has promising engineering application prospects and can be widely applied to various autonomous navigation scenarios such as intelligent shipping, marine monitoring, and search and rescue missions.

[0216] The present invention also provides an electronic device, including a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method described above.

[0217] The present invention also provides a computer program product, including a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of the method described above.

[0218] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for smooth collision avoidance of unmanned surface vessels considering marine environmental disturbances, characterized in that, include: Acquire the observation data of the current ship and surrounding target ships, perform preprocessing operations, and store the data in the historical status cache; Iterate through the historical cache data for each moment in the historical state cache, perform geometric and kinematic calculations, generate the feature vector for the corresponding moment, and stack the feature vectors of each moment in chronological order. The stacked result of the feature vectors at each time step is input into the SAC algorithm based on Mamba to obtain the collision avoidance control command at the current time step, and then sent to the unmanned ship's underlying control system for execution.

2. The method for smooth collision avoidance of unmanned surface vessels considering marine environmental disturbances according to claim 1, characterized in that, Acquire observation data of the current ship and surrounding target ships, and perform preprocessing operations, specifically including: The ship's position in the geographic coordinate system was obtained using a dual-antenna GPS system. and heading angle ; Combine IMU to obtain the ship's pitch speed sway speed and bow angular velocity ; Millimeter-wave radar is used to detect surrounding target ships in real time and obtain their relative distances. Relative azimuth and radial velocity ; The data output from the dual-antenna GPS, IMU, and millimeter-wave radar are unified to the system clock reference, and data time alignment is achieved through interpolation or extrapolation; the relative coordinates measured by the millimeter-wave radar are converted into absolute position and velocity information in a geodetic coordinate system consistent with the GPS position; Physical constraint verification and statistical filtering were performed on the acquired observation data of the ship and surrounding target ships to remove outliers and noise; for data lost in a short period of time, a constant speed or constant rotation rate model was used for prediction and compensation.

3. The method for smooth collision avoidance of unmanned surface vessels considering marine environmental disturbances according to claim 1, characterized in that, Before storing the preprocessed observation data of the current ship and surrounding target ships in the historical state cache, the most dangerous target ship must be identified. The identification method is as follows: The collision risk between the two ships is assessed based on the current observation data of both the ship and the target ship. Based on the assessment results, all detected target ships are ranked, and the target ship with the highest collision risk index is selected as the most dangerous target ship. A five-factor collision risk assessment model is used to calculate the collision risk index of the target ship based on the relative distance, relative speed, relative heading, minimum collision distance, and time to closest point. The calculation formula is as follows: In the formula This is the weight vector for each factor, reflecting its relative contribution to collision risk. Let be the fuzzy membership function for each risk factor, with a value range of . , , , , , These represent the weights of nearest encounter distance (DCPA), nearest encounter time (TCPA), relative bearing, relative distance, and relative heading in the collision risk, respectively. , , , and These represent the risk membership degrees corresponding to the nearest encounter distance (DCPA), nearest encounter time (TCPA), relative bearing, relative distance, and relative heading, respectively. Indicates transpose; After identifying the most dangerous target vessel, the pre-processed observation data of both the current vessel and the most dangerous target vessel are stored in a file with a maximum length of [length missing]. Historical state cache Among them Indicates the first The observation data is spliced ​​together from the observation data of this ship and the most dangerous target ship at any given moment. This indicates the ship's position in the x-direction, y-direction, heading angle, pitch speed, sway speed, and bow turning angular velocity in the global coordinate system. This indicates the position of the most dangerous target ship in the x-direction, y-direction, heading angle, pitch speed, sway speed, and bow turning speed in the global coordinate system.

4. The method for smooth collision avoidance of unmanned surface vessels considering marine environmental disturbances according to claim 1, characterized in that, Before iterating through the historical cache data at each moment in the historical state cache, it is necessary to determine whether the sequence length of the historical state cache is less than L. If so, it is padded to the length L by copying adjacent frames.

5. A method for smooth collision avoidance of unmanned surface vessels considering marine environmental disturbances according to claim 1, characterized in that, Iterate through the historical cache data for each time step in the historical state cache, perform geometric and kinematic calculations to generate the feature vector for that time step, and then stack the feature vectors from each time step in chronological order. The specific method is as follows: Traverse the historical state cache data for each time step in the historical state cache of sequence length L, and perform geometric and kinematic calculations to generate the feature vector for the corresponding time step. The calculation formula is as follows: in , , These are the ship's pitch speed, sway speed, and bow turning angular velocity, respectively. , These are the relative bearing and relative heading angles between the ship and the target point, respectively. , , , , These include the relative distance, relative bearing, relative heading, collision risk index, and the resulting encounter situation between the vessel and the target vessel. The values ​​1, 2, 3, 4, and 5 represent the situations in which the ship and the target ship encounter each other, cross to the left, cross to the right, overtake, and are overtaken, respectively. The target ship's speed; After calculating the feature vector for each time step in the historical state cache of sequence length L, the feature vectors from each time step are stacked sequentially in chronological order, using the following calculation formula: in Represents the state sequence matrix, Indicates the first The feature vector at time step, Indicates the sequence length. Represents the state vector of a single frame Dimensions.

6. A method for smooth collision avoidance of unmanned surface vessels considering marine environmental disturbances according to claim 1, characterized in that, The stacked result of the feature vectors at each time step is input into the SAC algorithm based on Mamba to obtain the collision avoidance control command at the current time step, which specifically includes: The stacked state sequence matrix is ​​processed through a fully connected layer. Projected latent space dimension of Mamba module : in , These are the weight matrix and bias vector of the input linear mapping layer, respectively. is the sequence feature representation after projection, and L is the sequence length of the historical state cache; Projected sequence features Input the Mamba module to perform temporal modeling and information fusion of sequence features: in, This represents time series modeling based on a selective state-space model. This represents the output sequence of the Mamba module; Through the linear mapping layer Mapping to the policy latent space dimension : in , These are the weight matrix and bias vector of the linear mapping layer, respectively. This represents the output of the linear mapping layer; In the policy latent space, the mean of the action distribution is output through two independent fully connected layers. With log standard deviation : in , These are the weight matrix and bias vector of the mean head, respectively. , These are the weight matrix and bias vector of the standard deviation correlation branch, respectively. This represents a truncation function used to truncate... The value is limited to the range [-10, 2]; logarithmic standard deviation Indexing yields the standard deviation : The action is obtained by sampling from a Gaussian distribution using the reparameterization method in the SAC algorithm. : in This represents element-wise multiplication. To obtain from the standard multivariate normal distribution The noise vector obtained from sampling, Represents the identity matrix; Then, the tanh activation function is used to... Mapping to the (-1,1) interval yields the action to be executed. : Based on the physical constraints of the unmanned vessel actuators, Linear mapping to actual control range : in This is the collision avoidance control command for the current moment, which is then sent to the unmanned vessel's underlying control system for execution. They represent The maximum and minimum values.

7. A method for smooth collision avoidance of unmanned surface vessels considering marine environmental disturbances according to claim 1, characterized in that, After the unmanned surface vessel's underlying control system executes the collision avoidance control command for the current moment, the system continues to collect observation data of the vessel and surrounding target vessels in real time through multi-source sensors and update the historical state cache to update the input sequence for the next control cycle, thereby realizing a closed-loop control process of perception-decision-execution.

8. A method for smooth collision avoidance of unmanned surface vessels considering marine environmental disturbances according to claim 1, characterized in that, The SAC algorithm based on Mamba also introduces a smoothness constraint reward to guide the policy network of the Mamba-improved SAC algorithm to output smooth and executable collision avoidance control commands.

9. An electronic device, characterized in that, It includes a processor and a memory, wherein the memory stores a computer program that, when executed by the processor, causes the processor to perform the steps of the method as described in any one of claims 1-7.

10. A computer program product, characterized in that, The method includes a computer program stored in a computer-readable storage medium; when a processor of an electronic device reads the computer program from the computer-readable storage medium, the processor executes the computer program, causing the electronic device to perform the steps of the method as described in any one of claims 1-7.

Images