Smooth collision avoidance method for unmanned ship considering marine environment disturbance

Abstract

The present application relates to a kind of unmanned ship smooth collision avoidance method considering marine environment disturbance, belong to unmanned ship collision avoidance field.The method includes: obtaining the observation data of current time's ship and surrounding target ship, and after pre-processing operation, store in historical state cache;Each time's historical cache data in historical state cache is traversed, and geometry and kinematics calculation is carried out, corresponding time's feature vector is generated, and the feature vector of each time is sequentially stacked in time sequence;The stacking result of the feature vector of each time is input into strategy network to obtain the collision avoidance control instruction of current time, and is sent to the bottom layer control system of unmanned ship and is executed.The present application models the ship state of multiple times by introducing Mamba sequence encoder, extracts the time sequence characteristics reflecting the historical dynamic change trend;And increase action change rate penalty term in reward function, guide strategy output smooth executable collision avoidance action, realize the decision optimization of safety and execution stability.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of unmanned ship collision avoidance, and particularly relates to a smooth unmanned ship collision avoidance method considering ocean environment disturbance. BACKGROUND

[0002] With the wide application of unmanned surface vehicles (USVs) in the fields of marine patrol, hydrological measurement, environmental monitoring and military reconnaissance, the safety and stability problems in the process of collision avoidance have been increasingly valued. In the actual collision avoidance process, frequent and severe adjustment of the rudder angle or torque of the unmanned ship will cause serious wear to the actuator, and even lead to system failure, so it is urgent to design an intelligent collision avoidance strategy that takes into account safety and control stability.

[0003] In recent years, autonomous collision avoidance methods based on deep reinforcement learning (DRL) have gradually become a research hotspot. Through continuous interaction with the environment, these methods use reward signals to guide policy learning, and can realize end-to-end collision avoidance control from perception to decision-making, with good environmental adaptability and policy optimization ability. In complex and variable marine traffic scenarios, DRL shows stronger generalization ability and dynamic response ability compared with traditional methods. However, existing researches mostly focus on collision avoidance success rate and rule compliance in static water conditions, and pay little attention to the safety of the strategy and the stability of the control execution level in the context of ocean environment disturbance. Especially under the action of wind, wave, current and other complex disturbances, if the learned strategy has obvious fluctuations in action output, it may not only cause controller oscillation and system fatigue, but also increase the risk of structural damage, thereby seriously threatening the safety of unmanned ship navigation and equipment life.

[0004] To improve the practical applicability of DRL strategy, some studies attempt to introduce a control smoothing term into the reward function to punish excessive action changes, so as to encourage the strategy to generate more gentle control instructions. However, experiments show that simply relying on the reward term to balance the strategy performance and control stability has obvious limitations: on the one hand, too strong smoothing punishment may inhibit the reaction ability of the strategy and reduce the collision avoidance success rate; on the other hand, in dynamic multi-objective scenarios, there is a significant conflict between smoothness and safety, and it is difficult to achieve a balance between the two through static weight design. Therefore, how to improve collision avoidance safety while effectively suppressing the violent fluctuations of strategy output is still a key problem to be solved in current DRL unmanned ship collision avoidance control.

[0005] Existing technologies fail to adequately consider the impact of the smoothness of unmanned surface vessels' (USVs) action outputs during collision avoidance on system stability and operational safety. They typically prioritize collision avoidance success rate and rule compliance as optimization objectives, neglecting the dynamic smoothness requirements of collision avoidance strategies during continuous navigation. In practical applications, USVs operate under long-term conditions of multi-source disturbances such as wind, waves, and currents. If the strategy output exhibits drastic fluctuations, it can easily lead to frequent and significant adjustments in servo motors, abnormal propeller loads, and in severe cases, even control system oscillations, mechanical fatigue, or hardware failures. While some methods have attempted to constrain the magnitude of strategy changes by adding action smoothing terms to the reward function, this approach alone fails to balance safety and control smoothness in collision avoidance tasks, resulting in performance degradation when facing high-risk interactions or complex disturbances. Summary of the Invention

[0006] The purpose of this invention is to address the problems of existing solutions neglecting motion smoothness modeling during unmanned surface vessel (USV) collision avoidance, leading to drastic fluctuations in control commands, accelerated actuator wear, and insufficient strategy stability and reliability under complex ocean disturbance conditions. This invention provides a smooth collision avoidance method for USVs that considers ocean environmental disturbances. This method introduces a motion smoothness reward function into a deep reinforcement learning framework, guiding the strategy to output smooth control commands while achieving effective collision avoidance. Simultaneously, it improves the Actor-Critic network by incorporating the Mamba module, which has advantages in sequence modeling, to perform temporal modeling of historical dynamic states during navigation. This allows the strategy to perceive the evolution trend of states over time, enhancing its understanding and adaptability to target behavior patterns and disturbed environments, thereby significantly improving the success rate and robustness of the collision avoidance strategy. Through this method, while ensuring collision avoidance success rate and rule compliance, drastic fluctuations in motion output can be effectively reduced, improving the safe operation and control smoothness of USVs under real and complex sea conditions such as wind, waves, and currents.

[0007] To achieve the above objectives, the technical solution of the present invention is: a smooth collision avoidance method for unmanned surface vessels considering marine environmental disturbances, comprising:

[0008] Acquire the observation data of the current ship and surrounding target ships, perform preprocessing operations, and store the data in the historical status cache;

[0009] Iterate through the historical cache data for each moment in the historical state cache, perform geometric and kinematic calculations, generate the feature vector for the corresponding moment, and stack the feature vectors of each moment in chronological order.

[0010] The stacked result of the feature vectors at each time step is input into the SAC algorithm based on Mamba to obtain the collision avoidance control command at the current time step, and then sent to the unmanned ship's underlying control system for execution.

[0011] Furthermore, acquire the current observation data of the ship and surrounding target ships, and perform preprocessing operations, specifically including:

[0012] The ship's position in the geographic coordinate system was obtained using a dual-antenna GPS system. and heading ; Combine IMU to obtain the ship's pitch speed sway speed and bow angular velocity ;

[0013] Millimeter-wave radar is used to detect surrounding target ships in real time and obtain their relative distances. Relative azimuth and radial velocity ;

[0014] The data output from the dual-antenna GPS, IMU, and millimeter-wave radar are unified to the system clock reference, and data time alignment is achieved through interpolation or extrapolation; the relative coordinates measured by the millimeter-wave radar are converted into absolute position and velocity information in a geodetic coordinate system consistent with the GPS position;

[0015] Physical constraint verification and statistical filtering were performed on the acquired observation data of the ship and surrounding target ships to remove outliers and noise; for data lost in a short period of time, a constant speed or constant rotation rate model was used for prediction and compensation.

[0016] Furthermore, before storing the preprocessed observation data of the current ship and surrounding target ships in the historical state cache, the most dangerous target ship needs to be identified. The identification method is as follows:

[0017] The collision risk between the two ships is assessed based on the current observation data of both the ship and the target ship. Based on the assessment results, all detected target ships are ranked, and the target ship with the highest collision risk index is selected as the most dangerous target ship. A five-factor collision risk assessment model is used to calculate the collision risk index of the target ship based on the relative distance, relative speed, relative heading, minimum collision distance, and time to closest point. The calculation formula is as follows:

[0018]

[0019] In the formula This is the weight vector for each factor, reflecting its relative contribution to collision risk. Let be the fuzzy membership function for each risk factor, with a value range of . , , , , , respectively represent the weight of the closest point of approach distance DCPA, the closest point of approach time TCPA, the relative bearing, the relative distance and the relative heading in the collision risk, , , , and respectively represent the risk membership degree corresponding to the closest point of approach distance DCPA, the closest point of approach time TCPA, the relative bearing, the relative distance and the relative heading, denotes the transpose;

[0020] After identifying the most dangerous target ship, the observation data of the pre-processed own ship and the most dangerous target ship are stored in a history state cache with a maximum length of , wherein represents the observation data of the own ship and the most dangerous target ship spliced at the time. represents the x-direction position, the y-direction position, the heading angle, the surge velocity, the sway velocity and the yaw rate of the own ship in the global coordinate system, represents the x-direction position, the y-direction position, the heading angle, the surge velocity, the sway velocity and the yaw rate of the most dangerous target ship in the global coordinate system. Further, before traversing the history cache data at each time in the history state cache, it is necessary to determine whether the sequence length of the history state cache is less than L. If so, the length is filled up to L by means of adjacent frame copying.

[0021] Further, the history cache data at each time in the history state cache is traversed, and geometric and kinematic calculations are performed to generate the feature vector at the corresponding time. The feature vectors at each time are sequentially stacked in time sequence, and the specific way is as follows:

[0022] The history cache data at each time in the history state cache with a sequence length of L is traversed, and geometric and kinematic calculations are performed to generate the feature vector at the corresponding time. The calculation formula is as follows:

[0023]

[0024]

[0025] , wherein , , are the surge velocity, the sway velocity and the yaw rate of the own ship, , are the relative bearing angle and the relative heading angle between the own ship and the target point, , , , , ​The relative distance, relative bearing angle, relative heading angle, collision risk index and the composed meeting situation between the ship and the target ship, wherein The values are 1, 2, 3, 4 and 5, respectively, indicating that the ship and the other ship constitute a meeting, left crossing, right crossing, overtaking and being overtaken meeting situation; The speed of the ship;

[0026] After completing the feature vector calculation of each time in the history state buffer with a sequence length of L, the feature vectors of each time are stacked in time sequence, and the calculation formula is as follows:

[0027]

[0028] Wherein represents the state sequence matrix, represents the feature vector at the time, represents the sequence length, represents the dimension of the single-frame state vector .

[0029] Further, the stacking result of the feature vectors at each time is input into the SAC algorithm based on the improved Mamba to obtain the collision avoidance control instruction at the current time, which specifically includes:

[0030] The stacked state sequence matrix is projected to the hidden space dimension of the Mamba module through the full connection layer:

[0031]

[0032] Wherein , are the weight matrix and bias vector of the input linear mapping layer, is the projected sequence feature representation, and L is the sequence length of the history state buffer;

[0033] The projected sequence feature is input into the Mamba module to perform time series modeling and information fusion on the sequence feature:

[0034]

[0035] Wherein, represents the time series modeling based on the selective state space model, is the output sequence representation of the Mamba module.

[0036] The is mapped to the policy hidden space dimension through the linear mapping layer:

[0037]

[0038] where , are the weight matrix and bias vector of the linear mapping layer, respectively, denotes the output of the linear mapping layer.

[0039] In the policy latent space, the mean and log standard deviation of the action distribution are output by two independent fully connected layers, respectively:

[0040]

[0041] where , are the weight matrix and bias vector of the mean head, respectively, , are the weight matrix and bias vector of the standard deviation branch, respectively, denotes the truncation function, which limits the value of to the interval [-10, 2].

[0042] The log standard deviation is exponentiated to obtain the standard deviation :

[0043]

[0044] The action is sampled from the Gaussian distribution using the reparameterization method in the SAC algorithm:

[0045]

[0046] where denotes the element-wise multiplication, is the noise vector sampled from the standard multivariate normal distribution , denotes the identity matrix.

[0047] The is then mapped to the interval (-1, 1) by the tanh activation function to obtain the execution action :

[0048]

[0049] According to the physical constraints of the unmanned ship actuators, the is linearly mapped to the actual control quantity interval :

[0050]

[0051] wherein is the collision avoidance control instruction of the current moment, and is sent to the unmanned ship bottom layer control system for execution. respectively represent The maximum and minimum values of

[0052] Further, after the unmanned ship bottom layer control system executes the collision avoidance control instruction of the current moment, the system continues to collect observation data of the ship and surrounding target ships in real time through the multi-source sensor and updates the historical state cache, which is used to update the input sequence of the next control period, to realize the closed-loop control process of perception-decision-execution.

[0053] Further, in the SAC algorithm improved based on Mamba, a smoothness constraint reward is introduced to guide the policy network of the SAC algorithm improved based on Mamba to output a smooth executable collision avoidance control instruction.

[0054] The application also provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method described above.

[0055] The application also provides a computer program product comprising a computer program stored in a computer readable storage medium, and when the processor of an electronic device reads the computer program from the computer readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of the method described above.

[0056] Compared with the prior art, the application has the following beneficial effects:

[0057] The smooth collision avoidance decision method for unmanned ships based on smooth reward and Mamba time sequence modeling provided by the application solves the problems of large fluctuation of control instructions, fast wear of actuators and difficulty in fully utilizing historical dynamic information in the existing collision avoidance strategies, designs an action smoothness constraint reward function, and introduces a Mamba structure with long-time dependence modeling capability to improve the actor and critic networks in the existing SAC algorithm, so that the actor and critic networks have time sequence perception capability. The method effectively suppresses the violent change of collision avoidance actions while maintaining the collision avoidance success rate.

[0058] Through the smoothness reward constraint, the policy tends to generate control instructions with smaller change amplitude and good continuity in the training process, thereby improving the smoothness of the collision avoidance actions; through the Mamba time sequence modeling, the policy can perceive the evolution trend of the relative situation between the ship and the target ship over time, form more forward-looking collision avoidance actions, and improve the collision avoidance success rate. Compared with the traditional DRL method, the application can significantly improve the smoothness of the collision avoidance actions and the safety of the overall operation. BRIEF DESCRIPTION OF DRAWINGS

[0059] Figure 1 Flowchart of collision avoidance decision-making process for unmanned ship.

[0060] Figure 2 Flowchart of smooth collision avoidance strategy reasoning.

[0061] Figure 3 Flowchart of algorithm training. DETAILED DESCRIPTION

[0062] The technical solutions of the present application will be specifically described below in combination with the drawings.

[0063] The present application provides a smooth collision avoidance method for unmanned ship considering ocean environmental disturbance, comprising:

[0064] Obtain the observation data of the current time of the ship and the surrounding target ship, and store it in the history state cache after preprocessing operation;

[0065] Traverse the historical cache data of each time in the historical state cache, and perform geometric and kinematic calculation to generate the feature vector of the corresponding time, and stack the feature vectors of each time in time sequence;

[0066] The stacking result of the feature vectors of each time is input into the SAC algorithm based on Mamba improvement to obtain the collision avoidance control instruction of the current time, and sent to the bottom layer control system of the unmanned ship for execution.

[0067] The following is the specific implementation process of the present application.

[0068] The present application provides a smooth collision avoidance method for unmanned ship considering ocean environmental disturbance, comprising: Figure 1 As shown in the figure. Mainly includes the following steps:

[0069] Step 1: environmental perception and data preprocessing;

[0070] Step 2: history sequence arrangement;

[0071] Step 3: input of calculation model;

[0072] Step 4: strategy reasoning;

[0073] Step 5: instruction issuing and closed loop updating.

[0074] For step 1: the system first perceives the navigation environment through multi-source sensors to obtain the navigation state of the ship and the surrounding target ship and the disturbance elements of the marine environment. The perception data mainly includes the position, speed, heading, surge and sway velocity, and turning angle velocity of the ship, as well as the relative position, relative velocity, and heading angle of the target ship. To ensure the consistency and reliability of the data in subsequent processing, the system pre-processes various types of collected information to form a structured, non-redundant, and dimensionally unified observation data set. These pre-processed environmental perception data will serve as the input basis for subsequent strategy reasoning. The specific steps are as follows:

[0075] Step 1.1: Obtain the ship state.

[0076] Obtain the ship's position in the geographic coordinate system through dual-antenna GPS , heading ; obtain motion state parameters such as surge velocity , sway velocity , and turning angle velocity through IMU, and improve the positioning and attitude solving accuracy through sensor fusion algorithm.

[0077] Step 1.2: Obtain the target ship state.

[0078] Use millimeter wave radar to detect surrounding target ships in real time and obtain their relative distance , relative azimuth , and radial velocity ; through multi-target tracking algorithm, associate and maintain the radar track, and obtain the continuous motion state estimation of the target ship.

[0079] Step 1.3: Time synchronization and coordinate conversion

[0080] Unify the data output by dual-antenna GPS, IMU, and millimeter wave radar to the system clock reference, and realize data time alignment through interpolation or extrapolation; convert the relative coordinates measured by the radar into absolute position and velocity information in the geodetic coordinate system consistent with the GPS position.

[0081] Step 1.4: Data cleaning and abnormality rejection

[0082] Physically constrain and statistically filter the position, velocity, and heading, etc. to reject outliers and noise; for short-term missing data, use constant velocity or constant rotation rate model for prediction compensation to ensure data continuity and stability.

[0083] For step 2: after processing the observation data of the current time of the ship and the target ship, the system stores it in the historical state cache for extracting time series information of a certain length to provide data basis for the input of subsequent calculation models. The specific steps are as follows:

[0084] Step 2.1: Identify the most dangerous target ship

[0085] According to the observation data of the current time between the ship and the target ship, the collision risk between the two ships is evaluated, and according to the evaluation result, all the target ships currently detected are sorted, and the target ship with the highest collision risk index is selected as the main tracking object in this period. When the number of detected target ships is insufficient, record the empty slot and use zero vector to occupy. The specific evaluation method is a five-factor collision risk evaluation model, which considers the relative distance, relative speed, relative heading, minimum collision distance (DCPA) and time to closest point (TCPA) between the two ships, and calculates the collision risk index of the target ship. The overall risk index is calculated by the following linear weighting model:

[0086]

[0087] In the formula is the weight vector of each factor, reflecting its relative contribution to the collision risk, is the fuzzy membership function of each risk factor, with a value range of , , , , , respectively represent the weight of the closest encounter distance (DCPA), the closest encounter time (TCPA), the relative bearing, the relative distance and the relative heading in the collision risk, , , , and respectively represent the risk membership degree of the above five factors, represents the transpose;

[0088] Step 2.2: History cache maintenance

[0089] The observation data of the selected target ship at the current time and the ship observation data are stored in the state sequence cache with the maximum length of , where represents the observation data of the ship and the target ship at the time. represents the x direction position, y direction position, heading angle, surge speed, sway speed and turning angle speed of the ship in the global coordinate system, The x-direction position, y-direction position, heading angle, surge velocity, sway velocity, and turning angle velocity of the opposite ship in the global coordinate system are represented. The cache adopts a first-in, first-out strategy. When new data enters, the oldest historical state is removed to ensure a constant cache length.

[0090] For step 3: After completing steps 1 and 2, the system reads the latest observation sequence from the historical cache at the current time The observation data of the own ship and the target ship are calculated to generate a fixed-length input for the strategy network inference. The specific steps are as follows:

[0091] Step 3.1: Historical sequence length check and padding

[0092] From the historical cache of step 2 , the latest observation sequence is read. If the sequence length of the historical cache at the current time is less than the minimum sequence length required by the Mamba time encoding module , it is padded by adjacent frame duplication. Specifically: the adjacent time of the current time , i.e. , the available data is repeated and inserted into the back end of the sequence until the length requirement is met. This operation ensures that the improved strategy module of Mamba can still receive input sequences with consistent dimensions and fixed lengths even in the case of insufficient data at the initial stage of system startup.

[0093] Step 3.2: Calculate model input

[0094] After completing the historical sequence padding of step 3.1, the system iterates through each time of the padded historical cache data, and performs geometric and kinematic calculations on the observation data of the own ship and the target ship to generate the feature vector at the corresponding time. The specific calculation formula of each time feature vector is as follows:

[0095]

[0096] where , , are the surge velocity, sway velocity, and turning angle velocity of the own ship, , are the relative bearing angle and relative heading angle between the own ship and the target point, , , , , are the relative distance, relative bearing angle, relative heading angle, collision risk index, and the formed encounter situation between the own ship and the target ship, where takes the values 1, 2, 3, 4, and 5, respectively, representing the encounter, left crossing, right crossing, overtaking, and being overtaken encounter situations between the own ship and the opposite ship. the target ship's speed;

[0097] After the feature vector calculation of each time point in the history state buffer with sequence length L is completed, the feature vectors of each time point are stacked in time sequence to form the model input, and the calculation formula is as follows:

[0098]

[0099] wherein denotes the state sequence matrix, denotes the state at the time point, denotes the sequence length, denotes the dimension of the single-frame state vector.

[0100] For step 4: after the model input calculation of step 3 is completed, the result is input into the strategy network to obtain the real-time collision avoidance action. The specific calculation process is shown in Figure 2 . The specific steps are as follows:

[0101] Step 4.1: linear projection

[0102] The stacked state sequence matrix is projected to the hidden space dimension of the Mamba module through the fully connected layer:

[0103]

[0104] wherein , are the weight matrix and bias vector of the input linear mapping layer, respectively, is the projected sequence feature representation, and L is the sequence length of the history state buffer H;

[0105] Step 4.2: Mamba time sequence encoding

[0106] The projected sequence feature is input into the Mamba module to model the sequence feature and fuse the information:

[0107]

[0108] wherein, denotes the time sequence modeling based on the selective state space model, is the output sequence representation of the Mamba.

[0109] Step 4.3: Mamba encoder output mapping

[0110] The ​Mapping to policy latent space dimensions :

[0111]

[0112] where , are the weight matrix and bias vector of the linear mapping layer, respectively, denotes the linear mapping layer output.

[0113] Step 4.4: Gaussian policy distribution generation

[0114] In the policy latent space, the mean and log standard deviation of the action distribution are outputted by two independent fully connected layers, respectively:

[0115]

[0116] where , are the weight matrix and bias vector of the mean head, respectively, , are the weight matrix and bias vector of the standard deviation related branch, respectively, denotes the clamp function, which limits the value of to the interval [-10, 2].

[0117] The log standard deviation is exponentiated to obtain the standard deviation :

[0118]

[0119] Step 4.5: reparameterization sampling and action squashing

[0120] The action is sampled from the Gaussian distribution using the reparameterization method in the SAC algorithm:

[0121]

[0122] where denotes the element-wise multiplication, is the noise vector sampled from the standard multivariate normal distribution , denotes the identity matrix.

[0123] The action is then mapped to the interval (-1, 1) by the tanh activation function to obtain the executed action :

[0124]

[0125] Step 4.6: Action mapping to control commands

[0126] According to the physical constraints of the unmanned ship executor, linear mapping to the actual control quantity interval :

[0127]

[0128] where is the collision avoidance control command at the current time, and is sent to the unmanned ship bottom layer control system for execution, respectively represent the maximum and minimum values of .

[0129] For step 5: After completing the strategy reasoning of step 4 and obtaining the current time control action vector, the control command is sent to the bottom layer execution unit of the unmanned ship in real time through the communication interface. The bottom layer control system adjusts the ship heading, propulsion speed and other motion parameters according to the received command, so as to implement the collision avoidance action at the physical level.

[0130] After the completion of the command execution, the system continues to collect the ship and environmental state information in real time through multiple source sensors and updates the historical cache observation data, which is used to update the input sequence of the next control period, realizing the closed-loop control process of perception-decision-execution. Through this closed-loop mechanism, the system can continuously correct the collision avoidance strategy in the presence of dynamic factors such as sea state changes and target ship maneuvers, ensuring the navigation safety and path smoothness of the unmanned ship.

[0131] In order to improve the smoothness of unmanned ship in dynamic multi-target ship encounter and real sea state disturbance conditions, the present application based on Soft Actor-Critic (SAC) algorithm, proposes an unmanned ship collision avoidance decision method which integrates Mamba time series modeling structure and action smoothness constraint. This method aims at the performance degradation problem of existing collision avoidance strategy when facing multi-target interaction and control command stability requirements, by introducing Mamba sequence encoder in Actor-Critic network, the ship state at multiple times is modeled, so as to extract the time series features reflecting the historical dynamic change trend; and increase the action change rate penalty term in the reward function, guide the policy output to execute smooth collision avoidance action, realize the decision optimization considering safety and execution stability. The specific training steps are as follows:

[0132] Step 1: Dynamics modeling and environmental disturbance design

[0133] Step 2: State and action space design

[0134] Step 3: Constructing reward function

[0135] Step 4: Actor-Critic network design

[0136] Step 5: Loss function design

[0137] Step 6: Time buffer pool design

[0138] Step 7: Training process design

[0139] For step 1: Fossen's three-degree-of-freedom ship motion mathematical model is used to describe the unmanned ship motion during training; to simulate the external disturbances such as wind, wave, and flow in the real sea state, Ornstein-Uhlenbeck (OU) noise is used to simulate the interference of the ocean environment on the unmanned ship motion.

[0140] The three-degree-of-freedom ship motion mathematical model used for training is as follows:

[0141]

[0142] where is the velocity vector in the ship coordinate system, denotes the derivative of , is the sum of the rigid body and the added mass, is the Coriolis and centripetal force matrix, is the viscous damping matrix, is the control input (thrust and yaw moment), is the environmental disturbance force / moment. The relationship between the ship attitude and position vector and the velocity is:

[0143]

[0144] where is the ship-to-ground coordinate system conversion matrix.

[0145] To efficiently generate time-varying disturbance signals during training, OU noise is used to model :

[0146]

[0147] where is the OU state vector at time t, is the mean, is the regression rate, is the fluctuation amplitude, is the standard Wiener process. After discretization, it can be written as:

[0148]

[0149] where​ For discrete time steps, Let be independent and identically distributed standard Gaussian random variables, satisfying , Represents the identity matrix.

[0150] Environmental disturbance at time t Through the proportionality coefficient The matrix is ​​mapped from the OU state.

[0151]

[0152] The generated disturbance signal can maintain a certain smoothness while adjusting the amplitude and correlation time, making it convenient to simulate different sea conditions.

[0153] Regarding step 2: During training, the agent's state vector is composed of information about the current vessel, the most dangerous target vessel, and the target point. Features are calculated and combined using the specific formula for calculating the feature vector at each moment, as described above, ensuring a fixed input dimension and collision risk perception capability. A continuous action space is employed, with the unmanned vessel's yaw moment as the sole control input, controlling heading changes to achieve collision avoidance and navigation tasks.

[0154] For step 3: To achieve effective training of the autonomous collision avoidance strategy, a composite reward function is constructed, comprehensively considering target distance reward, heading reward, collision penalty, collision risk penalty, speed obstacle penalty, international collision avoidance rule penalty, and smoothing reward. The overall reward function is calculated as follows:

[0155]

[0156] in These represent the weighted coefficients for distance bonus, heading bonus, collision risk penalty, collision penalty, violation of collision avoidance rules penalty, speed obstacle penalty, and smoothness bonus, respectively. The specific numerical values ​​for each reward component, and the detailed meanings of each sub-item, are as follows:

[0157] (1) Distance Reward Items ( )

[0158] The unmanned surface vessel is encouraged to continue moving towards the target point, specifically calculated as follows:

[0159]

[0160] in , These represent the Euclidean distances from the unmanned surface vessel to the target point at the previous and current moments, respectively. This indicates the speed of the unmanned surface vessel at the previous moment. This represents the simulation step size, set to 1.

[0161] (2) Heading bonus items ( )

[0162] The unmanned surface vessel is encouraged to adjust its course toward the target point, and the calculation formula is as follows:

[0163]

[0164] (3) Collision risk penalty items ( )

[0165] This item is used to penalize unmanned surface vessels for collision avoidance behaviors that increase navigational risks, and its calculation is shown in the following formula:

[0166]

[0167] in This indicates the collision risk between this vessel and the most dangerous target vessel.

[0168] (4) Collision penalty items ( )

[0169] Triggered when the agent collides with another object or successfully reaches the target point, the calculation is as follows:

[0170]

[0171] in , , and Represent the distance between the unmanned surface vessel (USV) and the target point, respectively, and the distance between the USV and the target point. Distance between the two opposing vessels, distance threshold for determining whether the target point has been reached, and the first The radius of the safety zone of the other vessel.

[0172] (5) Penalties for violating collision avoidance rules ( )

[0173] To improve the model's compliance with the International Regulations for Preventing Collisions at Sea (COCR), this penalty term is added to the single-vessel encounter scenario. This term guides the unmanned surface vessel (USV) to take avoidance measures that comply with the COCR. Its calculation is shown in the following formula:

[0174]

[0175] (6) Speed ​​obstacle penalty item ( )

[0176] In ship collision avoidance, VO is a speed space-based collision avoidance modeling method, which defines that if the speed of the own ship (OS) falls into the speed obstacle region formed by the target ship or the opposite ship (TS) within a given time window, the two ships are likely to collide. Therefore, the VO penalty term is introduced in the reward function to punish the action selection that may lead to entering the collision trajectory during the reinforcement learning training process, so as to drive the strategy to automatically avoid the unsafe speed region. The calculation is as follows:

[0177]

[0178] Wherein represents the speed vector of the own ship; It is explained that there is no predicted collision risk under the current speed selection, and the reward term is 0 at this time; It is explained that the speed of the own ship falls into the speed obstacle region, and the future may collide, so a negative reward is given.

[0179] (7) Smooth reward (R) )

[0180] According to the action second-order difference, the smooth reward is designed to suppress the action mutation:

[0181]

[0182] Wherein , and represent the actions calculated by the Actor network at , and .

[0183] For step 4: In SAC, the Actor network outputs continuous action distribution with state input, which is used to generate yawing torque control instructions; the Critic receives state-action pairs and outputs Q value estimates, which are used to evaluate the strategy. Through the automatic adjustment mechanism of the temperature parameter, the balance between exploration and optimality is achieved.

[0184] In order to enhance the modeling ability of the time sequence characteristics of the target ship interaction, the Mamba sequence modeling module is introduced into the Actor and Critic networks, so that it can process state sequence input, extract the dynamic evolution law of the state space in the navigation process, and thus improve the safety and smoothness of decision-making.

[0185] Actor network: the Mamba module first performs dynamic modeling on the input state sequence, extracts the time sequence evolution law in the navigation process, and outputs the last frame as the time sequence feature representation of the current decision, which is mapped to the policy hidden space to generate the control distribution for outputting the yawing torque action.

[0186] Critic network: The Critic network also inputs the state sequence into the Mamba module for temporal feature extraction, and splices the current candidate action at the feature layer, and then calculates the corresponding Q value estimate through a multi-layer perception. Such a structure ensures that the Critic's evaluation of the action's pros and cons not only depends on the instantaneous state, but also combines the dynamic context of historical interactions, improving the ability to evaluate the value of complex ship interaction scenarios.

[0187] The network structure is consistent with the inference process in the "policy inference" part of the summary: in the training phase, the Mamba module is also responsible for extracting temporal features from the state sequence, and the last frame is taken as the input for the current decision, and the action is output by the Actor, while the Critic calculates the Q value based on the same temporal features and actions, realizing the functional unity with the actual inference phase, thereby ensuring the consistency of training and inference and the transferability of policy effect.

[0188] For step 5: In the present application, the Actor loss function remains consistent with the standard SAC, still mainly maximizing the entropy regularization policy target, without additional modifications.

[0189] For the Critic network, in order to improve the evaluation ability of the long-term effect of actions in complex interaction scenarios, the present application introduces the weighted combination of n-step cumulative rewards on the basis of traditional single-step rewards. The target value of the Critic is given by the following formula:

[0190]

[0191] wherein is a weighting coefficient for balancing between short-term stability and long-term effect modeling; is a discount factor; , represent the rewards at and time; , represent the outputs of the Mamba module at and time; , represent the outputs of the Actor network at and time; represents the output of the Critic network in and cases; represents the output of the Critic network in and cases.

[0192] This design makes the Critic's value evaluation not only rely on single-step rewards, but also combine multi-step cumulative returns, thereby improving learning efficiency and stability under dynamic interaction and complex environmental disturbances.

[0193] For step 6: To meet the input requirements of the Mamba temporal encoder for consecutive historical frames, and to utilize n-step temporal information in the Critic target, the invention designs a temporal experience replay pool that supports fixed-length sequence sampling, n-step return calculation, and GPU vectorization based on the conventional experience pool. This buffer pool not only stores single-step transitions, but also ensures that the samples sampled in batches are continuous in time, facilitating direct input to the sequence network of Actor-Critic.

[0194] First, during the interaction process, the buffer pool stores the state of the unmanned ship, the control action, the reward value, the completion flag, and the next time state in turn, forming a continuous trajectory segment. Unlike traditional single-step storage, this buffer pool automatically maintains a fixed-length state sequence during storage, ensuring that each sampled data contains historical information during the navigation process, thereby meeting the input requirements of the Mamba module.

[0195] Second, to improve training efficiency, the n-step return calculation mechanism is directly introduced into the buffer pool. In the sampled state sequence, the buffer pool calculates the multi-step cumulative reward at the end of the sequence and directly obtains the reward through sampling, thereby avoiding additional computational overhead during training.

[0196] Finally, the buffer pool fully utilizes parallel processing and vectorization operations in implementation, allowing high throughput even in large-scale training. Through this design, the Actor-Critic network can obtain single-step feedback for short-term fine optimization and utilize multi-step returns to capture global behavior trends during training, thereby improving the stability of decision-making and the success rate of collision avoidance.

[0197] For step 7: The training process mainly includes five steps: environment interaction, data storage, sequence sampling, network updating, and parameter optimization. The overall process is shown in Figure 3 The specific steps are as follows:

[0198] Step 7.1: Environment interaction and data collection

[0199] The unmanned ship agent selects actions according to the current strategy in the simulation environment. The state space includes the ship's position, heading, speed, target point information, and the kinematic characteristics of surrounding target ships. The action is generated by the policy network and mapped to a control torque signal to drive the unmanned ship to complete the navigation. The environment feeds back the reward value and the next time state, and gives the termination flag.

[0200] Step 7.2: Experience storage and n-step cumulative return pre-computation

[0201] The resulting state, action, reward, next state, and terminal flag are stored in the experience buffer. When storing sequence data, the buffer directly computes the multi-step cumulative return at the end frame position of the sequence:

[0202] Step 7.3: Sequence sampling and temporal modeling

[0203] In the parameter update phase, the buffer samples fixed-length sequences (length 3) according to a batch size of 256. The sampled sequences are fed into the policy network and the value network, where the sequence part is processed by the Mamba module for temporal feature extraction. It includes an input projection layer (dimension 256), a state convolution kernel size of 4, a hidden state dimension of 64, and a gating unit. The Mamba module dynamically models historical features and effectively captures the time correlation of target ship behavior patterns.

[0204] Step 7.4: Policy network and value network update

[0205] The policy network (Actor) consists of two fully connected layers with 256 neurons each, using the ReLU activation function, and outputs the mean and standard deviation of the action distribution. Sampling is performed through the reparameterization trick, and the corresponding policy loss is calculated. The value network (Critic) uses a double Q structure, with each Q network being a two-layer fully connected network (256 neurons per layer). It estimates the value of state-action pairs and updates them in comparison with the target network to reduce overestimation bias.

[0206] Step 7.5: Parameter optimization and target network soft update

[0207] The Adam optimizer is used for gradient descent, with a learning rate set to The target network replaces parameters through a soft update mechanism.

[0208] Step 7.6: Automatic temperature adjustment mechanism

[0209] During training, the temperature parameter is automatically adjusted to balance exploration and exploitation, ensuring that the policy can both explore sufficiently and converge to a stable control strategy.

[0210] Step 7.7: Iterative loop and policy convergence

[0211] The above process continues to iterate until the agent achieves stable high success rates and smooth control performance in the collision avoidance task, enabling the unmanned boat to safely navigate in complex dynamic environments.

[0212] By the provided technical solutions, the Mamba timing modeling module and the action smoothness reward design are introduced into the SAC framework when the Mamba is introduced into the application, so that the collision avoidance performance and the control stability of the unmanned ship in a complex dynamic environment can be significantly improved.

[0213] Specifically, the Mamba module is embedded into the structure of the policy network and the value network as an efficient sequence modeling unit, and replaces the traditional fully connected layer to solve the problem of time dependence. The module can effectively capture the time evolution law of the motion characteristics of the target ship and the historical interaction state, so that the network has stronger dynamic modeling capability and generalization capability when processing the multi-target ship scene and uncertain behavior mode.

[0214] In the reward function design, the action smoothness reward term is designed, the change rate of the control instruction at adjacent time steps is constrained, the policy output is guided to keep stable in the collision avoidance process, and the risk of excessive wear of the actuator and ship maneuvering caused by action oscillation is avoided. The reward term and the safety and rule compliance reward jointly constitute a composite incentive system, so that the unmanned ship can avoid collision while realizing the stability and compliance of the maneuvering action in the training process.

[0215] In combination with the above technical solutions, the application not only enhances the time sequence modeling capability through the Mamba in the network structure, but also optimizes the action stability through the smoothness reward in the training target, so that the unmanned ship has high success rate and low oscillation under the conditions of multi-target encounter and uncertain disturbance. The method has good engineering application prospect and can be widely applied to intelligent shipping, ocean monitoring, search and rescue tasks and other autonomous navigation scenes.

[0216] The application further provides an electronic device including a processor and a memory, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method described above.

[0217] The application further provides a computer program product including a computer program stored in a computer readable storage medium, and when a processor of an electronic device reads the computer program from the computer readable storage medium, the processor executes the computer program, so that the electronic device executes the steps of the method described above.

[0218] The above is the preferred embodiment of the application, and any change made according to the technical solutions of the application, as long as the generated function does not exceed the range of the technical solutions of the application, belongs to the protection scope of the application.

Claims

1. A method for smooth collision avoidance of an unmanned ship considering ocean environment disturbance, characterized in that, The method comprises: Obtain the observation data of the current time of the ship and the surrounding target ship, and store it in the historical state cache after preprocessing operation; Traverse the historical cache data of each time in the historical state cache, and perform geometric and kinematic calculation to generate the feature vector of the corresponding time, and stack the feature vectors of each time in time sequence; The stacking result of the feature vectors of each time is input into the SAC algorithm based on Mamba improvement to obtain the collision avoidance control instruction of the current time, and sent to the unmanned ship bottom layer control system for execution; Obtain the observation data of the current time of the ship and the surrounding target ship, and perform preprocessing operation, specifically including: Obtain the position of the ship in the geographical coordinate system by dual-antenna GPS and the heading angle ; obtain the surge velocity , sway velocity and yaw angular velocity of the ship in combination with the IMU Real-time detection of surrounding target ships using millimeter wave radar to obtain the relative distance , relative azimuth angle and radial velocity of the target ship; Align the data time of the data output by the dual-antenna GPS, IMU and millimeter wave radar to the system clock reference, and realize data time alignment by interpolation or extrapolation; Convert the relative coordinates measured by the millimeter wave radar into absolute position and velocity information in the geodetic coordinate system consistent with the GPS position; Physically constrained test and statistical filtering are performed on the obtained observation data of the ship and the surrounding target ship to eliminate abnormal values and noise; for the short-time missing data, a constant speed or constant rotation rate model is used for prediction compensation.

2. The method of claim 1, wherein the ocean environment disturbance is a sea state. The observation data of the ship and the surrounding target ship after preprocessing operation at the current time needs to be identified as the most dangerous target ship before being stored in the historical state cache, and the identification method is: According to the observation data of the current time of the ship and the target ship, the collision risk between the two ships is evaluated, and according to the evaluation result, all the target ships detected at the current time are sorted, and the target ship with the highest collision risk index is selected as the most dangerous target ship; the collision risk adopts a five-factor collision risk evaluation model, and the collision risk index of the target ship is calculated through the relative distance, relative speed, relative heading, minimum collision distance and time to closest point between the two ships , the calculation formula is as follows: In the formula is the weight vector of each factor, reflecting its relative contribution to the collision risk, is the fuzzy membership function of each risk factor, with a value range of , , , , , respectively represent the weight of the closest approach distance DCPA, the closest approach time TCPA, the relative bearing, the relative distance and the relative heading in the collision risk, , , , and respectively represent the risk membership corresponding to the closest approach distance DCPA, the closest approach time TCPA, the relative bearing, the relative distance and the relative heading, represents the transpose; After the most dangerous target ship is identified, the observation data of the pre-processed own ship and the most dangerous target ship are stored in a history state buffer with a maximum length of , wherein represents the observation data of the own ship and the most dangerous target ship spliced at the th moment, represents the x-direction position, the y-direction position, the heading angle, the surge velocity, the sway velocity and the yaw rate of the own ship in the global coordinate system, represents the x-direction position, the y-direction position, the heading angle, the surge velocity, the sway velocity and the yaw rate of the most dangerous target ship in the global coordinate system.​ 3. The method of claim 1, wherein the ocean environment disturbance is a sea state. Before traversing the historical cache data of each time in the historical state cache, it is necessary to determine whether the sequence length of the historical state cache is less than L, if so, the length is filled to L by adjacent frame replication.

4. The method of claim 1, wherein the ocean environment disturbance is a sea state. Traverse the historical cache data of each time in the historical state cache, and perform geometric and kinematic calculation to generate the feature vector of the corresponding time, and stack the feature vectors of each time in time sequence, specifically: Traverse the historical cache data of each time in the historical state cache with sequence length L, and perform geometric and kinematic calculation to generate the feature vector of the corresponding time, the calculation formula is as follows: wherein , , are the surge velocity, sway velocity and yaw rate of the own ship respectively, , are the relative bearing angle and relative course angle between the own ship and the target point respectively, , , , , are the relative distance, relative bearing angle, relative course angle, collision risk index and the composed encounter situation between the own ship and the target ship respectively, wherein takes the values 1, 2, 3, 4, 5, and represents the encounter, left crossing, right crossing, overtake and be overtaken encounter situation between the own ship and the target ship respectively; is the speed of the target ship; After completing the calculation of the feature vector of each time in the historical state cache with sequence length L, the feature vectors of each time are stacked in time sequence, the calculation formula is as follows: wherein denotes the state sequence matrix, denotes the feature vector at time denotes the feature vector at time denotes the sequence length, denotes the dimension of the single frame state vector .

5. The method of claim 1, wherein the ocean environment disturbance is a sea state. The stacking result of the feature vectors of each time is input into the SAC algorithm based on Mamba improvement to obtain the collision avoidance control instruction of the current time, specifically including: The stacked state sequence matrix is projected to the hidden space dimension of the Mamba module by a fully connected layer projection to the hidden space dimension of the Mamba module : wherein , are the weight matrix and bias vector of the input linear mapping layer, respectively, is the projected sequence feature representation, and L is the sequence length of the history state buffer. projected sequence features inputting the mamba module, which performs temporal modeling and information fusion on the sequence features: wherein, denotes a time series modeling based on a selective state space model, is an output sequence representation for the Mamba module; by a linear mapping layer to policy latent space dimensions : wherein , are the weight matrix and bias vector of the linear mapping layer, respectively, denotes the output of the linear mapping layer; In the policy latent space, the mean of the action distribution is output by two independent fully connected layers and the log standard deviation : where , are the weight matrix and bias vector of the mean head, respectively, , are the weight matrix and bias vector of the standard deviation related branch, respectively, denotes a clipping function that limits the value of to the interval [-10, 2]; Log standard deviation Exponentialized to standard deviation : The reparameterization method in SAC algorithm is adopted to sample actions from Gaussian distribution : wherein denotes element-wise multiplication, is a noise vector sampled from a standard multivariate normal distribution with mean zero and identity covariance matrix, denotes the identity matrix; Subsequently, the mapping is performed to the interval (-1, 1) by a tanh activation function to obtain the performed action : According to the physical constraints of the unmanned ship actuators, the linear mapping to the actual control quantity interval : wherein is the collision avoidance control instruction at the current time, and is sent to the unmanned ship bottom layer control system for execution, respectively represent the maximum and minimum values of 6. The method of claim 1, wherein the ocean environment disturbance is a sea state. After the unmanned ship bottom layer control system executes the collision avoidance control instruction of the current time, the system continues to collect the observation data of the ship and the surrounding target ship through multiple sensors in real time and updates the historical state cache, which is used to update the input sequence of the next control period, realizing the closed-loop control process of perception-decision-execution.

7. The method of claim 1, wherein the ocean environment disturbance is a sea state. In the SAC algorithm based on Mamba improvement, a smoothness constraint reward is also introduced to guide the policy network of Mamba improved SAC algorithm to output smooth executable collision avoidance control instruction.

8. An electronic device, comprising: The processor and the memory, wherein the memory stores a computer program, when the computer program is executed by the processor, the processor executes the steps of the method as claimed in any one of claims 1-7.

9. A computer program product, characterised in that, The computer program is stored in a computer readable storage medium. When an electronic device reads the computer program from the computer readable storage medium, a processor of the electronic device executes the computer program, so that the electronic device performs the steps of the method according to any one of claims 1-7.

Images