Multi-sensor fusion intelligent anti-collision method and system

Abstract

The invention belongs to the technical field of ocean detection, belongs to a multi-sensor fusion intelligent anti-collision method and system, comprises a sensing layer, a processing layer and an application layer, and provides an intelligent anti-collision and evidence recording ocean monitoring floating system integrating computer vision, target ranging, satellite positioning and ship automatic recognition system multi-sensor fusion. According to the invention, YOLOv8 target detection, Transform data fusion and a Kalman filtering algorithm are adopted, so that accurate detection and anti-collision early warning of ships and floating objects on the sea are realized. The system has an AIS failure processing mechanism and an evidence encryption storage function, and ensures reliable operation and data compliance under complex sea conditions.

Description

Technical Field

[0001] This application relates to the field of marine detection technology, specifically to an intelligent collision avoidance method and system that integrates multiple sensors. Background Technology

[0002] Marine buoys are widely used in marine environmental monitoring, meteorological observation, and navigation safety. However, current marine buoy systems have significant shortcomings in ship collision prevention and accident evidence recording: Single sensor reliance: Traditional buoys mainly rely on AIS (Automatic Identification System) to monitor ship dynamics. However, AIS is susceptible to signal interference, deliberate shutdown, or tampering, preventing buoys from obtaining crucial information such as the hull number, position, heading, and speed of the offending vessel; Incomplete evidence chain: After a collision, existing buoys lack the ability to fuse and record multi-source data, failing to provide a legally valid complete evidence chain (such as spatiotemporally synchronized data like ship attitude, position, images, and radar point clouds). Summary of the Invention

[0003] A multi-sensor fusion method and system for intelligent buoys addresses the aforementioned problems through high-precision target detection, multi-source data fusion, and collision evidence chain generation. The technical solution is as follows: A multi-sensor fusion-based intelligent collision avoidance method includes the following steps: S1. Obtain environmental information and preprocess the data; S2. Feature Extraction: Using a lightweight detection framework, ship feature values ​​are extracted; S3. Multi-sensor data fusion: Employs a two-way information flow closed loop formed between the Transformer and Kalman filter prediction modules to fuse data from vision, radar, and AIS; S4. Conduct a risk assessment.

[0004] Preferably, data preprocessing includes: image preprocessing: performing distortion correction and stereo matching processing on the images acquired by the binocular cameras; Radar data preprocessing: The lidar point cloud data is filtered, and the attitude data calculated by the IMU is used to perform motion compensation on the lidar and millimeter-wave radar data to eliminate the measurement error caused by the buoy due to the waves. Position calculation: The coordinates of the buoy are calculated based on quaternions, and the coordinates calculated by the IMU are corrected by Kalman filtering in combination with Beidou satellite coordinate data; Time calibration: Multi-sensor time synchronization is achieved through RTC and the BeiDou system; AIS Status Detection: Detects the status of the AIS signal to determine whether nearby vessels have activated AIS; When AIS is normal: AIS data is given high weight to correct radar and visual feature extraction; When AIS is malfunctioning: AIS data weight is reduced and radar and visual feature extraction weight is increased.

[0005] Preferably, the feature extraction steps are as follows: S21. The feature dataset includes ship outline chords and floating objects; the activation function used is the SiLU function. The hull number recognition adopts a CRNN-CTC network architecture, uses PSMNet to generate synthetic data to simulate interference caused by the external environment (fog / waves), and uses the SE module to enhance the weight of the character region. S22. A fusion strategy is adopted, primarily using visual data and secondarily using radar data. The steps are as follows: Binocular disparity calculation: The target distance is calculated by the pixel offset between the left and right images. The raw data acquired by the binocular cameras is processed by the Brown-Conrady model to eliminate lens distortion, and then the Bouguet algorithm outputs horizontally aligned images. The left and right images are then matched by the SGM algorithm to generate a disparity map. Finally, based on the camera calibration parameters, the disparity map is converted into depth values. The calculation formula is: ; Camera focal length; Binocular baseline distance; : disparity value; Radar data: Point cloud cluster data matrix after background filtering and motion compensation, optimized by merging neighboring points using the DBSCAN algorithm; depth calibration is performed on the disparity map generated by binocular vision calculation; Fusion computing module: Radar depth data and visual depth data are corrected using formulas, and the corrected depth data is output. The fusion formula is: ; Laser reference depth; Original visual depth; Radar depth.

[0006] Preferably, the weight of AIS is determined based on the AIS status detection value. If AIS fails, a lightweight detection framework and Kalman filtering algorithm are used based on the features acquired by radar. The AIS health assessment model is as follows: ; , , These are the weighting coefficients. For received signal strength, For actual reporting frequency, Maximum allowed reporting frequency (configurable) For log consistency evaluation (empirical value), assess the logic and rationality of the data reported by AIS devices.

[0007] Preferably, the Transformer encoder fuses different heterogeneous data into a unified feature representation: Feature input: Mapping features from heterogeneous sensors such as vision, radar, and AIS to a unified semantic space; Feature embedding stage: Sensor features of different dimensions are linearly projected into a unified vector space, and all modal features are unified to 512 dimensions; Through a multi-head attention mechanism, all sensor features interact, complement, and enhance each other, learning the deep relationships between them. The processing steps are as follows: Feature concatenation: The four 512-dimensional feature vectors are concatenated along the sequence dimension to form a larger feature matrix; Location encoding: For each location in the concatenated sequence, i.e. each sensor feature, a unique 512-dimensional location vector is generated and added to the feature. The unified feature sequence with added location information is then fed into an encoder composed of multiple stacked Transformer blocks for depth processing. Multi-head self-attention mechanism: Calculates the correlation weight between each feature in the sequence and all other features in the sequence; The self-attention model is: ; Query, key, value, The dimension of the key vector; The multi-head self-attention model is as follows: ; ; Where h is the number of heads. Let be the feature transformation matrix of the i-th head. The transformation matrix is ​​output for multi-head features; Feedforward Neural Network (FNN): Performs nonlinear transformations and enhancements on each attention-weighted feature. ; , It is a learnable weight matrix. It is the input feature matrix. , For the bias of the first and second layers; Residual connection and layer normalization: The residual connection is calculated by adding the sub-layer input and sub-layer output. The mean and variance of the result after residual connection are calculated and layer normalization is performed.

[0008] Preferably, the target motion state is predicted based on the fused features, and the steps are as follows: Step 1: Initialize state estimates ; ; Position (x, y, z), velocity Bow Speed Turning rate ; Step 2: Standard Kalman filter prediction: State prediction: ; It is the state transition matrix. It is a control input. It is a control matrix; Error covariance predict: ; It is the adaptively adjusted process noise covariance matrix from the previous time step; Step 3: Standard Kalman filter update: Calculate the Kalman gain: ; It is the adaptively adjusted observation noise covariance matrix from the previous time step. The observation matrix is ​​used to map the 9-dimensional state space to the 6-dimensional observation space. Status Update: ; in, These are enhanced observations from the Transformer; Step 4: Adaptive Adjustment

[0009] ; New sequence It is a function with zero mean and covariance of A white noise sequence; Estimating the actual covariance of the innovation: Calculate the actual covariance of the innovation sequence using a sliding window of length N. ; in ; Adjust the noise covariance matrix: Adjusting observation noise If the actual covariance of the new information Greater than its theoretical value ， Need to increase , to generate a new ; ; Step 5: Iterate through the loop. Forward fusion: ; in It is the fused feature output by the Transformer. Let k be the feature vector at time k; It is a projection matrix. Let be the original observation vector of the sensor at time k; Feedback optimization: Kalman filtering provides The temperature coefficient used to adjust the attention calculation in the next frame of the Transformer. : ； ； The larger the value, the more evenly attention is distributed, reducing reliance on unreliable historical information.

[0010] Preferably, if the innovation sequence shows correlation, it indicates that the adaptively adjusted process noise covariance matrix is ​​inaccurate and needs adjustment. To adjust the latest observation matrix Weights: ; Apply the adjusted noise matrix: The newly estimated and The smoothed value is used for prediction and update at the next time step (k+1): ; ; in , It is a smoothing factor.

[0011] Preferably, collision risk is calculated based on the Time of Closest Approach (TCPA) and Distance of Closest Approach (DCPA) models for vessels or floating objects: The formula for calculating the nearest proximity distance (DCPA) is as follows: ; ; Relative position vector, for x The relative position vector in the direction, yes y The relative position vector in the direction; Relative velocity vector for x The relative velocity vector in the direction, for y The relative velocity vector in the direction; The closest time TCPA: ; The two ships are approaching, but have not yet reached their closest point; It is exactly at the closest point; The two ships have passed their closest point and are now moving away from each other.

[0012] Preferably, the steps for calculating the relative motion vector are as follows: From the Kalman filter, accurate state estimates of the target (T) and buoy (O) were obtained: Buoy (O) status: ; ( ): The position of this ship in the Northeast ENU coordinate system; ( ): The components of the ship's velocity vector in the east and north directions; Own ship speed: ; This vessel's course: ; Target (T) state: ; ( ): The location of the target ship; ( : The velocity vector of the target ship; Target ship speed: ; Target ship's course: ; Calculate the relative motion vector: Relative position vector: ; Relative velocity vector: .

[0013] A multi-sensor fusion intelligent collision avoidance system, comprising a perception layer, a processing layer, and an application layer; Perception layer: This is the system's data input layer, responsible for collecting environmental information from various sensors; Processing layer: responsible for data preprocessing, feature extraction, multi-sensor fusion, collision risk assessment, and collision evidence chain data preparation; Application layer: Responsible for system output and data storage.

[0014] Compared with the prior art, the beneficial effects of this application are as follows: Multimodal sensor fusion: Efficient fusion of heterogeneous data is achieved through YOLOv8+Transformer and Kalman filtering.

[0015] Dynamic target tracking: Target motion modeling based on LiDAR point cloud and visual data, supporting trajectory prediction in complex scenarios.

[0016] Collision evidence chain generation: Synchronously store video, radar point cloud, AIS data and timestamps to form an immutable evidence package. Attached Figure Description

[0017] Figure 1 Depth estimation flowchart; Figure 2 Flowchart for the fusion module; Figure 3 Here is a flowchart of the Transformer encoder; Figure 4 Here is the flowchart for the adaptive Kalman filter; Figure 5 The Kalman filter trajectory effect diagram; Figure 6 This is a comparison chart of the fusion algorithm errors; Figure 7 This is a comparison chart of the effects of different systems. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] The purpose of this invention is to provide an intelligent collision avoidance and evidence recording marine monitoring floating system that integrates computer vision, target ranging, satellite positioning, and automatic identification system (AIS) multi-sensor fusion. To achieve this purpose, the invention employs YOLOv8 target detection, Transformer data fusion, and Kalman filtering algorithms to achieve accurate detection and collision avoidance early warning of ships and floating objects at sea. The system has an AIS failure handling mechanism and encrypted evidence storage function to ensure reliable operation and data compliance under complex sea conditions.

[0020] This invention is divided into a sensing layer, a processing layer, and an application layer; 1. Perception Layer (Data Acquisition) The perception layer is the system's data input layer, responsible for acquiring environmental information from various sensors: Binocular camera: responsible for acquiring RGB and infrared images for visual target recognition. The intrinsic parameters (focal length, optical center, distortion parameters, etc.) and extrinsic parameters (rotation matrix and translation vector of the left and right cameras, etc.) of the binocular camera have been calibrated. LiDAR: Acquires high-precision point cloud data, constructs a 3D model of the surrounding environment, and supplements the target features extracted by vision, such as the structural characteristics of ships or floating objects, such as ship length and height. Millimeter-wave radar: detects the velocity matrix of ships or floating objects within the detection area, providing motion status information; AIS receiver: Receives data from the Automatic Identification System of ships, including information such as MMSI, heading, and speed; IMU: Acquires buoy attitude data, used to obtain buoy attitude and position data, and provides continuous buoy coordinate data when there is no satellite signal; BeiDou system: provides time synchronization, buoy position information, blockchain hash value, and shore station communication; RTC Clock: Buoy System Clock Metronome, providing high-precision timekeeping; Processing Layer (Core Algorithm Processing): The processing layer is the core of the system, responsible for data preprocessing, feature extraction, multi-sensor fusion, collision risk assessment, and collision evidence chain data preparation. 3. Application Layer (Output and Storage) The application layer is responsible for system output and data storage: 3.1 Collision Avoidance Warning Collision avoidance warning module: Generates warning signals based on risk assessment results and controls the audible and visual alarm module and VHF broadcast module to notify the vessel to avoid obstacles; if the visual module identifies a floating object, the module is disabled. Audible and visual alarm system: Triggers the audible and visual alarm device; VHF broadcasting system: broadcasts early warning information via VHF radio; 3.2 Evidence Encryption and Storage Evidence Encryption Module: This processor-based encryption module generates encryption keys, encrypts the evidence data using the national standard SM4 algorithm, and calculates the hash value of the encrypted data using the SHA-256 module. The generated evidence data structure includes (timestamp, float ID, encrypted data, and data hash).

[0021] Local storage: Storing encrypted evidence locally and using blockchain hashing technology to ensure data integrity; Cloud storage: Encrypted evidence is transmitted to the cloud via the BeiDou / Iridium communication system, using the national cryptographic algorithm SM4 for encryption; 3.3 Legal Compliance Guarantee Legal Compliance Assurance (LEGAL): Local storage complies with the Maritime Investigation Evidence Rules. Cloud storage complies with the "Guidelines for Encryption Protection of Cloud Storage Data".

[0022] A multi-sensor fusion-based intelligent collision avoidance method includes the following steps: Step 1: Data Acquisition and Synchronization: Each sensor acquires data.

[0023] Step 2: Data Preprocessing 2.1 Data Preprocessing Stage: Image preprocessing: Distortion correction and stereo matching are performed on the images acquired by the binocular cameras.

[0024] Radar data preprocessing: The lidar point cloud data is filtered, and the attitude data calculated by the IMU is used to perform motion compensation on the lidar and millimeter-wave radar data to eliminate the measurement error caused by the buoy due to the waves.

[0025] Position calculation: The coordinates of the buoy are calculated based on quaternions, and the coordinates calculated by the IMU are corrected by Kalman filtering in combination with Beidou satellite coordinate data.

[0026] Time calibration: Multi-sensor time synchronization is achieved through RTC and the BeiDou system.

[0027] AIS Status Detection: Detects the status of the AIS signal to determine whether nearby vessels have activated AIS; When AIS is normal: AIS data is given high weight to correct radar and visual feature extraction; When AIS is malfunctioning: AIS data weight is reduced and radar and visual feature extraction weight is increased.

[0028] 2.2 Feature Extraction Stage: YOLOv8 Object Detection: Visual object detection based on deep learning, using the lightweight detection framework of YOLOv8s, integrating a ship-specific recognition model and a hull number OCR engine to extract ship hull number, feature values, and floating object information; the implementation steps are as follows: Implementation steps: 1. Ship target detection: The training dataset for ships uses the COCO format dataset, which includes ship outlines, chords, and floating objects; the activation function is the SiLU function, and the image resolution is set to 640*512.

[0029] Hull number recognition: The CRNN-CTC network architecture is adopted, and the process is as follows: output image - 7-layer convolution - BiLSTM - CTC decoding - hull number text.

[0030] PSMNet is used to generate synthetic data to simulate interference caused by fog / waves, and the SE module is used to enhance the weight of character regions.

[0031] Depth estimation: A fusion strategy is adopted, primarily using visual data and secondarily using radar data, to achieve high-precision ranging in all weather conditions. Figure 1 As shown.

[0032] Binocular disparity calculation: The principle is to calculate the target distance by the pixel offset between the left and right images; the raw data collected by the binocular cameras is processed by the Brown-Conrady model to eliminate lens distortion, and then the Bouguet algorithm outputs horizontally aligned images. The SGM algorithm is then used to match the left and right images to generate a disparity map. Finally, based on the camera calibration parameters, the disparity map is converted into depth values; the calculation formula is: ; Camera focal length (unit: mm); Binocular baseline distance (200mm for this system); : Parallax value (unit: pixels).

[0033] Radar data: Point cloud cluster data matrix after background filtering and motion compensation, optimized by merging neighboring points using the DBSCAN algorithm; depth calibration is performed on the disparity map generated by binocular vision calculation.

[0034] Fusion computing module: Radar depth data and visual depth data are corrected using formulas, and the corrected depth data is output. Figure 2 As shown; the fusion formula is: ; Laser reference depth; Original visual depth.

[0035] Step 3, Multi-sensor fusion stage: The core technology employs a two-way information flow closed loop formed between the Transformer and Kalman filter prediction modules to fuse data from vision, radar, and AIS. The Transformer performs deep fusion of multi-sensor features and outputs an enhanced observation vector. This is used for Kalman filtering. Kalman filtering feeds back the target's motion state (position, velocity, acceleration, and their uncertainties) predicted by the dynamic model to the Transformer, guiding it to adjust its internal attention distribution. For high-speed maneuvering targets, the model instructs the Transformer to focus more on the velocity information from the millimeter-wave radar; for targets with high motion uncertainty (covariance P), it instructs the model to reduce its reliance on historical features. The algorithm's processing cycle is set to 100ms.

[0036] Implementation steps: Spacetime synchronization: Time synchronization: RTC time is calibrated using BeiDou satellite time synchronization; Spatial alignment: Convert LiDAR point cloud and visual data into ENU coordinate system (East-North-Sky coordinate system).

[0037] Dynamic weight allocation: Weighting strategy: Based on AIS status detection values, the weight of AIS in the module is determined. If AIS fails, the fusion module uses YOLOv8 and radar-acquired features for Kalman filtering. The AIS health assessment model is as follows: ; in:( =0.4, =0.3, =0.3).

[0038] The weighting strategy is as follows: Table 1 shows the weight allocation strategy. .

[0039] The Transformer encoder's process of fusing different heterogeneous data into a unified feature representation can be divided into four stages: input features, feature embedding, Transformer encoding, and feature output. The flowchart is as follows: Figure 3 As shown.

[0040] The implementation steps are as follows: Feature input: Features from heterogeneous sensors such as vision, radar, and AIS are mapped to a unified semantic space. The feature engineering of the input is shown in Table 2. Table 2 Input Feature Table .

[0041] Feature embedding stage: Sensor features of different dimensions are linearly projected into a unified vector space. 256-dimensional visual features are linearly projected into 512-dimensional space; 128-dimensional LiDAR features are linearly projected into 512-dimensional space; 64-dimensional millimeter-wave radar features are linearly projected into 512-dimensional space; 32-dimensional AIS features are linearly projected into 512-dimensional space; all modal features are unified into 512-dimensional space.

[0042] Transformer encoder: Through a multi-head attention mechanism, all sensor features interact, complement, and enhance each other, learning the deep relationships between them. The processing steps are as follows: Feature concatenation: The four 512-dimensional feature vectors are concatenated along the sequence dimension to form a larger feature matrix. Assuming each sensor inputs one feature vector, the concatenated sequence will have a length of 4 and a dimension of 512.

[0043] Position Encoding: Since the Transformer itself lacks the ability to perceive sequence order, positional information must be injected. PosEnc [Position Encoding] generates a unique 512-dimensional position vector for each position (i.e., each sensor feature) in the concatenated sequence and adds it to the feature. This allows the model to understand order information such as "visual features first, AIS features last". The unified feature sequence with added positional information is then fed into an encoder consisting of G stacked Transformer blocks for depth processing. G can be set to 4.

[0044] Multi-head self-attention mechanism: Calculate the correlation weight of each feature in the sequence with all other features in the sequence (including itself).

[0045] The self-attention model is: ; ; in: is the input feature matrix, and W is the learnable weight matrix.

[0046] The multi-head self-attention model is as follows: ; ; Where h is the number of heads, and in this system h=8.

[0047] Feedforward Neural Network (FNN): Performs nonlinear transformation and enhancement on each feature after attention weighting.

[0048] The FNN model is: ; This system uses a two-layer network and includes the ReLU activation function.

[0049] Residual connection and layer normalization: The residual connection is calculated by adding the sub-layer input and sub-layer output. The mean and variance of the result after residual connection are calculated and layer normalization is performed.

[0050] Features are passed sequentially through TransformerBlock1, TransformerBlock2, ..., with each layer learning more complex and abstract cross-modal interaction patterns.

[0051] Feature output stage: Outputs a comprehensive target state vector that fuses features from multiple sensors, used for Kalman filtering prediction or update; Outputs the enhanced features from the four sensors, which are saved as high-level features; The processing steps are as follows: the last TransformerBlock outputs the fused feature sequence (4*512 dimensional vector), which is then transformed by a neural network to map the fused features to the target state estimate. (6-dimensional state vector, including position, velocity, and heading) and feature enhancement output; the feature enhancement output of vision is 256-dimensional, the feature enhancement output of lidar is 128-dimensional, the feature enhancement output of millimeter-wave radar is 64-dimensional, and the feature enhancement output of AIS is 32-dimensional.

[0052] Adaptive Kalman Filtering: Based on the fused features, the target motion state is predicted, a state-space model is established, and an observation model is established. The adaptive Kalman filtering process is as follows: Figure 4 As shown.

[0053] The implementation steps are as follows: Step 1: Initialization: 1) Set the initial state estimate The state vector contains all the state information I need to estimate for the target; in the ENU coordinate system, the 9-dimensional state vector is defined as: ; Location (x, y, z): The result of fusion of lidar ranging, visual bounding box depth estimation, and BeiDou / GPS data, where z is typically very small or 0 (sea level height); speed : Decomposition of radial velocity from millimeter-wave radar and fusion of AIS / SOG (ground speed) data. Typically close to 0; bow towards The main source of the COG (Course of Gauss) is AIS, but the orientation of visual and lidar point clouds can also be used for auxiliary estimation. speed : Primarily from AIS SOG or millimeter-wave radar speed measurement; Turning rate : It is usually initialized to 0, or estimated by the rate of change of the bow direction of consecutive frames.

[0054] The initial state estimate is: ; If an AIS message is received first, the system is initialized using the latitude and longitude (converted to ENU coordinates), SOG (Speed ​​Over Ground), and COG (Course Over Ground) provided by AIS. x,y,v,ψ The remaining velocity components and turning rate are initialized to 0.

[0055] 2) Set the initial error covariance matrix : This represents the variance of each component in the state estimate and the covariance between different components. This represents the variance of the initial state estimate. A larger diagonal element (variance) indicates that the initial estimate is highly uncertain.

[0056] .

[0057] The initial position has significant uncertainty (±50 meters); Same as above; The altitude uncertainty is relatively small (±5 meters); Velocity uncertainty (±10 m / s); Uncertainty regarding the heading (±0.5 rad, approximately ±28.6 degrees); Turning rate uncertainty (±0.1 rad / s).

[0058] Off-diagonal elements: These are typically set to 0 initially, assuming that the state components are initially uncorrelated. The filter will automatically estimate these correlations during operation.

[0059] 3) Set the initial values ​​for process noise and observation noise. , Preliminary settings are made based on sensor characteristics and the general motion behavior of the target.

[0060] It is a 9x9 matrix, usually designed as a diagonal matrix, with the initial values ​​set as follows: ; Standard deviation of maximum plane acceleration; Assuming the ship's maximum acceleration is approximately ; Vertical acceleration standard deviation (much smaller than that of the plane, because ships move slowly in the vertical direction); Standard deviation of turning angle acceleration; Standard deviation of heading acceleration; Standard deviation of the rate of change of turning rate; Filtering period.

[0061] It is a 6x6 diagonal matrix, with initial values ​​determined by the accuracy of each sensor. The R matrix represents the uncertainty of the observations, i.e., sensor error, which comes from the Transformer module and characterizes the confidence level of the fusion result.

[0062] .

[0063] Step 2: Standard Kalman filter prediction: State prediction: ; in, It is the state transition matrix: ; The sampling time of the filter (e.g., 0.1 seconds, 0.5 seconds, or 1 second, depending on the sensor data update rate).

[0064] Lines 1-3: Position = Original Position + Velocity × Time. For example: .

[0065] Lines 4-6: Assume that the speed is constant over a short period of time. .

[0066] Line 7: Bow direction = Original bow direction + Turning rate × Time. .

[0067] Line 8: Assume constant speed. .

[0068] Line 9: Assume the turning rate is constant. .

[0069] It is a control input. This is the control matrix, primarily used to control systems with known external inputs, such as controlling one's own vessel. In this system, tracking the vessel or floating object is required, and since no external control input is typically assumed, the control term is set to 0.

[0070] ; Error covariance prediction: ; The use here It is the process noise covariance matrix after adaptive adjustment at the previous time step.

[0071] Step 3: Standard Kalman Filter Update Step: Calculate the Kalman gain: ; The use here It is the observation noise covariance matrix after adaptive adjustment at the previous time step. It is a 6x9 matrix used to map a 9-dimensional state space to a 6-dimensional observation space.

[0072] .

[0073] Status Update: ; in, These are enhanced observations from the Transformer module.

[0074] ; Transformer integrates the pixel coordinates of binocular vision, the point cloud distance of LiDAR, and the BeiDou self-position to calculate the optimal estimated position of the target in the ENU coordinate system.

[0075] The Transformer integrates the radial velocity of the millimeter-wave radar (in the sensor coordinate system) and transforms it to the ENU coordinate system. It may also incorporate the velocity components decomposed from SOG and COG of AIS.

[0076] This is a key function of the Transformer. It can combine the COG from AIS, the ship's heading angle identified in the visual image, and the major axis direction of the LiDAR point cloud to provide a more reliable estimate of the ship's heading. For example, when the AIS signal is lost, the Transformer can rely entirely on the visual and LiDAR outputs for this value.

[0077] Similarly, SOG integrates millimeter-wave radar speed measurement and AIS.

[0078] Error covariance update: .

[0079] Step 4: Adaptive Adjustment Calculate the innovation sequence: The innovation sequence is the difference between the predicted and actual observed values. ; New sequence It is a function with zero mean and covariance of A white noise sequence.

[0080] Estimating the actual covariance of the innovation: Calculate the actual covariance of the innovation sequence using a sliding window of length N. ; in .

[0081] Adjust the noise covariance matrix: Adjusting observation noise If the actual covariance of the new information Greater than its theoretical value , This usually indicates that we Over-reliance on observations Therefore, it needs to be increased. , produce a : ; Need to ensure It must be a positive definite matrix; otherwise, special handling is required.

[0082] Adjustment process noise : If the new sequence shows correlation (non-white noise), it indicates that the process model is inaccurate and needs to be increased. To increase the weighting of the most recent observations. A common approach is: ; This is equivalent to attributing the model's uncertainty to process noise.

[0083] Apply the adjusted noise matrix: The newly estimated and (or the smoothed value) is used for prediction and update at the next time step (k+1).

[0084] ; ; in , It is a smoothing factor used to prevent adjustments from being too drastic.

[0085] Step 5: Iteration Loop Set the time index k=k+1 and jump back to step two to continue the next round of prediction, update and adaptive adjustment.

[0086] The mathematical model for the collaborative interface is: Forward fusion (Transformer→KF): ; in It is the fused feature output by the Transformer. It is a projection matrix.

[0087] Feedback optimization (KF→Transformer): Kalman filtering provides motion uncertainty The temperature coefficient used to adjust the attention calculation in the next frame of the Transformer. : ； in The greater the uncertainty, The larger the value, the more evenly attention is distributed, reducing reliance on unreliable historical information.

[0088] Step 4: Risk Assessment and Evidence Preprocessing Collision risk assessment: Collision risk is calculated based on the Time of Closest Approach (TCPA) and Distance of Closest Approach (DCPA) models of vessels or floating objects.

[0089] The calculation steps are as follows: Step 1): Define the state vector: Accurate state estimates of the target (T) and buoy (O) were obtained from the Kalman filter.

[0090] Buoy (O) status: ; The position of this ship in the Northeastern Sky (ENU) coordinate system; The components of the ship's velocity vector in the east and north directions; Own ship speed ; This ship's course (The angle of counterclockwise rotation from the east).

[0091] Target (T) state: ; The location of the target ship; : The velocity vector of the target ship; target ship speed ; target ship's course .

[0092] Step 2): Calculate the relative motion vector: Relative position vector: ; Relative velocity vector: ; relative speed .

[0093] Step 3): Calculation and : (Nearest distance): Mathematically, it is equal to the relative position vector. With relative velocity vector The absolute value of the cross product is divided by the magnitude of the relative velocity. Geometrically, it represents the distance from the origin to the line of relative motion.

[0094] .

[0095] (Recently expected time): TCPA is mathematically equal to the relative position vector. With relative velocity vector The dot product of the relative velocity divided by the square of the relative velocity. Its geometric meaning is the time required for the relative motion to reach the nearest point, which can be negative (indicating that the nearest point has been passed).

[0096] ; The meaning of plus and minus signs: The two ships are approaching, but have not yet reached their closest point; It is exactly at the closest point;

[0097] The two ships have passed their closest point and are now moving away from each other.

[0098] After obtaining the DCPA and TCPA, a risk assessment function needs to be established to quantify the collision hazard level. A two-dimensional risk matrix is ​​typically used to define the risk level, as shown in Table 3.

[0099] Table 3 Risk Levels .

[0100] from Figure 5 , Figure 6 and Figure 7 As can be seen, this application integrates multi-sensor fusion data and adopts YOLOv8 target detection, Transformer data fusion and Kalman filtering algorithm to achieve accurate detection and collision avoidance warning of ships and floating objects at sea, and its accuracy has been greatly improved.

Claims

1. A multi-sensor fusion intelligent collision avoidance method, characterized in that, Includes the following steps: S1. Obtain environmental information and preprocess the data; S2. Feature Extraction: Using a lightweight detection framework, ship feature values ​​are extracted; S3. Multi-sensor data fusion: Employs a two-way information flow closed loop formed between the Transformer and Kalman filter prediction modules to fuse data from vision, radar, and AIS; S4. Conduct a risk assessment.

2. The intelligent collision avoidance method based on multi-sensor fusion according to claim 1, characterized in that, Data preprocessing includes: Image preprocessing: distortion correction and stereo matching processing of images acquired by the binocular cameras; Radar data preprocessing: Filter the lidar point cloud data and use the attitude data calculated by the IMU to perform motion compensation on the lidar and millimeter-wave radar data; Position calculation: The coordinates of the buoy are calculated based on quaternions, and the coordinates calculated by the IMU are corrected by Kalman filtering in combination with Beidou satellite coordinate data; Time calibration: Multi-sensor time synchronization is achieved through RTC and the BeiDou system; AIS Status Detection: Detects the status of the AIS signal to determine whether nearby vessels have activated AIS; When AIS is normal: AIS data is given high weight to correct radar and visual feature extraction; When AIS is malfunctioning: AIS data weight is reduced and radar and visual feature extraction weight is increased.

3. The intelligent collision avoidance method based on multi-sensor fusion according to claim 1, characterized in that, The feature extraction steps are as follows: S21. The feature dataset includes ship outline chords and floating objects; the activation function used is the SiLU function. The hull number recognition adopts a CRNN-CTC network architecture, uses PSMNet to generate synthetic data to simulate interference from the external environment, and uses the SE module to enhance the weight of the character region. S22. A fusion strategy is adopted, primarily using visual data and secondarily using radar data. The steps are as follows: Binocular disparity calculation: The target distance is calculated by the pixel offset between the left and right images. The raw data acquired by the binocular cameras is processed by the Brown-Conrady model to eliminate lens distortion, and then the Bouguet algorithm outputs horizontally aligned images. The left and right images are then matched by the SGM algorithm to generate a disparity map. Finally, based on the camera calibration parameters, the disparity map is converted into depth values. The calculation formula is: ； Camera focal length; Binocular baseline distance; disparity value; Radar data: Point cloud cluster data matrix after background filtering and motion compensation, optimized by merging neighboring points using the DBSCAN algorithm; Depth calibration is performed on the disparity map generated by binocular vision computation; Fusion computing module: Radar depth data and visual depth data are corrected using formulas, and the corrected depth data is output. The fusion formula is: ； Laser reference depth; Original visual depth; Radar depth.

4. The intelligent collision avoidance method based on multi-sensor fusion according to claim 1, characterized in that, Based on the AIS status detection values, the weights of the AIS are determined. If the AIS fails, a lightweight detection framework and the features acquired by the radar are used for Kalman filtering. The AIS health assessment model is as follows: ； These are the weighting coefficients. For received signal strength, For actual reporting frequency, For the maximum allowed reporting frequency, To evaluate log consistency, assess the logic and rationality of the data reported by AIS devices.

5. The intelligent collision avoidance method based on multi-sensor fusion according to claim 1, characterized in that, The Transformer encoder fuses different heterogeneous data into a unified feature representation: Feature input: Mapping features from heterogeneous sensors such as vision, radar, and AIS to a unified semantic space; Feature embedding stage: Sensor features of different dimensions are linearly projected into a unified vector space, and all modal features are unified to 512 dimensions; Through a multi-head attention mechanism, all sensor features interact, complement, and enhance each other, learning the deep relationships between them. The processing steps are as follows: Feature concatenation: The four 512-dimensional feature vectors are concatenated along the sequence dimension to form a larger feature matrix; Location encoding: For each location in the concatenated sequence, i.e. each sensor feature, a unique 512-dimensional location vector is generated and added to the feature. The unified feature sequence with added location information is then fed into an encoder composed of multiple stacked Transformer blocks for depth processing. Multi-head self-attention mechanism: Calculates the correlation weight between each feature in the sequence and all other features in the sequence; Feedforward Neural Network (FNN): Performs nonlinear transformations and enhancements on each attention-weighted feature. Residual connection and layer normalization: The residual connection is calculated by adding the sub-layer input and sub-layer output. The mean and variance of the result after residual connection are calculated and layer normalization is performed.

6. The intelligent collision avoidance method based on multi-sensor fusion according to claim 1, characterized in that, The steps for predicting the target's motion state based on the fused features are as follows: Step 1: Initialize state estimates : ； Position (x, y, z), velocity bow towards speed Turning rate ; Step 2: Standard Kalman filter prediction: State prediction: ； It is the state transition matrix. It is a control input. It is a control matrix; Error covariance predict: ； It is the adaptively adjusted process noise covariance matrix from the previous time step; Step 3: Standard Kalman filter update: Calculate the Kalman gain: ； It is the adaptively adjusted observation noise covariance matrix from the previous time step. The observation matrix is ​​used to map the 9-dimensional state space to the 6-dimensional observation space. Status Update: ； in, These are enhanced observations from the Transformer module; Step 4: Adaptive Adjustment ； New sequence It is a function with zero mean and covariance of A white noise sequence; estimating the actual covariance of the innovation: calculating the actual covariance of the innovation sequence through a sliding window of length N: ； in ; Adjust the noise covariance matrix: Adjusting observation noise If the actual covariance of the new information Greater than its theoretical value It needs to be increased , to generate a new ; ； Step 5: Iterate through the loop. ； in It is the fused feature of the Transformer output. Let k be the feature vector at time k; It is a projection matrix. Let be the original observation vector of the sensor at time k; Feedback optimization: Kalman filtering provides The temperature coefficient used to adjust the attention calculation in the next frame of the Transformer. : ； ； The larger the value, the more evenly attention is distributed, reducing reliance on unreliable historical information.

7. The intelligent collision avoidance method based on multi-sensor fusion according to claim 6, characterized in that, If the innovation sequence shows correlation, it indicates that the adaptively adjusted process noise covariance matrix is ​​inaccurate and needs adjustment. To adjust the latest observation matrix Weights: ； Apply the adjusted noise matrix: The newly estimated and The smoothed value is used for prediction and update at the next time step (k+1): ； ； in , It is a smoothing factor.

8. The intelligent collision avoidance method based on multi-sensor fusion according to claim 1, characterized in that, Collision risk calculation based on the Time of Closest Approach (TCPA) and Distance of Closest Approach (DCPA) models for ships or floating objects: The formula for calculating the nearest proximity distance (DCPA) is as follows: ； ； Relative position vector, for x The relative position vector in the direction, yes y The relative position vector in the direction, Relative velocity vector for x The relative velocity vector in the direction, for y The relative velocity vector in the direction; The closest time TCPA: ； The two ships are approaching, but have not yet reached their closest point; It is exactly at the closest point; The two ships have passed their closest point and are now moving away from each other.

9. The intelligent collision avoidance method based on multi-sensor fusion according to claim 8, characterized in that, The steps for calculating the relative motion vector are as follows: From the Kalman filter, accurate state estimates of the target (T) and buoy (O) were obtained: Buoy (O) status: ； ( ): The position of this ship in the Northeast ENU coordinate system; ( ): The components of the ship's velocity vector in the east and north directions; Vessel speed: ; This vessel's course: ; Target (T) state: ； ( ): The location of the target ship; ( : The velocity vector of the target ship; Target ship speed: ; Target ship's course: ; Calculate the relative motion vector: Relative position vector: ; Relative velocity vector: .

10. A multi-sensor fusion intelligent collision avoidance system, employing the multi-sensor fusion intelligent collision avoidance method as described in any one of claims 1-9, characterized in that, It includes the perception layer, processing layer, and application layer; Perception layer: This is the system's data input layer, responsible for collecting environmental information from various sensors; Processing layer: responsible for data preprocessing, feature extraction, multi-sensor fusion, collision risk assessment, and collision evidence chain data preparation; Application layer: Responsible for system output and data storage.

Images