Intelligent anti-collision method and system based on multi-sensor fusion

Abstract

The application belongs to the technical field of marine detection and is a multi-sensor fusion intelligent anti-collision method and system, which comprises a perception layer, a processing layer and an application layer, and provides a multi-sensor fusion intelligent anti-collision and evidence recording marine monitoring floating system integrating computer vision, target ranging, satellite positioning and ship automatic identification system, for the purpose, the application adopts YOLOv8 target detection, Transformer data fusion and Kalman filtering algorithm to realize accurate detection and anti-collision warning of marine ships and floating objects. The system has an AIS failure processing mechanism and an evidence encryption storage function, ensuring reliable operation and data compliance in complex sea conditions.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of marine detection and is a kind of intelligent anti-collision method and system based on multi-sensor fusion. BACKGROUND

[0002] Marine detection buoys are widely used in marine environment monitoring, weather observation, channel safety and other fields. However, the current marine buoy system has significant defects in ship collision prevention and accident evidence recording: single sensor dependence: traditional buoys mainly rely on AIS (Automatic Identification System) to monitor ship dynamics, but AIS is susceptible to signal interference, intentional shutdown or tampering, resulting in the inability of the buoy to obtain the key information of the offending ship, such as the MMSI number, position, heading, speed, etc.; incomplete evidence chain: after a collision accident occurs, the existing buoy lacks the ability to record multi-source data fusion, and cannot provide a complete evidence chain with legal effect (such as ship attitude, position, image, radar point cloud, etc.). SUMMARY

[0003] An intelligent buoy method and system based on multi-sensor fusion solves the above problems through high-precision target detection, multi-source data fusion and collision evidence chain generation. The technical solution is as follows:

[0004] An intelligent anti-collision method based on multi-sensor fusion, comprising the following steps:

[0005] S1. Obtain environmental information and pre-process the data;

[0006] S2. Feature extraction: use a lightweight detection framework to extract ship feature values;

[0007] S3. Multi-sensor data fusion: use the bidirectional information flow closed loop formed between the Transformer and the Kalman filter prediction module to fuse data from vision, radar and AIS;

[0008] S4. Risk assessment.

[0009] Preferably, the data preprocessing includes: image preprocessing: performing distortion correction and stereo matching processing on the images collected by the binocular camera;

[0010] Radar data preprocessing: filtering the laser radar point cloud data, using the attitude data calculated by the IMU to perform motion compensation on the laser radar and millimeter wave radar data, and eliminating the measurement error caused by the buoy due to sea waves;

[0011] Position calculation: based on the quaternion, the coordinates of the buoy are calculated, and the coordinates calculated by the IMU are corrected using Kalman filter combined with Beidou satellite coordinate data;

[0012] Time calibration: realize multi-sensor time synchronization through RTC and Beidou system;

[0013] AIS state detection: detect AIS signal state, judge whether the nearby ship opens AIS; when AIS is normal: give AIS data high weight, used to correct radar and visual feature extraction; when AIS fails: reduce the weight of AIS data, increase the weight of radar and visual feature extraction.

[0014] Preferably, the feature extraction step is as follows:

[0015] S21. The feature data set contains the ship profile chord number and the float; the activation function uses the SiLU function;

[0016] The hull number recognition uses the CRNN-CTC network architecture, uses the PSMNet to generate synthetic data to simulate the interference caused by the external environment (fog / wave), and uses the SE module to enhance the character area weight;

[0017] S22. The fusion strategy is mainly based on number vision and supplemented by radar data, and the steps are as follows:

[0018] Binocular disparity calculation: calculate the target distance through the pixel offset of left and right images; the original data collected by the binocular camera is processed through the Brown-Conrady model to eliminate lens distortion, and then the left and right horizontal alignment views are output by the Bouguet algorithm, and then the left and right image matching is performed through the SGM algorithm to generate a disparity map, and then the disparity map is converted into a depth value according to the camera calibration parameters , the calculation formula is:

[0019] ;

[0020] : camera focal length; : binocular baseline distance; : disparity value;

[0021] Radar data: point cloud cluster data matrix after background filtering and motion compensation, using DBSCAN algorithm to merge adjacent points for optimization; depth calibration is performed on the disparity map generated by binocular vision calculation;

[0022] Fusion calculation module: radar depth data and visual depth are corrected by formula, and the corrected output depth data ; the fusion formula is:

[0023] ;

[0024] : laser reference depth; : visual original depth; : radar depth.

[0025] Preferably, according to the AIS state detection value, the weight of the AIS is determined, if the AIS fails, the light detection framework and the features obtained by the radar are used for Kalman filtering algorithm; the AIS health degree evaluation model is:

[0026] ;

[0027] 、 、 is a weight coefficient, is a received signal strength, is an actual reporting frequency, is a maximum allowed reporting frequency (can be set), is a log consistency evaluation (empirical value), which evaluates the logic and rationality of AIS equipment reporting data.

[0028] Preferably, the Transformer encoder fuses different heterogeneous data into a unified feature representation:

[0029] Feature input: map visual, radar and AIS heterogeneous sensor features to a unified semantic space;

[0030] Feature embedding stage: linearly project sensor features of different dimensions to a unified vector space, and unify all modal features to 512 dimensions;

[0031] Through the multi-head attention mechanism, all sensor features interact, supplement and enhance each other, learn the deep correlation between them, and the processing steps are:

[0032] Feature splicing: splice the four 512-dimensional feature vectors in the sequence dimension to form a larger feature matrix;

[0033] Position encoding: a unique 512-dimensional position vector is generated for each position in the spliced sequence, i.e. each sensor feature, and added to the feature. The unified feature sequence with position information is sent to an encoder stacked by multiple Transformer blocks for deep processing;

[0034] Multi-head self-attention mechanism: calculate the correlation weight of each feature in the sequence with all other features in the sequence;

[0035] Self-attention model is:

[0036] ;

[0037] Query, Key, Value, Key vector dimension;

[0038] The multi-head self-attention model is:

[0039]

[0040]

[0041] where h is the number of heads, is the feature transformation matrix of the i-th head, is the multi-head feature output transformation matrix;

[0042] The feedforward neural network FNN: each feature after attention weighting is nonlinearly transformed and enhanced:

[0043]

[0044] , is a learnable weight matrix, is an input feature matrix, , are the biases of the first and second layers;

[0045] Residual connection and layer normalization: the residual connection is calculated by adding the sublayer input and the sublayer output, and the mean and variance of the result after the residual connection are calculated for layer normalization.

[0046] Preferably, the target motion state is predicted based on the fused features, and the steps are as follows:

[0047] Step one, initialize state estimation value

[0048]

[0049] Position (x, y, z), velocity , heading , speed , rudder rate

[0050] Step two, standard Kalman filter prediction:

[0051] State prediction:

[0052]

[0053] is the state transition matrix, is the control input, is the control matrix;

[0054] Error covariance prediction:​​​​​​​

[0055] ;

[0056] is the process noise covariance matrix adjusted adaptively at the last time step;

[0057] Step three, standard Kalman filter update:

[0058] Compute the Kalman gain:

[0059] ;

[0060] is the observation noise covariance matrix adjusted adaptively at the last time step, is the observation matrix that maps the 9-dimensional state space to the 6-dimensional observation space;

[0061] State update:

[0062] ;

[0063] where, is the enhanced observation value from the Transformer;

[0064] Step four, adaptive adjustment:

[0065] ;

[0066] Innovation sequence is a white noise sequence with mean zero and covariance ;

[0067] Estimate the actual covariance of the innovation: Calculate the actual covariance of the innovation sequence through a sliding window of length N:

[0068] ;

[0069] where ;

[0070] Adjust the noise covariance matrix:

[0071] Adjust the observation noise : If the actual covariance of the innovation is greater than its theoretical value ， , increase , and generate a new ;

[0072] ;

[0073] Fifth step, loop iteration:

[0074] Forward fusion: ;

[0075] where is the fused feature output by the Transformer, is the feature vector at time k; is the projection matrix, is the raw observation vector of the sensor at time k;

[0076] Feedback optimization: the innovation provided by the Kalman filter is used to adjust the temperature coefficient in the attention computation of the next frame by the Transformer :

[0077] ；

[0078] ；

[0079] The larger, the more attention distribution is flat, and the dependence on unreliable historical information is reduced.

[0080] Preferably, if the innovation sequence shows correlation, it indicates that the adaptive adjusted process noise covariance matrix is inaccurate, and the needs to be adjusted to adjust the weight of the latest observation matrix :

[0081] ;

[0082] Apply the adjusted noise matrix: use the newly estimated and the smoothed value of for prediction and update at the next time step (k+1):

[0083] ;

[0084] ;

[0085] where , is the smoothing factor.

[0086] Preferably, the collision risk calculation is based on the closest point of approach TCPA and the closest point of approach DCPA model of the ship or floating object:

[0087] The closest point of approach DCPA calculation formula is as follows:

[0088] ; ​

[0089] ;

[0090] relative position vector in the direction of the relative velocity vector, x relative position vector in the direction of the relative velocity vector, y relative position vector in the direction of the relative velocity vector; relative velocity vector, x relative velocity vector in the direction of the relative position vector, y relative velocity vector in the direction of the relative position vector;

[0091] time to closest point of approach (TCPA):

[0092] ;

[0093] : two vessels are approaching, not yet at closest point;

[0094] : exactly at closest point;

[0095] : two vessels have passed closest point, moving away.

[0096] Preferably, the step of calculating the relative motion vector is as follows:

[0097] From the Kalman filter, the accurate state estimates of the target (T) and the buoy (O) are obtained:

[0098] Buoy (O) state:

[0099] ;

[0100] ( ): position of the own ship in the East-North-Up (ENU) coordinate system;

[0101] ( ): components of the own ship's velocity vector in the East and North directions;

[0102] own ship's speed: ;

[0103] own ship's heading: ;

[0104] Target (T) state:

[0105] ;

[0106] ( ): position of the target ship; ​​​​

[0107] ): target ship's velocity vector;

[0108] Target ship speed:

[0109] Target ship heading:

[0110] Calculate relative motion vector:

[0111] Relative position vector:

[0112]

[0113] Relative velocity vector:

[0114]

[0115] A multi-sensor fusion intelligent anti-collision system, comprising a perception layer, a processing layer and an application layer;

[0116] The perception layer is the data input layer of the system, responsible for collecting environmental information from various sensors;

[0117] The processing layer is responsible for data preprocessing, feature extraction, multi-sensor fusion, collision risk assessment and collision evidence chain data preparation;

[0118] The application layer is responsible for system output and data storage.

[0119] Compared with the prior art, the application has the following beneficial effects:

[0120] Multi-modal sensor fusion: efficient fusion of heterogeneous data through YOLOv8+Transformer and Kalman filtering.

[0121] Dynamic target tracking: target motion modeling based on laser radar point cloud and visual data, supporting trajectory prediction in complex scenes.

[0122] Collision evidence chain generation: synchronous storage of video, radar point cloud, AIS data and timestamp, forming an unalterable evidence package. BRIEF DESCRIPTION OF DRAWINGS

[0123] Figure 1 Depth estimation flowchart;

[0124] Figure 2 Fusion module flowchart;

[0125] Figure 3 Transformer encoder flowchart;

[0126] Figure 4 ​​​​​Adaptive Kalman filter flowchart;

[0127] Figure 5 Kalman filter trajectory effect diagram;

[0128] Figure 6 Fusion algorithm error comparison diagram;

[0129] Figure 7 Different system effect comparison diagram. DETAILED DESCRIPTION

[0130] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0131] The purpose of the present application is to provide an intelligent anti-collision and evidence recording marine monitoring floating system integrating computer vision, target ranging, satellite positioning, and ship automatic identification system multi-sensor fusion. To achieve the purpose, the present application adopts YOLOv8 target detection, Transformer data fusion, and Kalman filtering algorithm to realize accurate detection and anti-collision warning of marine ships and floating objects. The system has an AIS failure processing mechanism and evidence encryption storage function, ensuring reliable operation and data compliance in complex sea conditions.

[0132] The present application is divided into a perception layer, a processing layer, and an application layer;

[0133] 1. Perception layer (data acquisition) The perception layer is the data input layer of the system, responsible for collecting environmental information from various sensors:

[0134] Binocular camera: responsible for collecting RGB and infrared images for visual target recognition. The intrinsic parameters (focal length, optical center, distortion parameters, etc.) and extrinsic parameters (rotation matrix and translation vector of left and right cameras, etc.) of the binocular camera have been calibrated;

[0135] Laser radar: obtains high-precision point cloud data to construct a three-dimensional model of the surrounding environment, supplementing the target features extracted by vision, such as the structure characteristics of ships or floating objects such as ship length and ship width;

[0136] Millimeter wave radar: detects the speed matrix of ships or floating objects in the detection area, providing motion state information;

[0137] AIS receiver: receives ship automatic identification system data, including MMSI, heading, speed, etc.

[0138] IMU: Obtain the attitude data of the buoy, which is used to obtain the attitude and position data of the buoy, and provide continuous buoy coordinate data when there is no satellite signal;

[0139] Beidou system: Provide time synchronization, buoy position information, blockchain hash value, and shore station communication;

[0140] RTC clock: Buoy system clock tick, providing high-precision time counting;

[0141] Processing layer (core algorithm processing): The processing layer is the core of the system, responsible for data preprocessing, feature extraction, multi-sensor fusion, collision risk assessment, and collision evidence chain data preparation:

[0142] 3. Application layer (output and storage): The application layer is responsible for system output and data storage:

[0143] 3.1 Anti-collision warning

[0144] Anti-collision warning module: Generate warning signals based on risk assessment results, and control the sound and light alarm module. The VHF broadcast module notifies the ship to avoid obstacles. If the visual module identifies it as a floating object, the module is disabled;

[0145] Sound and light alarm system: Trigger the sound and light alarm device;

[0146] VHF broadcast system: Broadcast warning information through VHF radio;

[0147] 3.2 Evidence encryption and storage

[0148] Evidence encryption module: Based on the processor's encryption module, generate an encryption key, encrypt the evidence data using the national standard SM4 algorithm, and calculate the hash value of the encrypted data using the SHA-256 module. The generated evidence data structure includes (timestamp, buoy ID, encrypted data, data hash).

[0149] Local storage: Store encrypted evidence locally and use blockchain hash technology to ensure data integrity;

[0150] Cloud storage: Transmit encrypted evidence to the cloud through Beidou / Iridium communication systems and use the national SM4 algorithm for encryption;

[0151] 3.3 Legal compliance guarantee

[0152] Legal compliance guarantee (LEGAL): Local storage complies with the "Maritime Investigation Evidence Rules"

[0153] Cloud storage complies with the "Cloud Storage Data Encryption Protection Guidelines".

[0154] A multi-sensor fusion intelligent anti-collision method, comprising the following steps:

[0155] Step 1: Data collection and synchronization: Each sensor collects data.

[0156] Step 2: Data preprocessing:

[0157] 2.1 Data preprocessing stage:

[0158] Image preprocessing: Distortion correction and stereo matching processing are performed on the images collected by the binocular camera.

[0159] Radar data preprocessing: Filtering processing is performed on the laser radar point cloud data, and the attitude data calculated by the IMU is used to compensate the motion of the laser radar and millimeter wave radar data, eliminating the measurement error caused by the sea wave.

[0160] Position calculation: The coordinates of the buoy are calculated based on the quaternion, and the coordinates calculated by the IMU are corrected using Kalman filtering combined with the Beidou satellite coordinate data.

[0161] Time calibration: Multi-sensor time synchronization is realized through RTC and Beidou system.

[0162] AIS state detection: Detect the AIS signal state to determine whether the nearby ship has turned on the AIS; when the AIS is normal: give high weight to the AIS data, which is used to correct the radar and visual feature extraction; when the AIS is invalid: reduce the weight of the AIS data, and increase the weight of the radar and visual feature extraction.

[0163] 2.2 Feature extraction stage:

[0164] YOLOv8 target detection: Visual target detection based on deep learning, using the lightweight detection framework of YOLOv8s, integrating ship-specific recognition models and string number OCR engine, extracting ship side number, feature value and floating object information; the implementation steps are:

[0165] Implementation steps:

[0166] 1. Ship target detection: The training data set of the ship uses the COCO format data set, which contains the ship contour string number and floating object; the activation function uses SiLU function, and the resolution of the picture is set to 640*512.

[0167] Side number recognition: CRNN-CTC network architecture is used, and the process is: output image-7 layer convolution-BiLSTM-CTC decoding-side number text.

[0168] PSMNet is used to generate synthetic data to simulate the interference caused by fog / wave, and SE module is used to enhance the weight of character area.

[0169] Depth estimation: the fusion strategy of number vision as the main and radar data as the auxiliary is adopted to realize all-weather high-precision ranging, Figure 1 as shown.

[0170] Binocular parallax calculation: the principle is to calculate the target distance through the pixel offset of left and right images; the original data collected by the binocular camera is subjected to Brown-Conrady model to eliminate lens distortion, and then the left and right horizontal alignment views are output by Bouguet algorithm, and then the left and right image matching is performed by SGM algorithm to generate a parallax map, and then the parallax map is converted into a depth value according to the calibration parameters of the camera; the calculation formula is:

[0171] ;

[0172] : camera focal length (unit: mm); : binocular baseline distance (200mm is adopted in the system); : parallax value (unit: pixel).

[0173] Radar data: the point cloud cluster data matrix subjected to background filtering and motion compensation is adopted to optimize the adjacent points by DBSCAN algorithm; the parallax map generated by binocular vision calculation is subjected to depth calibration.

[0174] Fusion calculation module: the radar depth data and the visual depth are corrected by a formula, and the corrected depth data is output, Figure 2 as shown; the fusion formula is:

[0175] ;

[0176] : laser reference depth; : visual original depth.

[0177] Step 3, multi-sensor fusion stage:

[0178] The core is to form a bidirectional information flow closed loop between the Transformer and the Kalman filter prediction module, and fuse the data from vision, radar and AIS; the Transformer performs deep fusion on the multi-sensor features to output an enhanced observation vector for the Kalman filter. The Kalman filter predicts the target motion state (position, velocity, acceleration and its uncertainty) based on the dynamics model, feeds back to the Transformer, and guides it to adjust the internal attention distribution. For high-speed maneuvering targets, the model will guide the Transformer to pay more attention to the speed information of the millimeter wave radar; for targets with large motion uncertainty (covariance P), the model will guide it to reduce the dependence on historical features. The processing period of the algorithm is set to 100ms.

[0179] Implementation steps:

[0180] Space-time synchronization:

[0181] Time synchronization: use Beidou satellite time to calibrate RTC time;

[0182] Spatial alignment: convert lidar point cloud and vision data into ENU coordinate system (East-North-Sky coordinate system).

[0183] Dynamic weight distribution:

[0184] Weight distribution strategy: according to the AIS state detection value, the weight of AIS in the module is decided, if AIS fails, the fusion module adopts YOLOv8 and the features obtained by radar to perform Kalman filtering algorithm; the AIS health assessment model is:

[0185] ;

[0186] Wherein: =0.4, =0.3, =0.3).

[0187] The weight distribution strategy is:

[0188] Table 1 is the weight distribution strategy

[0189] .

[0190] The completion process of the Transformer encoder to fuse different heterogeneous data into a unified feature representation can be divided into four stages: input features, feature embedding, Transformer encoding, and feature output. The flow chart is shown in Figure 3 .

[0191] The implementation steps are:

[0192] Feature input: map the heterogeneous sensor features of vision, radar, and AIS to a unified semantic space. The input feature engineering is shown in Table 2:

[0193] Table 2 Input feature table

[0194] .

[0195] Feature embedding stage: linear projection of different dimensional sensor features to a unified vector space. 256-dimensional vision features are linearly projected to 512-dimensional; 128-dimensional lidar features are linearly projected to 512-dimensional; 64-dimensional millimeter wave radar is linearly projected to 512-dimensional; 32-dimensional AIS is linearly projected to 512-dimensional; all modal features are unified to 512-dimensional.

[0196] Transformer encoder: Through multi-head attention mechanism, all sensor features interact, complement and enhance each other, learn the deep correlation between them, and the processing steps are:

[0197] Feature concatenation: The four 512-dimensional feature vectors are concatenated in the sequence dimension to form a larger feature matrix. Assuming that each sensor inputs a feature vector, the length of the concatenated sequence is 4 and the dimension is 512.

[0198] Position encoding: Since the Transformer itself does not have the ability to perceive the sequence order, position information must be injected. PosEnc[Position Encoding] generates a unique 512-dimensional position vector for each position (i.e., each sensor feature) in the concatenated sequence and adds it to the feature. This allows the model to understand the order information such as "visual features are in the first position and AIS features are in the last position". The unified feature sequence with position information is sent to an encoder composed of G stacked Transformer blocks for deep processing. G can be set to 4.

[0199] Multi-head self-attention mechanism: Calculate the correlation weight of each feature in the sequence with all other features (including itself) in the sequence.

[0200] The self-attention model is:

[0201] ;

[0202] ;

[0203] Where: is the input feature matrix, and W is a learnable weight matrix.

[0204] The multi-head self-attention model is:

[0205] ;

[0206] ;

[0207] Where h is the number of heads, and in this system h = 8.

[0208] Feedforward neural network (FNN): Non-linear transformation and enhancement of each feature after attention weighting.

[0209] The FNN model is: ;

[0210] This system uses a 2-layer network with a ReLU activation function.

[0211] Residual connection and layer normalization: the residual connection is calculated by adding the sub-layer input and the sub-layer output, and the mean and variance of the result after the residual connection are calculated for layer normalization.

[0212] The features will pass through TransformerBlock1, TransformerBlock2,..., and each layer will learn more complex and abstract cross-modal interaction patterns.

[0213] Feature output stage: output a comprehensive target state vector that integrates multi-sensor features, which is used for prediction or update in Kalman filtering; output four sensor-enhanced features as advanced features for storage;

[0214] Processing steps: the fused feature sequence (4*512-dimensional vector) output in the last TransformerBlock is mapped to the target state estimate (6-dimensional state vector containing position, velocity, and heading) and feature enhancement output through linear transformation of the neural network; the visual feature enhancement output is 256-dimensional, the laser radar feature enhancement output is 128-dimensional, the millimeter wave radar feature enhancement output is 64-dimensional, and the AIS feature enhancement output is 32-dimensional.

[0215] Adaptive Kalman filtering: based on the fused features, the target motion state is predicted, the state space model is established, and the observation model is established, and the adaptive Kalman filtering process is as shown in Figure 4 .

[0216] Implementation steps are:

[0217] Step 1: initialization

[0218] 1) Set the initial state estimate value : the state vector contains all the state information of the target that needs to be estimated; in the East-North-Up (ENU) coordinate system, the 9-dimensional state vector is defined as:

[0219] ;

[0220] Position (x, y, z): the fusion result of laser radar ranging, visual detection frame depth estimation, and Beidou / GPS data, z is usually small or 0 (sea level height);

[0221] Velocity : the fusion of millimeter wave radar radial velocity decomposition and AIS / SOG (speed over ground), which is usually close to 0;

[0222] Ship heading ​: COG (Course Over Ground) from AIS, direction of visual and lidar point clouds can also be used to assist in the estimation;

[0223] : SOG (Speed Over Ground) from AIS or millimeter wave radar;

[0224] : Usually initialized to 0, or estimated from the rate of change of heading between successive frames.

[0225] The initial state estimate is:

[0226] ;

[0227] If an AIS message is received first, the latitude and longitude provided by the AIS (converted to ENU coordinates), SOG (Speed Over Ground) and COG (Course Over Ground) are used to initialize x, y, v, ψ The remaining velocity components and rate of turn are initialized to 0.

[0228] 2) Set the initial error covariance matrix : represents the variance of each component in the state estimate and the covariance between different components, The diagonal elements (variances) are set to be large, indicating that the initial estimate is very uncertain.

[0229] .

[0230] The initial position uncertainty is large (±50 meters);

[0231] As above;

[0232] The height uncertainty is small (±5 meters);

[0233] The velocity uncertainty (±10 m / s);

[0234] The heading uncertainty (±0.5 rad, about ±28.6 degrees);

[0235] The rate of turn uncertainty (±0.1 rad / s).

[0236] ​​Off-diagonal elements: usually set to 0 initially, assuming no initial correlation between state components. The filter will automatically estimate these correlations during operation.

[0237] 3) Set initial values for process noise and observation noise 、 : preliminary setting based on sensor characteristics and target general motion behavior.

[0238] is a 9x9 matrix, usually designed as a diagonal matrix, with initial values set as follows:

[0239] ;

[0240] Standard deviation of maximum acceleration in the plane;

[0241] Assuming the maximum acceleration of the ship is about ;

[0242] Standard deviation of vertical acceleration (much smaller than the plane, because the ship moves slowly in the vertical direction);

[0243] Standard deviation of yaw acceleration;

[0244] Standard deviation of heading acceleration;

[0245] Standard deviation of yaw rate change rate;

[0246] Filtering period.

[0247] is a 6x6 diagonal matrix, and the initial value is determined by the accuracy of each sensor. The R matrix represents the uncertainty of the observation value, that is, the sensor error, which comes from the Transformer module, representing the confidence of the fusion result.

[0248] .

[0249] Step 2: Standard Kalman filter prediction:

[0250] State prediction: ;

[0251] where, is the state transition matrix:

[0252] ;

[0253] Sampling time of filter (e.g. 0.1 sec, 0.5 sec or 1 sec, depending on sensor data update rate).

[0254] Lines 1-3: Position = original position + velocity x time. For example: .

[0255] Lines 4-6: Assume velocity is constant over a short time. .

[0256] Line 7: Heading = original heading + turn rate x time. .

[0257] Line 8: Assume speed is constant. .

[0258] Line 9: Assume turn rate is constant. .

[0259] is the control input, is the control matrix, mainly used to control known external input systems, such as controlling one's own ship. In this system, the ship or floating object to be tracked is generally considered to have no external control input, so the control term is set to 0.

[0260] ;

[0261] Error covariance prediction:

[0262] ;

[0263] Here, the is the process noise covariance matrix adjusted adaptively at the last time.

[0264] Step 3: Standard Kalman filter update step:

[0265] Calculate Kalman gain:

[0266] ;

[0267] Here, the is the observation noise covariance matrix adjusted adaptively at the last time. is a 6x9 matrix used to map the 9-dimensional state space to the 6-dimensional observation space.

[0268] .

[0269] State update:

[0270] ;

[0271] where, is the enhanced observation from the Transformer module.

[0272] ;

[0273] : The Transformer synthesizes the pixel coordinates from binocular vision, the point cloud distances from lidar, and the own position from Beidou, to calculate the optimal estimated position of the target in the ENU coordinate system.

[0274] : The Transformer synthesizes the radial velocity from the millimeter wave radar (in the sensor coordinate system) and converts it to the ENU coordinate system, and may also incorporate the velocity components decomposed from the AIS SOG and COG.

[0275] : An important function of the Transformer. It may synthesize the COG from AIS, the recognition result of the ship heading angle in the visual image, and the long-axis direction of the lidar point cloud, to give a more reliable estimate of the ship heading. For example, when the AIS signal is lost, the Transformer can completely rely on the visual and lidar to output this value.

[0276] : Similarly, it synthesizes the millimeter wave radar speed measurement and the AIS SOG.

[0277] Error covariance update:

[0278] .

[0279] Step four: adaptive adjustment:

[0280] Calculate the innovation sequence: the innovation sequence is the difference between the predicted value of the observation and the actual observed value:

[0281] ;

[0282] The innovation sequence is a white noise sequence with mean zero and covariance .

[0283] Estimate the actual covariance of the innovation: calculate the actual covariance of the innovation sequence through a sliding window of length N:

[0284] ;

[0285] where .

[0286] Adjust the noise covariance matrix:

[0287] Adjustment of observation noise : If the actual covariance of innovation greater than its theoretical value , generally indicates that we is too much trusted therefore need to increase , a :

[0288] ;

[0289] needs to be ensured to be positive definite, otherwise special treatment is needed.

[0290] Adjustment of process noise :

[0291] If the innovation sequence shows correlation (non-white noise), it indicates that the process model is not accurate, and needs to be increased to increase the weight of the latest observation. A commonly used method is:

[0292] ;

[0293] This is equivalent to attributing the uncertainty of the model to the process noise.

[0294] Apply the adjusted noise matrix: use the newly estimated and (or the smoothed values) for prediction and update at the next time step (k+1).

[0295] ;

[0296] ;

[0297] where , is a smoothing factor to prevent excessive adjustment.

[0298] Step five: iterative loop:

[0299] Time index k=k+1, and jump back to step two to continue the prediction, update and adaptive adjustment of the next round.

[0300] The mathematical model of the cooperative interface is:

[0301] Forward fusion (Transformer→KF): ;

[0302] where is the fusion feature output by the Transformer, is the projection matrix.

[0303] Feedback optimization (KF→Transformer): Kalman filter provides motion uncertainty is used to adjust the temperature coefficient in attention computation for the next frame of the Transformer :

[0304] ；

[0305] where the larger the uncertainty, the larger the temperature, the flatter the attention distribution, reducing the reliance on unreliable historical information.

[0306] Step 4, risk assessment and evidence preprocessing:

[0307] Collision risk assessment: Calculate the collision risk based on the closest point of approach (TCPA) and the closest distance of approach (DCPA) model of the ship or floating object.

[0308] The implementation steps are:

[0309] Step 1): Define the state vector:

[0310] From the Kalman filter, we obtain the accurate state estimation of the target (T) and the buoy (O).

[0311] Buoy (O) state: ;

[0312] : Position of the ship in the East-North-Up (ENU) coordinate system;

[0313] : Components of the ship's velocity vector in the East and North directions;

[0314] Ship speed ;

[0315] Ship heading (angle of rotation counterclockwise from the East direction).

[0316] Target (T) state: ;

[0317] : Position of the target ship;

[0318] : Velocity vector of the target ship;

[0319] Target ship speed ;

[0320] Target ship heading .

[0321] Step 2) Calculate the relative motion vector:

[0322] Relative position vector: ;

[0323] Relative velocity vector: ;

[0324] Relative speed .

[0325] Step 3) Calculate and :

[0326] Closest point of approach (CPA): Mathematically, it is equal to the absolute value of the cross product of the relative position vector and the relative velocity vector divided by the magnitude of the relative velocity. Its geometric meaning is the distance of the relative motion from the origin.

[0327] .

[0328] Time to closest point of approach (TCPA): Mathematically, it is equal to the dot product of the relative position vector and the relative velocity vector divided by the square of the magnitude of the relative velocity. Its geometric meaning is the time required for the relative motion to reach the closest point, which can be negative (indicating that the closest point has already passed).

[0329] ;

[0330] Significance of the sign:

[0331] The two ships are approaching and have not yet reached the closest point;

[0332] Exactly at the closest point;

[0333] The two ships have passed the closest point and are moving away.

[0334] After obtaining DCPA and TCPA, a risk assessment function needs to be established to quantify the degree of collision danger. A two-dimensional risk matrix is usually used to define the risk level, as shown in Table 3.

[0335] Table 3 Risk Level

[0336] .

[0337] FromFigure 5 、 Figure 6 and Figure 7 It can be seen that the application integrates multi-sensor fusion data, adopts YOLOv8 target detection, Transformer data fusion and Kalman filtering algorithm, realizes accurate detection and anti-collision warning of marine ships and floating objects, and the accuracy is greatly improved.

Claims

1. A multi-sensor fusion intelligent anti-collision method, characterized in that, The method comprises the following steps: S1. Obtain environmental information and pre-process the data; S2. Feature extraction: use a lightweight detection framework to extract ship feature values; According to the AIS state detection value, the weight of the AIS is decided, if the AIS fails, the Kalman filtering algorithm is used for the features obtained by the lightweight detection framework and the radar; The AIS health assessment model is: ; , , is a weight coefficient, is a received signal strength, is an actual reporting frequency, is a maximum allowed reporting frequency, is a log consistency evaluation, evaluating the logic and reasonableness of AIS device reporting data; S3. Multi-sensor data fusion: a bidirectional information flow closed loop is formed between the Transformer and the Kalman filtering prediction module, and data from vision, radar and AIS is fused; predicting a target motion state based on the fused features, including initializing a state estimation value , a standard Kalman filter prediction step, and a standard Kalman filter update and adaptive adjustment Adaptive adjustment: ; innovation sequence is a white noise sequence, is an enhanced observation from a Transformer module, an observation matrix, state prediction; S4. Risk assessment.

2. The intelligent anti-collision method of multi-sensor fusion according to claim 1, characterized in that, Data preprocessing includes: image preprocessing: distortion correction and stereo matching processing are performed on the images collected by the binocular camera; Radar data preprocessing: filter the laser radar point cloud data, and use the attitude data calculated by the IMU to perform motion compensation on the laser radar and millimeter wave radar data; Position calculation: the coordinates of the buoy are calculated based on quaternions, and the coordinates calculated by the IMU are corrected using Kalman filtering combined with Beidou satellite coordinate data; Time calibration: realize multi-sensor time synchronization through RTC and Beidou system; AIS state detection: detect the AIS signal state, and judge whether the nearby ship opens the AIS; when the AIS is normal: give high weight to the AIS data, which is used to correct the radar and vision feature extraction; when the AIS fails: reduce the weight of the AIS data, and increase the weight of the radar and vision feature extraction. 3.The intelligent anti-collision method of multi-sensor fusion according to claim 1, characterized in that, The feature extraction steps are as follows: S21. The feature data set contains the ship profile chord number and floating objects; the activation function uses the SiLU function; The ship number recognition uses the CRNN-CTC network architecture, uses the PSMNet to generate synthetic data to simulate the interference brought by the external environment, and uses the SE module to enhance the character area weight; S22. The fusion strategy mainly uses number vision and radar data as follows: Binocular parallax calculation: the target distance is calculated by the pixel offset amount of left and right images; the original data collected by the binocular camera is subjected to Brown-Conrady model to eliminate lens distortion, and then subjected to Bouguet algorithm to output left and right horizontally aligned views, subjected to SGM algorithm to match left and right images, to generate a parallax map, and then the parallax map is converted into a depth value according to the calibration parameters of the camera ; the calculation formula is: ； : camera focal length; : binocular baseline distance; d : parallax value; Radar data: after background filtering and motion compensation, the point cloud cluster data matrix is merged and optimized using the DBSCAN algorithm; Depth calibration is performed on the disparity map generated by binocular vision calculation; Fusion calculation module: radar depth data and visual depth are corrected by formula, and the corrected depth data is output ; the fusion formula is: ； : laser reference depth; : visual raw depth; : Radar depth.

4. The intelligent anti-collision method of multi-sensor fusion as claimed in claim 1, wherein, The Transformer encoder fuses different heterogeneous data into a unified feature representation: Feature input: map the vision, radar and AIS heterogeneous sensor features to a unified semantic space; Feature embedding stage: linearly project sensor features of different dimensions to a unified vector space, and unify all modal features to 512 dimensions; Through the multi-head attention mechanism, all sensor features interact, supplement and enhance each other, learn the deep correlation between them, and the processing steps are as follows: Feature splicing: splice the four 512-dimensional feature vectors in the sequence dimension to form a larger feature matrix; Position encoding: a unique 512-dimensional position vector is generated for each position in the spliced sequence, i.e. each sensor feature, and added to the feature. The unified feature sequence with position information is sent to an encoder stacked by multiple Transformer blocks for deep processing; Multi-head self-attention mechanism: calculate the correlation weight of each feature in the sequence with all other features in the sequence; Feedforward neural network FNN: Non-linear transformation and enhancement of each feature after attention weighting: Residual connection and layer normalization: Calculate the residual connection by adding the sub-layer input and the sub-layer output, and calculate the mean and variance of the result after the residual connection to perform layer normalization. 5.The intelligent anti-collision method of multi-sensor fusion according to claim 1, wherein, Based on the fused features, the target motion state is predicted, and the steps are as follows: Step one, initialize state estimate : ； position (x, y, z), velocity heading speed turning rate ; Step 2, standard Kalman filter prediction: State prediction: ； is a state transition matrix, is a control input matrix, is a control matrix; Error covariance Prediction: ； is the process noise covariance matrix after the last time adaptive adjustment; Step 3, standard Kalman filter update: Calculate the Kalman gain: ； is the observation noise covariance matrix adapted at the last time instant, an observation matrix for mapping the 9-dimensional state space to the 6-dimensional observation space; State update: ； wherein, is an enhanced observation from the Transformer module; Step 4: Adaptive adjustment: innovation sequence is a white noise sequence with zero mean and covariance The actual covariance of the innovation sequence is estimated: The actual covariance of the innovation sequence is computed by a sliding window of length N: ； wherein ; Adjust the noise covariance matrix: Adjusting observation noise : If the actual covariance of the innovations Greater than its theoretical value , need to increase , produce a new ； ； Step 5, loop iteration: Forward fusion: ； wherein is a fused feature of the Transformer output, is a feature vector at time k; is a projection matrix, is a raw observation vector of the sensor at time k; Feedback optimization: The Kalman filter provides The temperature coefficient used to adjust the attention computation of the next frame of the Transformer : ； ； The larger, the more attention distribution is flat, reduce the dependence on unreliable historical information.

6. The intelligent anti-collision method of multi-sensor fusion according to claim 5, characterized in that, If the innovation sequence shows correlation, it indicates that the process noise covariance matrix adjusted adaptively is not accurate and needs to be adjusted to adjust the weight of the latest observation matrix ​ ； Apply the adjusted noise matrix: use the newly estimated and or the smoothed values thereof, for prediction and update at the next time step (k+1): ； ； wherein , is a smoothing factor.

7. The intelligent anticollision method with multisensor fusion as claimed in claim 1 wherein, Collision risk calculation based on the closest point of approach TCPA and the closest distance of approach DCPA model of the ship or floating object: The formula for calculating the closest point of approach TCPA is as follows: ； ； a relative position vector, is x a relative position vector in the direction of, is y a relative position vector in the direction of; a relative velocity vector, is x a relative velocity vector in the direction of, is y a relative velocity vector in the direction of; Closest point of approach TCPA: ； : Two ships are approaching, not yet at closest point. : just at the point of closest approach; : The two ships have passed the closest point and are moving away.

8. The intelligent anti-collision method of multi-sensor fusion according to claim 7, characterized in that, The steps for calculating the relative motion vector are as follows: From the Kalman filter, the accurate state estimation of the target (T) and the buoy (O) is obtained: Buoy (O) state: ； ): position of the ship in the ENU coordinate system in the northeast sky​ ): the east and north components of the velocity vector of the ship;​ Vessel speed: ; Ship's heading: ; Target (T) state: ； ): position of the target ship;​ ): velocity vector of the target ship;​ Target ship speed: ​ Target ship course: ​ Calculate the relative motion vector: Relative position vector: ； Relative velocity vector: 。 9. A multi-sensor fusion intelligent anti-collision system, using the method according to any one of claims 1-8, characterized in that, It includes the perception layer, processing layer and application layer; Perception layer: It is the data input layer of the system, responsible for collecting environmental information from various sensors; Processing layer: Responsible for data preprocessing, feature extraction, multi-sensor fusion, collision risk assessment and collision evidence chain data preparation; Application layer: Responsible for system output and data storage.

Images