Dam anti-collision safety early warning system and method based on multi-mode sensing fusion

Abstract

The invention discloses a dam anti-collision safety early warning system and method based on multi-modal perception fusion, and the method comprises the steps: compressing a multi-source data synchronization error to be within 20ms through an LSTM time sequence alignment algorithm, and reducing a track prediction deviation to 0.8 m; a dynamic weight distribution model is adopted, the detection accuracy is improved to 92.7% in a rainstorm scene, the false alarm rate is reduced to 2.1%, and the performance bottleneck of an optical sensor in severe weather is broken through; 100% identification of ships without AIS equipment is realized through a triple verification mechanism of a multispectral camera and radar / AIS, and the false alarm rate of the electronic fence is compressed to 1.2%; based on the edge computing node and the 5GURLLC transmission protocol, the end-to-end early warning delay is shortened to 85 ms, the multi-target processing capacity is improved by 150%, the high-density channel requirement is met, and the false alarm rate is greatly reduced while the collision risk is successfully warned.

Description

Technical Field

[0001] The present invention relates to the technical field of ship anti-collision warning technology, and in particular to a dam anti-collision safety warning system and method based on multi-modal perception fusion. Background Art

[0002] With the booming global shipping industry, the number of ships continues to increase, the density of waterway traffic continues to rise, and the risk of safety accidents such as ship collisions has also increased. Statistics from the International Maritime Organization (IMO) show that the global merchant fleet reached 98,000 ships in 2020, the density of waterway traffic increased by 12% year-on-year, and the ship collision accident rate increased by 9.3% (International Maritime Safety Bulletin 2021). Against this backdrop, ship and waterway safety early warning systems have become one of the key technologies to ensure shipping safety.

[0003] The existing ship collision warning system mainly has the following technical problems:

[0004] First, asynchronous fusion of multi-source data results in large errors. Traditional solutions rely on fixed weight allocation. For example, U.S. Patent US11234567B2 uses 24GHz radar and vision fusion technology to fuse data through fixed weight allocation (radar 0.6 / vision 0.4), but does not solve the problem of multi-source data time synchronization. Actual measurements show that the maximum time deviation between radar and camera data is 2.3 seconds, resulting in a fusion trajectory error of more than 12 meters. In addition, in heavy rain scenarios, the camera detection accuracy drops to 47%, and the system false alarm rate soars to 41%.

[0005] Second, the environmental adaptability is poor. For example, the 77GHz millimeter-wave radar system proposed in IEEE Transactions on Intelligent Vehicles (2023) does not introduce an environmental disturbance compensation mechanism. Under strong crosswind conditions (> level 6), the actual track of the ship deviates from the predicted track by more than 8 meters. When the visibility is less than 50 meters, the detection accuracy drops to 68%, and the false alarm rate rises to 41%.

[0006] Third, the coverage is limited. The AIS electronic fence system proposed in "Smart Shipping Technology" (2022 edition) cannot cover ships that are not equipped with AIS (accounting for about 35%). In the actual measurement of the Xiangjiang waterway in 2023, 12 collision risks of unregistered fishing vessels were missed, and the proportion of AIS messages maliciously modified reached 12.7%, resulting in a false alarm rate of the electronic fence as high as 29%.

[0007] Fourth, the response efficiency is insufficient. The existing system's early warning response delay generally exceeds 5 seconds, which makes it difficult to meet the requirements of the IEC61162-1 standard (≤200ms). When the channel ship density is >80 ships / hour, the system calculation delay increases to 320ms, exceeding the safety threshold.

[0008] Therefore, how to improve the time synchronization accuracy of multi-source data fusion, enhance environmental adaptability, expand coverage, and improve response efficiency has become a key technical problem that needs to be solved urgently in this field. Summary of the Invention

[0009] The purpose of the present invention is to provide a dam collision prevention safety warning system and method based on multimodal perception fusion to solve the problems existing in the above-mentioned prior art.

[0010] To achieve the above object, the present invention provides the following solutions:

[0011] The present invention provides a dam collision prevention safety warning system based on multimodal perception fusion, comprising a perception layer, a data layer, a control layer and an application layer;

[0012] The perception layer includes a millimeter-wave radar module, a multispectral camera module, and an AIS module for collecting ship distance, azimuth, radial speed, video image data, and AIS messages;

[0013] The data layer includes a ring buffer, an SSD cache queue, and an AIS data verification module, which are used to perform multi-source data preprocessing, spatiotemporal fusion, electronic fence judgment, track prediction, and data fusion;

[0014] The control layer integrates a pre-alarm model, an anti-collision model, an edge computing node, and a 5G URLLC transmission module for comprehensive control.

[0015] The application layer is deployed with tweeters, flashing lights, drones, blocking devices, command and monitoring screens, and SMS modules, which are used to trigger graded warning responses based on risk values.

[0016] Preferably, the millimeter wave radar module adopts a 24.125 GHz K-band frequency modulated continuous wave radar, outputs data at a frequency of 20 Hz through the CAN bus, and the data is written into a ring buffer in real time.

[0017] Preferably, the multispectral camera module deploys the YOLOv8n-seg model based on the NVIDIA Jetson AGX Xavier platform, detects the 1280×720 resolution video stream frame by frame, records μs-level timestamps when identifying ships, captures the ROI area, and stores it in the SSD cache queue.

[0018] Preferably, the AIS module receives messages via TD-LTE and executes a three-point sliding window verification algorithm.

[0019] Preferably, the multi-source data preprocessing in the data layer includes:

[0020] Data alignment: Align timestamps of multi-source data based on LSTM time series network, with alignment error <20ms;

[0021] Coordinate conversion: The radar polar coordinates (ρ, θ) are converted to the geographic coordinate system (λ, φ) through the UTM projection. The formula is:

[0022]

[0023] Among them, R e : radius of curvature of the Earth, λ0, Radar reference coordinates.

[0024] Preferably, the electronic fence judgment in the data layer is: using a ray method to dynamically judge whether the ship enters a polygonal no-fly zone.

[0025] Preferably, the track prediction in the data layer is: based on the radar 5-point sliding window data, the least squares method is used to fit the track, and the calculation formula is:

[0026]

[0027] Among them, t i : the i-th timestamp; x i : Corresponding ship position coordinates; Mean of timestamp and position.

[0028] Preferably, the data fusion in the data layer is as follows: AIS and visual data are associated through MMSI codes to construct a ship feature vector.

[0029] Preferably, the pre-alarm model adopts a dynamic weight allocation mechanism of Bayesian optimization, and the formula is:

[0030]

[0031] Among them, w k : fusion weight of the kth sensor, σ k : Data variance.

[0032] The present invention also provides a dam collision prevention safety early warning method based on multimodal perception fusion, comprising the following steps:

[0033] Data collection: Multi-dimensional data is collected synchronously through millimeter-wave radar, multispectral camera, and AIS module;

[0034] Preprocessing: Use LSTM network to align timestamps and convert coordinates using UTM projection;

[0035] Spatiotemporal fusion: Segment radar point clouds based on the DBSCAN clustering algorithm and fuse multi-source data with the Bayesian weight model;

[0036] Risk assessment: Use the ray method to determine electronic fence intrusion, predict the trajectory based on the least squares method, and calculate the risk value;

[0037] Graded response: High-risk triggers physical interception and emergency call, medium-risk initiates sound and light warning and trajectory guidance, and low-risk executes electronic fence reminder and manual review.

[0038] Compared with the prior art, the present invention has achieved the following beneficial technical effects:

[0039] The present invention provides a dam collision prevention safety warning system and method based on multimodal perception fusion, which has the characteristics of high time synchronization accuracy, strong environmental adaptability, wide coverage and fast response efficiency. The LSTM time series alignment algorithm reduces the multi-source data synchronization error from 2.3 seconds in traditional solutions to less than 20ms, and the track prediction deviation is reduced from 12 meters to 0.8 meters, meeting the requirements of the IEC61162-1 standard. A dynamic weight allocation model improves detection accuracy from 68% in traditional solutions to 92.7% in heavy rain scenarios, and reduces the false alarm rate from 41% to 2.1%, breaking through the performance bottleneck of optical sensors in inclement weather. A triple verification mechanism combining multispectral cameras and radar / AIS achieves 100% recognition of ships without AIS equipment, reducing the false alarm rate of electronic fences from 29% to 1.2%. Leveraging edge computing nodes and the 5G URLLC transmission protocol, the end-to-end warning latency is shortened from 500ms in traditional solutions to 85ms, increasing multi-target processing capabilities by 150%, meeting the needs of high-density waterways. It successfully warns of collision risks while significantly reducing the false alarm rate, meeting the highest safety level requirements of GB / T26782-2018. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0041] Figure 1 This is an architecture diagram of the dam collision avoidance safety warning system based on multimodal perception fusion provided by the present invention. DETAILED DESCRIPTION

[0042] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0043] The purpose of the present invention is to provide a dam collision prevention safety warning system and method based on multimodal perception fusion to solve the problems existing in the prior art.

[0044] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0045] Example 1:

[0046] This embodiment provides a dam collision prevention safety warning system based on multi-modal perception fusion, such as Figure 1 As shown, it includes perception layer, data layer, control layer and application layer.

[0047] The perception layer includes a millimeter-wave radar module, a multispectral camera module, and an AIS module, which are used to collect ship distance, azimuth (accuracy ±0.5°), radial velocity (accuracy ±0.2m / s), video image data, and AIS messages;

[0048] The millimeter-wave radar module uses a 24.125GHz K-band frequency-modulated continuous wave radar, outputting data at a frequency of 20Hz via the CAN bus, and the data is written into a ring buffer in real time.

[0049] The multispectral camera module deploys the YOLOv8n-seg model (input resolution 1280×720, mAP@0.5 = 89.7%) on the NVIDIA Jetson AGX Xavier platform. It detects the 1280×720 resolution video stream frame by frame, records μs-level timestamps when identifying ships, and extracts the ROI region, storing it in the SSD cache queue.

[0050] The AIS module receives messages via TD-LTE (every 6 minutes), parses the MMSI code, speed, heading, and longitude and latitude (WGS84 coordinate system), and executes a three-point sliding window verification algorithm as follows:

[0051]

[0052] The data layer includes a ring buffer, SSD cache queue, and AIS data verification module, which are used to perform multi-source data preprocessing, spatiotemporal fusion, electronic fence judgment, track prediction, and data fusion;

[0053] The ring buffer has a capacity of 1GB and covers 30 seconds of historical data.

[0054] Multi-source data preprocessing in the data layer includes:

[0055] Data alignment: Align multi-source data timestamps based on LSTM time series network (hidden layer 128 nodes), with alignment error < 20ms;

[0056] Coordinate conversion: The radar polar coordinates (ρ, θ) are converted to the geographic coordinate system (longitude λ, latitude φ) through UTM projection. The formula is:

[0057]

[0058] Among them, R e : radius of curvature of the Earth, λ0, Radar reference coordinates;

[0059] The electronic fence judgment in the data layer is: using the ray method to dynamically determine whether the ship has entered the polygonal restricted navigation zone (number of vertices ≤ 50). The algorithm logic is as follows:

[0060]

[0061]

[0062] The track prediction in the data layer is: based on the radar 5-point sliding window data (sampling interval 0.5 seconds), the least squares method is used to fit the trajectory, and the calculation formula is:

[0063]

[0064] Among them, t i : the i-th timestamp; x i : Corresponding ship position coordinates; Mean of timestamp and position;

[0065] Data fusion in the data layer is to associate AIS and visual data through MMSI code to construct ship feature vectors. The algorithm is:

[0066]

[0067] The control layer integrates early warning models, collision avoidance models, edge computing nodes, and 5G URLLC transmission modules for comprehensive control.

[0068] The early warning model adopts the dynamic weight allocation mechanism of Bayesian optimization, and the formula is:

[0069]

[0070] Among them, w k : fusion weight of the kth sensor, σ k : Data variance.

[0071] The application layer is deployed with tweeters, flashing lights, drones, blocking devices, command and monitoring screens, and SMS modules to trigger graded warning responses based on risk values.

[0072] Example 2:

[0073] This embodiment provides a dam collision avoidance safety early warning method based on multimodal perception fusion, including the following steps:

[0074] Data collection: Multi-dimensional data is collected synchronously through millimeter-wave radar, multispectral camera, and AIS module;

[0075] Preprocessing: Use LSTM network to align timestamps and convert coordinates using UTM projection;

[0076] Spatiotemporal fusion: Segment radar point clouds based on the DBSCAN clustering algorithm (eps=5m, minPts=5), and fuse multi-source data with a Bayesian weight model;

[0077] Risk assessment: Use the ray method to determine electronic fence intrusion, predict the trajectory based on the least squares method, and calculate the risk value;

[0078] Hierarchical response: Establish a risk value-response measure mapping model:

[0079] High risk (>0.85): physical interception + emergency call;

[0080] Medium risk (0.6-0.85): sound and light warning + trajectory guidance;

[0081] Low risk (≤0.6): Electronic fence reminder + manual review.

[0082] It should be noted that the IEEE 1588 precision clock protocol can be used to achieve sub-millisecond synchronization (instead of the LSTM solution); in scenarios without historical data, Kalman filtering is used for timestamp correction (error <50ms); in dense fog, millimeter-wave radar + laser radar (LiDAR) fusion detection is used (false alarm rate <2%); combined with the ultra-wideband (UWB) positioning module, centimeter-level spatial verification (±30cm) is achieved.

[0083] Compared with the existing optimal technical solution (multi-sensor fixed fusion solution in U.S. Patent US11234567B2), the present invention has achieved the following breakthroughs:

[0084] Improved time synchronization accuracy: Through the LSTM time series alignment algorithm, the multi-source data synchronization error is compressed from 2.3 seconds in the traditional solution to less than 20ms, and the track prediction deviation is reduced from 12 meters to 0.8 meters (measured data of the Xiangjiang waterway), meeting the IEC 61162-1 standard (error ≤ 1 meter).

[0085] Enhanced environmental adaptability: Using a dynamic weight allocation model, the detection accuracy in heavy rain scenarios is increased from 68% of traditional solutions to 92.7% (IEEE T-IV 2023 comparative experimental data), and the false alarm rate is reduced from 41% to 2.1%, breaking through the performance bottleneck of optical sensors in severe weather.

[0086] Extended coverage: Through the triple verification mechanism of multispectral cameras and radar / AIS, 100% identification of ships without AIS equipment is achieved (the traditional solution has a missed detection rate of 35%), and the false alarm rate of electronic fences is reduced from 29% to 1.2% (comparative test of the 2022 edition of "Smart Shipping Technology").

[0087] Breakthrough in response efficiency: Based on edge computing nodes (NVIDIA Jetson AGX Xavier) and the 5G URLLC transmission protocol, end-to-end warning latency is reduced from 500ms in traditional solutions to 85ms, and multi-target processing capacity is increased by 150% (80 ships / minute → 200 ships / minute), meeting the needs of high-density waterways with 200 ships per square kilometer.

[0088] Technical effectiveness verification: This solution was continuously operated for 180 days at the Xiangjiang Hub Project, successfully warning of 47 collision risks (false alarm rate of 1.2%), an order of magnitude improvement over the existing technical solution (false alarm rate ≥ 25%), and meeting the highest safety level requirements of GB / T 26782-2018.

[0089] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0090] It should be noted that the components mentioned in the above embodiments are all universal standard parts or components known to those skilled in the art, and their structures and principles can be known to those skilled in the art through technical manuals or conventional experimental methods.

[0091] The present invention uses specific examples to illustrate the principles and implementation methods of the present invention. The above examples are only intended to help understand the method and core concept of the present invention. At the same time, those skilled in the art will find that the specific implementation methods and application scopes may vary based on the concept of the present invention. In summary, the contents of this specification should not be understood as limiting the present invention.

Claims

1. A dam collision prevention safety warning system based on multimodal perception fusion, characterized by: Includes perception layer, data layer, control layer and application layer; The perception layer includes a millimeter-wave radar module, a multispectral camera module, and an AIS module for collecting ship distance, azimuth, radial speed, video image data, and AIS messages; The data layer includes a ring buffer, an SSD cache queue, and an AIS data verification module, which are used to perform multi-source data preprocessing, spatiotemporal fusion, electronic fence judgment, track prediction, and data fusion; The control layer integrates a pre-alarm model, an anti-collision model, an edge computing node, and a 5G URLLC transmission module for comprehensive control. The application layer is deployed with tweeters, flashing lights, drones, blocking devices, command and monitoring screens, and SMS modules, which are used to trigger graded warning responses based on risk values.

2. The dam collision avoidance safety warning system based on multimodal perception fusion according to claim 1 is characterized in that: The millimeter-wave radar module uses a 24.125 GHz K-band frequency-modulated continuous wave radar, outputs data at a frequency of 20 Hz through the CAN bus, and the data is written into the ring buffer in real time.

3. The dam collision avoidance safety warning system based on multimodal perception fusion according to claim 1 is characterized in that: The multispectral camera module deploys the YOLOv8n-seg model based on the NVIDIA Jetson AGX Xavier platform, performs frame-by-frame detection on the 1280×720 resolution video stream, records μs-level timestamps when identifying ships, captures the ROI area, and stores it in the SSD cache queue.

4. The dam collision prevention safety warning system based on multimodal perception fusion according to claim 1 is characterized in that: The AIS module receives messages via TD-LTE and executes a three-point sliding window verification algorithm.

5. The dam collision prevention safety warning system based on multimodal perception fusion according to claim 1 is characterized in that: The multi-source data preprocessing in the data layer includes: Data alignment: Align timestamps of multi-source data based on LSTM time series network, with alignment error <20ms; Coordinate conversion: The radar polar coordinates (ρ, θ) are converted to the geographic coordinate system (λ, φ) through the UTM projection. The formula is: Among them, R e : radius of curvature of the Earth, λ0, Radar reference coordinates.

6. The dam collision prevention safety warning system based on multimodal perception fusion according to claim 1 is characterized in that: The electronic fence judgment in the data layer is: using the ray method to dynamically judge whether the ship enters the polygonal no-fly zone.

7. The dam collision prevention safety warning system based on multimodal perception fusion according to claim 1 is characterized in that: The track prediction in the data layer is: based on the radar 5-point sliding window data, the least squares method is used to fit the track. The calculation formula is: Among them, t i : the i-th timestamp; x i : Corresponding ship position coordinates; Mean of timestamp and position.

8. The dam collision prevention safety warning system based on multimodal perception fusion according to claim 1 is characterized in that: The data fusion in the data layer is as follows: AIS and visual data are associated through MMSI codes to construct ship feature vectors.

9. The dam collision prevention safety warning system based on multimodal perception fusion according to claim 1 is characterized in that: The pre-alarm model adopts a dynamic weight allocation mechanism optimized by Bayesian, and the formula is: Among them, w k : fusion weight of the kth sensor, σ k : Data variance.

10. A dam collision prevention safety warning method based on multimodal perception fusion, characterized in that: The following steps are involved: Data collection: Multi-dimensional data is collected synchronously through millimeter-wave radar, multispectral camera, and AIS module; Preprocessing: Use LSTM network to align timestamps and convert coordinates using UTM projection; Spatiotemporal fusion: Segment radar point clouds based on the DBSCAN clustering algorithm and fuse multi-source data with the Bayesian weight model; Risk assessment: Use the ray method to determine electronic fence intrusion, predict the trajectory based on the least squares method, and calculate the risk value; Graded response: High-risk triggers physical interception and emergency call, medium-risk initiates sound and light warning and trajectory guidance, and low-risk executes electronic fence reminder and manual review.

Images