Ship collision monitoring and early warning method and system for dam and navigation facilities

Abstract

The invention provides a ship collision monitoring and early warning method and system for dams and navigation facilities, and relates to the technical field of shipping traffic. The method comprises the following steps: respectively acquiring images, messages and radar point cloud data through a camera, a ship automatic identification system and a millimeter wave radar at a main monitoring point and an auxiliary monitoring point, projecting the radar data to an image plane to form a radar image after time-space synchronization processing, sending the radar image to a ship identification network, and identifying a two-dimensional surrounding frame and a type of a ship; then, obtaining a three-dimensional bounding box and a type of the ship by utilizing epipolar geometric processing, comparing the three-dimensional bounding box and the type with message data, and issuing third-level early warning when the three-dimensional bounding box and the type are inconsistent with the message data or no related type exists; meanwhile, the future navigational speed and course of the ship are predicted according to a plurality of three-dimensional bounding boxes of the ship, and if the future navigational speed and course exceed a set threshold value or the risk of impacting a dam exists, secondary early warning is issued; if the ship triggers third-level and second-level early warning at the same time, first-level early warning is issued. According to the scheme, the accuracy of early warning and the completeness of an early warning mechanism can be greatly improved.

Description

Technical Field

[0001] This application relates to the technical field of shipping traffic, and more particularly, to a method and system for ship collision monitoring and early warning of dams and navigation facilities. Background Art

[0002] As important components of water conservancy projects, dams and navigation facilities have inestimable value for flood control and regional economic development. However, with the booming development of waterborne transportation, the number of ships has increased sharply, and the safety challenges of dams and their navigation facilities have also intensified. Especially in the reservoir area upstream of the dam and the navigation facility area, various fishing boats, cargo ships, and passenger ships shuttle through. Once a ship gets out of control due to human operation errors or adverse weather conditions, it may pose a serious threat to the dam and navigation facilities.

[0003] To effectively monitor the navigation status of ships and prevent potential ship-dam collision accidents, ship collision monitoring and early warning systems have emerged. However, most current ship collision monitoring and early warning systems adopt a single sensor technology, such as relying solely on cameras for image recognition or only using the AIS system to obtain ship message data. This single monitoring method is difficult to meet the complex and changeable monitoring requirements due to its inherent limitations. For example, although high-definition cameras can identify the position, size, and heading of ships, their recognition effect is extremely vulnerable to lighting and weather conditions; radar monitoring can scan the water surface and cluster analyze to obtain moving target information, but there are blind spots and it cannot identify the static information of ships; the AIS Automatic Identification System for Ships can perceive the navigation situation of ships, but its effectiveness completely depends on whether the ship turns on the relevant equipment as required.

[0004] To overcome the deficiencies of the single monitoring method, some existing technologies have tried to adopt a combination of multiple monitoring methods. However, most of these fusion methods only perform a simple superposition of multi-source data at the decision-making level, with limited room for performance improvement, and once a certain monitoring method fails, it will have a significant impact on the effect of the entire method. For example, some methods fuse the position information obtained by combining radar and the AIS Automatic Identification System for Ships to obtain the dynamic information of ships, and then use computer vision technology to calculate the shape and size of ships. However, this method still fails to fully utilize the complementary advantages of multi-source data and lacks in-depth processing and precise matching in the data fusion process.

[0005] In addition, there are also some methods that try to obtain two types of ship target frames by separately processing video and millimeter-wave radar information, and then fuse the data by a simple addition method. However, this method also has problems such as insufficient data fusion and low accuracy, and it is difficult to achieve precise prediction of ship behavior and timely early warning of dangerous behaviors.

[0006] In summary, there are many deficiencies in the existing ship collision monitoring and early warning technologies in aspects such as sensor technology, data fusion, and early warning mechanisms. Especially in the specific application scenario of dams and navigation facilities, the existing technologies are difficult to effectively meet the complex and changing monitoring requirements, unable to achieve deep fusion and precise matching of multi-source data, thus limiting the accuracy and practicality of the early warning system. Therefore, developing a ship collision monitoring and early warning method that can comprehensively apply various sensor technologies, achieve efficient fusion and precise matching of multi-source data, and have a perfect early warning mechanism is of great significance for ensuring the safe operation of dams and navigation facilities. Summary of the Invention

[0007] The purpose of this application is to provide a ship collision monitoring and early warning method and system for dams and navigation facilities, which can comprehensively apply various sensor technologies and achieve efficient fusion and precise matching of multi-source data, thereby greatly improving the accuracy of early warning and the perfection of the early warning mechanism.

[0008] This application is implemented as follows:

[0009] In the first aspect, this application provides a ship collision monitoring and early warning method for dams and navigation facilities, including the following steps: At the main monitoring point and the secondary monitoring point respectively, use cameras, Automatic Identification Systems (AIS) of ships, and millimeter-wave radars to obtain image data, ship message data, and radar point cloud data respectively, and perform spatio-temporal synchronization processing to project the radar point cloud data onto the image plane to obtain a radar image, and send the image data and the radar image into a ship recognition network to identify the two-dimensional bounding boxes and the first ship types of all ships within the ship recognition area. Use the two-dimensional bounding boxes and the first ship types of all ships within the ship recognition area identified by the main monitoring point and the secondary monitoring point to perform epipolar geometry processing to obtain the three-dimensional bounding boxes and the second ship types of the corresponding ships. Traverse and compare whether the second ship type is consistent with the ship type in the ship message data. If they are inconsistent or there is no relevant type in the ship message data, issue a level-three early warning. Based on the multiple three-dimensional bounding boxes when the ship passes through the ship recognition area, predict its speed result and heading result in a future period of time. In response to the speed result exceeding the speed threshold set by the navigation facility and / or the heading result indicating a possible collision with the dam, issue a level-two early warning; and in response to any ship simultaneously triggering a level-three early warning and a level-two early warning, issue a level-one early warning.

[0010] In the second aspect, this application provides a ship collision monitoring and early warning system for dams and navigation facilities, which includes:

[0011] The first recognition module is configured to: respectively at the main monitoring point and the secondary monitoring point, use a camera, an Automatic Identification System (AIS) of ships, and a millimeter-wave radar to obtain image data, ship message data, and radar point cloud data respectively, and perform spatio-temporal synchronization processing to project the radar point cloud data onto the image plane to obtain a radar image, and send the image data and the radar image into a ship recognition network to recognize the two-dimensional bounding boxes and the first ship type of all ships in the ship recognition area. The second recognition module is configured to: use the two-dimensional bounding boxes and the first ship type of all ships in the ship recognition area recognized by the main monitoring point and the secondary monitoring point to perform epipolar geometry processing to obtain the three-dimensional bounding boxes and the second ship type of the corresponding ships. The first warning module is configured to: traverse and compare whether the second ship type is consistent with the ship type in the ship message data, and if they are inconsistent or there is no relevant type in the ship message data, issue a level-three warning. The second warning module is configured to: based on multiple three-dimensional bounding boxes when a ship passes through the ship recognition area, predict the speed result and the heading result of the ship in a future period of time, and in response to the speed result exceeding the speed threshold set by the navigation facility, and / or, the heading result indicating a possible impact with the dam, issue a level-two warning; and in response to any ship triggering both a level-three warning and a level-two warning simultaneously, issue a level-one warning.

[0012] In a third aspect, the present application provides another ship collision monitoring and warning system for a dam and a navigation facility, which includes: a front-end monitoring module, disposed at the main monitoring point and the secondary monitoring point, including a fill light for providing illumination for the camera at night, a camera for obtaining image data, an Automatic Identification System (AIS) of ships for obtaining ship message data, and a millimeter-wave radar for obtaining radar point cloud data. A ship recognition module, configured to perform spatio-temporal synchronization processing on the image data, ship message data, and radar point cloud data obtained by the main monitoring point and the secondary monitoring point to project the radar point cloud data onto the image plane to obtain a radar image, and send the image data and the radar image into a ship recognition network to recognize the two-dimensional bounding boxes and the first ship type of all ships in the ship recognition area; and use the two-dimensional bounding boxes and the first ship type of all ships in the ship recognition area recognized by the main monitoring point and the secondary monitoring point to perform epipolar geometry processing to obtain the three-dimensional bounding boxes and the second ship type of the corresponding ships. A trajectory analysis module, configured to predict the speed result and the heading result of a ship in a future period of time based on multiple three-dimensional bounding boxes when the ship passes through the ship recognition area. A warning handling module, configured to traverse and compare whether the second ship type is consistent with the ship type in the ship message data, and if they are inconsistent or there is no relevant type in the ship message data, issue a level-three warning; and in response to the speed result exceeding the speed threshold set by the navigation facility, and / or, the heading result indicating a possible impact with the dam, issue a level-two warning; and in response to any ship triggering both a level-three warning and a level-two warning simultaneously, issue a level-one warning.

[0013] Fourthly, the present application provides an electronic device, which includes a memory for storing one or more programs, a processor; when the above one or more programs are executed by the above processor, the method described in any item of the above first aspect is implemented.

[0014] Fifthly, the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method described in any item of the above first aspect is implemented.

[0015] Compared with the prior art, the present application has at least the following advantages or beneficial effects:

[0016] (1) Improve the accuracy of early warning: By comprehensively applying a variety of sensor technologies, efficient fusion and precise matching of multi-source data are achieved, improving the accuracy and robustness of ship identification. At the same time, through steps such as epipolar geometry processing and type consistency checking, the accuracy of early warning information is further ensured.

[0017] (2) Enhance the real-time performance of early warning: The present application can obtain the position, type and dynamic state information of ships in real time, and predict the future speed and heading of ships based on this information. This enables early warning information to be issued before the ship approaches the dam or navigation facilities, providing sufficient time for operators to take preventive measures.

[0018] (3) Improve the efficiency of emergency response: By designing a multi-level early warning mechanism, the present application can take corresponding emergency response measures according to the actual situation and potential risks of ships. This helps operators quickly identify and respond to potential ship collision risks, reducing the possibility of accidents. Description of the Drawings

[0019] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings required to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, so they should not be regarded as limiting the scope. For those of ordinary skill in the art, other related drawings can also be obtained based on these drawings without creative efforts.

[0020] Figure 1 It is a flowchart of an embodiment of a ship collision monitoring and early warning method for a dam and navigation facilities of the present application;

[0021] Figure 2 It is a schematic diagram of the layout positions of the main monitoring points and sub-monitoring points and the ship identification area in an embodiment of the present application;

[0022] Figure 3 It is a schematic diagram of the device hardware framework of the main monitoring point in an embodiment of the present application;

[0023] Figure 4 Schematic diagram of the device hardware framework of the secondary monitoring point in an embodiment of the present application;

[0024] Figure 5 Flowchart of the ship recognition network based on the channel attention mechanism in an embodiment of the present application;

[0025] Figure 6 Schematic diagram of the image feature encoder / radar image feature encoder framework in an embodiment of the present application;

[0026] Figure 7 Schematic diagram of the framework of an embodiment of the ship collision monitoring and early warning method for dams and navigation facilities in the present application;

[0027] Figure 8 Block diagram of the structure of an electronic device provided in an embodiment of the present application.

[0028] Icons: 201, processor; 202, memory; 203, communication interface. Detailed implementation manners

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are some, but not all, of the embodiments of the present application. The components of the embodiments of the present application described and illustrated herein generally can be arranged and designed in various different configurations. In this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or sequence between these entities or operations.

[0030] The following will describe in detail some implementation manners of the present application with reference to the accompanying drawings. Without conflict, the following various embodiments and the various features in the embodiments can be combined with each other.

[0031] Embodiment 1

[0032] As a key part of water conservancy projects, the safety of dams and navigation facilities is severely threatened by the rapid growth of waterborne transportation. Existing ship collision monitoring and early warning systems mainly rely on single-sensor technologies such as cameras, AIS, etc., but these technologies have obvious limitations, such as being vulnerable to weather, having blind spots, and relying on the activation of ship equipment. Although some technologies attempt to integrate multiple monitoring methods, most of them only perform simple data superposition at the decision-making level and fail to fully utilize the complementary advantages of multi-source data, resulting in limited accuracy and practicality of the early warning system.

[0033] In this regard, the embodiments of the present application provide a ship collision monitoring and early warning method for dams and navigation facilities, which can comprehensively utilize a variety of sensor technologies and achieve efficient fusion and precise matching of multi-source data, thereby greatly improving the accuracy of early warning and the perfection of the early warning mechanism.

[0034] Please refer to Figure 1 , the ship collision monitoring and early warning method for dams and navigation facilities includes the following steps:

[0035] Step S101: At the main monitoring point and the secondary monitoring point respectively, use a camera, an Automatic Identification System (AIS) of ships, and a millimeter-wave radar to obtain image data, ship message data, and radar point cloud data respectively, and perform spatio-temporal synchronization processing to project the radar point cloud data onto the image plane to obtain a radar image, and send the image data and the radar image into a ship recognition network to identify the two-dimensional bounding boxes and the first ship types of all ships within the ship recognition area;

[0036] In step S101, at the main monitoring point and the secondary monitoring point, a camera, an Automatic Identification System (AIS) of ships, and a millimeter-wave radar are used to obtain image data, ship message data, and radar point cloud data respectively. These data are aligned through spatio-temporal synchronization processing to ensure that they are within the same time and space framework. Then, the radar point cloud data is projected onto the image plane to form a radar image. These image data and the radar image are then sent into a ship recognition network, which uses deep learning algorithms to identify the two-dimensional bounding boxes (i.e., the position range of the ship in the image) and the first ship types (such as cargo ships, passenger ships, etc.) of all ships within the ship recognition area. In this way, by comprehensively using a variety of sensor technologies, the system will be able to obtain rich ship information, including position, type, and dynamic state. The fusion of this multi-source data improves the accuracy and robustness of ship recognition, providing a reliable data basis for subsequent steps.

[0037] It should be noted that the ship message data includes ship static data and ship dynamic data. The ship static data includes ship type and hull number, and the ship dynamic data includes ship longitude and latitude, course, and speed.

[0038] Step S102: Use the two-dimensional bounding boxes and the first ship types of all ships within the ship recognition area identified by the main monitoring point and the secondary monitoring point to perform epipolar geometry processing to obtain the three-dimensional bounding boxes and the second ship types of the corresponding ships;

[0039] In step S102, the two-dimensional bounding box and the first ship type obtained in step S101 are used to obtain the three-dimensional bounding box of the corresponding ship (i.e., the position range of the ship in three-dimensional space) and the second ship type through epipolar geometry processing (a geometric transformation method based on image matching). This step is actually a further refinement and verification of the ship information to ensure the accuracy of ship type identification. The extraction of its three-dimensional bounding box provides the system with the specific position information of the ship in space, which is crucial for subsequent speed and heading predictions. Among them, when the same ship type identified by the main monitoring point and the secondary monitoring point is inconsistent, the second ship type takes the first ship type identified by the secondary monitoring point.

[0040] Step S103: traverse and compare whether the second ship type is consistent with the ship type in the ship message data. If they are inconsistent or there is no relevant type in the ship message data, issue a third-level warning;

[0041] In step S103, the second ship type obtained in step S102 is compared to the ship type in the ship message data to see if they are consistent. If they are inconsistent, or there is no relevant type information in the ship message data (perhaps because the ship has not turned on the AIS device, etc.), the system issues a level 3 warning. The issuance of a level 3 warning is a preliminary response to potential safety risks. Through the type consistency check, the system can promptly detect type inconsistencies or missing message data, and provide warning information to operators so that they can take necessary preventive measures.

[0042] Step S104: Based on multiple three-dimensional bounding boxes of the ship when it passes through the ship identification area, predict its speed and heading results in the future period of time (exemplarily, the improved Kalman filter can be used for prediction), and in response to the speed result exceeding the speed threshold set for the navigation facilities, and / or the heading result indicating the possibility of collision with the dam, issue a second-level warning; and in response to any ship triggering the third-level warning and the second-level warning at the same time, issue a first-level warning.

[0043] In step S104, the prediction of speed and heading enables real-time assessment of the ship's navigation status and timely detection of dangerous behaviors such as speeding or deviation from the channel. The introduction of a multi-level warning mechanism enables appropriate emergency response measures to be taken according to different situations, improving the efficiency and accuracy of emergency response. The issuance of a first-level warning is an emergency response to serious safety risks, ensuring that operators can take immediate action to avoid collision accidents.

[0044] In summary, the technical solutions of the above embodiments can achieve the following beneficial effects:

[0045] (1) Improve the accuracy of early warning: By comprehensively applying various sensor technologies, the efficient fusion and precise matching of multi-source data are achieved, improving the accuracy and robustness of ship identification. At the same time, through steps such as epipolar geometry processing and type consistency checking, the accuracy of early warning information is further ensured.

[0046] (2) Enhance the real-time performance of early warning: This method can obtain the position, type, and dynamic state information of ships in real time, and predict the future speed and course of ships based on this information. This enables early warning information to be issued before the ship approaches the dam or navigation facilities, providing sufficient time for operators to take preventive measures.

[0047] (3) Improve the efficiency of emergency response: By designing a multi-level early warning mechanism, this method can take corresponding emergency response measures according to the actual situation and potential risks of the ship. This helps operators quickly identify and respond to potential ship collision risks, reducing the likelihood of accidents.

[0048] Based on the foregoing solution, as Figure 2 shown, in some implementation manners of the present application, the main monitoring points are arranged on the high points of the navigation facilities, and the secondary monitoring points are arranged on the high points on the upstream side of the dam, i.e., the navigation facilities. The ship identification area is set in the overlapping area of the monitoring ranges of the main monitoring points and the secondary monitoring points.

[0049] It should be noted that the existing application scenarios of research on active anti-collision early warning technology are mostly bridges, and the monitoring range is generally a section of the river channel, which is much smaller than the monitoring range required for dams and navigation facilities. Moreover, the monitoring direction is generally towards the river channel direction. This method can effectively judge whether the ship deviates from the channel, but the accuracy of ship type identification is not high. The present invention proposes to arrange main monitoring points on the navigation facilities, arrange secondary monitoring points on the upstream side of the dam and the navigation facilities, and set a ship identification area in the overlapping area of the monitoring ranges of the main and secondary monitoring points. It can efficiently and comprehensively monitor the navigation status of ships upstream of the dam and the navigation facilities. Combining the monitoring data of the main and secondary monitoring points, it is also possible to more comprehensively obtain the ship form of the ships passing through the ship identification area, effectively identify the ship type, and obtain a more accurate three-dimensional bounding box of the ship. Correspondingly, more accurate ship position, course, and speed can be calculated.

[0050] Among them, the main monitoring points are carefully arranged at the high points of the navigation facilities. This location selection has several significant advantages: Firstly, the high point location can provide a broader field of vision for the main monitoring point, ensuring that it can cover the key parts of the navigation facilities and their surrounding areas; Secondly, since the navigation facilities are usually key nodes for ship navigation, setting the main monitoring point here can capture ship navigation information more effectively; Finally, the high point location also helps to reduce the interference of ground obstacles and improve the clarity and accuracy of monitoring data. The secondary monitoring points are arranged at the high points on the upstream side of the dam, which is also the navigation facility. This location arrangement also has its unique advantages: On the one hand, the secondary monitoring points can capture the ship information about to enter the navigation facility area in advance, providing sufficient warning time for the system; On the other hand, the high point location on the upstream side enables the secondary monitoring points to observe the ship's navigation trajectory and status more comprehensively, helping the system to predict the future behavior of the ship more accurately; In addition, the arrangement of the secondary monitoring points helps to form a complement to the main monitoring points, jointly improving the coverage and accuracy of the entire monitoring and warning system.

[0051] Exemplarily, as Figure 3 and Figure 4 shown, the equipment installed at the main monitoring point and the secondary monitoring point can include a monitoring module, a communication module, an auxiliary module, and a processing module, and the equipment needs to be connected to the mains power. Compared with the secondary monitoring point, the monitoring module of the equipment installed at the main monitoring point is equipped with an AIS system (Automatic Identification System for Ships), which is suitable for obtaining ship message data within a range of 10 km around the dam and the navigation facilities. The auxiliary module is equipped with an audible and visual alarm module, which is suitable for directly warning the dam duty officers and the navigation facility management personnel after triggering an alarm. The main monitoring point is arranged at a certain high place of the navigation facility, with no obstruction around, and there is no suitable point on the site to set up a column for installing the equipment. The secondary monitoring point is arranged at a certain high place on the right bank upstream of the dam, with no obstruction around, and there is no suitable point on the site to set up a column for installing the equipment. The overlapping area of the monitoring ranges of the main monitoring point and the secondary monitoring point refers to the overlapping part of the monitoring ranges of the millimeter-wave radar and the high-definition camera. The ship identification area generally needs to cover at least 100 m along the river direction.

[0052] Based on the foregoing solution, in some implementation manners of the present application, the spatio-temporal synchronization processing includes: Spatial synchronization processing steps: Before the device installation, calibrate the same set of cameras and millimeter-wave radars to the same coordinate system; after the device installation, select one or more known control points within the monitoring ranges of the main monitoring point and the secondary monitoring point, and use the stereo matching algorithm to calibrate the geometric relationship of the cameras at the main monitoring point and the secondary monitoring point to obtain the calibration result; convert the ship dynamic data obtained by the Automatic Identification System (AIS) to a preset three-dimensional rectangular coordinate system, and calculate the coordinate transformation matrix of the cameras and millimeter-wave radars at the main monitoring point and the secondary monitoring point according to the calibration result to transform them to this three-dimensional rectangular coordinate system. Temporal synchronization processing steps: Use timestamps to synchronize the image data, ship message data, and radar point cloud data in terms of time.

[0053] In the above implementation manner, through spatial synchronization processing and temporal synchronization processing, it is possible to ensure that data from different monitoring points and different sensor types are fused and analyzed within a unified spatial and temporal framework. This helps to reduce data deviation and error, and improve the accuracy and reliability of the data. And this enables subsequent data from multiple data sources to be fused and analyzed within a unified spatial and temporal framework, so that even if a certain data source fails or has data anomalies, the data of other data sources can still be used for early warning and decision-making. This enhances the robustness and fault tolerance.

[0054] Exemplarily, when using timestamps to synchronize the image data, ship message data, and radar point cloud data, the following steps may be included: First, align the timestamps of all data, find the earliest and latest timestamps, and determine the time range of the data. Then, divide the data into different time windows according to the time resolution (such as per second, per half second, etc.), and the data within each time window comes from the same time point or adjacent time points. Next, within each time window, fuse the data from different data sources, which includes the integration and association of image data, radar point cloud data, and ship message data. Then, if a certain data source has no data within the time window (such as the camera stops working due to a fault), perform anomaly processing, such as using interpolation to estimate the missing data or marking it as abnormal data.

[0055] As Figure 5 shown, based on the foregoing solution, in some implementation manners of the present application, the ship recognition network includes: an image feature encoder for extracting features from the input image data to obtain a first feature; a radar image encoder for extracting features from the input radar image to obtain a second feature; a feature fuser for fusing the first feature and the second feature to obtain a fused feature; and a decoder for decoding the fused feature to obtain the two-dimensional bounding boxes of all ships within the ship recognition area and the first ship type.

[0056] In the above implementation, through the image data and radar images, the ship recognition network can capture more information about the ship, thereby improving the recognition accuracy. Among them, the image data provides the appearance features of the ship, while the radar image provides the position and speed information of the ship. The two complement each other and jointly improve the recognition accuracy. Moreover, the radar image has the ability to penetrate bad weather conditions such as clouds, rain, and fog. Therefore, the ship recognition network can still maintain a high recognition performance under bad weather conditions.

[0057] It should be noted that single-modal data often has limitations or noise, while the ship recognition network designed in the above implementation can integrate information from two different modalities, namely images and radar, enabling the network model to obtain more feature expressions, which helps to enhance the robustness of the ship recognition network to noise and abnormal situations and improve the final performance of ship recognition. In addition, compared with the decision-level fusion strategy commonly used in the prior art, the present application adopts a feature-level fusion strategy. This method has a higher degree of information integration, can avoid information loss, and fuses information from multiple modalities together at an early stage of data processing. It enables the ship recognition network to learn the correlation information of all modalities simultaneously during the training process and can more deeply explore the internal connections and synergistic effects between modalities.

[0058] Exemplarily, the image feature encoder can adopt a convolutional neural network (CNN) architecture. Through multi-layer convolution, pooling, and activation function operations, high-dimensional feature vectors, i.e., the first features, are extracted from the images. The radar image encoder, which works in parallel with the image feature encoder, can also adopt deep learning techniques to extract features from the radar images, obtaining the second features. These features contain key information such as the position, speed, and heading of the ship, complementing the image features. The feature fusion device is responsible for fusing the first features and the second features output by the image feature encoder and the radar image encoder. The fusion strategy can adopt various methods such as splicing, weighted summation, and attention mechanism. Through feature fusion, the network can comprehensively utilize information from different sensors to improve the accuracy and robustness of ship recognition. The decoder can adopt structures such as transposed convolutional layers and upsampling layers to convert the high-dimensional feature vectors into a low-dimensional image space to achieve precise recognition and positioning of the ship.

[0059] Based on the foregoing solution, in some implementation manners of the present application, the image feature encoder and the radar image encoder include: based on the MobileNet V2 network, retaining the first 5 convolutional blocks of the MobileNet V2 network, adding an atrous pyramid pooling layer composed of three parallel atrous convolutional layers, using convolutional layers with three aperture sizes of 3, 5, and 7 to extract features of different scales, and splicing them with the unpooled features in the channel dimension to obtain the encoded first feature or second feature.

[0060] Among them, the first 5 convolutional blocks of the MobileNet V2 network have undergone a large amount of training and verification, and have been proven to be highly effective in image feature extraction. Therefore, the first 5 convolutional blocks of the MobileNet V2 network are retained in the above implementation manners. These convolutional blocks can capture basic features in the image, such as edges, textures, and shapes, which are the basis for subsequent image processing and recognition tasks.

[0061] Next, an atrous pyramid pooling layer composed of three parallel atrous convolutional layers is added in the above implementation manners. Atrous convolution (also known as dilated convolution) is a method that can increase the receptive field of the convolutional kernel without increasing the computational amount. In this layer, convolutional layers with three different aperture sizes of 3, 5, and 7 are used. The purpose of doing this is to extract features of different scales. Features of different scales are very important for understanding and recognizing complex scenes because they can capture objects or structures of different sizes and shapes.

[0062] Then, the features of these different scales extracted are subsequently spliced with the unpooled features in the channel dimension. The splicing operation allows the network to simultaneously utilize the feature information from different scales and the original convolutional blocks, thereby generating a richer and more comprehensive feature representation. These spliced features are used as the encoded first feature or second feature, specifically depending on whether it is an ordinary image or a radar image being processed.

[0063] In summary, in the above implementation, by introducing the dilated pyramid pooling layer, multi-scale features can be extracted. This multi-scale feature extraction ability helps to improve the recognition accuracy of the model for objects of different sizes and shapes in the image. Although dilated convolutional layers are added, since these layers are parallel and are extended based on the lightweight architecture of MobileNet V2, the technical solution can still maintain a high computational efficiency. Additionally, such a design captures information at different scales by changing the receptive field size of the convolutional kernel. In the dilated pyramid pooling layer, each convolutional layer has a different aperture, enabling the learning of different-scale features of the image data and radar images. In this way, the designed ship recognition network can simultaneously focus on local details and global context information, enabling it to effectively capture features at multiple scales when processing tasks with objects of different sizes and avoid the problem of insufficient information at a single scale.

[0064] As Figure 6 shown, based on the foregoing solution, in some implementations of the present application, the feature fuser uses a channel attention mechanism to fuse the first feature and the second feature, specifically including: concatenating the first feature and the second feature in the channel dimension, and sending the concatenated feature into a convolutional layer with a kernel size of 1 to compress the channels, obtaining a first compressed feature; sending the first compressed feature into a global average pooling layer to compress the first compressed feature into statistical information at the channel level, obtaining a second compressed feature; using two non-linear fully connected layers to adjust the dimensions of the first feature and the first compressed feature, and adding them element-wise to obtain an image-radar semantic feature; using a multi-layer perceptron with a sigmoid activation function to calculate the channel attention weight of the image-radar semantic feature, and multiplying the obtained attention weight element-wise with the first compressed feature to obtain a fused feature.

[0065] In the above implementation, first, the first feature and the second feature are concatenated in the channel dimension. This is done to combine information from different sources and provide a rich feature set for subsequent processing. Then, the concatenated feature is sent into a convolutional layer with a kernel size of 1. The role of this layer is to compress the channels, reducing the number of channels of the feature (compressing 2n channels to n), thereby obtaining the first compressed feature This step is to reduce the computational complexity while retaining key information. Next, the first compressed feature is sent into a global average pooling layer P. The role of this layer is to compress the feature into statistical information at the channel level, that is, to average all elements of each channel, obtaining the second compressed feature. This step is to extract global context information and provide a basis for calculating the channel attention weight later. Then, two non-linear fully connected layers f1 and f2 are used to process the first feature and the first compressed feature Perform dimensionality adjustment on them to make their dimensions match. Then, add these two features element-wise to obtain the image-radar semantic feature. This step is to further fuse features from different sources and extract their common semantic information. Then, use a multi-layer perceptron U with a sigmoid activation function to calculate the channel attention weight W of the image-radar semantic feature n . The role of this MLP is to assign a weight to each channel according to its importance. The above process can be explained by the following equation:

[0066]

[0067] Finally, multiply the obtained attention weight W n element-wise with the first compressed feature to obtain the fused feature. This step is to weight the features according to the channel attention weight, so as to emphasize important channels and suppress unimportant channels.

[0068] In summary, through the technical solution of the above implementation method, it is possible to extract and fuse features from different sources, and emphasize important channel features, so the performance of the model can be enhanced. This is manifested in aspects such as improving the recognition accuracy and reducing the false alarm rate. And, by using the channel attention mechanism, this technical solution can automatically learn and emphasize important channel features while suppressing unimportant channel features. This helps to improve the effect of feature fusion, making the fused features more accurate and robust.

[0069] In addition, in the above implementation method, by adaptively weighting the channel dimensions of the features and dynamically assigning different weights to each modal feature, it can help the ship recognition network focus on features that make important contributions to the ship recognition task while suppressing irrelevant or redundant features. Compared with the commonly used channel attention modules in the prior art, this application uses the first compressed feature and the first feature to process and obtain the image-radar semantic feature. Since the radar image lacks semantic information while the image data has rich semantic information, such a design can make more effective use of the semantic information in the first feature to enhance the expression of features related to the ship recognition task in the first compressed feature in the final fused feature.

[0070] Embodiment 2

[0071] The embodiment of this application provides a ship collision monitoring and warning system for dams and navigation facilities, which includes:

[0072] The first recognition module is configured to: respectively at the main monitoring point and the secondary monitoring point, use a camera, an Automatic Identification System (AIS) of ships, and a millimeter-wave radar to obtain image data, ship message data, and radar point cloud data respectively, and perform spatio-temporal synchronization processing to project the radar point cloud data onto the image plane to obtain a radar image, and send the image data and the radar image into a ship recognition network to recognize the two-dimensional bounding boxes and the first ship types of all ships in the ship recognition area. The second recognition module is configured to: use the two-dimensional bounding boxes and the first ship types of all ships in the ship recognition area recognized by the main monitoring point and the secondary monitoring point to perform epipolar geometry processing to obtain the three-dimensional bounding boxes and the second ship types of the corresponding ships. The first warning module is configured to: traverse and compare whether the second ship type is consistent with the ship type in the ship message data, and issue a level-three warning if they are inconsistent or there is no relevant type in the ship message data. The second warning module is configured to: predict the speed result and the course result of a ship in a future period of time based on multiple three-dimensional bounding boxes when the ship passes through the ship recognition area, and issue a level-two warning in response to the speed result exceeding the speed threshold set by the navigation facility and / or the course result indicating a possible collision with the dam; and issue a level-one warning in response to any ship triggering both a level-three warning and a level-two warning.

[0073] For the specific implementation process of the above system, please refer to the ship collision monitoring and warning method for a dam and a navigation facility provided in Embodiment 1, which will not be elaborated here.

[0074] Please refer to Figure 7 , this embodiment of the present application also provides a ship collision monitoring and warning system for a dam and a navigation facility, which includes:

[0075] The front-end monitoring module is installed at the main monitoring point and the secondary monitoring point, and includes a supplementary light for providing illumination for the camera at night, a camera for acquiring image data, an Automatic Identification System (AIS) for acquiring ship message data, and a millimeter-wave radar for acquiring radar point cloud data. The ship identification module is used to perform spatio-temporal synchronization processing on the image data, ship message data, and radar point cloud data acquired at the main monitoring point and the secondary monitoring point, so as to project the radar point cloud data onto the image plane to obtain a radar image, and send the image data and the radar image into a ship identification network to identify the two-dimensional bounding boxes and the first ship types of all ships within the ship identification area; and use the two-dimensional bounding boxes and the first ship types of all ships within the ship identification area identified at the main monitoring point and the secondary monitoring point to perform epipolar geometry processing to obtain the three-dimensional bounding boxes and the second ship types of the corresponding ships. The trajectory analysis module is used to predict the speed result and the heading result of a ship within a future period of time based on multiple three-dimensional bounding boxes when the ship passes through the ship identification area. The warning and disposal module is used to traverse and compare whether the second ship type is consistent with the ship type in the ship message data. If they are inconsistent or there is no relevant type in the ship message data, a level-three warning is issued; and in response to the speed result exceeding the speed threshold set by the navigation facility, and / or the heading result indicating a possible impact with the dam, a level-two warning is issued; and in response to any ship triggering both a level-three warning and a level-two warning simultaneously, a level-one warning is issued.

[0076] In the above embodiment, by integrating a variety of sensor technologies, all-round and multi-angle monitoring of ships is achieved, improving the accuracy of ship identification and the precision of trajectory analysis. By setting up a multi-level warning mechanism, potential ship impact risks can be detected and processed in a timely manner, effectively preventing the occurrence of safety accidents. Moreover, the system can automatically trigger warnings and take corresponding disposal measures, reducing the delay of manual intervention and enhancing the speed and efficiency of emergency response.

[0077] Based on the foregoing solution, in some implementation manners of the present application, the ship identification module is further used to traverse and compare whether the position information corresponding to the three-dimensional bounding box and the second ship type are consistent with the ship type in the ship message data, so as to determine whether the target ship has correctly activated the Automatic Identification System (AIS).

[0078] In the above implementation manner, the function of the ship identification module is further enhanced. It is used to traverse and compare whether the position information corresponding to the three-dimensional bounding box and the second ship type are consistent with the ship type in the ship message data, and this is used as the basis for determining whether the target ship has correctly activated and used the Automatic Identification System (AIS).

[0079] Specifically, the vessel identification module will compare the position information provided by the three-dimensional bounding box with the position information in the vessel message data. Since the AIS system can send the position information of vessels in real time, the operating status of the AIS system can be preliminarily judged by comparing the differences between the two. In addition to the position information, the vessel identification module will also compare the second vessel type obtained through image recognition and radar data fusion with the vessel type declared in the vessel message data. If the two are inconsistent, it may mean that the AIS system is misconfigured or deliberately tampered with. Through automatic comparison and verification, it can timely detect and report vessels with the AIS system not properly turned on, improving the compliance and safety of vessel navigation.

[0080] Based on the foregoing solution, in some implementation manners of the present application, the early warning disposal module includes responses to first-level early warnings, second-level early warnings, and third-level early warnings, specifically including active handling and passive alarms. Among them, active disposal includes arranging active anti-collision devices, dispatching patrol boats to approach the vessel for disposal, and temporarily closing navigation facilities, and passive alarms include audible and visual alarms, platform alarms, and SMS alarms.

[0081] In the above implementation manner, by combining active handling and passive alarms, early warning information can be quickly responded to, effective disposal measures can be taken, and the risk of vessels hitting the dam can be reduced. The diverse disposal means adopted (such as active anti-collision devices, dispatching patrol boats to approach the vessel for disposal, adjusting navigation rules, etc.) and alarm methods (such as audible and visual alarms, platform alarms, SMS alarms, etc.) together constitute a comprehensive safety guarantee system.

[0082] Specifically, arranging active anti-collision devices means that when the system issues a first-level early warning, immediately activate or adjust active anti-collision devices, such as automatic sprinkler devices, sound warning systems, etc., to physically or audibly prevent or warn vessels that may hit the dam.

[0083] Dispatching patrol boats to approach the vessel for disposal means dispatching patrol boats to quickly approach and intercept vessels that may hit the dam, and ensuring the safe departure of the vessels from the dangerous area through manual intervention. Temporarily closing navigation facilities means that in extreme cases, such as when multiple vessels simultaneously trigger a first-level early warning and the active anti-collision devices and patrol boats cannot effectively handle it, consider temporarily closing the navigation facilities until the danger is lifted.

[0084] Adjusting navigation rules means adjusting the navigation rules in the navigation area according to the early warning information, such as restricting the speed, changing the waterway, etc., to reduce the risk of vessels hitting the dam.

[0085] Strengthening monitoring and notification means increasing the monitoring intensity of the early warning area and notifying the relevant vessels of the early warning information through the platform or SMS to remind them to pay attention to safety.

[0086] Vessel identity verification refers to the verification of the identity of vessels that trigger a level-three warning, confirming whether they are engaged in legitimate navigation and checking whether their AIS systems are operating normally.

[0087] Strengthening safety publicity means sending safety publicity information to vessels that trigger a level-three warning, reminding them to comply with navigation regulations and ensuring navigation safety.

[0088] Acoustic and optical alarm means installing acoustic and optical alarm devices at key positions of the dam and navigation facilities. When the system issues a warning, the acoustic and optical alarm is immediately activated to visually alert on-site personnel.

[0089] Platform alarm means displaying warning information on the platform of the monitoring center or relevant management departments, including the warning level, information of the vessel that triggered the warning, the warning area, etc., so that management personnel can quickly respond.

[0090] SMS alarm means sending SMS alarm information to preset recipients (such as management personnel, crew of patrol boats, etc.), including the warning level, information of the vessel that triggered the warning, recommended disposal measures, etc., to ensure that relevant personnel can receive the warning information in a timely manner and take corresponding actions.

[0091] Please refer to Figure 7 , based on the foregoing solution, in some implementation manners of the present application, the system further includes a real-time monitoring module for using a camera to real-time monitor the navigation conditions of vessels within the monitoring range, and / or, further includes a data management module for saving the original data and processing results obtained during the operation of the system.

[0092] In the above implementation manner, the functions of the system are further expanded by introducing a real-time monitoring module and a data management module to achieve continuous monitoring of the navigation conditions of vessels and comprehensive management of data. For the convenience of communication between modules, a data transmission module can also be set up, which is composed of 4G communication and optical fiber communication, and is used to transmit the data obtained by the front-end monitoring module to the real-time monitoring module, vessel identification module, trajectory analysis module, data management module and warning disposal module.

[0093] Among them, the real-time monitoring module uses the camera in the front-end monitoring module to real-time monitor the navigation conditions of vessels within the monitoring range. This module can capture and display real-time images of vessels, helping management personnel visually understand the navigation status, position and potential safety hazards of vessels. The data management module is used to save the original data and processing results obtained during the operation of the system, including image data, radar point cloud data, vessel message data, vessel identification results, trajectory analysis results and warning disposal records, etc. This module can realize functions such as classified storage, query, export and backup of data, providing data support for the subsequent analysis and optimization of the system.

[0094] Embodiment 3

[0095] Please refer to Figure 8 Figure 8 , an embodiment of the present application provides an electronic device, which includes at least one processor 201 and at least one memory 202; wherein, the processor 201 is directly connected to the memory 202, or communicates with each other through a communication interface 203, or is electrically connected through one or more communication buses or signal lines to realize data transmission or interaction; the memory 202 stores program instructions executable by the processor 201, and the processor 201 calls the program instructions to execute a ship collision monitoring and early warning method for dams and navigation facilities. For example, it realizes:

[0096]

[0096] At the main monitoring point and the secondary monitoring point respectively, image data, ship message data and radar point cloud data are obtained by using a camera, an Automatic Identification System (AIS) and a millimeter-wave radar respectively, and spatio-temporal synchronization processing is performed to project the radar point cloud data onto the image plane to obtain a radar image, and the image data and the radar image are sent into a ship recognition network to identify the two-dimensional bounding boxes and the first ship types of all ships in the ship recognition area. Using the two-dimensional bounding boxes and the first ship types of all ships in the ship recognition area identified by the main monitoring point and the secondary monitoring point, epipolar geometry processing is performed to obtain the three-dimensional bounding boxes and the second ship types of the corresponding ships. Traverse and compare whether the second ship type is consistent with the ship type in the ship message data. If they are inconsistent or there is no relevant type in the ship message data, a level-3 early warning is issued. According to the multiple three-dimensional bounding boxes when the ship passes through the ship recognition area, the speed result and the heading result of the ship in a future period of time are predicted. In response to the speed result exceeding the speed threshold set by the navigation facility, and / or, the heading result indicating a possible collision with the dam, a level-2 early warning is issued; and in response to any ship triggering both a level-3 early warning and a level-2 early warning at the same time, a level-1 early warning is issued.

[0097] Among them, the memory 202 can be, but is not limited to, Random Access Memory (RAM), Read Only Memory (ROM), Programmable Read-Only Memory (PROM), Erasable Programmable Read-Only Memory (EPROM), Electrically Erasable Programmable Read-Only Memory (EEPROM), etc.

[0098] The processor 201 may be an integrated circuit chip with signal processing capabilities. The processor 201 may be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it may also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0099] It can be understood that Figure 8 the structure shown is only schematic, and the electronic device may also include more or fewer components than those shown Figure 8 in it, or have a different configuration from that shown Figure 8 in it. Figure 8 Each component shown in it may be implemented by hardware, software, or a combination thereof.

[0100] Embodiment 4

[0101] This application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the processor 201, it implements a method for ship collision monitoring and early warning of a dam and navigation facilities. For example, it implements:

[0102] At the main monitoring point and the secondary monitoring point respectively, use cameras, Automatic Identification Systems (AIS) of ships, and millimeter-wave radars to obtain image data, ship message data, and radar point cloud data respectively, and perform spatio-temporal synchronization processing to project the radar point cloud data onto the image plane to obtain a radar image, and send the image data and the radar image into a ship recognition network to identify the two-dimensional bounding boxes and the first ship types of all ships in the ship recognition area. Use the two-dimensional bounding boxes and the first ship types of all ships in the ship recognition area identified by the main monitoring point and the secondary monitoring point to perform epipolar geometry processing to obtain the three-dimensional bounding boxes and the second ship types of the corresponding ships. Traverse and compare whether the second ship type is consistent with the ship type in the ship message data. If they are inconsistent or there is no relevant type in the ship message data, issue a level-three early warning. According to the multiple three-dimensional bounding boxes when the ship passes through the ship recognition area, predict the speed result and the heading result of the ship in a future period of time. In response to the speed result exceeding the speed threshold set by the navigation facility, and / or, the heading result indicating a possible collision with the dam, issue a level-two early warning; and in response to any ship simultaneously triggering a level-three early warning and a level-two early warning, issue a level-one early warning.

[0103] When the above functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of this technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in various embodiments of this application. The foregoing storage medium includes: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM, Read-Only Memory), random access memories (RAM, Random Access Memory), magnetic disks, or optical discs.

[0104] For those skilled in the art, it is obvious that this application is not limited to the details of the above exemplary embodiments, and without departing from the spirit or basic characteristics of this application, this application can be implemented in other specific forms. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of this application is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be encompassed in this application. Any reference signs in the claims should not be regarded as limiting the claimed rights.

Claims

1. A ship collision monitoring and early warning method for dams and navigation facilities, characterized in that: The following steps are involved: At the primary monitoring point and the secondary monitoring point, respectively, using a camera, a ship automatic identification system and a millimeter wave radar, image data, ship message data and radar point cloud data are acquired, and spatiotemporal synchronization processing is performed to project the radar point cloud data onto an image plane to obtain a radar image, and the image data and the radar image are sent to a ship identification network to identify a two-dimensional bounding box of all ships in the ship identification area and a first ship type; Using the two-dimensional bounding boxes and the first ship type of all ships in the ship identification area identified by the primary monitoring point and the secondary monitoring point, epipolar geometry processing is performed to obtain a three-dimensional bounding box of the corresponding ship and a second ship type; Traverse and compare whether the second ship type is consistent with the ship type in the ship message data. If they are inconsistent or there is no relevant type in the ship message data, issue a level 3 warning; Based on multiple three-dimensional bounding boxes of a ship passing through the ship identification zone, the ship's speed and heading results in the future are predicted, so that in response to the speed result exceeding the speed threshold set by the navigation facilities and / or the heading result indicating the possibility of collision with the dam, a secondary warning is issued; And in response to any ship triggering both a Level 3 warning and a Level 2 warning simultaneously, a Level 1 warning is issued.

2. The method according to claim 1, characterized in that The main monitoring point is arranged at a high point of the navigation facility, the secondary monitoring point is arranged at a high point on the upstream side of the dam, i.e. the navigation facility, and the ship identification area is arranged in the overlapping area of ​​the monitoring range of the main monitoring point and the secondary monitoring point.

3. The method according to claim 1, characterized in that The time-space synchronization process comprises: Spatial synchronization processing steps: before the equipment is installed, the same set of cameras and millimeter-wave radars are calibrated to the same coordinate system; after the equipment is installed, one or more known control points are selected within the monitoring range of the main monitoring point and the secondary monitoring point, and the cameras in the main monitoring point and the secondary monitoring point are calibrated using a stereo matching algorithm to obtain the geometric relationship of the equipment, and the calibration results are obtained; the ship dynamic data obtained by the ship automatic identification system is converted to a preset three-dimensional rectangular coordinate system, and the coordinate system conversion matrix of the cameras and millimeter-wave radars at the main monitoring point and the secondary monitoring point is calculated according to the calibration results to convert them to the three-dimensional rectangular coordinate system; Time synchronization processing steps: Use timestamps to synchronize image data, ship message data, and radar point cloud data.

4. The method according to claim 1, characterized in that: The ship identification network comprises: An image feature encoder, used for extracting features from input image data to obtain a first feature; A radar image encoder, used for extracting features from the input radar image to obtain a second feature; A feature fuser, used for fusing the first feature and the second feature to obtain a fused feature; The decoder is used to decode the fused features to obtain the two-dimensional bounding boxes and the first ship type of all ships in the ship identification area.

5. The method according to claim 4, characterized in that The image feature encoder and the radar image encoder include: Based on the MobileNet V2 network, the first five convolution blocks of the MobileNet V2 network are retained, and a perforated pyramid pooling layer consisting of three parallel perforated convolutional layers is added. Convolutional layers with three aperture sizes of 3, 5, and 7 are used to extract features of different scales, and they are concatenated with the unpooled features in the channel dimension to obtain the encoded first feature or second feature.

6. The method according to claim 4, characterized in that The feature fuser uses a channel attention mechanism to fuse the first feature and the second feature, specifically including: The first feature and the second feature are concatenated in the channel dimension, and the concatenated feature is sent to a convolution layer with a convolution kernel of 1 to compress the channel to obtain a first compressed feature; Sending the first compressed feature to a global average pooling layer to compress the first compressed feature into channel-level statistical information to obtain a second compressed feature; The first feature and the first compressed feature are dimensionally adjusted using two nonlinear fully connected layers, and are added element by element to obtain the image-radar semantic feature; The channel attention weights of the image-radar semantic features are calculated using a multilayer perceptron with a sigmoid activation function, and the obtained attention weights are multiplied element-by-element with the first compressed feature to obtain the fused feature.

7. A ship collision monitoring and early warning system for dams and navigation facilities, characterized in that: include: The front-end monitoring module is installed at the main monitoring point and the auxiliary monitoring point, including a fill light for providing illumination for the camera at night, a camera for acquiring image data, a ship automatic identification system for acquiring ship message data, and a millimeter-wave radar for acquiring radar point cloud data; A ship identification module is used to perform spatiotemporal synchronous processing on the image data, ship message data and radar point cloud data acquired by the primary monitoring point and the secondary monitoring point, so as to project the radar point cloud data onto an image plane to obtain a radar image, and to send the image data and the radar image into a ship identification network to identify a two-dimensional bounding box and a first ship type of all ships in the ship identification area; and performing epipolar geometry processing on the two-dimensional bounding boxes and the first ship type of all ships in the ship identification area identified by the primary monitoring point and the secondary monitoring point to obtain a three-dimensional bounding box of the corresponding ship and a second ship type; The trajectory analysis module is used to predict the speed and heading of a ship in the future based on multiple three-dimensional bounding boxes when the ship passes through the ship identification area; An early warning handling module is used to traverse and compare whether the second ship type is consistent with the ship type in the ship message data. If they are inconsistent or there is no relevant type in the ship message data, a third-level early warning is issued; and issuing a level 2 warning in response to the speed result exceeding the speed threshold set by the navigation facility, and / or the heading result indicating a possible collision with the dam; And in response to any ship triggering both a Level 3 warning and a Level 2 warning simultaneously, a Level 1 warning is issued.

8. The system according to claim 7, characterized in that The ship identification module is also used to traverse and compare the position information corresponding to the three-dimensional bounding box and the second ship type to see whether they are consistent with the ship type in the ship message data, so as to determine whether the target ship has correctly turned on the ship automatic identification system.

9. The system according to claim 7, characterized in that The early warning and disposal module includes responses to level one, level two and level three warnings, specifically active processing and passive alarms, among which active processing includes the deployment of active anti-collision devices, patrol ship berthing and temporary closure of navigation facilities, and passive alarms include sound and light alarms, platform alarms and SMS alarms.

10. The system according to claim 7, characterized in that The system also includes a real-time monitoring module for monitoring the navigation conditions of ships within the monitoring range in real time using a camera, and / or a data management module for storing raw data and processing results obtained during the operation of the system.

Images