A method and system for monitoring and early warning of ship collision with a dam and navigation facilities

Abstract

The application provides a dam and navigation facility ship collision monitoring and early warning method and system, and relates to the technical field of shipping traffic. It comprises: at the main and auxiliary monitoring points, image, message and radar point cloud data are obtained through a camera, an automatic ship identification system and a millimeter wave radar respectively, after time and space synchronization processing, the radar data is projected to the image plane to form a radar image, and then sent to a ship identification network to identify the two-dimensional bounding box and type of the ship; then, the three-dimensional bounding box and type of the ship are obtained by using epipolar geometry processing, and compared with the message data, if they are inconsistent or have no related type, a three-level early warning is issued; at the same time, the future speed and heading of the ship are predicted according to the multiple three-dimensional bounding boxes of the ship, if the set threshold is exceeded or there is a risk of collision with the dam, a two-level early warning is issued; if the ship triggers the three-level and two-level early warnings at the same time, a one-level early warning is issued. The scheme can greatly improve the accuracy of early warning and the perfection of the early warning mechanism.

Description

Technical Field

[0001] This application relates to the field of shipping and transportation technology, and more specifically, to a method and system for monitoring and warning of ship collisions at dams and navigation facilities. Background Technology

[0002] Dams and navigation facilities, as crucial components of water conservancy projects, hold immeasurable value for flood control and regional economic development. However, with the booming development of water transportation and the surge in the number of vessels, the safety challenges facing dams and their navigation facilities have intensified. Especially in the upstream reservoir area and navigation facility zone, various fishing boats, cargo ships, and passenger vessels traverse these waters. If personnel error or severe weather causes vessels to lose control, it could pose a serious threat to the dam and navigation facilities.

[0003] To effectively monitor the navigation status of ships and prevent potential collisions with dams, ship collision monitoring and early warning systems have emerged. However, current ship collision monitoring and early warning systems mostly employ single sensor technologies, such as relying solely on cameras for image recognition or using only AIS systems to acquire ship message data. This single monitoring method, due to its inherent limitations, is difficult to meet complex and ever-changing monitoring needs. For example, while high-definition cameras can identify the position, size, and course of a ship, their recognition effectiveness is highly susceptible to the influence of lighting and weather conditions; radar monitoring can scan the water surface and perform cluster analysis to obtain information on moving targets, but it has blind spots and cannot identify static information about ships; while the AIS automatic identification system can sense the navigation status of ships, its effectiveness depends entirely on whether the ship has activated the relevant equipment as required.

[0004] To overcome the shortcomings of single monitoring methods, some existing technologies attempt to fuse multiple monitoring methods. However, most of these fusion methods merely perform a simple overlay of multi-source data at the decision-making level, limiting their performance improvement potential. Furthermore, the failure of any monitoring method significantly impacts the effectiveness of the entire approach. For example, some methods fuse position information obtained from radar and the Automatic Identification System (AIS) to acquire dynamic information about the vessel, then use computer vision technology to calculate the vessel's shape and size. However, this method still fails to fully utilize the complementary advantages of multi-source data and lacks in-depth processing and precise matching during the data fusion process.

[0005] In addition, some methods attempt to obtain two types of ship target bounding boxes by processing video and millimeter-wave radar information separately, and then fuse the data through a simple addition method. However, this method also suffers from insufficient data fusion and low accuracy, and it is difficult to achieve accurate prediction of ship behavior and timely early warning of dangerous behaviors.

[0006] In summary, existing ship collision monitoring and early warning technologies have many shortcomings in terms of sensor technology, data fusion, and early warning mechanisms. Especially in the specific application scenario of dams and navigation facilities, existing technologies struggle to effectively address complex and ever-changing monitoring needs, and cannot achieve deep fusion and accurate matching of multi-source data, thus limiting the accuracy and practicality of the early warning system. Therefore, developing a ship collision monitoring and early warning method that can comprehensively utilize multiple sensor technologies, achieve efficient fusion and accurate matching of multi-source data, and possess a comprehensive early warning mechanism is of great significance for ensuring the safe operation of dams and navigation facilities. Summary of the Invention

[0007] The purpose of this application is to provide a method and system for monitoring and warning of ship collisions in dams and navigation facilities. This system can comprehensively utilize multiple sensor technologies and achieve efficient fusion and accurate matching of multi-source data, thereby significantly improving the accuracy of the warning and the completeness of the warning mechanism.

[0008] This application is implemented as follows:

[0009] Firstly, this application provides a method for monitoring and early warning of ship collisions at dams and navigation facilities, comprising the following steps: acquiring image data, ship message data, and radar point cloud data respectively at a main monitoring point and a secondary monitoring point using cameras, an automatic ship identification system, and millimeter-wave radar, and performing spatiotemporal synchronization processing to project the radar point cloud data onto an image plane to obtain a radar image; and sending the image data and radar image into a ship identification network to identify the two-dimensional bounding boxes and first ship types of all ships within the ship identification area. Using the two-dimensional bounding boxes and first ship types of all ships within the ship identification area identified by the main and secondary monitoring points, performing epipolar geometry processing to obtain the three-dimensional bounding boxes and second ship types of the corresponding ships. Iterating and comparing whether the second ship type matches the ship type in the ship message data; if they do not match or there is no relevant type in the ship message data, issuing a level three warning. Based on multiple three-dimensional bounding boxes when a vessel passes through the vessel identification zone, its speed and course results for a future period of time are predicted. In response to the speed result exceeding the speed threshold set by the navigation facility, and / or the course result indicating a possible collision with the dam, a Level II warning is issued; and in response to any vessel triggering both the Level III and Level II warnings simultaneously, a Level I warning is issued.

[0010] Secondly, this application provides a ship collision monitoring and early warning system for dams and navigation facilities, comprising:

[0011] The first identification module is configured to: acquire image data, ship message data, and radar point cloud data respectively at the main monitoring point and the secondary monitoring point using cameras, an automatic ship identification system, and millimeter-wave radar; perform spatiotemporal synchronization processing to project the radar point cloud data onto an image plane to obtain a radar image; and send the image data and radar image into the ship identification network to identify the two-dimensional bounding boxes and first ship types of all ships within the ship identification area. The second identification module is configured to: use the two-dimensional bounding boxes and first ship types of all ships within the ship identification area identified by the main and secondary monitoring points to perform epipolar geometry processing to obtain the three-dimensional bounding boxes and second ship types of the corresponding ships. The first early warning module is configured to: iterate and compare the second ship type with the ship type in the ship message data; if they do not match or the ship message data does not contain a relevant type, issue a level three early warning. The second early warning module is configured to: predict the speed and heading of a vessel over a future period based on multiple three-dimensional bounding boxes when the vessel passes through the vessel identification zone; issue a level-two early warning in response to the speed exceeding the speed threshold set by the navigation facility and / or the heading indicating a possible collision with the dam; and issue a level-one early warning in response to any vessel triggering both a level-three and a level-two early warning simultaneously.

[0012] Thirdly, this application provides another ship collision monitoring and early warning system for dams and navigation facilities, comprising: a front-end monitoring module, located at a main monitoring point and a secondary monitoring point, including supplementary lighting for providing illumination to cameras at night, cameras for acquiring image data, an automatic ship identification system for acquiring ship message data, and millimeter-wave radar for acquiring radar point cloud data. The ship identification module performs spatiotemporal synchronization processing on the image data, ship message data, and radar point cloud data acquired by the main and secondary monitoring points to project the radar point cloud data onto an image plane to obtain a radar image, and sends the image data and radar image into a ship identification network to identify the two-dimensional bounding boxes and first ship types of all ships within the ship identification area; and uses the two-dimensional bounding boxes and first ship types of all ships within the ship identification area identified by the main and secondary monitoring points to perform epipolar geometry processing to obtain the three-dimensional bounding boxes and second ship types of the corresponding ships. The trajectory analysis module predicts the speed and heading of a ship over a future period based on multiple three-dimensional bounding boxes when the ship passes through the ship identification area. The early warning and handling module is used to iterate and compare whether the second vessel type is consistent with the vessel type in the vessel message data. If they are inconsistent or there is no relevant type in the vessel message data, a level 3 early warning is issued. In response to the speed result exceeding the speed threshold set by the navigation facility, and / or the heading result indicating a possible collision with the dam, a level 2 early warning is issued. In response to any vessel triggering both the level 3 and level 2 early warnings at the same time, a level 1 early warning is issued.

[0013] Fourthly, this application provides an electronic device including a memory for storing one or more programs; a processor; and, when the one or more programs are executed by the processor, implementing the method as described in any one of the first aspects above.

[0014] Fifthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any one of the first aspects above.

[0015] Compared with the prior art, this application has at least the following advantages or beneficial effects:

[0016] (1) Improved early warning accuracy: By comprehensively utilizing multiple sensor technologies, efficient fusion and accurate matching of multi-source data were achieved, improving the accuracy and robustness of ship identification. At the same time, through steps such as polar geometry processing and type consistency checks, the accuracy of early warning information was further ensured.

[0017] (2) Enhanced real-time early warning: This application is able to acquire the location, type, and dynamic status information of ships in real time, and predict the future speed and course of ships based on this information. This enables early warning information to be issued before ships approach dams or navigation facilities, providing operators with sufficient time to take preventive measures.

[0018] (3) Improved emergency response efficiency: By designing a multi-level early warning mechanism, this application can take corresponding emergency response measures based on the actual situation and potential risks of the vessel. This helps operators quickly identify and respond to potential collision risks, reducing the likelihood of accidents. Attached Figure Description

[0019] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of an embodiment of a ship collision monitoring and early warning method for dams and navigation facilities according to this application;

[0021] Figure 2 This is a schematic diagram showing the layout locations of the main monitoring point and the auxiliary monitoring point, as well as the ship identification zone, in one embodiment of this application.

[0022] Figure 3 This is a schematic diagram of the hardware framework of the main monitoring point in one embodiment of this application;

[0023] Figure 4 This is a schematic diagram of the hardware framework of the secondary monitoring point in one embodiment of this application;

[0024] Figure 5 This is a flowchart of a ship identification network based on a channel attention mechanism in one embodiment of this application;

[0025] Figure 6 This is a schematic diagram of the image feature encoder / radar image feature encoder framework in one embodiment of this application;

[0026] Figure 7 This is a schematic diagram of the framework of an embodiment of a ship collision monitoring and early warning method for dams and navigation facilities according to this application;

[0027] Figure 8 This is a structural block diagram of an electronic device provided in an embodiment of this application.

[0028] Icons: 201, Processor; 202, Memory; 203, Communication Interface. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0030] The following detailed description of some embodiments of this application is provided in conjunction with the accompanying drawings. Unless otherwise specified, the various embodiments and features described below can be combined with each other.

[0031] Example 1

[0032] Dams and navigation facilities, as crucial components of water conservancy projects, are severely threatened by the rapid growth of water traffic. Existing ship collision monitoring and early warning systems mainly rely on single-sensor technologies, such as cameras and AIS, but these technologies have significant limitations, such as susceptibility to weather conditions, blind spots, and dependence on the activation of ship equipment. Although some technologies attempt to integrate multiple monitoring methods, most only perform simple data overlay at the decision-making level, failing to fully utilize the complementary advantages of multi-source data, thus limiting the accuracy and practicality of the early warning system.

[0033] In response, this application provides a method for monitoring and early warning of ship collisions in dams and navigation facilities. This method can comprehensively utilize multiple sensor technologies and achieve efficient fusion and accurate matching of multi-source data, thereby significantly improving the accuracy of early warning and the completeness of the early warning mechanism.

[0034] Please see Figure 1 The method for monitoring and early warning of ship collisions in dams and navigation facilities includes the following steps:

[0035] Step S101: At the main monitoring point and the secondary monitoring point, image data, ship message data and radar point cloud data are acquired respectively using cameras, automatic identification system for ships and millimeter-wave radar, and spatiotemporal synchronization processing is performed to project the radar point cloud data onto the image plane to obtain radar images. The image data and radar images are then sent into the ship identification network to identify the two-dimensional bounding boxes of all ships in the ship identification area and the first ship type.

[0036] In step S101, image data, ship message data, and radar point cloud data are acquired at the main monitoring point and secondary monitoring point using cameras, an Automatic Identification System (AIS), and millimeter-wave radar, respectively. These data are aligned through spatiotemporal synchronization processing to ensure they exist within the same temporal and spatial framework. Next, the radar point cloud data is projected onto the image plane to form a radar image. This image data and radar image are then fed into a ship identification network, which uses deep learning algorithms to identify the two-dimensional bounding boxes (i.e., the location range of the ship in the image) and the first ship type (e.g., cargo ship, passenger ship) of all ships within the ship identification area. By comprehensively utilizing multiple sensor technologies, the system can acquire rich ship information, including location, type, and dynamic status. This fusion of multi-source data improves the accuracy and robustness of ship identification, providing a reliable data foundation for subsequent steps.

[0037] It should be noted that the ship message data includes ship static data and ship dynamic data. The ship static data includes the ship type and hull number, and the ship dynamic data includes the ship's latitude and longitude, heading and speed.

[0038] Step S102: Using the two-dimensional bounding boxes of all ships in the ship identification area identified by the main monitoring point and the secondary monitoring point and the first ship type, perform epipolar geometry processing to obtain the three-dimensional bounding box of the corresponding ship and the second ship type.

[0039] In step S102, using the two-dimensional bounding box obtained in step S101 and the first ship type, a three-dimensional bounding box (i.e., the ship's position range in three-dimensional space) and a second ship type are obtained through epipolar geometry processing (a geometric transformation method based on image matching). This step is essentially a further refinement and verification of the ship information, ensuring the accuracy of ship type identification. The extraction of the three-dimensional bounding box provides the system with specific spatial position information of the ship, which is crucial for subsequent speed and heading prediction. Specifically, when the same ship type identified by the main monitoring point and the secondary monitoring point is inconsistent, the second ship type is the first ship type identified by the secondary monitoring point.

[0040] Step S103: Iterate and compare whether the second vessel type is consistent with the vessel type in the vessel message data. If they are inconsistent or there is no relevant type in the vessel message data, issue a level 3 warning.

[0041] In step S103, the system iterates through and compares the second vessel type obtained in step S102 with the vessel type in the vessel message data. If they do not match, or if the vessel message data does not contain information about the relevant type (possibly due to reasons such as the vessel not having its AIS equipment turned on), the system issues a Level 3 warning. The issuance of a Level 3 warning is an initial response to potential safety risks. Through type consistency checks, the system can promptly detect type mismatches or missing message data, providing operators with warning information so they can take necessary preventative measures.

[0042] Step S104: Based on multiple three-dimensional bounding boxes when a vessel passes through the vessel identification zone, predict its speed and heading results for a future period of time (for example, an improved Kalman filter can be used for prediction). In response to the speed result exceeding the speed threshold set by the navigation facility, and / or the heading result indicating a possible collision with the dam, issue a Level II warning; and in response to any vessel triggering both a Level III warning and a Level II warning simultaneously, issue a Level I warning.

[0043] In step S104, the prediction of speed and heading enables real-time assessment of the vessel's navigation status, allowing for the timely detection of dangerous behaviors such as speeding or deviation from the course. The introduction of a multi-level early warning mechanism allows for appropriate emergency response measures to be taken based on different situations, improving the efficiency and accuracy of emergency response. The issuance of a Level 1 early warning is an emergency response to serious safety risks, ensuring that operators can take immediate action to avoid collisions.

[0044] In summary, the technical solutions of the above embodiments can achieve the following beneficial effects:

[0045] (1) Improved early warning accuracy: By comprehensively utilizing multiple sensor technologies, efficient fusion and accurate matching of multi-source data were achieved, improving the accuracy and robustness of ship identification. At the same time, through steps such as polar geometry processing and type consistency checks, the accuracy of early warning information was further ensured.

[0046] (2) Enhanced real-time early warning: This method can acquire real-time information on the location, type, and dynamic status of ships, and predict their future speed and course based on this information. This enables early warning information to be issued before ships approach dams or navigation facilities, providing operators with sufficient time to take preventative measures.

[0047] (3) Improved emergency response efficiency: By designing a multi-level early warning mechanism, this method can take corresponding emergency response measures based on the actual situation and potential risks of the vessel. This helps operators quickly identify and respond to potential collision risks, reducing the likelihood of accidents.

[0048] Based on the aforementioned solutions, such as Figure 2 As shown, in some implementations of this application, the main monitoring point is located at a high point of the navigation facility, the secondary monitoring point is located at a high point on the upstream side of the dam, i.e., the navigation facility, and the ship identification zone is located in the overlapping area of ​​the monitoring ranges of the main monitoring point and the secondary monitoring point.

[0049] It should be noted that existing research on active collision avoidance warning technology is mostly applied to bridges, with the monitoring range generally limited to a section of waterway, much smaller than the monitoring range required for dams and navigation facilities. Furthermore, the monitoring direction is typically towards the waterway. While this method can effectively determine whether a vessel has deviated from its course, the accuracy of vessel type identification is not high. This invention proposes deploying a main monitoring point at the navigation facility and a secondary monitoring point upstream of the dam and navigation facility. A vessel identification zone is set up in the overlapping area of ​​the main and secondary monitoring points. This allows for efficient and comprehensive monitoring of the navigation status of vessels upstream of the dam and navigation facility. Combining the monitoring data from the main and secondary monitoring points also enables a more comprehensive acquisition of the vessel morphology passing through the vessel identification zone, effectively identifying vessel types and obtaining more accurate three-dimensional bounding boxes for the vessels. Consequently, more precise vessel positions, headings, and speeds can be calculated.

[0050] The main monitoring points are carefully positioned at high points along the navigation facilities. This placement offers several significant advantages: First, the high points provide a wider field of view, ensuring coverage of the navigation facilities and key areas surrounding them; second, since navigation facilities are typically critical points for ship navigation, placing the main monitoring points here allows for more effective capture of ship navigation information; and finally, the high points also help reduce interference from ground obstacles, improving the clarity and accuracy of monitoring data. The secondary monitoring points are positioned at high points upstream of the dam, i.e., the navigation facilities. This placement also offers unique advantages: on the one hand, the secondary monitoring points can capture information about ships about to enter the navigation facility area in advance, providing the system with ample warning time; on the other hand, the upstream high points allow the secondary monitoring points to observe ship trajectories and states more comprehensively, helping the system to more accurately predict future ship behavior; furthermore, the placement of the secondary monitoring points complements the main monitoring points, jointly enhancing the coverage and accuracy of the entire monitoring and early warning system.

[0051] For example, such as Figure 3 and Figure 4 As shown, the equipment installed at the main and secondary monitoring points can include monitoring modules, communication modules, auxiliary modules, and processing modules. The equipment requires connection to mains power. Compared to the secondary monitoring points, the monitoring module at the main monitoring point is equipped with an AIS (Automatic Identification System) for acquiring ship message data within a 10km radius of the dam and navigation facilities. The auxiliary module includes an audible and visual alarm module, which directly alerts dam duty personnel and navigation facility managers upon triggering an early warning. The main monitoring point is located at a high point on the navigation facility with unobstructed views. If no suitable location exists on-site, a single pillar can be used for equipment installation. The secondary monitoring point is located at a high point on the right bank upstream of the dam with unobstructed views. If no suitable location exists on-site, a single pillar can be used for equipment installation. The overlapping monitoring range of the main and secondary monitoring points refers to the area where the millimeter-wave radar and high-definition camera monitoring ranges coincide. The ship identification zone generally needs to cover at least 100m along the river channel.

[0052] Based on the aforementioned scheme, in some implementations of this application, the spatiotemporal synchronization processing includes: Spatial synchronization processing steps: Before equipment installation, the same set of cameras and millimeter-wave radar are calibrated to the same coordinate system; after equipment installation, one or more known control points within the monitoring range of the main monitoring point and the secondary monitoring point are selected, and the geometric relationship of the cameras in the main monitoring point and the secondary monitoring point is calibrated using a stereo matching algorithm to obtain calibration results; the ship dynamic data acquired by the Automatic Identification System is converted to a preset three-dimensional Cartesian coordinate system, and the coordinate system transformation matrix of the cameras and millimeter-wave radar at the main monitoring point and the secondary monitoring point is calculated based on the calibration results to transform them to the three-dimensional Cartesian coordinate system. Time synchronization processing steps: Image data, ship message data, and radar point cloud data are synchronized in time using timestamps.

[0053] In the above implementation, spatial and temporal synchronization processing ensures that data from different monitoring points and sensor types are fused and analyzed within a unified spatial and temporal framework. This helps reduce data bias and errors, improving data accuracy and reliability. Furthermore, it allows for subsequent fusion and analysis of data from multiple data sources within a unified spatial and temporal framework, enabling early warning and decision-making even if one data source fails or exhibits data anomalies. This enhances robustness and fault tolerance.

[0054] For example, time synchronization of image data, ship message data, and radar point cloud data using timestamps may include the following steps: First, align the timestamps of all data to find the earliest and latest timestamps, determining the time range of the data. Then, divide the data into different time windows according to time resolution (e.g., per second, per half-second, etc.), with data within each time window originating from the same or similar time points. Next, within each time window, fuse the data from different data sources, including integrating and correlating image data, radar point cloud data, and ship message data. Finally, if a data source has no data within a time window (e.g., a camera malfunctions and stops working), anomaly handling is performed, such as using interpolation to estimate missing data or marking it as anomalous data.

[0055] like Figure 5 As shown, based on the aforementioned scheme, in some implementations of this application, the ship identification network includes: an image feature encoder, used to extract features from input image data to obtain a first feature; a radar image encoder, used to extract features from input radar images to obtain a second feature; a feature fusion processor, used to fuse the first feature and the second feature to obtain a fused feature; and a decoder, used to decode the fused feature to obtain a two-dimensional bounding box of all ships within the ship identification area and a first ship type.

[0056] In the above implementation, the ship identification network can capture more information about the ship through image data and radar images, thereby improving the accuracy of identification. Image data provides the ship's external features, while radar images provide its position and speed information; the two complement each other to enhance identification accuracy. Furthermore, radar images have the ability to penetrate adverse weather conditions such as clouds, fog, and rain, allowing the ship identification network to maintain high identification performance even in adverse weather conditions.

[0057] It should be noted that single-modal data often has limitations or is noisy. The ship identification network designed using the above implementation method can integrate information from both image and radar modalities, enabling the network model to acquire more feature representations. This helps enhance the robustness of the ship identification network to noise and anomalies, improving the final performance of ship identification. Furthermore, compared to the decision-level fusion strategy commonly used in existing technologies, this application adopts a feature-level fusion strategy. This approach offers a higher degree of information integration, avoids information loss, and integrates information from multiple modalities early in the data processing stage. This allows the ship identification network to learn the correlation information of all modalities simultaneously during training, enabling a deeper exploration of the intrinsic connections and synergistic effects between modalities.

[0058] For example, the image feature encoder can employ a convolutional neural network (CNN) architecture, extracting high-dimensional feature vectors, i.e., the first features, from the image through multiple layers of convolution, pooling, and activation function operations. The radar image encoder, operating in parallel with the image feature encoder, can also employ deep learning techniques to extract features from the radar image, obtaining the second features. These features contain key information such as the ship's position, speed, and heading, complementing the image features. The feature fusion unit is responsible for fusing the first and second features output from the image feature encoder and the radar image encoder. Fusion strategies can employ various methods such as concatenation, weighted summation, and attention mechanisms. Through feature fusion, the network can comprehensively utilize information from different sensors, improving the accuracy and robustness of ship identification. The decoder can employ structures such as convolutional transpose layers and upsampling layers to convert the high-dimensional feature vectors into a low-dimensional image space, achieving accurate ship identification and positioning.

[0059] Based on the aforementioned scheme, in some implementations of this application, the image feature encoder and radar image encoder include: based on the MobileNet V2 network, retaining the first 5 convolutional blocks of the MobileNet V2 network, adding a perforated pyramid pooling layer composed of three parallel perforated convolutional layers, using convolutional layers with aperture sizes of 3, 5 and 7 to extract features at different scales, and concatenating them with the unpooled features in the channel dimension to obtain the encoded first feature or second feature.

[0060] The first five convolutional blocks of the MobileNet V2 network have undergone extensive training and validation, proving their high effectiveness in image feature extraction. Therefore, the above implementation retains these five convolutional blocks. These blocks capture fundamental features in images, such as edges, textures, and shapes, which form the basis for subsequent image processing and recognition tasks.

[0061] Next, a dilated pyramid pooling layer consisting of three parallel dilated convolutional layers was added to the above implementation. Dilated convolution (also known as attenuated convolution) is a method that can increase the receptive field of the convolutional kernel without increasing computational cost. This layer uses convolutional layers with 3, 5, and 7 different aperture sizes to extract features at different scales. Features at different scales are crucial for understanding and recognizing complex scenes because they can capture objects or structures of different sizes and shapes.

[0062] The extracted features at different scales are then concatenated with the unpooled features along the channel dimension. This concatenation operation allows the network to utilize feature information from different scales and the original convolutional blocks simultaneously, resulting in richer and more comprehensive feature representations. These concatenated features are used as either the pre-encoded first or second features, depending on whether a regular image or a radar image is being processed.

[0063] In summary, the above implementation utilizes attenuated pyramid pooling layers to extract multi-scale features, which helps improve the model's accuracy in recognizing objects of different sizes and shapes in images. Although attenuated convolutional layers are added, the solution maintains high computational efficiency because these layers are parallel and extended from the lightweight architecture of MobileNet V2. Furthermore, this design captures information at different scales by varying the receptive field size of the convolutional kernels. In the attenuated pyramid pooling layers, each convolutional layer has a different aperture, allowing it to learn features at different scales from both image data and radar images. In this way, the designed ship recognition network can simultaneously focus on local details and global contextual information, enabling it to effectively capture features at multiple scales when handling tasks involving objects of different sizes, avoiding the problem of insufficient information at a single scale.

[0064] like Figure 6 As shown, based on the aforementioned scheme, in some implementations of this application, the feature fusion processor utilizes a channel attention mechanism to fuse the first feature and the second feature. Specifically, this includes: concatenating the first feature and the second feature along the channel dimension, and feeding the concatenated feature into a convolutional layer with a kernel of 1 to compress the channels, obtaining a first compressed feature; feeding the first compressed feature into a global averaging pooling layer to compress the first compressed feature to channel-level statistical information, obtaining a second compressed feature; using two nonlinear fully connected layers to adjust the dimensions of the first feature and the first compressed feature, and adding them element-wise to obtain image-radar semantic features; and using a multilayer perceptron with a sigmoid activation function to calculate the channel attention weights of the image-radar semantic features, and multiplying the obtained attention weights element-wise with the first compressed feature to obtain the fused feature.

[0065] In the above implementation method, the first feature is first... Second feature Concatenation is performed along the channel dimension. This is done to merge information from different sources, providing a rich feature set for subsequent processing. Next, the concatenated features are fed into a convolutional layer with a kernel of 1. This layer compresses the channels, reducing the number of channels in the feature (compressing 2n to n channels), thus obtaining the first compressed feature. This step aims to reduce computational complexity while preserving key information. Next, the first compressed feature... The data is fed into a global average pooling layer P. This layer compresses the features to channel-level statistics, i.e., it averages all elements over each channel to obtain the second compressed feature. This step is to extract global contextual information, providing a foundation for subsequent calculation of channel attention weights. Then, two non-linear fully connected layers f1 and f2 are used to process the first feature. and the first compression feature Dimensional adjustments are made to match the dimensions of the two features. Then, the two features are added element-wise to obtain the image-radar semantic features. This step is to further fuse features from different sources and extract their common semantic information. Next, a multilayer perceptron U with a sigmoid activation function is used to calculate the channel attention weights W of the image-radar semantic features. n The function of this MLP is to assign a weight to each channel based on its importance. The above process can be explained by the following equation:

[0066]

[0067] Finally, the obtained attention weights W n With the first compression feature Element-wise multiplication yields the fused features. This step is to weight the features according to the channel attention weights, thereby emphasizing important channels and suppressing unimportant ones.

[0068] In summary, the technical solution described above can extract and fuse features from different sources, emphasizing important channel features and thus enhancing model performance. This is reflected in improved recognition accuracy and reduced false positive rate. Furthermore, by utilizing a channel attention mechanism, this solution can automatically learn and emphasize important channel features while suppressing unimportant ones. This helps improve the effectiveness of feature fusion, making the fused features more accurate and robust.

[0069] Furthermore, by adaptively weighting the channel dimensions of the features and dynamically assigning different weights to each modality feature in the above implementation, the ship identification network can focus on features that significantly contribute to the ship identification task while suppressing irrelevant or redundant features. Compared with the channel attention modules commonly used in existing technologies, this application employs a first compressed feature... and first feature The image-radar semantic features are obtained through processing. Because radar images lack semantic information while image data contains rich semantic information, this design can more effectively utilize the first feature. Use semantic information to enhance the first compressed feature The representation of features relevant to the ship identification task in the final fused features.

[0070] Example 2

[0071] This application provides a ship collision monitoring and early warning system for dams and navigation facilities, comprising:

[0072] The first identification module is configured to: acquire image data, ship message data, and radar point cloud data respectively at the main monitoring point and the secondary monitoring point using cameras, an automatic ship identification system, and millimeter-wave radar; perform spatiotemporal synchronization processing to project the radar point cloud data onto an image plane to obtain a radar image; and send the image data and radar image into the ship identification network to identify the two-dimensional bounding boxes and first ship types of all ships within the ship identification area. The second identification module is configured to: use the two-dimensional bounding boxes and first ship types of all ships within the ship identification area identified by the main and secondary monitoring points to perform epipolar geometry processing to obtain the three-dimensional bounding boxes and second ship types of the corresponding ships. The first early warning module is configured to: iterate and compare the second ship type with the ship type in the ship message data; if they do not match or the ship message data does not contain a relevant type, issue a level three early warning. The second early warning module is configured to: predict the speed and heading of a vessel over a future period based on multiple three-dimensional bounding boxes when the vessel passes through the vessel identification zone; issue a level-two early warning in response to the speed exceeding the speed threshold set by the navigation facility and / or the heading indicating a possible collision with the dam; and issue a level-one early warning in response to any vessel triggering both a level-three and a level-two early warning simultaneously.

[0073] For the specific implementation process of the above system, please refer to the ship collision monitoring and early warning method for dams and navigation facilities provided in Example 1, which will not be repeated here.

[0074] Please see Figure 7 This application also provides a ship collision monitoring and early warning system for dams and navigation facilities, comprising:

[0075] The front-end monitoring module, located at the main and secondary monitoring points, includes supplementary lighting for cameras at night, cameras for acquiring image data, an automatic ship identification system (AIS) for acquiring ship message data, and a millimeter-wave radar for acquiring radar point cloud data. The ship identification module performs spatiotemporal synchronization processing on the image data, ship message data, and radar point cloud data acquired from the main and secondary monitoring points to project the radar point cloud data onto an image plane to obtain a radar image. This image data and radar image are then sent to the ship identification network to identify the two-dimensional bounding boxes and first ship types of all ships within the ship identification area. Furthermore, using the two-dimensional bounding boxes and first ship types identified by the main and secondary monitoring points, epipolar geometry processing is performed to obtain the corresponding three-dimensional bounding boxes and second ship types. The trajectory analysis module predicts the speed and heading of a ship over a future period based on multiple three-dimensional bounding boxes as it passes through the ship identification area. The early warning and handling module is used to iterate and compare whether the second vessel type is consistent with the vessel type in the vessel message data. If they are inconsistent or there is no relevant type in the vessel message data, a level 3 early warning is issued. In response to the speed result exceeding the speed threshold set by the navigation facility, and / or the heading result indicating a possible collision with the dam, a level 2 early warning is issued. In response to any vessel triggering both the level 3 and level 2 early warnings at the same time, a level 1 early warning is issued.

[0076] In the above embodiments, by integrating multiple sensor technologies, comprehensive and multi-angle monitoring of ships is achieved, improving the accuracy of ship identification and the precision of trajectory analysis. By setting up a multi-level early warning mechanism, potential ship collision risks can be detected and addressed in a timely manner, effectively preventing safety accidents. Furthermore, the system can automatically trigger early warnings and take corresponding measures, reducing the delay of manual intervention and improving the speed and efficiency of emergency response.

[0077] Based on the aforementioned scheme, in some implementations of this application, the ship identification module is also used to traverse and compare the position information and the second ship type corresponding to the three-dimensional bounding box, and whether they are consistent with the ship type in the ship message data, so as to determine whether the target ship has correctly activated the automatic identification system.

[0078] In the above implementation, the ship identification module is further enhanced to traverse and compare the position information corresponding to the three-dimensional bounding box and whether the second ship type is consistent with the ship type in the ship message data, thereby determining whether the target ship has correctly started and used the Automatic Identification System (AIS).

[0079] Specifically, the vessel identification module compares the location information provided by the 3D bounding box with the location information in the vessel's message data. Since the AIS system can transmit vessel location information in real time, comparing the differences allows for a preliminary assessment of the AIS system's operational status. In addition to location information, the vessel identification module also compares the second vessel type, obtained through image recognition and radar data fusion, with the vessel type declared in the vessel message data. If they do not match, it may indicate that the AIS system has been misconfigured or deliberately tampered with. Through automatic comparison and verification, it can promptly detect and report vessels that have not correctly activated their AIS systems, improving the compliance and safety of vessel navigation.

[0080] Based on the aforementioned scheme, in some implementations of this application, the early warning and handling module includes responses to Level 1, Level 2, and Level 3 early warnings, specifically including active handling and passive alarms. Active handling includes deploying active collision avoidance devices, patrol boat docking, and temporarily closing navigation facilities. Passive alarms include audible and visual alarms, platform alarms, and SMS alarms.

[0081] In the above-mentioned implementation method, the combination of active processing and passive alarms enables rapid response to early warning information and the implementation of effective measures to reduce the risk of ships colliding with dams. The diverse handling methods (such as active collision avoidance devices, patrol boat docking, and adjustments to navigation rules) and alarm methods (such as audible and visual alarms, platform alarms, and SMS alarms) together constitute a comprehensive safety assurance system.

[0082] Specifically, deploying active collision avoidance devices means that when the system issues a Level 1 warning, active collision avoidance devices, such as automatic water spray devices and audible warning systems, are immediately activated or adjusted to physically or audibly prevent or warn ships that may collide with the dam.

[0083] Patrol vessel berthing refers to dispatching patrol vessels to quickly approach and intercept vessels that may collide with the dam, ensuring their safe departure from the danger zone through manual intervention. Temporary closure of navigation facilities refers to situations where, in extreme circumstances, such as multiple vessels simultaneously triggering a Level 1 warning, and active collision avoidance devices and patrol vessels are unable to effectively handle the situation, the navigation facilities may be temporarily closed until the danger has passed.

[0084] Adjusting navigation rules refers to adjusting navigation rules in a navigation area based on early warning information, such as limiting speed or changing channels, in order to reduce the risk of ships colliding with dams.

[0085] Strengthening monitoring and notification refers to increasing the monitoring of the warning area and notifying relevant vessels of the warning information through platforms or SMS to remind them to pay attention to safety.

[0086] Vessel identity verification refers to verifying the identity of a vessel that has triggered a Level 3 warning, confirming whether it is sailing legally, and checking whether its AIS system is functioning properly.

[0087] Strengthening safety awareness means sending safety information to vessels that have triggered a Level 3 warning, reminding them to comply with navigation regulations and ensure navigational safety.

[0088] Audible and visual alarms refer to the installation of audible and visual alarm devices at key locations on dams and navigation facilities. When the system issues a warning, the audible and visual alarms are activated immediately to alert on-site personnel in a visually intuitive way.

[0089] Platform alarms refer to the display of early warning information on the monitoring center or relevant management department's platform, including the early warning level, information about the vessel that triggered the early warning, and the early warning area, so that management personnel can respond quickly.

[0090] SMS alerts are sent to pre-defined recipients (such as managers, patrol boat crew members, etc.) with alert information including the alert level, information about the vessel that triggered the alert, and suggested handling measures, ensuring that relevant personnel can receive the alert information in a timely manner and take appropriate actions.

[0091] Please see Figure 7 Based on the aforementioned scheme, in some implementations of this application, the system further includes a real-time monitoring module for using a camera to monitor the navigation of ships within the monitoring range in real time, and / or includes a data management module for storing the raw data and processing results acquired during system operation.

[0092] The above implementation further expands the system's functionality by introducing a real-time monitoring module and a data management module to achieve continuous monitoring of ship navigation and comprehensive data management. To facilitate communication between modules, a data transmission module, consisting of 4G and fiber optic communication, can also be included to transmit data acquired by the front-end monitoring module to the real-time monitoring module, ship identification module, trajectory analysis module, data management module, and early warning and response module.

[0093] The real-time monitoring module utilizes cameras in the front-end monitoring module to monitor the navigation of vessels within its monitoring range in real time. This module can capture and display real-time images of vessels, helping managers intuitively understand their navigation status, location, and potential safety hazards. The data management module stores the raw data and processing results acquired during system operation, including image data, radar point cloud data, vessel message data, vessel identification results, trajectory analysis results, and early warning and response records. This module enables data classification, storage, querying, exporting, and backup, providing data support for subsequent system analysis and optimization.

[0094] Example 3

[0095] Please see Figure 8 This application provides an electronic device including at least one processor 201 and at least one memory 202. The processor 201 and memory 202 are directly connected to each other, or communicate with each other through a communication interface 203, or are electrically connected through one or more communication buses or signal lines to achieve data transmission or interaction. The memory 202 stores program instructions executable by the processor 201, which calls the program instructions to execute a ship collision monitoring and early warning method for dams and navigation facilities. For example, it can be implemented as follows:

[0096] At the main monitoring point and secondary monitoring point, image data, ship message data, and radar point cloud data are acquired using cameras, an Automatic Identification System (AIS), and millimeter-wave radar, respectively. These data are then processed in a spatiotemporal synchronization manner to project the radar point cloud data onto an image plane, generating a radar image. The image data and radar image are then fed into a ship identification network to identify the two-dimensional bounding boxes and first ship types of all ships within the ship identification area. Using the two-dimensional bounding boxes and first ship types identified at the main and secondary monitoring points, epipolar geometry processing is performed to obtain the corresponding three-dimensional bounding boxes and second ship types. The second ship type is then compared with the ship type in the ship message data. If they do not match or the ship message data does not contain a relevant type, a Level 3 warning is issued. Based on multiple three-dimensional bounding boxes when a vessel passes through the vessel identification zone, its speed and course results for a future period of time are predicted. In response to the speed result exceeding the speed threshold set by the navigation facility, and / or the course result indicating a possible collision with the dam, a Level II warning is issued; and in response to any vessel triggering both the Level III and Level II warnings simultaneously, a Level I warning is issued.

[0097] The memory 202 may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0098] The processor 201 can be an integrated circuit chip with signal processing capabilities. The processor 201 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0099] Understandable. Figure 8 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 8 The more or fewer components shown, or having the same Figure 8 The different configurations shown. Figure 8 The components shown can be implemented using hardware, software, or a combination thereof.

[0100] Example 4

[0101] This application provides a computer-readable storage medium storing a computer program thereon. When executed by a processor 201, the computer program implements a method for monitoring and warning of ship collisions in dams and navigation facilities. For example, it implements:

[0102] At the main monitoring point and secondary monitoring point, image data, ship message data, and radar point cloud data are acquired using cameras, an Automatic Identification System (AIS), and millimeter-wave radar, respectively. These data are then processed in a spatiotemporal synchronization manner to project the radar point cloud data onto an image plane, generating a radar image. The image data and radar image are then fed into a ship identification network to identify the two-dimensional bounding boxes and first ship types of all ships within the ship identification area. Using the two-dimensional bounding boxes and first ship types identified at the main and secondary monitoring points, epipolar geometry processing is performed to obtain the corresponding three-dimensional bounding boxes and second ship types. The second ship type is then compared with the ship type in the ship message data. If they do not match or the ship message data does not contain a relevant type, a Level 3 warning is issued. Based on multiple three-dimensional bounding boxes when a vessel passes through the vessel identification zone, its speed and course results for a future period of time are predicted. In response to the speed result exceeding the speed threshold set by the navigation facility, and / or the course result indicating a possible collision with the dam, a Level II warning is issued; and in response to any vessel triggering both the Level III and Level II warnings simultaneously, a Level I warning is issued.

[0103] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] It will be apparent to those skilled in the art that this application is not limited to the details of the exemplary embodiments described above, and that this application can be implemented in other specific forms without departing from the spirit or essential characteristics of this application. Therefore, the embodiments should be considered illustrative and non-limiting in all respects, and the scope of this application is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within this application. No reference numerals in the claims should be construed as limiting the scope of the claims.

Claims

1. A method for monitoring and early warning of ship collisions in dams and navigation facilities, characterized in that, Includes the following steps: At the main monitoring point and the secondary monitoring point, image data, ship message data and radar point cloud data are acquired by cameras, automatic identification system for ships and millimeter-wave radar respectively, and spatiotemporal synchronization processing is performed to project the radar point cloud data onto the image plane to obtain radar images. The image data and radar images are then sent into the ship identification network to identify the two-dimensional bounding boxes of all ships in the ship identification area and the first ship type. Using the two-dimensional bounding boxes of all ships within the ship identification area identified by the main monitoring point and the secondary monitoring point, and the first ship type, epipolar geometry processing is performed to obtain the three-dimensional bounding box of the corresponding ship and the second ship type. The system iterates through and compares the second vessel type with the vessel type in the vessel message data. If they do not match or there is no relevant type in the vessel message data, a Level 3 warning is issued. Based on multiple three-dimensional bounding boxes when a vessel passes through the vessel identification zone, its speed and heading results for a future period of time are predicted. In response to the speed result exceeding the speed threshold set by the navigation facility, and / or the heading result indicating a possible collision with the dam, a level-two warning is issued. And in response to any vessel simultaneously triggering both a Level 3 and a Level 2 warning, a Level 1 warning will be issued.

2. The method according to claim 1, characterized in that, The main monitoring point is located at a high point on the navigation facility, and the secondary monitoring point is located at a high point on the upstream side of the dam, i.e., the navigation facility. The vessel identification zone is located in the overlapping area of ​​the monitoring ranges of the main monitoring point and the secondary monitoring point.

3. The method according to claim 1, characterized in that, The spatiotemporal synchronization process includes: Spatial synchronization processing steps: Before equipment installation, calibrate the same set of cameras and millimeter-wave radar to the same coordinate system; after equipment installation, select one or more known control points within the monitoring range of the main monitoring point and the secondary monitoring point, and use a stereo matching algorithm to calibrate the geometric relationship of the cameras in the main monitoring point and the secondary monitoring point to obtain calibration results; convert the ship dynamic data acquired by the Automatic Identification System to a preset three-dimensional rectangular coordinate system, and calculate the coordinate system transformation matrix of the cameras and millimeter-wave radar at the main monitoring point and the secondary monitoring point based on the calibration results to transform them to the three-dimensional rectangular coordinate system; Time synchronization processing steps: Use timestamps to synchronize image data, ship message data, and radar point cloud data.

4. The method according to claim 1, characterized in that, The ship identification network includes: An image feature encoder is used to extract features from input image data to obtain the first feature; A radar image encoder is used to extract features from an input radar image to obtain a second feature; The feature fusion processor is used to fuse the first feature and the second feature to obtain the fused feature. The decoder is used to decode the fused features to obtain the two-dimensional bounding boxes of all ships within the ship identification area and the first ship type.

5. The method according to claim 4, characterized in that, The image feature encoder and radar image encoder include: Based on the MobileNet V2 network, the first 5 convolutional blocks of the MobileNet V2 network are retained, and a dilated pyramid pooling layer consisting of three parallel dilated convolutional layers is added. Convolutional layers with aperture sizes of 3, 5 and 7 are used to extract features at different scales, and these features are concatenated with the unpooled features in the channel dimension to obtain the encoded first or second features.

6. The method according to claim 4, characterized in that, The feature fusion processor utilizes a channel attention mechanism to fuse the first feature and the second feature, specifically including: The first feature and the second feature are concatenated along the channel dimension, and the concatenated feature is fed into a convolutional layer with a kernel of 1 to compress the channels, thus obtaining the first compressed feature. The first compressed feature is fed into a global averaging pooling layer to compress the first compressed feature to channel-level statistical information, thus obtaining the second compressed feature; Two nonlinear fully connected layers are used to adjust the dimensions of the first feature and the first compressed feature, and then they are added element by element to obtain the image-radar semantic features; The channel attention weights of the image-radar semantic features are calculated using a multilayer perceptron with a sigmoid activation function. The obtained attention weights are then multiplied element-wise with the first compressed feature to obtain the fused feature.

7. A ship collision monitoring and early warning system for dams and navigation facilities, characterized in that, include: The front-end monitoring module, located at the main monitoring point and the secondary monitoring point, includes a supplementary light for providing illumination to the camera at night, a camera for acquiring image data, an automatic identification system for acquiring ship message data, and a millimeter-wave radar for acquiring radar point cloud data. The ship identification module is used to perform spatiotemporal synchronization processing on image data, ship message data and radar point cloud data acquired by the main monitoring point and the secondary monitoring point, so as to project the radar point cloud data onto the image plane to obtain a radar image, and send the image data and radar image into the ship identification network to identify the two-dimensional bounding box and the first ship type of all ships in the ship identification area. And by using the two-dimensional bounding boxes of all ships in the ship identification area identified by the main monitoring point and the secondary monitoring point and the first ship type, the epipolar geometry is processed to obtain the three-dimensional bounding box of the corresponding ship and the second ship type. The trajectory analysis module is used to predict the ship's speed and heading over a future period of time based on multiple three-dimensional bounding boxes when the ship passes through the ship identification zone. The early warning and handling module is used to iterate and compare whether the second vessel type is consistent with the vessel type in the vessel message data. If they are inconsistent or there is no relevant type in the vessel message data, a level 3 early warning is issued. And in response to the speed result exceeding the speed threshold set by the navigation facility, and / or the heading result indicating a possible collision with the dam, a Level II warning shall be issued; And in response to any vessel simultaneously triggering both a Level 3 and a Level 2 warning, a Level 1 warning will be issued.

8. The system according to claim 7, characterized in that, The ship identification module is also used to traverse and compare the position information and second ship type corresponding to the three-dimensional bounding box with the ship type in the ship message data to determine whether the target ship has correctly activated the automatic identification system.

9. The system according to claim 7, characterized in that, The early warning and response module includes responses to Level 1, Level 2, and Level 3 early warnings, specifically including proactive handling and passive alarms. Proactive handling includes deploying active collision avoidance devices, patrol boat docking, and temporarily closing navigation facilities. Passive alarms include audible and visual alarms, platform alarms, and SMS alarms.

10. The system according to claim 7, characterized in that, The system also includes a real-time monitoring module for monitoring the navigation of ships within the monitoring range using cameras, and / or a data management module for storing the raw data and processing results acquired during system operation.

Images