A ship collision warning method and system based on collaborative perception of vision and AIS data

Abstract

This invention belongs to the field of ship collision avoidance technology and discloses a ship collision avoidance early warning method and system based on collaborative perception of visual and AIS data. First, an enhanced YOLO (E-YOLO) target detector is proposed, capable of high-precision, real-time ship detection in complex environments, combined with the DeepSORT algorithm for ship tracking and visual trajectory extraction. Second, a homography coordinate transformation method based on multi-frame static images is proposed, which can achieve the conversion between pixel coordinate system and world coordinate system without precise camera calibration, improving the fusion accuracy of visual and AIS data. The K-D tree algorithm is used to match AIS and visual information, projecting AIS data onto visual coordinates to achieve accurate matching of heterogeneous data. The system sets rules for judging ship violations and collision warnings, which can identify behaviors such as speeding and abnormal AIS shutdown and trigger warnings. Experiments demonstrate that the proposed method has strong adaptability and robustness, and can effectively improve the level of ship monitoring and traffic management in bridge areas.

Description

Technical Field

[0001] This invention belongs to the field of ship collision avoidance technology, and particularly relates to a ship collision avoidance early warning method and system based on visual and AIS data collaborative perception. Background Technology

[0002] Currently, waterway transportation, as a crucial mode of logistics, carries nearly 90% of the world's bulk cargo volume. The United Nations and the Conference on Development predict that maritime trade will grow at an annual rate of 2.4% between 2022 and 2026. Therefore, strengthening the situational awareness of vessels in navigation, maintaining order on the waterways, and issuing warnings to vessels at risk of collision can significantly reduce the occurrence of maritime accidents.

[0003] The key to safe navigation lies in timely and accurate perception of changes in the navigation environment, guiding ship operators to make correct maneuvers and decisions. Vessel Traffic Service (VTS) utilizes technologies such as traffic situational awareness, analysis, and monitoring to effectively monitor ship movement, significantly reducing the probability of traffic accidents. However, the inland waterway navigation environment is more complex than open sea waters, including coastal structures, bridges across rivers, and navigation aids within the waterway, posing greater challenges to safe navigation. Acquiring ship navigation information is crucial for collision prevention. Currently, patrol boats, synthetic aperture radar (SAR), closed-circuit television (CCTV), and automatic identification systems (AIS) are widely used to obtain ship navigation information. While patrol boats provide the most direct ship navigation information, they waste significant manpower and financial resources of maritime authorities. SAR imagery is unaffected by weather conditions and can operate continuously day and night, but its significant time lag makes it unsuitable for real-time monitoring of inland waterway vessels. CCTV provides richer visual information about ships compared to SAR, allowing for a more intuitive understanding of traffic conditions on the water. However, it cannot acquire information about the spatial state and other intrinsic attributes of ships, such as dimensions, speed, and navigation. Nevertheless, visual sensors offer advantages such as long detection range, intuitiveness, and real-time monitoring. In the context of intelligent shipping development, video surveillance systems have been widely applied in inland waterways, significantly improving situational awareness and the ability to monitor abnormal ship behavior. AIS is a self-reporting surveillance system for dynamic and static information about ships. According to the International Convention for the Safety of Life at Sea (SOLAS) established by the International Maritime Organization, all international vessels with a gross tonnage exceeding 300 and all passenger ships must be equipped with AIS. However, most AIS devices have low reporting frequency and accuracy, and there are even instances of unauthorized AIS shutdown, failing to meet the need for real-time collision avoidance warnings for ships in monitored waters. Despite these issues with reliability and operability, AIS information still provides valuable information for ship navigation. Based on the advantages and disadvantages of different maritime sensor equipment, our work tends to combine visual and ship AIS data to help maritime regulators judge the movement status of ships in the waters around bridges and achieve real-time collision avoidance warnings for ships in the waters around bridges.

[0004] Based on the above analysis, the problems and shortcomings of the existing technology are as follows:

[0005] (1) Data from a single sensor has limitations and is difficult to meet the requirements for real-time and accurate perception. Data from a single sensor is also difficult to meet the requirements for real-time collision avoidance and early warning of ships in complex inland waterway navigation environments.

[0006] (2) Existing ship situation awareness systems are poorly adapted to the complex environment of inland waterways. Due to the more complex navigation environment of inland waterways, the influence of terrain and waterway restrictions is greater, and the ship navigation path and dynamic characteristics are more difficult to predict than in the marine environment. This makes it impossible for VTS systems based on a single data source to effectively improve collision avoidance and early warning capabilities.

[0007] (3) The reliability and real-time performance of AIS data are insufficient, making it difficult to support accurate collision avoidance warnings. Although AIS provides valuable information in ship navigation, its data update frequency is limited by the ship's speed. For example, at low speeds, the update interval of AIS data is long, making it difficult to meet the requirements of high-precision situational awareness. In addition, AIS information may be lost due to equipment failure, malicious shutdown, or signal interference, further reducing the reliability of ship dynamic information. Therefore, relying solely on AIS for ship situational analysis is insufficient to achieve high-precision, real-time collision avoidance warnings.

[0008] (4) Existing collision avoidance warning methods lack multi-sensor fusion strategies, resulting in incomplete information. Warning systems that rely on a single data source cannot comprehensively assess the risks of ship navigation and cannot provide accurate and reliable collision avoidance decision support.

[0009] (5) Existing methods do not fully combine visual and AIS data for deep collaborative perception. Existing technologies fail to effectively combine these two types of data, resulting in a low level of intelligence in collision avoidance warnings, which cannot meet the needs of intelligent shipping in complex environments. Summary of the Invention

[0010] To address the problems existing in the prior art, this invention provides a ship collision avoidance early warning method and system based on collaborative perception of visual and AIS data.

[0011] This invention is implemented as follows: a ship collision avoidance early warning method and system based on collaborative perception of visual and AIS data. The ship collision avoidance early warning method based on collaborative perception of visual and AIS data includes:

[0012] S1. The E-YOLO target detection model is used to detect ships, and the DeepSORT algorithm is combined to track ships and obtain visual trajectory data.

[0013] S2 performs AIS data preprocessing, including data cleaning, regulatory area filtering, and navigation status filtering;

[0014] S3, through a coordinate transformation method based on homography, projects the preprocessed AIS data into the image coordinate system to achieve the fusion of visual trajectory data and AIS data;

[0015] S4. A collision warning strategy is adopted to establish a matching relationship between AIS data and visual trajectory data in order to improve the accuracy of ship motion trajectory analysis.

[0016] S5, based on the vessel's navigation behavior, the rules of the regulated waters, and the preset collision risk assessment model, issues an early warning to vessels at risk of collision and triggers an audible and visual alarm.

[0017] Furthermore, the E-YOLO object detection model adopts the FasterNetF structure and integrates the SA mechanism; it uses WIoU as the bounding box regression loss function, and uses weighted nonmaximum suppression to calculate the scores between bounding boxes and remove prediction results with lower scores.

[0018] The FasterNetF structure adopts a lightweight design, which introduces the FReLU activation function by integrating PWConv and the optimized FasterNet architecture;

[0019] The FasterNetF architecture consists of PWConv, FasterNet modules, and FreLU activation layers; PWConv is used to increase the number of channels in the feature map, the FasterNet modules are used to provide feature information, and the FreLU activation function is used for pixel-level spatial information modeling.

[0020] The FasterNet module contains one PConv and two PWConv layers, which integrate feature information through skip connections; only batch normalization and ReLU layers are added between the PWConv layers.

[0021] The FReLU activation function is expressed as:

[0022]

[0023] Where T(x) c,i,j () represents the defined funnel condition, and (i,j) is used to describe the spatial location information of pixels in the input feature map, where i and j represent the row index and column index in the feature map, respectively. This represents a window on the c-th channel centered at (i,j). This represents the parameters shared in the c-th channel;

[0024] The SA module consists of four key parts:

[0025] (1) Feature grouping

[0026] set up The input feature map is divided into G groups along the channel direction, i.e., X = [X1, X2, ..., X...]. G ], Then each group is further divided into two different branches along the channel direction. The first branch generates a channel attention map by utilizing the interrelationships between different channels of the input features, while the other branch generates a spatial attention map by utilizing the spatial relationships between features.

[0027] (2) Channel attention mechanism

[0028] The channel attention mechanism aims to capture the interdependencies between input feature channels. In the SA module, global average pooling (GAP) and the sigmoid activation function are combined. The GAP operation aggregates global feature information, while the sigmoid function guides adaptive learning within the model. The calculation...

[0029] The process can be summarized as follows:

[0030]

[0031] X i ′1=σ(f c (s))·X i1 =σ(W1s+b1)·X i1 (3)

[0032] Where f gp (·) represents the operation of Global Average Pooling (GAP), σ(·) is the Sigmoid activation function, and f c (·)=W·x+b, These are the learnable parameters of the network;

[0033] (3) Spatial attention

[0034] Spatial attention complements channel attention; group normalization is used to compute spatial statistics.

[0035] X i ′2=σ(W2·GN(X i2 )+b1)·X i2 (4)

[0036] in These are the learnable parameters of the network;

[0037] (4) Aggregation

[0038] After obtaining the channel attention map and spatial attention map of the input features, these two branches are concatenated and fused to obtain the output feature map of each group, i.e. The features of each group are aggregated, and the final feature map is obtained by channel shuffling.

[0039] Furthermore, the E-YOLO network is used to detect ship targets in vision and obtain a series of bounding boxes, which are represented as follows:

[0040] BBox = {bbox1, bbox2, ..., bbox} l ...bbox L} (5)

[0041] Where L represents the number of bounding boxes, bbox l It is the position of the i-th bounding box, where (x tl ,y tl ) and (x br ,y br These are the pixel indices of the top-left and bottom-right corners of the bounding box, respectively.

[0042] Each bounding box is fed into DeepSORT, and based on Kalman filtering and the Hungarian algorithm, the state of the target in the next frame is updated and associated. Finally, the bounding box information of the target in each frame is output (x tl ,y tl The bounding box is defined as follows: (w, h), confidence level (conf); where (w, h) are the width and height of the bounding box, respectively; then a series of visual trajectories of ships are obtained, represented as:

[0043]

[0044] Furthermore, the WIoU constructs a distance attention mechanism loss and introduces a dynamic non-monotomic focusing mechanism. By assigning corresponding gradient gains to anchor boxes with different outlier degrees, it effectively balances the model's fitting ability to low-quality and high-quality samples; the calculation formula is shown in (7):

[0045]

[0046] Among them, R WIoU It describes the distance between the center points of the two boxes, and R WIoU ∈[1,e),R WIoU The focus will be on learning the L-shaped anchor frame of ordinary mass. IoU W g and H g Calculate using constants, eliminate R WIoU The gradient that hinders convergence is generated; β is the outlier, the smaller the value, the higher the quality of the anchor box; r is the gradient gain; by assigning different sizes of r, the impact of low-quality samples on training is effectively reduced. The moving average of momentum m; and since It is dynamic, and the anchor frame mass division is also dynamic, which makes L WIoU During training, it can make the gradient gain allocation strategy that best suits the current situation at every moment;

[0047] The weighted nonmaximum suppression (WNMS) is expressed as follows:

[0048]

[0049] Among them, Score i Represents the i-th bounding box b i The detection scores, where Max represents the highest-scoring bounding box, and N represents the detection score. t It is the preset IoU threshold, while WIoU is the similarity calculation between two bounding boxes.

[0050] Furthermore, the specific process of AIS data preprocessing is as follows:

[0051] (1.1) Data cleaning; A data cleaning filter (DCF) was designed. clean To clean the data; the processed AIS data D clean It can be represented in the following form:

[0052] D clean =F clean (D row (9)

[0053] Among them, D row This represents the raw AIS data collected in the experiment. T represents the time span of data collection. E represents the necessary elements required for the recorded AIS data, including MMSI, LON, LAT, COG, and SOG. ε is the tolerance range for each element. First, AIS data that is too old is deleted, and then those AIS records that are missing E or have a deviation greater than ε are deleted.

[0054] (1.2) Surveillance Area Filtering; A surveillance area filter (SAF) was designed based on the camera's location, filtering through the surveillance area F. SAF Filtered AIS data D SAF It is expressed as follows:

[0055] D SAF =F SAF (D clean ,L camera ,θ) (10)

[0056] Among them, L camera The location of the camera is indicated by θ, which represents the radius of the restricted area, i.e., the threshold for rejecting ships that are more than θ nautical miles away from the camera. The distance between the ship and the camera is calculated using the following formula:

[0057]

[0058] Where R represents the radius of the Earth, LON V ,LAT V,LON C and LAT C These are the longitude and latitude coordinates of the ship and the camera, respectively.

[0059] (1.3) Navigation status filtering; extract the COG and SOG of the ship from the AIS data; design a navigation status filter (NSF) to remove ship records that are moored or in irrelevant navigation status.

[0060] Furthermore, the homography-based coordinate transformation method includes:

[0061] (2.1) Feature point selection: Manually select several ships with known AIS data from the image;

[0062] (2.2) Homography matrix estimation; the homography matrix H is calculated based on the corresponding AIS data and image coordinates using the Direct Linear Transform (DLT) method; the mathematical model of the homography matrix H is:

[0063] Where (lon,lat) and (x,y) represent geographic coordinates and their corresponding image coordinates, respectively; H is the homography matrix, h ij Let be the corresponding parameter in H, and s be the scale factor; it can be seen that at least 4 sets of non-collinear control points are needed to estimate the homography matrix H, and can be solved using equation (13);

[0064]

[0065] To accurately estimate the homography matrix H, several ships will be manually selected, and their visual and positioning data will be easily extracted from camera and AIS messages, respectively; at this point, the two-dimensional pixel coordinates [x] will be obtained. pixel ,y pixel ] T and the corresponding two-dimensional world coordinates [lon] world ,lat world ] T Then, the parameters of the homography matrix H are obtained according to formulas (12) and (13);

[0066] (2.3) Coordinate projection; The calculated homography matrix H is used to project the AIS data into the pixel space to align it with the visual tracking results.

[0067] Furthermore, the collision warning strategy establishes a matching relationship between AIS data and visual trajectory data based on the KD tree algorithm;

[0068] The received AIS information is mapped to image coordinates using a homography matrix and matched with visually detected targets; historical AIS data with kinematic information is used to predict the trajectory; the ship's real-time position P(LON) is calculated.rt ,LAT rt The following is represented:

[0069] P(LON rt ,LAT rt ) = [LON h ,LAT h ]+[SOG lon SOG lat ]·(T rt -T h (14)

[0070] Among them [LON h ,LAT h [SOG] is the location in the latest received AIS information. lon SOG lat [T] is the ground velocity component along the longitude and latitude directions, calculated based on the ship's heading relative to the ground; rt and T h These are real-time and historical timestamps, respectively.

[0071] AIS data will be predicted using formula (14) to match the timestamps of the video, thereby aligning the two heterogeneous pieces of information in time and space. Then, the Euclidean distance will be used directly as the similarity between the two pieces of information, and the nearest neighbor search based on the KD tree algorithm will be used to assign the AIS data to the nearest bounding box in the image to obtain more complete information on the ship's motion state. After matching, the system will update the ship's motion trajectory in real time, including position, speed (SOG), heading (COG), and acceleration dynamic data.

[0072] Furthermore, the warning rules of the collision warning strategy are as follows:

[0073] (3.1) Warning of abnormal AIS device shutdown: Abnormal shutdown of AIS device will prevent the ship from being effectively perceived in terms of navigation dynamics and static information; if the ship is detected in the visual monitoring system, but the corresponding AIS data cannot be matched after the continuous monitoring time exceeds 60 seconds, it will be judged as "No AIS" and a warning will be triggered; in this case, the dynamic position information of the ship can be effectively estimated from the visual data and the ship's navigation status can be captured in a timely manner.

[0074] (3.2) Warning of navigation prohibition under severe weather conditions: Based on the atmospheric scattering model and Retinex theory, visual surveillance scenarios in foggy and low-light environments were simulated respectively. The received AIS data was directly projected onto the visual surveillance water area through coordinate transformation. If a vessel is found to have a dynamic navigation status, a "navigation prohibition warning" is issued and the identity and navigation information of the illegal vessel are recorded.

[0075] (3.3) Ship overspeed warning: The normal speed for upstream ships is set to [3,8] knots and the normal speed for downstream ships is set to [5,13] knots; by acquiring the ship's speed SOG information in real time and comparing the ship's current speed with the set safe speed range, if the speed exceeds the threshold, it is determined as an "overspeed warning"; if the overspeed duration exceeds the threshold for more than 30 seconds, a "high-priority overspeed warning" is issued.

[0076] (3.4) Vessel Collision Warning: Based on the inland waterway navigation rules, the geodetic distance between vessels is calculated in real time using global coordinate information, and the vessel distance is set. Dis The threshold is 50 meters; if the real-time distance between vessels is less than this threshold, it is considered a "collision warning"; if the warning lasts for more than 20 seconds, a "high-priority collision warning" is issued; the scale information of the encountering vessels is comprehensively considered, and the vessel distance is set accordingly. Dis The calculation rules are as follows:

[0077] Vessel Dis =Dis - (L1 + L2) / 2 (15)

[0078] Where Dis is the real-time geodetic distance between ships, which can be calculated by equation (11); L1 and L2 are the scale information of the encountering ships, referring to the ship lengths.

[0079] Another object of the present invention is to provide a ship collision warning system based on visual and AIS data collaborative perception, which applies the aforementioned ship collision warning method based on visual and AIS data collaborative perception, the system comprising:

[0080] The ship target detection and tracking module is used to detect ship targets based on the E-YOLO model and track the ship's trajectory by combining the DeepSORT algorithm.

[0081] The AIS data preprocessing module is used to clean up abnormal AIS data, filter valid data within the monitored area, and filter data that is irrelevant to navigation status, so as to ensure the accuracy of AIS information.

[0082] The data fusion module is used to map AIS data to the image coordinate system based on the homography transformation method, and to match AIS data with visual detection results using the KD tree algorithm;

[0083] The early warning decision module is used to analyze ship behavior, detect abnormal navigation conditions, and trigger corresponding collision warnings.

[0084] Another object of the present invention is to provide a computer device comprising a memory and a processor, the memory storing a computer program, which, when executed by the processor, causes the processor to perform the following steps:

[0085] S1. The E-YOLO target detection model is used to detect ships, and the DeepSORT algorithm is combined to track ships and obtain visual trajectory data.

[0086] S2 performs AIS data preprocessing, including data cleaning, regulatory area filtering, and navigation status filtering;

[0087] S3, through a coordinate transformation method based on homography, projects the preprocessed AIS data into the image coordinate system to achieve the fusion of visual trajectory data and AIS data;

[0088] S4. A collision warning strategy is adopted to establish a matching relationship between AIS data and visual trajectory data in order to improve the accuracy of ship motion trajectory analysis.

[0089] S5, based on the vessel's navigation behavior, the rules of the regulated waters, and the preset collision risk assessment model, issues an early warning to vessels at risk of collision and triggers an audible and visual alarm.

[0090] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solution to be protected by this invention are as follows:

[0091] First,

[0092] This invention proposes an enhanced YOLO (E-YOLO) target detection algorithm. By optimizing the YOLOv7 structure and introducing the FasterNetF module and ShuffleAttention (SA) mechanism, the detection accuracy is improved while reducing the computational complexity, making it suitable for ship detection in complex inland waterway environments.

[0093] The FasterNetF module reduces computational resource consumption while maintaining high detection accuracy, enabling this method to be deployed on shipboard computing equipment and achieve high real-time target detection.

[0094] The ShuffleAttention (SA) mechanism enhances the detection capability of small targets, improves the robustness of detection in complex backgrounds and occlusion conditions, and makes ship target detection more accurate.

[0095] The DeepSORT algorithm is used for ship tracking. Based on the Kalman filter and Hungarian matching algorithm, it can effectively deal with complex scenarios such as occlusion and ship intersection, and ensure the integrity of the ship target's trajectory in the visual data.

[0096] This innovation solves the problems of insufficient accuracy in single-target detection and poor adaptability to small targets in visual data, enabling more accurate acquisition of ship movement trajectories in complex water environments, and providing reliable data support for subsequent data fusion and collision warning.

[0097] This invention proposes a coordinate transformation method based on homography, which realizes the mapping between the world coordinate system (AIS data) and the pixel coordinate system (visual data) by calculating the homography matrix between pairs of ship feature points.

[0098] This method does not require precise camera calibration; it only requires acquiring multiple frames of still images and estimating the homography matrix using the Direct Linear Transform (DLT) method, thereby achieving coordinate unification between different data sources.

[0099] Calculating the homography matrix using multiple frames of images, compared to single-frame calculation methods, can effectively reduce projection errors caused by factors such as water surface fluctuations and lighting changes, and improve coordinate transformation accuracy.

[0100] In the error analysis experiment, when this method projects world coordinates to pixel coordinates, the average error is reduced by 61%, making the fusion of AIS and visual data more accurate.

[0101] This technical solution solves the problem of information fusion difficulty caused by the different coordinate systems of AIS data and visual data, improves the accuracy of the collision avoidance system, and enables the real-time position, speed and other information of ships to be presented intuitively in the visual system, thereby enhancing the situational awareness of maritime regulatory personnel.

[0102] Secondly, through an automated intelligent identification system, this technology can monitor the navigation status of vessels in real time, accurately identify abnormal behavior, and trigger early warnings. For example, if a vessel fails to match an AIS signal within 60 seconds, the system will trigger a "NoAIS" warning, alerting regulatory personnel to pay attention to abnormal vessels and preventing illegal navigation or loss of contact. Furthermore, in severe weather conditions, such as when visibility is below 1500 meters or winds exceed force 7, the system will automatically trigger a no-navigation warning and track violating vessels to ensure the safety of maritime traffic.

[0103] This invention is highly adaptable and applicable to various navigation environments, improving the stability of vessel monitoring. The system can operate accurately in bridge areas, narrow channels, and complex shipping environments, and integrates multiple data sources (AIS, visual recognition, meteorological data, etc.) to ensure highly robust vessel early warning capabilities. Even under abnormal conditions such as vessel obstruction, severe weather, or illegal AIS shutdown, the system can still operate stably, achieving continuous maritime monitoring, improving the reliability of collision avoidance warnings, and reducing sudden risks.

[0104] This invention improves maritime regulatory efficiency and significantly reduces labor costs. Through automated data analysis and intelligent early warning, it reduces reliance on manual patrols and enhances regulatory efficiency. Compared to traditional patrol boat methods, this system can reduce labor costs by more than 60%, while expanding regulatory coverage and ensuring the accuracy of maritime traffic management. Furthermore, early warning information can be automatically pushed to the maritime management system, supporting remote real-time monitoring and further improving the modernization level of waterway safety management.

[0105] Third, the technology of this invention can be widely applied in various fields such as port management, maritime supervision, shipping companies, and intelligent ships, meeting the market's urgent need for efficient and accurate ship monitoring and collision avoidance early warning technologies. The global ship monitoring and safety systems market is in a phase of rapid growth, and the market size is expected to grow at an average annual rate of over 10% in the next five years. The technical solution of this invention has enormous commercial potential and can be promoted and applied as intelligent monitoring systems, software platforms, and other products, empowering the development of the smart shipping industry.

[0106] Maritime accidents not only cause direct damage to ships and cargo, but can also lead to serious environmental pollution and economic losses ranging from millions to hundreds of millions of US dollars. The intelligent collision avoidance early warning system provided by this invention can effectively identify abnormal ship navigation behavior, issue real-time warnings, and help ports, shipping companies, and maritime management departments take timely measures to reduce the accident rate, minimize economic losses, and protect the marine ecological environment, thus possessing extremely high social value.

[0107] Currently, most existing collision avoidance warning technologies rely on a single data source, such as AIS, radar, or CCTV monitoring, which is insufficient to address data gaps, obstructions, and interference. This invention proposes for the first time a ship collision avoidance warning method based on collaborative perception of visual detection and AIS data. By comprehensively utilizing multi-sensor information fusion technology, it improves the ship's situational awareness capabilities and maintains high-precision warnings even in complex water environments, effectively solving the problems of incomplete information and low warning accuracy in traditional methods.

[0108] This invention innovatively proposes a coordinate transformation method based on homography matrices to improve data fusion accuracy. Traditional ship target positioning technology typically relies on precise camera calibration, a complex process with significant errors. This invention utilizes homography matrices to calculate the coordinate transformation relationship between ship target points, requiring only a few static images to complete the mapping from pixel coordinates to world coordinates. This improves data fusion accuracy, reduces coordinate transformation errors, and provides more accurate decision-making basis for ship monitoring and collision avoidance early warning. Attached Figure Description

[0109] Figure 1 This is a schematic diagram of a ship collision avoidance early warning framework provided in an embodiment of the present invention;

[0110] Figure 2This is a schematic diagram of the labeled convolution and partial convolution structure provided in the embodiments of the present invention;

[0111] Figure 3 This is a schematic diagram of the FasterNet module structure provided in an embodiment of the present invention;

[0112] Figure 4 This is a schematic diagram of the FasterNetF module structure provided in an embodiment of the present invention;

[0113] Figure 5 This is a schematic diagram of the SA module structure provided in an embodiment of the present invention;

[0114] Figure 6 This is a schematic diagram of the channel attention structure provided in an embodiment of the present invention;

[0115] Figure 7 This is a schematic diagram illustrating the IoU principle between the predicted bounding box and the ground truth bounding box provided in an embodiment of the present invention;

[0116] Figure 8 This is a schematic diagram of the E-YOLO network structure provided in an embodiment of the present invention;

[0117] Figure 9 This is a flowchart of the DeepSORT algorithm provided in an embodiment of the present invention;

[0118] Figure 10 This is a flowchart of ship target tracking based on E-YOLO provided in an embodiment of the present invention;

[0119] Figure 11 This is a schematic diagram of the pixel coordinates and world coordinates of a vessel in a regulated waterway provided in an embodiment of the present invention;

[0120] Figure 12 These are ship images from the dataset provided in this embodiment of the invention;

[0121] Figure 13 This is a visualization diagram of the detection results of a single target ship provided in an embodiment of the present invention;

[0122] Figure 14 This is a schematic diagram of the detection results of multi-target ships provided in an embodiment of the present invention;

[0123] Figure 15 This is a visual schematic diagram of the detection results of small-scale ships provided in an embodiment of the present invention;

[0124] Figure 16 This is a visual schematic diagram of the ship detection results under obstruction conditions provided in an embodiment of the present invention;

[0125] Figure 17This is a schematic diagram of the projection result between pixel coordinates and world coordinates provided in the embodiments of the present invention; (a) AIS to pixel coordinate conversion, (b) pixel to world coordinate conversion;

[0126] Figure 18 These are schematic diagrams illustrating the error distribution of different methods provided in the embodiments of the present invention; (a) error distribution of world coordinates projected onto pixel coordinates; (b) error distribution of pixel coordinates projected onto world coordinates.

[0127] Figure 19 This is a schematic diagram of the monitoring range of the bridge area waterway provided in an embodiment of the present invention;

[0128] Figure 20 This is a schematic diagram of the visual and AIS collaborative perception results provided in an embodiment of the present invention;

[0129] Figure 21 This is a schematic diagram of a navigation warning experiment provided in an embodiment of the present invention: (a) warning of abnormal shutdown of AIS equipment; (b) warning of ship navigation restriction under severe weather conditions; (c) warning of ship speeding; and (d) warning of collision between ships. Detailed Implementation

[0130] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0131] I. Explanatory and Illustrative Embodiments. To enable those skilled in the art to fully understand how the present invention is specifically implemented, this section provides an explanatory and illustrative description of the embodiments described in the claims.

[0132] like Figure 1 As shown, this embodiment of the invention provides a ship collision avoidance early warning method based on collaborative perception of visual and AIS data. The ship collision avoidance early warning method based on collaborative perception of visual and AIS data includes:

[0133] S1 uses an enhanced YOLO (E-YOLO) target detection model to detect ships and combines it with the DeepSORT algorithm to track ships and obtain visual trajectory data;

[0134] S2, by using a coordinate transformation method based on homography, AIS data is projected into the image coordinate system to achieve the fusion of visual trajectory data and AIS data;

[0135] S3 employs a collision warning strategy, establishing a matching relationship between AIS data and visual trajectory data to improve the accuracy of ship motion trajectory analysis;

[0136] S4, based on the vessel's navigation behavior, the rules of the regulated waters, and the preset collision risk assessment model, issues an early warning to vessels at risk of collision and triggers an audible and visual alarm.

[0137] Furthermore, the enhanced YOLO (E-YOLO) target detection model adopts the FasterNetF structure and integrates the Shuffle Attention (SA) mechanism to improve detection accuracy while reducing computational complexity, making it suitable for real-time detection by shipborne computing devices.

[0138] In the original YOLOv7 backbone network, the reuse of the E-ELAN module can lead to increased computational overhead and reduced efficiency. To address this issue, this invention replaces the E-ELAN module with the FasterNetF structure. FasterNetF employs a lightweight design, integrating PWConv and an optimized FasterNet architecture to maintain rich feature representations while reducing computational resources. Furthermore, the FReLU activation function is introduced to enhance pixel-level spatial information modeling capabilities and improve the model's nonlinear learning ability. By embedding the FasterNetF structure into the backbone network, the computational complexity of YOLOv7 is significantly reduced while maintaining or even improving detection performance.

[0139] Second, in the multi-scale detection stage of E-YOLO, the model introduces the Shuffle Attention (SA) mechanism to enhance detection performance at different scales. The SA mechanism is integrated into the detection framework, selectively emphasizing important features at different scales, thereby improving the model's ability to detect small targets and ships in complex backgrounds. By dynamically reallocating spatial and channel information, SA enables the network to focus on key regions while suppressing irrelevant background noise. This enhancement is particularly beneficial in maritime environments, where detecting small and distant ships is challenging, especially in complex backgrounds. By applying SA to the multi-scale detection process, the detection accuracy for ships at different scales is significantly improved while maintaining computational efficiency.

[0140] Third, given the presence of low-quality ship images in the collected samples, the model employs Wise-IoU (WIoU) as the bounding box regression (BBR) loss function to facilitate better model training. Simultaneously, weighted non-maximum suppression (WNMS) is used to calculate scores between bounding boxes, and predictions with lower scores are removed. This approach helps balance the localization and regression capabilities of E-YOLO across samples of varying quality, thereby significantly improving detection accuracy.

[0141] (I) YOLO Structure Reconstruction

[0142] Partial convolution (PConv) applies traditional convolution (Conv) to only a portion of the input channels to extract spatial features, while keeping the remaining channels unchanged, such as... Figure 2 As shown, this simultaneously reduces memory access and computational redundancy. Specifically, for an input feature map... Calculate the output feature map using a k×k convolution kernel. In this case, the floating-point operation of traditional convolution is h×w×k. 2 ×c 2 PConv's FLOPs are only Where c p This refers to the number of channels in traditional convolution operations. In the experiments, c... p = c / 4, therefore PConv's FLOPs are only 1 / 16 of those of traditional convolution, significantly reducing computational complexity and improving efficiency.

[0143] To fully utilize the information from all channels in the feature map, partial convolution (PConv) and point-wise convolution (PWConv) are combined. Based on this, a FasterNet module is constructed for fast and efficient processing, such as... Figure 3 As shown, various computer vision tasks such as detection, classification, and segmentation are effectively achieved by stacking FasterNet modules. Each FasterNet module contains one PConv and two PWConv layers, integrating feature information through skip connections. Furthermore, batch normalization (BN) and ReLU layers are added only between the PWConv layers to ensure the diversity of feature maps.

[0144] To reduce the computational and parameter overhead of the YOLOv7 model and fully utilize channel feature information, this invention draws on the concept of FasterNet and proposes a new FasterNetF structure, such as... Figure 4 As shown. This structure, embedded in the YOLOv7 object detection framework, effectively reduces the computational burden while maintaining detection accuracy. The FasterNetF structure consists of PWConv, FasterNet, and FReLU activation layers. PWConv increases the number of channels in the feature map, while the introduced FasterNet module provides richer feature information, thereby enhancing the model's learning ability. The FReLU activation function achieves pixel-level spatial information modeling, effectively improving the model's non-linear learning ability. It can be expressed as:

[0145]

[0146] Where T(x) c,i,j) represents the defined funnel condition, and (i,j) is used to describe the spatial location information of pixels in the input feature map, where i and j represent the row index and column index in the feature map, respectively. This represents a window on the c-th channel centered at (i,j). This indicates the parameters shared in the c-th channel.

[0147] (II) Strategies to Enhance Feature Extraction Capabilities

[0148] This invention enhances the model's ability to detect small targets, distant targets, and ships in complex backgrounds at different scales by integrating the SA module into the multi-scale detection stage of YOLOv7. For example... Figure 5 As shown, the SA module consists of four key parts:

[0149] (1) Feature grouping

[0150] set up The input feature map is divided into G groups along the channel direction, i.e., X = [X1, X2, ..., X...]. G ], Then each group is further divided into two different branches along the channel direction. The first branch generates a channel attention map by utilizing the interrelationships between different channels of the input features, while the other branch generates a spatial attention map by utilizing the spatial relationships between features.

[0151] (2) Channel attention

[0152] Channel attention mechanism, such as Figure 6 As shown. Its purpose is to capture the interdependencies between input feature channels. The SA module combines Global Average Pooling (GAP) and the Sigmoid activation function. The GAP operation aggregates global feature information, while the Sigmoid function guides adaptive learning within the model. The calculation process is summarized below:

[0153]

[0154] X i ′1=σ(f c (s))·X i1 =σ(W1s+b1)·X i1 (3)

[0155] Where f gp (·) represents the operation of Global Average Pooling (GAP), σ(·) is the Sigmoid activation function, and f c (·)=W·x+b, These are the learnable parameters of the network.

[0156] (3) Spatial attention

[0157] Spatial attention complements channel attention. Here, group normalization (GN) is used to compute spatial statistics:

[0158] X i ′2=σ(W2·GN(X i2 )+b2)·X i2 (4)

[0159] in These are the learnable parameters of the network.

[0160] (4) Aggregation

[0161] After obtaining the channel attention map and spatial attention map of the input features, these two branches are concatenated and fused to obtain the output feature map of each group, i.e. The features of each group are aggregated, and the final feature map is obtained by channel shuffling.

[0162] (III) Enhanced Nonmaximum Suppression Based on Wise-IoU

[0163] Traditional loss functions primarily rely on aggregation and bounding box regression (BBR) related metrics. CIoU specifically considers scale information and the aspect ratio of the bounding box, aiming to enhance the bounding box's fitting ability. However, in real-world applications, the collected samples contain low-quality targets, and the foreground and background information are unbalanced. Blindly enhancing the bounding box's fitting ability for low-quality samples, such as considering geometric information like the distance between the predicted box and the bounding box's center point, or aspect ratio, can suppress the overall model performance to some extent. To address this issue, this invention employs Wise-IoU as the model's BBR loss calculation, balancing the model's localization and regression capabilities for samples of different quality, thereby improving the model's generalization performance.

[0164] IoU characterizes the degree of overlap between the model's predicted bounding box and the ground truth bounding box, such as Figure 7 As shown. When using L IoU When used as the bounding box regression (BBR) loss in the detection model, it exhibits strong detection performance for both large and small targets in the sample. IoU The loss function formula is shown in equation (5):

[0165]

[0166] Among them, [w,h] and [w gt ,h gt ] corresponds to the width and height of the predicted bounding box and the ground truth bounding box, respectively, S u It is the sum of the areas of the two. However, when there is no overlapping area between the bounding boxes, L IoUIt will become ineffective and affect model training.

[0167] In order to overcome L IoU To address the drawbacks of traditional methods, Wise-IoU constructs a distance attention mechanism loss and introduces a dynamic non-monotomic focusing mechanism (FM). By assigning corresponding gradient gains to anchor boxes with different outlier degrees, it effectively balances the model's fitting ability to low-quality and high-quality samples. The calculation formula is shown in equation (6):

[0168]

[0169] Among them, R WIoU It describes the distance between the center points of the two boxes, and R WIoU ∈[1,e),R WIoU The focus will be on learning the L-shaped anchor frame of ordinary mass. IoU W g and H g Calculate using constants, eliminate R WIoU The gradient that hinders convergence is generated; β is the outlier, the smaller the value, the higher the quality of the anchor box; r is the gradient gain; by assigning different sizes of r, the impact of low-quality samples on training can be effectively reduced. Let be the sliding average value of momentum m. And since... It is dynamic, and the anchor frame mass division is also dynamic, which makes L WIoU During training, it can make the gradient gain allocation strategy that best suits the current situation at every moment.

[0170] Meanwhile, to achieve higher quality results during detection, Non-Maximum Suppression (NMS) has become an important component of object detection algorithms. It sorts predicted bounding boxes based on detection scores and removes unnecessary bounding boxes based on overlap constraints. However, in practical ship detection applications, challenges such as low-quality dataset images and excessively close target spacing lead to problems like target overlap and low detection accuracy. To alleviate these problems, this invention will... WIoU Combined with NMS, it forms a method called Weighted Non-Maximum Suppression (WNMS). By calculating the scores between bounding boxes, WNMS removes predicted boxes with lower scores. To better understand this, WNMS can be described as:

[0171]

[0172] Among them, Score i Represents the i-th bounding box b iThe detection scores, where Max represents the highest-scoring bounding box, and N represents the detection score. t It is the preset IoU threshold, while WIoU is the similarity calculation between two bounding boxes.

[0173] (iv) E-YOLO Network Structure

[0174] Based on the above analysis, although the YOLOv7 model exhibits superior performance, practical ship target detection applications require consideration of real-time performance and detection accuracy, as well as the deployment of shipborne computer equipment. Therefore, the model's complexity and number of parameters should be further reduced. Thus, this invention, based on the YOLOv7 model, constructs a novel ship target detection model, E-YOLO, for complex inland waterway environments. Its network structure is as follows: Figure 8 As shown.

[0175] In the original YOLOv7 backbone network, the reuse of the E-ELAN module can lead to increased computational overhead and reduced efficiency. To address this issue, this invention replaces the E-ELAN module with the FasterNetF structure. FasterNetF employs a lightweight design, integrating PWConv and an optimized FasterNet architecture to maintain rich feature representations while reducing computational resources. Furthermore, the FReLU activation function is introduced to enhance pixel-level spatial information modeling capabilities and improve the model's nonlinear learning ability. By embedding the FasterNetF structure into the backbone network, the computational complexity of YOLOv7 is significantly reduced while maintaining or even improving detection performance.

[0176] In the multi-scale detection phase of E-YOLO, the model introduces the Shuffle Attention (SA) mechanism to enhance detection performance at different scales. Integrated into the detection framework, SA selectively emphasizes important features at different scales, thereby improving the model's ability to detect small targets and ships in complex backgrounds. By dynamically reallocating spatial and channel information, SA enables the network to focus on key regions while suppressing irrelevant background noise. This enhancement is particularly beneficial in maritime environments, where detecting small, distant ships is challenging, especially in complex backgrounds. By applying SA to the multi-scale detection process, the detection accuracy for ships at different scales is significantly improved while maintaining computational efficiency.

[0177] Given the presence of low-quality ship images in the collected samples, the model employs Wise-IoU (WIoU) as the bounding box regression (BBR) loss function to facilitate better model training. Simultaneously, weighted nonmaximum suppression (WNMS) is used to compute scores between bounding boxes, and predictions with lower scores are removed. This approach helps balance the localization and regression capabilities of E-YOLO across samples of varying quality, thereby significantly improving detection accuracy.

[0178] (V) Ship target tracking based on E-YOLO

[0179] After detection, ship tracking is performed using the detected ship bounding boxes. This invention uses the DeepSORT algorithm to implement the tracking task. The core idea of ​​the DeepSORT algorithm is to use a uniform linear Kalman filter model to predict the target's motion state in the next frame, and to combine the (x...) in the state vector X... center ,y center The four values ​​(r, h) are used as direct observations. The predicted state of the target in the next frame is then updated based on the detection results. The Hungarian algorithm is used to correlate the predicted and detected targets, primarily focusing on motion and appearance information. A cascaded matching method is employed, assigning greater weight to more frequently occurring targets. For trackers, detectors, and trackers in unconfirmed states that still haven't matched after cascaded matching, Intersection over Union (IoU) matching is performed. This process is repeated for each frame until the end of the video frame sequence. The specific flow of this algorithm is as follows: Figure 9 As shown.

[0180] DeepSORT is a detection-and-track algorithm that extracts the appearance features of a target using a pre-trained convolutional neural network (CNN) and represents them as a high-dimensional tensor. Therefore, a robust detector is crucial for accurate target tracking, effectively reducing false positives and false negatives during the tracking process. To achieve high-performance tracking of ship targets, this invention uses E-YOLO as the detector and the detection results of each frame as input to DeepSORT. Through data association and matching, effective tracking of ship targets is achieved. The specific algorithm flow is as follows: Figure 10 As shown.

[0181] Figure 10 The E-YOLO-based ship target tracking process is described in detail below:

[0182] This paper proposes an enhanced ship target tracking algorithm based on the combination of E-YOLO and DeepSORT. The algorithm mainly consists of three core modules: target detection, data association, and trajectory management. It can achieve stable and efficient ship target tracking in complex water environments.

[0183] Step 1: Ship Target Detection Stage. First, the algorithm extracts consecutive image frames from the video stream and uses them as input for the detection stage. Then, E-YOLO performs deep feature extraction on the input ship targets and performs multi-scale feature fusion to improve detection accuracy. Finally, three output layers with different resolutions are used to achieve accurate detection of ship targets at different scales.

[0184] Step Two: Data Association Stage. First, the ship targets detected by E-YOLO are fed into the DeepSORT tracker for feature extraction, obtaining the appearance and motion features of each target. Then, based on Kalman filtering and the Hungarian matching algorithm, the matching degree between targets is calculated to match newly detected targets with existing tracked targets, ensuring the consistency of target IDs. Furthermore, when a target briefly disappears (e.g., due to occlusion by other ships), the algorithm uses Kalman filtering to predict its trajectory, enabling re-matching in the next frame and improving tracking stability.

[0185] Step 3: Trajectory Management Phase. Once a vessel target is correctly matched, its historical trajectory information is updated to ensure trajectory consistency. If a target is not detected for an extended period, the algorithm removes it from the tracking list to reduce false tracking.

[0186] Step 4: Output the tracking results of the ship target and overlay them onto the video frame for further ship visual trajectory analysis and extraction.

[0187] (vi) Mapping between pixel coordinates and world coordinates

[0188] Homography-based coordinate transformation methods include:

[0189] Feature point selection: Manually select several ships with known AIS coordinates from the image.

[0190] Homography matrix estimation. The homography matrix H is calculated using the Direct Linear Transform (DLT) method based on the corresponding AIS coordinates and image coordinates.

[0191] Coordinate projection. The calculated matrix H is used to project the AIS coordinates into the pixel space, aligning them with the visual tracking results.

[0192] Given that ship motion occurs on a nearly planar water surface, homography matrices provide an efficient and accurate method for converting data between world coordinates and pixel coordinates.

[0193] The mathematical model for using homography to obtain the transformation relationship between two images on the same plane can be expressed as:

[0194]

[0195] Where (lon,lat) and (x,y) represent the geographic coordinates and their corresponding image coordinates, respectively. H is the homography matrix, h ij Let be the corresponding parameter in H, and s be the scale factor. It can be seen that estimating the homography matrix H requires at least 4 sets of non-collinear control points, which can be solved using equation (9).

[0196]

[0197] To accurately estimate the homography matrix H, such as Figure 11 As shown, several ships will be manually selected in the experiment, and their visual and positioning data can be easily extracted from the camera and AIS messages, respectively. At this point, the two-dimensional pixel coordinates [x...] can be obtained. pixel ,y pixel ] T and the corresponding two-dimensional world coordinates [lon] world ,lat world ] T Then, according to formulas (8) and (9), the parameters of the homography matrix H can be easily obtained.

[0198] Furthermore, the collision warning strategy includes:

[0199] A rule-based collision warning system was implemented based on unified vessel trajectory data. This framework identifies potential risks based on vessel behavior and predefined maritime safety regulations. The warning strategy consists of three key modules:

[0200] AIS and Visual Data Fusion Based on KD-Tree: To enhance situational awareness, a K-Dimensional Tree (KD-tree) is employed to efficiently match AIS data with corresponding visual detection results, ensuring real-time fusion of multi-source ship information. The KD-tree is a data structure that efficiently processes multi-dimensional spatial information, enabling rapid retrieval and matching of high-dimensional data, making it suitable for scenarios involving the fusion of ship AIS data and visual detection results. This approach achieves precise alignment between AIS data and visual detection results, thereby providing more comprehensive and accurate information support for ship collision warning.

[0201] Violation detection. This framework identifies violations such as speeding and unauthorized AIS shutdown by monitoring vessel behavior.

[0202] Environmental Navigation Restrictions. Severe weather conditions, such as dense fog and strong winds, can significantly impact vessel safety. To address these challenges, this framework integrates real-time meteorological data to automatically implement navigation restrictions, such as suspending traffic in low visibility conditions. This automated weather monitoring and response mechanism effectively reduces navigation risks caused by deteriorating weather conditions, enhancing maritime traffic safety.

[0203] Collision Risk Assessment. To provide early warning of potential ship collisions, this framework analyzes AIS and visual trajectory data. Risk assessment is based on distance thresholds to ensure that vessels with a high probability of collision trigger early warnings.

[0204] Generate an alarm. When a potential collision risk is detected, an audible and visual alarm will be triggered to alert maritime authorities or vessel operators.

[0205] Through this integrated approach, the proposed framework enhances maritime situational awareness in inland bridge waterways and improves collision prevention effectiveness.

[0206] II. Application Examples. To demonstrate the inventiveness and technical value of the present invention, this section provides application examples of the claimed technical solutions applied to specific products or related technologies.

[0207] (I) Performance Evaluation of E-YOLO Target Detection

[0208] In this invention, the experiments were conducted using the GPU version of PyTorch. The specific computer configuration is shown in Table 1. To accelerate the convergence of the E-YOLO model during training, the Adam optimizer was used to optimize the network weights. Specifically, the momentum factor of the Adam optimizer was set to 0.937, and the decay factor was set to 0.0005. Considering the limitations of the computer hardware, the input image size was set to 640×640, and the batch size for model training was set to 8.

[0209] Table 1 Hardware and Software Environment

[0210]

[0211] (II) Description of Experimental Dataset

[0212] E-YOLO was evaluated using the publicly available ship dataset SeaShips. To increase the diversity of the dataset and capture a wider range of scenes, the experiment additionally collected 2925 ship images taken by the ChinaSat camera, such as... Figure 12As shown. This invention employs data augmentation techniques (DAT), such as horizontal flipping, random translation, random cropping, mosaicking, etc., to improve the training process and reduce network overfitting.

[0213] (III) Evaluation Indicators

[0214] To comprehensively evaluate the performance of E-YOLO, this invention uses mean average precision (mAP) and false alarm rate (FAR) as evaluation metrics. These metrics can provide valuable insights into the effectiveness of the established model. Their mathematical expressions are shown in formulas (10) to (13):

[0215]

[0216] Where mAP represents the average detection accuracy across all ship categories, and classes represent the ship categories. Furthermore, this invention also uses Average Frames Per Second (AVGFPS) as an evaluation metric for the model; a higher AVGFPS value indicates faster detection speed.

[0217] III. Evidence of the Relevant Effects of the Embodiments. The embodiments of the present invention have achieved some positive effects during research and development or use, and indeed possess significant advantages compared to existing technologies. The following description, in conjunction with data, charts, and other materials from the experimental process, illustrates these advantages.

[0218] (I) Comparative Analysis of Different Models

[0219] To evaluate the performance and effectiveness of E-YOLO, this invention provides a comprehensive comparison with other well-known and representative object detection models, including YOLOv3, YOLOv4, YOLOv5, YOLOv7, YOLOv8, and the Detection Transformer (DETR). To ensure fairness in the comparison, all experiments employing Data Augmentation Techniques (DAT) were conducted in the same PyTorch environment.

[0220] Table 2 presents the performance evaluation results of different models. The E-YOLO model achieved a significant improvement in mAP compared to YOLOv7, increasing by 4.5%. DETR, an end-to-end detector, achieved the lowest mAP value, especially in detecting fishing vessels. YOLOv3 and YOLOv4 exhibited similar detection performance, with low mAP values, particularly in detecting ore carriers. Compared to YOLOv4, YOLOv5 improved mAP by 1.5%, accurately detecting various types of vessels, but its detection capability is far inferior to E-YOLO. YOLOv8 achieved an mAP value of 0.975, ranking second. Regarding false alarm rate (FAR), E-YOLO achieved the lowest value, effectively mitigating the influence of background environmental factors. In contrast, DETR had the highest FAR, and due to its complex Transformer structure, its AVGFPS was only 12.5, making it unsuitable for practical maritime surveillance systems. Finally, E-YOLO achieved an AVGFPS of 10⁴, meeting the requirements for real-time detection and further highlighting its potential for practical applications.

[0221] Table 2 Experimental results of different models

[0222]

[0223] Next, this invention comprehensively evaluated the visualization detection results of several models under different monitoring conditions, such as Figure 13-16 As shown. In maritime visual surveillance applications, these detection models demonstrate strong detection performance for single-target vessels, such as... Figure 13 As shown. Figure 14 Multiple ship targets appeared in the middle. Figure 15 Chinese ships are relatively small in size. Figure 16 Significant overlap exists between vessels, increasing the difficulty of visual vessel detection and making maritime traffic monitoring systems unreliable. In inland waterway navigation environments, the background of images is often complex, including riverside buildings, bridges, and navigation aids, resulting in poor detection performance of YOLOv3, YOLOv4, and YOLOv5 for multi-target vessels. Figure 14 As shown. When the target ship is small in size, E-YOLO also exhibits strong detection performance, such as... Figure 15 As shown. In Figure 16 In practice, even when there are numerous ship targets and significant obstruction, E-YOLO can still accurately detect ship targets. In contrast, our method achieves robust and accurate monitoring of ships in different monitoring environments, which is more conducive to improving maritime traffic safety.

[0224] (II) Homography Matrix Estimation

[0225] Solving for the homography matrix H requires at least four sets of corresponding image and world coordinate data. Furthermore, the more coordinate control point pairs selected, the higher the accuracy of the matrix solution. Therefore, in the regulated waters, this invention randomly selected a scenario with a large number of vessels navigating the river (including upstream and downstream vessels) to calculate the homography matrix H.

[0226] Ships are affected by wind, waves, and currents during actual navigation, resulting in highly nonlinear motion. When calculating the homography matrix H, it is difficult to obtain accurate image coordinates due to the non-stationarity of the river surface, coordinate errors of the target detector, and slight camera jitter. To reduce projection errors, N images of ships in navigation are randomly taken in the monitored waters, and the corresponding H for each image is calculated according to formula (8). n In the experiment, N=10, and the calculated [H1,…,H] were selected. n The mean of the coordinates is used as the final homography matrix H. The homography matrix calculation results between the coordinates are shown in equations (14) and (15), where H AIS2Pixel H is the transformation matrix projected from world coordinates to pixel coordinates. Pixel2AIS The result is calculated by reversing the process. Figure 17 This demonstrates the projection effect between the corresponding AIS data and image coordinates.

[0227]

[0228] (III) Error Analysis

[0229] To verify the effectiveness of using multiple frames of images to calculate the homography matrix, this invention randomly selected multiple AIS and image coordinate point pairs for analysis. The coordinates of the target center point detected by E-YOLO were used as the true image coordinates for each ship. Euclidean distance was used to calculate the different spatial projection errors for each data point, and the mean projection error (MPE) was used to evaluate the homography matrix calculation results.

[0230]

[0231] Among them, x proj and y proj These are the coordinates after different spatial data transformations, x and y. real and y real The coordinates are the corresponding true coordinates, i.e., the center point coordinates of the E-YOLO detection result and the actual AIS position coordinates of the ship. Error is the projection error for each coordinate point pair, and N is the number of coordinate point pairs.

[0232] In the error calculation process, this invention selected 50 coordinate point pairs from different scenarios for analysis. Figure 18The projection error distribution for each coordinate point pair is shown in Table 3, and the error statistics are presented. It can be seen that using multiple frames of images to solve the homography matrix can significantly reduce the projection error between coordinates. When projecting world coordinates to pixel coordinates, the average error is only 39.2 pixels, which is about 61% lower than the single-frame image method, and the minimum error is only 6 pixels.

[0233] Similarly, when the world coordinates corresponding to the center point coordinates of the ship image are used to calculate the corresponding world coordinates, the average error is 0.000583, and the minimum error is only 0.000362, which further demonstrates the effectiveness of the method for calculating the homography matrix based on point pairs of multiple frames of images, and can be used for subsequent heterogeneous data collaborative sensing tasks.

[0234] Table 3 Error statistics for different methods

[0235]

[0236] (iv) Cooperative perception of vision and AIS

[0237] This invention uses the Wuhan Yangtze River Bridge No. 2 in Wuhan, Hubei Province as a case study. An industrial camera was installed in the center of the bridge area's navigation channel to continuously capture the real-time visual navigation status of vessels. For ease of calculation, the sampling period for each frame was set to 1 second. Simultaneously, an AIS receiver was deployed to synchronously collect the vessels' AIS navigation data and store it in a database for further analysis. Specifically, according to navigation regulations for the Wuhan section of the Yangtze River, a 2-kilometer area around the bridge was designated as a monitored navigation channel. Under normal navigation conditions, the trajectories of upstream and downstream vessels were used to define the navigation boundaries, such as... Figure 19 As shown.

[0238] As is well known, AIS data transmission has a wide reception range, uncertain time period, and low reliability. To reduce the computational complexity of trajectory interpolation and cooperative sensing methods, this invention first designs a series of AIS data filters.

[0239] (a) Data Cleaning. Due to the unreliability of AIS data transmission, a large amount of received AIS data contains "dirty data," including duplicate and garbled data, or lacks key information such as Maritime Mobile Service Identifier (MMSI), longitude (LON), latitude (LAT), speed to ground (SOG), and heading to ground (COG). Therefore, a Data Cleaning Filter (DCF) was designed. clean To clean the data. The processed AIS data (D) clean It can be represented in the following form:

[0240] D clean =F clean (D row(18)

[0241] Among them, D row This represents the raw AIS data collected in the experiment. T represents the time span of data collection, and E represents the necessary elements required for the recorded AIS data, including MMSI, LON, LAT, COG, and SOG. ε is the tolerance range for each element. Based on T, this invention first deletes AIS data that is too old. Then, it deletes AIS records that are missing E or have a deviation greater than ε.

[0242] (b) Surveillance Area Filtering. AIS receivers have a wide reception range, typically exceeding ten nautical miles. However, most of the received data is ineffective for visual surveillance. This invention designs a Surveillance Area Filter (SAF) based on the camera's location, such as... Figure 19 As shown. Within the regulatory scope F SAF Filtered AIS data D SAF It can be represented as follows:

[0243] D SAF =F SAF (D clean ,L camera ,θ) (19)

[0244] Among them, L camera Let θ represent the location of the camera, and let θ represent the radius of the restricted area, i.e., the threshold for rejecting ships that are more than θ nautical miles away from the camera. The distance between the ship and the camera can be calculated using the following formula:

[0245]

[0246] Where R represents the radius of the Earth, LON V ,LAT V ,LON C and LAT C These are the longitude and latitude coordinates of the ship and the camera, respectively.

[0247] (c) Navigation Status Filtering. A vessel's COG and SOG can be extracted from AIS data. In addition, a Navigation Status Filter (NSF) is designed to reject records of vessels that are not navigating within the monitoring range.

[0248] (V) Vision-based ship trajectory extraction

[0249] The DeepSORT algorithm has mature applications in visual target detection and tracking. Therefore, this invention tends to use E-YOLO as the detector and effectively combine it with DeepSORT to extract ship motion trajectories from visual information. Moreover, in the experiment, the camera was installed perpendicular to the bridge surface, so even if ships in the monitored waters crossed, overtook, or traversed, there was almost no serious occlusion between the ships, which is more conducive to extracting ship visual trajectory information.

[0250] Specifically, the E-YOLO network was used to detect ship targets in vision and obtain a series of bounding boxes, which can be represented as:

[0251] BBox = {bbox1, bbox2, ..., bbox} l ...bbox L} (twenty one)

[0252] Where L represents the number of bounding boxes, bbox l It is the position of the i-th bounding box, where (x tl ,y tl ) and (x br ,y br These are the pixel indices of the top left and bottom right corners of the bounding box, respectively.

[0253] Each bounding box is fed into DeepSORT, and based on Kalman filtering and the Hungarian algorithm, the state of the target in the next frame is updated and associated. Finally, the bounding box information of the target in each frame is output (x tl ,y tl The bounding box contains (w, h), confidence level (conf), and other state information. Here, (w, h) represent the width and height of the bounding box, respectively. This allows us to obtain a series of visual trajectories for the ships, which can be represented as:

[0254]

[0255] (vi) Visual and AIS Collaborative Perception Method

[0256] To provide early warnings for vessels at risk of collision and ensure their safe navigation in bridge-area waters, this invention proposes a simple and efficient method for collaborative perception of vessel vision and AIS information, which can be effectively used for vessel situation assessment and collision warning. Specifically, this invention maps the received AIS information to image coordinates using a homography matrix and matches it with visually detected targets, effectively displaying AIS information in the visual image and achieving the fusion and unification of heterogeneous information.

[0257] The time interval for AIS data transmission is variable, reaching up to 5 minutes or more. Therefore, to obtain the real-time position of each vessel within the monitoring range, this invention utilizes historical AIS data with kinematic information to predict its trajectory. The real-time position of the vessel P(LON) rt ,LAT rt This can be represented as follows:

[0258] P(LON rt ,LAT rt ) = [LON h ,LAT h ]+[SOG lon SOG lat ]·(T rt -T h ) (twenty three)

[0259] Among them [LON h ,LAT h [SOG] is the location in the latest received AIS information. lon SOG lat [ ] is the ground velocity (SOG) component along the longitude and latitude directions, which is calculated based on the ship's ground heading (COG). T rt and T h These are real-time and historical timestamps, respectively.

[0260] In this invention, AIS data is predicted using formula (23) to match the timestamp of the video, thereby aligning the two heterogeneous pieces of information in time and space. Then, this invention directly uses the Euclidean distance as the similarity between the two pieces of information and uses nearest neighbor search based on the k-Dimensional Tree algorithm to assign the AIS message to the nearest bounding box in the image.

[0261] (vii) Performance evaluation of collaborative sensing

[0262] To verify the matching accuracy of the proposed method, visual data from the AIS receiver was collected during the visual data acquisition process and used as the ground truth for comparison. The final output of the AIS and visual data matching is shown below. Figure 20 As shown, E-YOLO can accurately and robustly detect moving vessels in every frame of the video. In this case, this superior detection performance is mainly attributed to network architecture optimization and data augmentation. Finally, the detected vessels are associated with identity information extracted from AIS data. This successfully improves the level of maritime traffic monitoring in inland waterways and ensures the safety of vessel navigation.

[0263] Furthermore, to objectively evaluate the effectiveness of the proposed matching method, this invention randomly selected four cases from videos containing ship targets and calculated the matching accuracy by referring to simultaneously visualized AIS data. The accuracy value is calculated as shown in formula (24). The quantitative results are shown in Table 4. In most cases, this method can accurately associate AIS information with visual data with an accuracy of over 90%, and has the ability to provide auxiliary information for ship navigation risk warning to maritime monitoring departments.

[0264]

[0265] Table 4. Accuracy of successful matches in different cases

[0266]

[0267]

[0268] (viii) Collision prevention and early warning based on vision and AIS

[0269] Furthermore, in inland waterway bridge areas, some vessels may engage in illegal navigation activities. These include, for example, abnormally disabling AIS equipment, speeding, maintaining excessively close distances between vessels, and forcing passage under adverse weather conditions. These violations seriously impact the safe navigation of vessels in bridge areas. In the experiment, the rules for determining vessel violations and providing collision warnings are as follows:

[0270] Warning for Abnormal AIS Device Shutdown: Abnormal shutdown of the AIS device will prevent the effective perception of the vessel's dynamic and static navigation information. If a vessel is detected in the visual monitoring system, but no corresponding AIS data can be matched after continuous monitoring for more than 60 seconds, it will be judged as "No AIS" and a warning will be triggered. In this case, maritime regulatory personnel can still use the collaborative perception method proposed in this invention to effectively estimate the vessel's dynamic position information from visual data and promptly capture the vessel's navigation status.

[0271] Navigation Prohibition Warning under Severe Weather Conditions: In severe weather conditions such as fog and low light levels, the maritime traffic environment becomes more complex. Therefore, maritime authorities will prohibit vessels from navigating in bridge areas. For example, vessels are prohibited from passing through bridge areas when visibility on the river surface is less than 1500 meters; convoys are prohibited from passing through bridge areas when winds reach force 7 or above; vessels carrying explosives and vessels undergoing trials are prohibited from passing through bridge areas at night. However, under current traffic regulations, some vessels still illegally attempt to pass through bridge areas, posing a high risk of collision to both vessels and bridges. Based on this, this invention, using atmospheric scattering models and Retinex theory, simulates visual surveillance scenarios under foggy and low-light conditions. The received AIS data is directly projected onto the visually monitored water area through coordinate transformation. If a vessel is detected exhibiting dynamic navigation behavior, a "Navigation Prohibition Warning" is issued, and the identity and navigation information of the violating vessel are recorded.

[0272] Vessel Overspeed Warning: Inland waterway bridge areas typically restrict vessel speeds, and exceeding speed limits can lead to collisions. Based on relevant regulations from the Wuhan Maritime Safety Administration, this invention sets the normal speed for upstream vessels to [3,8] knots and for downstream vessels to [5,13] knots. The warning system acquires the vessel's speed of travel (SOG) in real time and compares it to the set safe speed range. If the speed exceeds the threshold, an "overspeed warning" is issued. If the overspeed duration exceeds the threshold by more than 30 seconds, a "high-priority overspeed warning" is issued.

[0273] Ship-to-ship collision avoidance warning: This technology has received widespread attention in the maritime field for practical applications, such as maritime surveillance and collision avoidance. It is typically defined as a two-dimensional area surrounding a target vessel, within which other vessels or obstacles are not permitted to intrude.

[0274] When the distance between ships is small, collisions may occur due to mismanagement. This invention, based on inland waterway navigation rules, uses global coordinate information to calculate the geodetic distance between ships in real time and sets the vessel distance (Vessel). Dis The threshold is 50 meters. If the real-time distance between ships is less than this threshold, a "collision warning" is issued; if the warning lasts for more than 20 seconds, a "high-priority collision warning" is issued. In the experiment, this invention comprehensively considered the dimensional information of the encountering ships and set the vessel distance between ships. Dis The calculation rules are as follows:

[0275] Vessel Dis =Dis - (L1 + L2) / 2 (25)

[0276] Where Dis is the real-time geodetic distance between ships, which can be calculated by equation (20). L1 and L2 are the scale information of the encountering ships, mainly referring to the ship length.

[0277] This section, based on the collaborative perception of vision and AIS, performs real-time detection and identification of vessel navigation status, promptly identifying and issuing warnings of illegal vessel behavior, thereby reducing the occurrence of vessel collisions. In the experiment, to enrich the maritime traffic scenario, this invention uses simulation to supplement existing AIS data, accurately reflecting illegal vessel behavior in bridge-area waters. The visualized warning results of illegal vessel behavior in the monitored waters are as follows: Figure 21 As shown. Under normal navigation conditions, using the proposed system, maritime authorities can directly identify illegal vessels, especially when the vessel's navigator deliberately turns off the AIS equipment; the maritime authorities can still obtain the vessel's navigation status through visual information.

[0278] Furthermore, even under complex and adverse weather conditions, where visual detection and tracking methods are ineffective in detecting moving vessels, the method of this invention can still effectively estimate and perceive the vessel's navigation status, achieving the regulatory goal of collaborative perception using visual and AIS data. Finally, the vessel collision avoidance and early warning method proposed in this invention has significant practical implications for maintaining maritime traffic order, reducing the risk of vessel collisions, and ensuring the safe navigation of vessels in bridge-area waterways.

[0279] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.

[0280] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A ship collision avoidance early warning method based on collaborative perception of visual and AIS data, characterized in that, Includes the following steps: Step S1: Use the E-YOLO target detection model to detect ship targets in the visual image, and combine it with the DeepSORT algorithm to obtain the visual trajectory data of the ship; Step S2: Preprocess the AIS data, including data cleaning, surveillance area filtering, and navigation status filtering; Step S3: By homography coordinate transformation, the preprocessed AIS data is projected into the image coordinate system to achieve the fusion of visual trajectory data and AIS data; Step S4: Based on the collision warning strategy, establish the matching relationship between AIS data and visual trajectory data, and obtain the fused ship motion trajectory; Step S5: Based on the fused ship motion trajectory and preset rules, issue early warnings to ships that may collide and trigger audible and visual alarms; The E-YOLO object detection model adopts the FasterNetF structure and the SA module. The FasterNetF structure consists of PWConv, the FasterNet module and the FReLU activation layer. The SA module divides the input features into several groups along the channel direction and calculates the channel attention and spatial attention maps respectively. Then, the channel attention maps and spatial attention maps are concatenated and fused, and the output feature map is obtained by channel shuffling.

2. The ship collision avoidance early warning method based on visual and AIS data collaborative perception according to claim 1, characterized in that, The DeepSORT algorithm combines Kalman filtering and the Hungarian algorithm to associate and update the state of targets, outputs the bounding box position and confidence of the target in each frame, and integrates the bounding box information of the ship in multiple frames into a visual trajectory sequence.

3. The ship collision avoidance early warning method based on visual and AIS data collaborative perception according to claim 1, characterized in that, The E-YOLO object detection model uses the WIoU loss function during training and employs weighted non-maximum suppression to calculate and filter bounding box scores for the detection results. WIoU adjusts the gradient gain for different anchor boxes through distance attention and dynamic non-monotomic focusing mechanism to balance the fitting ability for low-quality and high-quality samples.

4. The ship collision avoidance early warning method based on visual and AIS data collaborative perception according to claim 1, characterized in that, The AIS data preprocessing includes: AIS records that are too old, have missing elements, or have significant deviations are removed through data cleaning filters. A surveillance area filter based on camera location removes vessel data that is more than a preset threshold away from the camera. AIS records for anchored or irrelevant navigation states are removed based on the ship's navigation status filter.

5. The ship collision avoidance early warning method based on visual and AIS data collaborative perception according to claim 1, characterized in that, The homography coordinate transformation method includes: Select several AIS-known ships as feature points and obtain their corresponding pixel coordinates from the image; The homography matrix is ​​calculated using direct linear transformation, and then the AIS coordinates are mapped to image coordinates using this homography matrix. Align the mapped AIS coordinates with the ship bounding box obtained from visual detection in the image coordinate system.

6. The ship collision avoidance early warning method based on visual and AIS data collaborative perception according to claim 1, characterized in that, The collision warning strategy uses the KD tree algorithm to perform nearest neighbor search in the fused data, matches the AIS predicted location with the visual bounding box, and provides overspeed or collision warnings based on ship speed, heading, and distance thresholds. If the AIS device is detected to be off or in an abnormal state, an AIS-free warning will be triggered based on the visually detected ship information, and navigation restrictions will be detected and warned under severe weather or low light conditions.

7. A ship collision avoidance early warning system based on visual and AIS data collaborative perception, implementing the ship collision avoidance early warning method based on visual and AIS data collaborative perception as described in any one of claims 1 to 6, characterized in that, include: The visual acquisition and detection unit is equipped with a camera and an E-YOLO detection module to acquire images and detect ship targets within them; The target tracking unit, connected to the vision acquisition and detection unit, includes a DeepSORT algorithm module for associating and tracking the trajectory of the detected ship target based on the bounding boxes in multiple frames of images. The AIS data processing unit includes AIS data cleaning, surveillance area filtering, and navigation status filtering modules, which are used to preprocess the received AIS data. The coordinate transformation unit projects the AIS data onto the visual image coordinate system based on the homography matrix, thereby achieving the fusion of AIS information and visual trajectory data; The collision warning unit, based on the fused ship trajectory information, performs data matching through the KD tree algorithm and detects overspeed, abnormal AIS shutdown, and inter-ship distance according to pre-set rules, triggering a collision warning signal.

8. The ship collision avoidance early warning system based on visual and AIS data collaborative perception according to claim 7, characterized in that, The visual acquisition and detection unit includes: The E-YOLO object detection module adopts the FasterNetF structure and SA module, and uses the WIoU loss function and weighted nonmaximum suppression to process the detection results during training. The DeepSORT target tracking module combines Kalman filtering and the Hungarian algorithm to correlate and update the target bounding box position and confidence level of each frame of image, and output a visual trajectory sequence.

9. The ship collision avoidance early warning system based on visual and AIS data collaborative perception according to claim 7, characterized in that, The AIS data processing unit further includes: Data cleaning filters are used to delete AIS records that are too old or have missing features; The regulatory area filter removes vessel data that exceeds a preset distance threshold based on the camera's location; The navigation status filter removes AIS records of anchored or irrelevant navigation statuses based on the ship's heading and speed. The coordinate transformation module uses direct linear transformation to calculate the homography matrix and map AIS geographic coordinates to image coordinates, thereby aligning AIS data with visual inspection data.

Images