A ship abnormal behavior monitoring and early warning method based on multi-modal perception

Abstract

This invention belongs to the field of intelligent port navigation safety and maritime intelligent detection technology, and discloses a method for monitoring and early warning of abnormal ship behavior based on multimodal perception. It collects ship trajectory data, video surveillance image data, radar point cloud data, and hydrological and meteorological data to construct a global feature vector through spatiotemporal alignment. Employing an offline training and online inference architecture, the feature vector is input into the trained model to complete anomaly identification, and initial risk assessment is performed by combining factors such as ship type, waterway area, and meteorological tide level. A graded early warning system is achieved through quantitative calculation of the early warning risk index, employing a differentiated triggering strategy: extremely high-risk situations trigger an alarm directly, while medium and low-risk situations trigger after continuous frame verification, reducing false alarms. This invention also establishes a closed-loop linkage between early warning and monitoring, allowing for dynamic encryption of monitoring and iterative model optimization. This invention enables precise monitoring of abnormal ship behavior across the entire domain and highly reliable early warning at the millisecond level, significantly improving the efficiency of port and waterway navigation safety management.

Description

Technical Field

[0001] This invention relates to the field of intelligent port navigation safety and maritime intelligent detection technology, specifically to a method for monitoring and early warning of abnormal ship behavior based on multimodal perception. Background Technology

[0002] As core hubs of maritime logistics, large coastal port waterways are characterized by high vessel traffic density, complex waterway topology, intricate navigation environments, and high demand for passage from large and critical vessels. Abnormal vessel behaviors, such as speeding, deviation from course, illegal berthing, and navigation without AIS, can easily lead to waterway congestion and navigation accidents, severely impacting the safe and efficient operation of ports and waterways. With the rapid development of smart ports, digital twins, and intelligent navigation technologies, real-time perception, accurate identification, proactive early warning, and coordinated response to vessel behavior have become core requirements for intelligent port and waterway management. Monitoring and early warning are the two core components of navigation safety management. Monitoring provides comprehensive data support for early warning, while early warning provides valuable application directions for monitoring; the two complement each other and are indispensable.

[0003] Current ship abnormal behavior monitoring and early warning technologies are mainly based on single-mode data from AIS or radar. Some technologies attempt to perform simple data stitching or result-level fusion of AIS, radar, and video. However, none of these technologies address the core issues of monitoring and early warning technologies by starting from the essential needs of port and waterway scenarios. Furthermore, the logical boundaries between monitoring and early warning are vague, and the connection mechanism is lacking. The problems of existing technologies are as follows: 1. AIS data only provides digital information such as ship trajectory and speed, and is easily affected by human intervention or signal loss. It cannot reflect the ship's physical shape or relative position in the surrounding environment. Radar data can only capture target point cloud features, has low resolution, and cannot identify ship type or navigation attitude. Although video data can provide visual information, it is easily obscured by adverse environmental conditions such as fog, rain, and night navigation, and cannot obtain quantitative data such as accurate ship speed and heading. Single-modal data all have information blind spots, reflecting only a single dimension of the ship's navigation status, and cannot achieve a comprehensive and complete characterization of ship behavior, directly leading to a high rate of missed and false detections in abnormal behavior detection.

[0004] 2. Existing technologies simply use detected anomalies such as "deviation" and "speeding" directly as early warning criteria, without designing dedicated quantitative judgment models or dynamic triggering mechanisms for the early warning process. They fail to differentiate between environmental factors and human violations by incorporating dynamic hydrological and meteorological data such as wind, waves, currents, visibility, and tide levels, nor do they perform risk quantification and classification based on scenario elements such as vessel type and waterway area. For example, the same monitoring result of "speeding 10%" triggers the same level of early warning for both large dangerous goods vessels speeding during high tide in the main channel and small fishing boats speeding near anchorages. This results in high-risk anomaly warnings lacking priority and minor anomaly warnings being overemphasized.

[0005] 3. The existing detection models and judgment rules are designed for ordinary open waters and do not incorporate the complex navigation constraints unique to large coastal ports, such as tidal navigation, two-way navigation, priority for key vessels, speed limits in different areas of the waterway, and the designation of prohibited waterways. Therefore, they cannot meet the safety requirements of large coastal ports.

[0006] 4. Some technical solutions that combine multiple data sources only involve data layering or simple result-level fusion, without achieving deep feature-level fusion. This fails to leverage the complementary and corrective advantages of multiple data sources, resulting in situations where radar detects anomalies but it is difficult to match them with data transmitted from AIS. Consequently, different data are difficult to match and cannot provide data support for port and waterway digital twins, full-domain management, and multi-system collaboration. Summary of the Invention

[0007] The purpose of this invention is to overcome the problems of high false alarm rate and high risk of missed detection in the application of existing ship abnormal behavior monitoring and early warning methods, and to provide a ship abnormal behavior monitoring and early warning method based on multimodal perception.

[0008] The technical solution of the present invention is as follows: A method for monitoring and early warning of abnormal ship behavior based on multimodal perception, comprising the following steps: S1: Multimodal data acquisition and preprocessing, collecting ship trajectory data, video surveillance image data, radar point cloud data, and hydrological and meteorological data, and performing deduplication, completion, noise reduction, target extraction, timestamp unification and format standardization on each type of data to form a standardized dataset; S2: Multimodal data spatiotemporal alignment. Based on the standardized dataset formed by S1, a unified spatiotemporal coordinate system for ports and waterways is established to match and associate targets with ship trajectories, video surveillance images, and radar point cloud data in the same spatiotemporal dimension. S3: Based on the spatiotemporally aligned multimodal data from S2, construct an anomaly detection model adapted to the general aviation environment, including: S31: Construct a Transformer multimodal feature fusion network based on asymmetric twin structure, and set encoders of different depths for ship trajectory data, video image data, and radar point cloud data to extract and encode the feature vectors of their respective modes; S32: An environmental adaptive dynamic fusion mechanism is introduced, which inputs the preprocessed hydrological and meteorological data into a lightweight environmental encoder to generate dynamic allocation values ​​for AIS feature weights, video feature weights, and radar feature weights in real time. The feature vectors of each mode are weighted and fused to obtain weighted multimodal features. S33: A multi-head cross-attention mechanism embedding a port and waterway spatial constraint mask is used to jointly encode and interactively enhance the weighted multimodal features; the port and waterway spatial constraint mask is dynamically generated according to the relative relationship of region types to guide attention calculation; S34: The hydrological and meteorological data are fused with the interactively enhanced fusion features as environmental correction factors to form a global feature vector that includes ship status and environmental constraints. S35: Set an anomaly classification layer at the top level of the global feature vector to complete the offline training of the anomaly detection model; S4: Input the spatiotemporally aligned multimodal data from S2 into the anomaly detection model trained in S35 to identify abnormal behavior and classify risks; define the types of abnormal ship behavior according to port navigation rules, output the anomaly type, location and confidence level, and classify the anomalies into three risk levels: high, medium and low, based on scene elements. S5: Tiered early warning and air traffic scheduling linkage intervention. Based on the output results of S4, the early warning level is accurately determined by the early warning risk index. Combined with dynamic adaptive triggering rules, the early warning is intelligently triggered. Early warning information is pushed according to standardized protocols. The monitoring system is dynamically optimized through early warning-monitoring closed-loop linkage.

[0009] In S1, the ship trajectory data is AIS data, including ship position, heading, speed, ship type, and MMSI code, acquired at the actual receiving frequency and interpolated to 0.1 seconds / time; the video surveillance image data is 1920×1080 resolution high-definition footage, covering the entire port channel and anchorage area; the radar point cloud data includes target distance, bearing, and echo intensity, with a detection range covering 20 nautical miles of the port channel; the hydrological and meteorological data includes wind speed, wind direction, current speed, current direction, visibility, and tide level information, with an original sampling frequency of minutes, interpolated to 0.1 seconds / time, and then standardized to 0-1; preprocessing operations are performed on various types of data, including... The process includes: AIS data is processed using the 3σ criterion to remove anomalous jump points, and missing data of ≤3 consecutive frames is completed by linear interpolation. Trajectory smoothing is performed using the moving average method. Video surveillance images are processed using a dark channel prior dehazing algorithm to eliminate fog interference. After denoising with a 3×3 Gaussian filter, ship target bounding boxes are extracted using the YOLOv8 target detection model, with a confidence threshold of 0.8. Radar point clouds are processed using Euclidean filtering to remove noise points, followed by density clustering using the DBSCAN algorithm, and then ship target contours are extracted using the convex hull algorithm. All modal data are unified to the UTC time base, with timestamp accuracy to 0.1 seconds, and the data format is unified as JSON.

[0010] In S2, the unified spatiotemporal coordinate system of the port and waterway is a plane rectangular coordinate system with the port and waterway management center as the origin, the WGS-84 geographic coordinate system as the reference, and the Mercator projection. The unit is meters, and the elevation reference is the local mean sea level. The conversion between latitude and longitude and plane rectangular coordinates is realized through the PyProj library. Using the AIS ship's latitude and longitude position as the initial reference, the video target box and radar clustered target are mapped to the Cartesian coordinate system. Kalman filtering and Hungarian matching algorithm are used to complete the association and tracking of the same ship in multimodal data. The matching threshold of the Hungarian matching algorithm is 10 meters.

[0011] The encoders at different depths in S31 adopt an asymmetric twin structure: the encoder for extracting AIS trajectory motion features is a 2-layer Transformer encoder, the encoder for extracting video image visual features is a 6-layer Transformer encoder, and the encoder for extracting radar point cloud features is a 4-layer Transformer encoder; the three encoders share some low-level parameters, and the feature vectors output by each branch are mapped to a unified dimension d_model=512 through independent linear projection layers. The method for dynamically calculating the fusion weights of each modal feature in S32 includes: inputting wind speed, visibility, and tide level stage information from hydrological and meteorological data into a lightweight environmental encoder, and outputting dynamic allocation values ​​for each modal feature; when the wind speed is ≥10m / s, the radar point cloud feature weight is greater than the AIS trajectory motion feature weight and the video image visual feature weight; when the visibility is <1000 meters, the radar point cloud feature weight is greater than the video image visual feature weight. The method for generating the port channel spatial constraint mask in S33 includes: dividing the channel area into four categories—main channel, branch channel, anchorage, and prohibited channel—based on port channel geographic information system data; performing path search on the channel topology map based on the ship's current position to predict the set of channel areas the ship will navigate in future time steps; and constructing an attention mask matrix M, where M(i,j)=1 if and only if the channel areas corresponding to positions i and j both belong to the set of channel areas to be navigated or are adjacent areas, otherwise M(i,j)=0. In S34, the environmental correction factor F is calculated using the formula F = 0.2 × normalized wind speed + 0.3 × normalized current velocity + 0.2 × normalized visibility + 0.3 × tidal stage coefficient. The tidal stage coefficient is assigned values ​​of 0.2, 0.1, 0.4, and 0.8 for high tide, low tide, slack tide, and high tide, respectively. The environmental correction is completed by multiplying F with the fused features output from S33 element by element, generating a 512-dimensional global feature vector. In S4, the types of abnormal ship behavior include speeding, deviation, illegal berthing, navigation without AIS, dangerous encounter, sudden change in track, and occupation of prohibited channels. The logic for determining navigation without AIS is that the same ship target is detected for 5 consecutive frames and there is no AIS signal feedback. The confidence threshold for the anomaly detection model was set to 0.8. Based on vessel type, waterway area, traffic flow density, visibility, and tide level, a preliminary risk assessment was performed using quantified thresholds. The criteria for high, medium, and low risk were as follows: High risk: Large cargo ships deviating from the main channel by ≥50 meters, dangerous encounters when visibility is <500 meters, occupying prohibited channels, sudden changes in track angle ≥30°, or illegal anchoring in the main channel; Medium risk: Small vessels exceeding the speed limit by more than 20% in the channel, deviating from the course by 20-50 meters during non-tide periods, dangerous encounter distance ≥500 meters and <1000 meters, and sudden change in course angle of 15-30°. Low risk: Minor danger may be encountered when the vessel deviates less than 20 meters from the anchorage, experiences minor speed fluctuations of ±5%, illegally anchors in a temporary area of ​​the anchorage, or has visibility of ≥1000 meters.

[0012] In S5, the early warning risk index R is calculated by combining the anomaly confidence level C, the ship type quantification coefficient, the waterway area quantification coefficient, the hydro-meteorological environment quantification coefficient, the traffic flow density quantification coefficient, and the anomaly type hazard coefficient. The warning level is determined based on the R value: an R value ≥ 0.8 is a Level 1 warning, corresponding to an initial assessment of high risk; 0.5 ≤ R < 0.8 is a Level 2 warning, corresponding to an initial assessment of medium risk; and R < 0.5 is a Level 3 warning, corresponding to an initial assessment of low risk.

[0013] The dynamic adaptive triggering rule in S5 is as follows: Extremely high-risk anomalies in the Level 1 warning include occupying prohibited channels, navigating in the main channel without AIS, and dangerous encounters in low visibility. If the conditions are met, the warning will be triggered directly without the need for continuous frame verification. Other Level 1 warnings require 2-3 consecutive frames of stable verification before triggering; Level 2 and Level 3 warnings require stable verification for 3 or more consecutive frames before they can be triggered. When the wind speed is ≥10m / s, the R-value thresholds for Level II and Level III warnings will be lowered to 0.5 and 0.3, respectively.

[0014] In S5, the early warning information is generated according to a standardized structure, including basic ship monitoring data, early warning-specific data, and visualization support data. It is pushed to the shore-based VTS system, port dispatch platform, and maritime law enforcement terminal within 100 milliseconds using the TCP / IP industrial communication protocol, and also pushes warning information to other ships within 5 nautical miles of the ship. When multiple ships in the same area trigger warnings simultaneously, the warnings are sent out in descending order of R value. If the R values ​​are the same, the warnings are sent out to ships with higher risk levels first.

[0015] The specific operations of the early warning-monitoring closed-loop linkage in S5 include: After triggering the Level 1 warning, the monitoring sampling frequency of the target vessel will be increased to 0.05 seconds / time, the weight of multimodal feature fusion will be increased by 20%, and the monitoring of the waterway area within 10 nautical miles around it will be intensified. When the abnormal behavior of the target vessel is detected and corrected, and the R value is less than 0.4 for 10 consecutive frames and the vessel status and hydrological and meteorological environment return to normal, the warning is lifted after the dual verification is completed, and the monitoring system restores normal parameters within 30 seconds. All early warning data are incorporated into the training sample library of the monitoring model, and the Transformer feature fusion network and anomaly detection model are iteratively optimized every quarter based on the early warning source tracing data.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. Multimodal collaborative perception and feature-level deep fusion enable accurate characterization of ship behavior across the entire domain. The system employs joint perception of four modal data sources: ship trajectory data, video surveillance image data, radar point cloud data, and hydrological and meteorological data. Through unified spatiotemporal alignment and Transformer feature-level deep fusion, combined with environmental correction factors, blind spots in single-modal information are eliminated, significantly reducing missed and false detections of abnormal behavior and adapting to complex port navigation environments.

[0017] 2. This invention employs asymmetric twin Transformer multimodal feature fusion, environment-adaptive dynamic weight allocation, and port and waterway spatial constraint masking. It can adaptively adjust the modal contribution based on hydrological and meteorological conditions and embed waterway navigation rules into feature encoding, significantly reducing the false positive and false negative rates of anomaly detection and improving detection accuracy and scene adaptability in complex environments.

[0018] 3. This invention accurately identifies and quantifies risk classification and calculates the early warning risk index R by identifying anomaly types and combining them with dynamic adaptive triggering rules to achieve high / medium / low risk classification early warning and differentiated response. The early warning judgment is more scientific and the response is more timely, meeting the real-time safety supervision needs of ports and waterways.

[0019] 4. This invention constructs a closed-loop linkage mechanism of early warning, monitoring, and scheduling, which can dynamically increase the sampling frequency of key targets, increase the density of monitoring range, and incorporate early warning data into the model iteration sample library, thereby realizing system self-optimization and long-term stable operation, and significantly improving the port's intelligent navigation management capabilities. Attached Figure Description

[0020] Figure 1 This is a simplified schematic diagram of the steps of the method of the present invention; Figure 2 This is a flowchart illustrating the method of the present invention; Figure 3 This is a network structure diagram of the anomaly detection model. Detailed Implementation

[0021] The present invention will now be described in further detail with reference to the accompanying drawings. These embodiments are for illustrative purposes only and do not constitute a limitation on the scope of protection of the present invention. Those skilled in the art, after reading this invention, can make various modifications and substitutions to these embodiments without creative effort, and all such equivalent forms fall within the scope of protection defined by the appended claims.

[0022] The method includes the following steps: S1: Multimodal data acquisition and preprocessing; S2: Spatiotemporal alignment of multimodal data; S3: Anomaly detection model training; S4: Identify and classify abnormal behavior based on a trained anomaly detection model; S5: Tiered early warning and coordinated intervention with general aviation scheduling.

[0023] In the above scheme, S1 includes: collecting ship trajectory data, video surveillance image data, radar point cloud data, and hydrological and meteorological data, and performing deduplication, completion, noise reduction, target extraction, timestamp unification and format standardization on each type of data to form a standardized dataset.

[0024] S11, Multimodal Data Acquisition: In this embodiment, multimodal data within the port cluster waterway is simultaneously collected using shore-based sensing devices, such as AIS receiving base stations, high-definition surveillance cameras, marine radar, and hydrological and meteorological monitoring stations, including: AIS vessel trajectory data, video surveillance image data, radar point cloud data, and hydrological and meteorological data.

[0025] The AIS data includes the ship's position, heading, speed, ship type, and MMSI code, which are obtained according to the actual receiving frequency and then uniformly interpolated to 0.1 seconds / time for time alignment. The video data is a high-definition monitoring image with a resolution of 1920×1080, covering the main channel, branch channels and the entire anchorage area; Radar data includes target range, azimuth, and echo intensity, with a detection range covering 20 nautical miles of the port channel; Hydrometeorological data includes wind speed, wind direction, current velocity, current direction, visibility, and tide level information. The original sampling frequency is on the minute level. When used for environmental correction, it is interpolated to 0.1 seconds / time according to UTC to match the time axis of the sensing data. The interpolation results are only used for time alignment and are not extrapolated in physical sense.

[0026] S12, Preprocess the collected raw data: AIS data is used to remove abnormal jump points, and missing points are filled with linear interpolation and trajectory smoothing is performed. The outlier removal operation uses the 3σ criterion to determine outliers; if the formula is satisfied, the outlier is determined to be an outlier and removed. For missing data of ≤3 consecutive frames, the missing point completion operation uses linear interpolation to complete the missing data; The trajectory smoothing operation uses the moving average method, which calculates the average of the last 5 frames of valid latitude and longitude data and uses it as the trajectory coordinates for the current frame. The video images are processed using a defogging algorithm to remove fog interference, and the ship target bounding boxes are extracted after noise reduction. The dehazing operation employs a dark channel prior dehazing algorithm. The noise reduction operation uses Gaussian filtering with a kernel size of 3×3 and a standard deviation σ=0.8. The scale normalization operation uniformly scales the image to 1920×1080 resolution, and bilinear interpolation is used to ensure that the image is distortion-free. The ship target bounding box extraction operation uses the YOLOv8 target detection model, with a training set of 100,000 port ship samples and a confidence threshold of 0.8 (which can be dynamically adjusted according to the actual deployment scenario). The output is the pixel coordinates of the upper left / lower right corner of the ship target bounding box.

[0027] The radar point cloud is filtered by Euclidean filtering to remove noise points, and the ship target outline is extracted after density clustering. Among them, the Euclidean filtering operation sets the neighborhood search radius to 0.5 meters, the minimum number of neighborhood points to 5, and removes noise points with less than 5 neighborhood points; The density clustering operation uses the DBSCAN algorithm with a neighborhood radius of ε=2 meters and a minimum number of points MinPts=8, to separate the ship target point cloud from the background point cloud. The target contour extraction operation uses the convex hull algorithm to extract the contour of the clustered ship point cloud, and outputs the minimum bounding rectangle parameters of the contour, including center coordinates, length, width, rotation angle, etc.

[0028] Hydrological and meteorological data have undergone 0-1 standardization processing; Among them, the time alignment operation interpolates the hydrological and meteorological data to 0.1 seconds / time according to the UTC timestamp, and synchronizes it with the AIS / radar data; The 0-1 standardization operation performs minimum-maximum standardization on all hydrological and meteorological indicators. After standardization, the value range of all indicators is [0,1]. The normalization basis is the extreme value range of each indicator in the historical observation data of the port.

[0029] All modal data were standardized to UTC time base, with timestamp accuracy down to 0.1 seconds, and the data format was standardized to JSON, forming a standardized dataset.

[0030] In the above scheme, S2 includes: establishing a unified spatiotemporal coordinate system for the port and waterway, and completing target matching and association of AIS, video, and radar data in the same spatiotemporal dimension.

[0031] S21, Establish a unified spatiotemporal coordinate system: Establish a unified spatiotemporal coordinate system for ports and waterways, using a plane rectangular coordinate system (unit: meters) with the port and waterway management center as the origin, the WGS-84 geographic coordinate system as the reference, and the Mercator projection. Parameter settings: Central Meridian: Longitude of the port; Projection scale factor: 0.9996; Coordinate unit: meter (m); Elevation datum: Local mean sea level.

[0032] This demonstrates the conversion between latitude and longitude (B, L) and Cartesian coordinates (X, Y) using the PyProj library. The core formula is: N is the radius of curvature of the ramusoidal loop, t=tanB; S22, Multimodal data coordinate mapping and propagation target association and tracking: Using the AIS ship's latitude and longitude position as the initial reference, the video target box and radar clustered target are mapped to this coordinate system and matched with the ship's position through coordinate transformation; Kalman filtering (state vector includes position x / y, speed v, and heading θ, observation noise covariance matrix Q=diag(0.1,0.1,0.05,0.05)) and Hungarian matching algorithm (cost matrix constructed based on ship position distance and size similarity, matching threshold 10 meters) are used to complete the association and tracking of the same ship in multimodal data, eliminating time offset and spatial misalignment.

[0033] In the above scheme, S3 includes: constructing an anomaly detection model that is adaptive to the general aviation environment based on spatiotemporally aligned multimodal data.

[0034] S31. Construct a Transformer multimodal feature fusion network based on an asymmetric twin structure. Set encoders of different depths for ship trajectory data, video image data, and radar point cloud data. The AIS trajectory encoder is a 2-layer Transformer, the video image encoder is a 6-layer Transformer, and the radar point cloud encoder is a 4-layer Transformer. The three encoders share some low-level parameters. The output features of each branch are mapped to a unified dimension d_model=512 through independent linear projection layers to complete the feature extraction and encoding of each modality.

[0035] S32 introduces an environment-adaptive dynamic fusion mechanism, which inputs preprocessed hydrological and meteorological data into a lightweight environmental encoder to generate dynamic allocation values ​​for AIS, video, and radar feature weights in real time. When the wind speed is ≥10m / s or the visibility is <1000m, the radar feature weights are automatically increased, and the feature vectors of each mode are weighted and fused to obtain weighted multimodal features.

[0036] S33 is a multi-head cross-attention mechanism that embeds port and waterway spatial constraint masks. Based on the waterway topology, the region is divided into four categories: main channel, branch channel, anchorage, and prohibited channel. An attention mask matrix is ​​dynamically generated based on the area where the ship will sail in the future to guide attention calculation and perform joint encoding and interactive enhancement of weighted multimodal features.

[0037] S34 uses hydrological and meteorological data as environmental correction factors, calculated according to the formula F=0.2×normalized wind speed + 0.3×normalized current speed + 0.2×normalized visibility + 0.3×tide stage coefficient, and multiplies it element-wise with the fusion features after interactive enhancement to form a 512-dimensional global feature vector that includes ship status and environmental constraints.

[0038] S35 sets up an anomaly classification layer at the top level of the global feature vector, and uses labeled samples to complete the offline training of the anomaly detection model, providing model support for subsequent online real-time anomaly recognition.

[0039] In the above scheme, S4 includes: defining abnormal behavior types according to port navigation rules, including speeding, deviation, illegal berthing, navigation without AIS, dangerous encounters, sudden changes in track, and occupation of prohibited channels; inputting global feature vectors into the trained anomaly detection model and outputting anomaly type, location, and confidence level; classifying anomalies into three risk levels—high, medium, and low—based on vessel type, channel area, traffic flow density, visibility, and tide level. This preliminary judgment provides basic parameters and initial coefficient values ​​for the calculation of the early warning risk index in S5.

[0040] According to the navigation rules of major coastal ports, there are seven types of abnormal behaviors: speeding, deviation, illegal berthing, navigation without AIS, dangerous encounter, sudden change in track, and occupation of prohibited channels. The logic for determining navigation without AIS is as follows: if the same vessel target is detected for 5 consecutive frames and there is no AIS signal feedback, it is determined to be abnormal. The 512-dimensional global feature vector is input into the offline-trained anomaly detection model for online real-time inference, outputting the anomaly type, latitude and longitude location, and confidence score. The confidence score threshold is set to 0.8. By combining factors such as vessel type, waterway area, traffic flow density, visibility, and tide level, a preliminary risk level assessment is completed based on quantified thresholds. This result provides fundamental parameters and initial coefficient values ​​for the quantitative calculation of the S5 early warning risk index, and provides a classification basis for dynamic triggering rules, such as: High risk: Dangerous encounters with large cargo ships when they deviate from the main channel by ≥50 meters / visibility <500 meters / occupying prohibited channels / changing track angle by ≥30° / illegally anchoring in the main channel; Medium risk: Small vessels exceeding speed by more than 20% in the channel / deviating from course by 20-50 meters during non-tide periods / dangerous encounter distance ≥500 meters and <1000 meters / track change angle 15°-30°; Low risk: Minor danger may be encountered when the vessel deviates less than 20 meters near the anchorage, experiences slight speed fluctuations of ±5%, illegally anchors in a temporary area of ​​the anchorage, or has visibility of ≥1000 meters.

[0041] In the above scheme, S5 includes: based on the anomaly type, location, confidence level, and initial risk level judgment results output by S4, accurately determining the warning level through the warning risk index R (Level 1, Level 2, and Level 3 correspond to high risk, medium risk, and low risk initial judgment, respectively), and realizing intelligent warning triggering by combining dynamic adaptive triggering rules; among them, extreme high-risk anomalies such as occupying prohibited channels and navigating in the main channel without AIS in Level 1 directly trigger the warning, while other medium and low-risk anomalies in Level 1, Level 2, and Level 3 are triggered after continuous frame verification; the warning information is pushed to the designated terminal according to the standardized output protocol, and the monitoring system is dynamically optimized through the warning-monitoring closed-loop linkage, ultimately achieving accurate warning of abnormal ship behavior.

[0042] S51. Calculation of Early Warning Risk Index and Determination of Early Warning Level; Based on the anomaly confidence level C output by the model and the quantitative values ​​of port scene elements, the early warning risk index is calculated using the following formula: in For ship type quantification coefficient, For the quantification coefficient of the waterway area, For the quantitative coefficient of hydrological and meteorological environment, This is the traffic flow density quantification coefficient. This represents the risk factor for the abnormal type.

[0043] An R value ≥ 0.8 is considered a Level 1 warning, corresponding to a high-risk initial assessment in S4; 0.5 ≤ R < 0.8 is considered a Level 2 warning, corresponding to a medium-risk initial assessment; and R < 0.5 is considered a Level 3 warning, corresponding to a low-risk initial assessment.

[0044] For example, if a large cargo ship is detected ( ) in the main channel ( ) Yaw ≥ 50 meters At this time, the channel traffic flow density is 75%. Wind speed 8 m / s, visibility 1500 meters, non-tide navigation phase ( The model output confidence level C=0.95, and the calculated R=0.836, which is ≥0.8, so it is judged as a level 1 warning.

[0045] S52, Dynamic adaptive early warning trigger; The warning triggers a tiered differentiated strategy, using the video or AIS data sampling period as a baseline (0.1 seconds / time), and the continuous frame verification window is converted into a corresponding time window based on the number of frames: Level 1 Warning (Extremely High Risk): Such as anomalies that could lead to immediate accidents, such as occupying a prohibited channel, navigating in the main channel without AIS, or encountering danger in low visibility. If the conditions are met, it will be triggered directly without the need for continuous frame verification, ensuring zero-delay response in extremely dangerous scenarios. For example, if a vessel is detected occupying a prohibited channel and the confidence level of a single frame is C=0.92, a Level 1 warning will be triggered directly without the need for continuous frame verification. Other Level 1 warnings, such as deviations from the main channel, significant changes in the course, and dangerous encounters under normal visibility, are high-risk but not immediately dangerous anomalies. They need to be verified stably for 2 to 3 consecutive frames before being triggered, taking into account both real-time performance and anti-interference capabilities. For example, if a ship is detected to be deviating from its course for 3 consecutive frames and the confidence level C is stable at around 0.95 and the R value is stable at above 0.83, a Level 1 warning is officially triggered. Level 2 and Level 3 alerts: They need to be stably verified for 3 or more consecutive frames before being triggered to reduce the false alarm rate; In harsh environments, the warning thresholds are adaptively lowered to improve environmental adaptability. For example, when a port encounters strong winds with a wind speed of ≥10m / s, the R-value thresholds for Level II and Level III warnings are lowered to 0.5 and 0.3 respectively to achieve environmental adaptability.

[0046] S53. Standardized output and push of early warning information; The warning information is generated according to a standardized structure, including basic monitoring data such as the large cargo ship's MMSI code, ship type, and real-time latitude and longitude; warning-specific data such as the warning level, R-value, and calculated values ​​of each component; and visual support data such as video screenshots of the ship's yaw, radar point cloud images, and AIS trajectory curves. Using the TCP / IP industrial communication protocol, the warning information is simultaneously pushed to the shore-based VTS system, port dispatch platform, and maritime law enforcement terminal within 100 milliseconds, and warning information is also pushed to other vessels within 5 nautical miles of the ship.

[0047] When multiple ships in the same area trigger warnings simultaneously, the system pushes warnings in descending order of R value. If the R values ​​are the same, the system prioritizes ships with higher risk levels.

[0048] The shore-based VTS system and port dispatch platform display complete early warning information and visualized data, marking the vessel's position in real time on the electronic nautical chart; the maritime law enforcement terminal displays core information and includes navigation and positioning links, facilitating law enforcement personnel to quickly reach the scene; surrounding vessels only receive the position, anomaly type, and avoidance suggestions to avoid information overload.

[0049] S54. Early warning and monitoring closed-loop linkage; After triggering a Level 1 warning, the system automatically sends an instruction to the monitoring system to increase the monitoring sampling frequency of the large cargo ship to 0.05 seconds / time, increase the multimodal feature fusion weight by 20%, and intensify monitoring of the surrounding 10-nautical-mile waterway area. When the ship's deviation behavior is detected to be corrected, the R value is <0.4 for 10 consecutive frames, and the ship's status returns to normal with no abnormalities in the hydrological and meteorological environment, the warning is lifted after completing the dual verification. The monitoring system restores the normal sampling frequency and feature weight within 30 seconds.

[0050] Meanwhile, all data from this early warning will be included in the training sample library of the monitoring model. Every quarter, the Transformer feature fusion network and anomaly detection model will be iteratively optimized by combining all early warning source data to continuously improve the accuracy of the monitoring model.

[0051] like Figure 2 As shown, this embodiment achieves comprehensive and accurate monitoring and intelligent early warning of abnormal vessel behavior in the coastal large port channel through the above method, effectively reducing the probability of missed detection, false detection and false early warning, improving the monitoring accuracy to over 98%, and controlling the early warning response delay within 100 milliseconds, which greatly improves the navigation safety management level of the port channel.

Claims

1. A method for monitoring and early warning of abnormal ship behavior based on multimodal perception, characterized in that, Includes the following steps: S1: Multimodal data acquisition and preprocessing, collecting ship trajectory data, video surveillance image data, radar point cloud data, and hydrological and meteorological data, and performing deduplication, completion, noise reduction, target extraction, timestamp unification and format standardization on each type of data to form a standardized dataset; S2: Multimodal data spatiotemporal alignment. Based on the standardized dataset formed by S1, a unified spatiotemporal coordinate system for ports and waterways is established to match and associate targets with ship trajectories, video surveillance images, and radar point cloud data in the same spatiotemporal dimension. S3: Based on the spatiotemporally aligned multimodal data from S2, construct an anomaly detection model adapted to the general aviation environment, including: S31: Construct a Transformer multimodal feature fusion network based on asymmetric twin structure, and set encoders of different depths for ship trajectory data, video image data, and radar point cloud data to extract and encode the feature vectors of their respective modes; S32: An environmental adaptive dynamic fusion mechanism is introduced, which inputs the preprocessed hydrological and meteorological data into a lightweight environmental encoder to generate dynamic allocation values ​​for AIS feature weights, video feature weights, and radar feature weights in real time. The feature vectors of each mode are weighted and fused to obtain weighted multimodal features. S33: A multi-head cross-attention mechanism embedding a port and waterway spatial constraint mask is used to jointly encode and interactively enhance the weighted multimodal features; the port and waterway spatial constraint mask is dynamically generated according to the relative relationship of region types to guide attention calculation; S34: The hydrological and meteorological data are fused with the interactively enhanced fusion features as environmental correction factors to form a global feature vector that includes ship status and environmental constraints. S35: Set an anomaly classification layer at the top level of the global feature vector to complete the offline training of the anomaly detection model; S4: Input the spatiotemporally aligned multimodal data from S2 into the anomaly detection model trained in S35 to identify abnormal behavior and classify risks; define the types of abnormal ship behavior according to port navigation rules, output the anomaly type, location and confidence level, and classify the anomalies into three risk levels: high, medium and low, based on scene elements. S5: Tiered early warning and air traffic scheduling linkage intervention. Based on the output results of S4, the early warning level is accurately determined by the early warning risk index. Combined with dynamic adaptive triggering rules, the early warning is intelligently triggered. Early warning information is pushed according to standardized protocols. The monitoring system is dynamically optimized through early warning-monitoring closed-loop linkage.

2. The method for monitoring and early warning of abnormal ship behavior based on multimodal perception according to claim 1, characterized in that, In S1, the ship trajectory data is AIS data, including ship position, heading, speed, ship type, and MMSI code, which is obtained according to the actual receiving frequency and then interpolated to 0.1 seconds / time; the video surveillance image data is a 1920×1080 resolution high-definition image, covering the entire port channel and anchorage area; the radar point cloud data includes target distance, bearing, and echo intensity, with a detection range covering 20 nautical miles of the port channel; the hydrological and meteorological data includes wind speed, wind direction, current speed, current direction, visibility, and tide information, with the original sampling frequency at the minute level, interpolated to 0.1 seconds / time, and then subjected to 0-1 normalization processing; Preprocessing operations were performed on various types of data, including: AIS data was processed by removing abnormal jump points using the 3σ criterion, missing data for ≤3 consecutive frames was completed by linear interpolation, and trajectory smoothing was performed using the moving average method; video surveillance images were processed by the dark channel prior dehazing algorithm to eliminate fog interference, and after denoising by 3×3 Gaussian filtering, ship target boxes were extracted using the YOLOv8 target detection model, with a confidence threshold set to 0.8; After removing noise points from the radar point cloud using Euclidean filtering, density clustering is performed using the DBSCAN algorithm, and then the ship target outline is extracted using the convex hull algorithm. All modal data are unified to the UTC time base, with timestamp accuracy to 0.1 seconds, and the data format is unified to JSON.

3. The method for monitoring and early warning of abnormal ship behavior based on multimodal perception according to claim 1, characterized in that, In S2, the unified spatiotemporal coordinate system of the port and waterway is a plane rectangular coordinate system with the port and waterway management center as the origin, the WGS-84 geographic coordinate system as the reference, and the Mercator projection. The unit is meters, and the elevation reference is the local mean sea level. The conversion between latitude and longitude and plane rectangular coordinates is realized through the PyProj library. Using the AIS ship's latitude and longitude position as the initial reference, the video target box and radar clustered target are mapped to the Cartesian coordinate system. Kalman filtering and Hungarian matching algorithm are used to complete the association and tracking of the same ship in multimodal data. The matching threshold of the Hungarian matching algorithm is 10 meters.

4. The method for monitoring and early warning of abnormal ship behavior based on multimodal perception according to claim 1, characterized in that, The encoders at different depths in S31 adopt an asymmetric twin structure: the encoder for extracting AIS trajectory motion features is a 2-layer Transformer encoder, the encoder for extracting video image visual features is a 6-layer Transformer encoder, and the encoder for extracting radar point cloud features is a 4-layer Transformer encoder; the three encoders share some low-level parameters, and the feature vectors output by each branch are mapped to a unified dimension d_model=512 through independent linear projection layers. The method for dynamically calculating the fusion weights of each modal feature in S32 includes: inputting wind speed, visibility, and tide level stage information from hydrological and meteorological data into a lightweight environmental encoder, and outputting dynamic allocation values ​​for each modal feature; when the wind speed is ≥10m / s, the radar point cloud feature weight is greater than the AIS trajectory motion feature weight and the video image visual feature weight; when the visibility is <1000 meters, the radar point cloud feature weight is greater than the video image visual feature weight. The method for generating the port channel spatial constraint mask in S33 includes: dividing the channel area into four categories—main channel, branch channel, anchorage, and prohibited channel—based on port channel geographic information system data; performing path search on the channel topology map based on the ship's current position to predict the set of channel areas the ship will navigate in future time steps; and constructing an attention mask matrix M, where M(i,j)=1 if and only if the channel areas corresponding to positions i and j both belong to the set of channel areas to be navigated or are adjacent areas, otherwise M(i,j)=0. In S34, the environmental correction factor F is calculated using the formula F = 0.2 × normalized wind speed + 0.3 × normalized current velocity + 0.2 × normalized visibility + 0.3 × tidal stage coefficient. The tidal stage coefficient is assigned values ​​of 0.2, 0.1, 0.4, and 0.8 for high tide, low tide, slack tide, and high tide, respectively. The environmental correction is completed by multiplying F with the fused features output from S33 element by element, generating a 512-dimensional global feature vector.

5. The method for monitoring and early warning of abnormal ship behavior based on multimodal perception according to claim 1, characterized in that, In S4, the types of abnormal ship behavior include speeding, deviation, illegal berthing, navigation without AIS, dangerous encounter, sudden change in track, and occupation of prohibited channels. The logic for determining navigation without AIS is that the same ship target is detected for 5 consecutive frames and there is no AIS signal feedback. The confidence threshold for the anomaly detection model was set to 0.

8. Based on vessel type, waterway area, traffic flow density, visibility, and tide level, a preliminary risk assessment was performed using quantified thresholds. The criteria for high, medium, and low risk were as follows: High risk: Large cargo ships deviating from the main channel by ≥50 meters, dangerous encounters when visibility is <500 meters, occupying prohibited channels, sudden changes in track angle ≥30°, or illegal anchoring in the main channel; Medium risk: Small vessels exceeding the speed limit by more than 20% in the channel, deviating from the course by 20-50 meters during non-tide periods, dangerous encounter distance ≥500 meters and <1000 meters, and sudden change in course angle of 15-30°. Low risk: Minor danger may be encountered when the vessel deviates less than 20 meters from the anchorage, experiences minor speed fluctuations of ±5%, illegally anchors in a temporary area of ​​the anchorage, or has visibility of ≥1000 meters.

6. The method for monitoring and early warning of abnormal ship behavior based on multimodal perception according to claim 1, characterized in that, In S5, the early warning risk index R is calculated by combining the anomaly confidence level C, the ship type quantification coefficient, the waterway area quantification coefficient, the hydro-meteorological environment quantification coefficient, the traffic flow density quantification coefficient, and the anomaly type hazard coefficient. The warning level is determined based on the R value: an R value ≥ 0.8 indicates a Level 1 warning, corresponding to an initial assessment of high risk; a R value ≤ 0.5 < 0.8 indicates a Level 2 warning, corresponding to an initial assessment of medium risk. R < 0.5 indicates a Level 3 warning, corresponding to an initial assessment of low risk.

7. The method for monitoring and early warning of abnormal ship behavior based on multimodal perception according to claim 6, characterized in that, The dynamic adaptive triggering rule in S5 is as follows: Extremely high-risk anomalies in the Level 1 warning include occupying prohibited channels, navigating in the main channel without AIS, and dangerous encounters in low visibility. If the conditions are met, the warning will be triggered directly without the need for continuous frame verification. Other Level 1 warnings require 2-3 consecutive frames of stable verification before triggering; Level 2 and Level 3 warnings require stable verification for 3 or more consecutive frames before they can be triggered. When the wind speed is ≥10m / s, the R-value thresholds for Level II and Level III warnings will be lowered to 0.5 and 0.3, respectively.

8. The method for monitoring and early warning of abnormal ship behavior based on multimodal perception according to claim 1, characterized in that, In S5, the early warning information is generated according to a standardized structure, including basic ship monitoring data, early warning-specific data, and visualization support data. It is pushed to the shore-based VTS system, port dispatch platform, and maritime law enforcement terminal within 100 milliseconds using the TCP / IP industrial communication protocol, and also pushes warning information to other ships within 5 nautical miles of the ship. When multiple ships in the same area trigger warnings simultaneously, the warnings are sent out in descending order of R value. If the R values ​​are the same, the warnings are sent out to ships with higher risk levels first.

9. The method for monitoring and early warning of abnormal ship behavior based on multimodal perception according to claim 1, characterized in that, The specific operations of the early warning-monitoring closed-loop linkage in S5 include: After triggering the Level 1 warning, the monitoring sampling frequency of the target vessel will be increased to 0.05 seconds / time, the weight of multimodal feature fusion will be increased by 20%, and the monitoring of the waterway area within 10 nautical miles around it will be intensified. When the abnormal behavior of the target vessel is detected and corrected, and the R value is less than 0.4 for 10 consecutive frames and the vessel status and hydrological and meteorological environment return to normal, the warning is lifted after the dual verification is completed, and the monitoring system restores normal parameters within 30 seconds. All early warning data are incorporated into the training sample library of the monitoring model, and the Transformer feature fusion network and anomaly detection model are iteratively optimized every quarter based on the early warning source tracing data.

Images