A method for ship identification based on AIS and multi-source remote sensing collaboration

Abstract

This invention relates to the fields of remote sensing technology and ship monitoring, and discloses a ship identification method based on AIS and multi-source remote sensing collaboration. The method filters AIS signals through spatiotemporal alignment and compensates for the imaging timestamps of remote sensing images; it utilizes 15 arcsecond resolution DEM data provided by GEBCO and land polygon data from OpenStreetMap to mask land interference; it employs a ResNet50 model with embedded deformable convolutional layers to detect ships and wakes from remote sensing images, and estimates ship speed using a Faster R-CNN model; finally, it constructs and matches feature description vectors extracted from remote sensing images and AIS data, and determines ship identity by calculating Euclidean distance. This method is applicable to port traffic, ship tracking, and safety management applications.

Description

Technical Field

[0001] This invention belongs to the field of remote sensing technology and ship monitoring, specifically relating to a ship identification method based on AIS and multi-source remote sensing collaboration. Background Technology

[0002] With the rapid development of the marine economy, maritime vessel monitoring technology is facing increasing demands and challenges. Currently, the mainstream monitoring methods mainly include synthetic aperture radar (SAR), visible light imaging, and automatic identification systems (AIS). These technologies each have their own characteristics but also exhibit significant complementarity.

[0003] SAR technology, with its all-weather, all-time operating characteristics, has become an important tool for maritime monitoring. Its active microwave imaging mechanism allows it to reliably acquire information about sea surface targets regardless of lighting or weather conditions. However, SAR systems have inherent drawbacks such as long revisit periods and susceptibility to sea state interference due to the lack of backscattering characteristics. This makes it difficult for SAR imagery to provide continuous and complete ship monitoring data in certain situations.

[0004] Visible light imagery provides rich details about ship texture and appearance, which is crucial for ship identification and classification. However, the application of visible light imaging is severely limited by lighting conditions, becoming largely ineffective at night or in adverse weather. Furthermore, cloud cover significantly restricts the reliability of visible light imagery.

[0005] Most importantly, it is difficult to accurately obtain a ship's motion parameters and identity information by relying solely on SAR and visible light imagery.

[0006] As a proactive reporting system for ships, the AIS system provides accurate ship identification information (MMSI code, ship name, etc.) and motion parameters, playing an irreplaceable role in ship monitoring. However, the AIS system relies on the active cooperation of ships and suffers from problems such as signal shutdown and forgery.

[0007] Therefore, there is an urgent need to develop a multi-source data fusion solution. However, achieving effective multi-source fusion faces many technical challenges: First, there is a serious time asynchrony problem between the low-frequency acquisition characteristics of SAR and visible light imagery and the high-frequency updates of AIS data; second, the spatial registration error between different sensor coordinate systems cannot be ignored; in addition, there are significant characterization differences between remote sensing image features (backscattering intensity of SAR, texture information of visible light imagery) and AIS structured data.

[0008] Against this backdrop, the development of a multi-source collaborative matching method integrating AIS, SAR, and visible light imagery is particularly urgent. This method needs to overcome the following key technical challenges: designing an accurate spatiotemporal alignment algorithm to effectively compensate for the observation delays and coordinate deviations of different sensors; and establishing a cross-modal feature correlation model to achieve a unified representation of the SAR scattering characteristics and visible light texture features of ships. Through this innovative multi-source fusion technology, it is expected to overcome the limitations of single sensors and provide a more comprehensive and reliable ship monitoring solution for maritime surveillance. Summary of the Invention

[0009] To address the aforementioned issues, this invention proposes a ship identification method based on the collaboration of AIS and multi-source remote sensing, aiming to provide a reliable method for ship identification using multi-source remote sensing images.

[0010] The objective of this invention is achieved through the following solution:

[0011] A ship identification method based on AIS and multi-source remote sensing collaboration includes the following steps:

[0012] Spatiotemporal alignment of the collected AIS and remote sensing image data for the target sea area is performed. This includes extracting the geographic boundary of the remote sensing image and expanding the geographic boundary by 10km to generate an expanded boundary polygon. AIS signals located within the expanded boundary polygon are selected as AIS data spatially aligned with the remote sensing image. The AIS signals are interpolated to the imaging timestamp of the remote sensing image. The distance difference between the interpolation point and the ship's position in the remote sensing image is calculated. Combined with the speed information in the AIS signal, the imaging time of the remote sensing image is compensated to achieve time alignment.

[0013] Based on spatiotemporally aligned AIS data and remote sensing imagery, land-sea segmentation is applied to shield against land interference;

[0014] Interpolate the AIS signal to the compensated remote sensing image timestamp;

[0015] Deep learning convolutional neural networks and multi-task loss functions are used to detect ships and wakes from remote sensing images;

[0016] Based on the ship and wake detected by the convolutional neural network, including ship coordinates, heading and size information, the image block is cropped according to the preset scale with the ship and wake detection box as the center and input into the speed estimation module to obtain the ship speed information;

[0017] We construct feature description vectors for ship coordinates, heading, size, and speed extracted from remote sensing images. We then find the AIS feature description vector with the smallest Euclidean distance and calculate whether the Euclidean distance between the two is less than a threshold to obtain the final matching result.

[0018] A further solution is that the remote sensing images spatiotemporally aligned with AIS include optical images and SAR images, which have medium to high resolution and can present the characteristics of the moving vessel itself and its wake.

[0019] A further solution involves the following steps for land-sea segmentation based on spatiotemporally aligned AIS data and remote sensing imagery: acquiring seabed depth sounding data of the target sea area, filtering data in the AIS signal that are above the sea level threshold to achieve land masking of the AIS signal; acquiring land polygon data of the target sea area, and performing georeferencing unification of the land polygon data with the remote sensing imagery; performing image registration of the georeferencing unified land polygon data with the remote sensing imagery; and performing land masking processing on the remote sensing imagery based on the registered land polygon data.

[0020] A further approach involves the land polygon data and remote sensing image registration method, which constructs orientation phase consistency histograms for both the land polygon data and the remote sensing image, calculates the sum of squared differences between the high-dimensional vectors formed by stitching the orientation phase consistency histograms of the two images, and uses this sum as the standard for matching similarity measurement. A template matching method is then used to achieve cross-modal corresponding point identification.

[0021] A further approach is to use a ResNet50 with embedded deformable convolutional layers in the deep learning convolutional neural network.

[0022] A further solution involves a speed estimation module that employs a Faster R-CNN model, using real AIS data as labels, and trains it on multi-scene, multi-modal remote sensing images. The input consists of remote sensing image slices containing ships and wakes. By iteratively passing images and adjusting model weights, the loss function is minimized to achieve end-to-end ship speed estimation.

[0023] The beneficial effects of this invention are: by using a spatiotemporal alignment algorithm, it effectively compensates for the observation delay of remote sensing satellite sensors, thereby improving the consistency and accuracy of data; by using a direction phase consistency feature map to perform image registration of land polygon data and remote sensing images, pixel-level aligned land masks are obtained; and by using a deep learning convolutional neural network to extract ship coordinates, size, heading, and speed information from remote sensing images end-to-end, it reduces manual intervention, improves the system's automation level, and achieves unified representation of ship features from multi-source remote sensing images and AIS signals.

[0024] Any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description

[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are merely some embodiments of the present invention and are not intended to limit the present invention.

[0026] Figure 1 This is a flowchart of a ship identification method based on the collaboration of AIS and multi-source remote sensing. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the specific embodiments of this invention will be further described in detail below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of this invention and should not be construed as limiting the scope of protection of this invention.

[0028] It should be understood that the reference to "embodiment" herein means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0029] This invention provides a ship identification method based on the collaboration of AIS and multi-source remote sensing, which will be described in detail below.

[0030] Figure 1 The flowchart shows a method for ship identification based on AIS and multi-source remote sensing collaboration, such as... Figure 1 As shown, the method includes the following steps:

[0031] Step 1: Perform spatiotemporal alignment on the collected AIS and remote sensing image data of the target sea area:

[0032] Spatial alignment first extracts the geographic boundary of the remote sensing image and then extends the geographic boundary by 10km to generate an extended boundary polygon. AIS signals located within the extended boundary polygon are then selected as AIS data spatially aligned with the remote sensing image.

[0033] Spatiotemporal alignment of the collected AIS and remote sensing imagery data for the target sea area:

[0034] Spatial alignment first extracts the geographic boundary of the remote sensing image and then extends this boundary by 10 km to generate an expanded boundary polygon. AIS signals located within this expanded boundary polygon are then selected as AIS data spatially aligned with the remote sensing image. This ensures that AIS signals of ships located at the edge of the remote sensing image have sufficient AIS sample points before and after the timestamp of the compensated remote sensing image, thereby improving the reasonableness of the interpolation and ultimately enhancing the accuracy of ship identification.

[0035] Due to the time delay in satellite sensor imaging, typically ranging from several minutes to over twenty minutes, a discrepancy arises between the AIS signal point corresponding to the timestamp recorded in the remote sensing image metadata and the actual position of the ship in the image. Therefore, it is necessary to compensate for the imaging timestamp of the remote sensing image. This embodiment achieves time alignment through the following steps: calculating the distance between the ship's detected position in the remote sensing image and the ship's position recorded by the AIS signal point corresponding to the metadata timestamp; combining the ship's speed information in the AIS, calculating the compensation value for the metadata timestamp; and averaging all compensation values ​​to compensate for the actual imaging time of the remote sensing image. This achieves time alignment between the AIS signal and the remote sensing image.

[0036] Step 2: Based on spatiotemporally aligned AIS data and remote sensing imagery, apply land-sea segmentation to shield against land interference;

[0037] The seabed bathymetry data used in this embodiment comes from the 2022 global 15-arcsecond resolution DEM (Digital Elevation Model) data provided by GEBCO (General Bathymetric Chart of the Oceans). After acquiring the AIS data, the geographic coordinates of each record are extracted, and the corresponding elevation value is matched from the GEBCO DEM data. This value is then compared with a preset sea level height threshold, and AIS signals higher than the threshold are removed to ensure that subsequent processing is only performed on ships in the ocean.

[0038] The land polygon data used in this embodiment is obtained from OpenStreetMap. This data provides land boundary information within the target sea area for subsequent land masking processing. After georeferencing the acquired land polygon data with the remote sensing image, due to geometric distortions in satellite sensor imaging, the land contours of the land polygon data and the remote sensing image do not perfectly match. Therefore, directional phase consistency histograms are constructed separately. Specifically, the response of each pixel in the image at different directions and scales is calculated using a set of Log-Gabor filters with different directions and scales, and the phase consistency amplitude is calculated.

[0039]

[0040] in, It is the first The response of each filter, Indicates scale. Indicates direction, It represents the number of filters.

[0041] Calculate the direction of phase consistency:

[0042]

[0043] use Pixel windows, each window is divided into Overlapping unit blocks, each unit block is For each pixel, calculate the magnitude and direction of phase coherence of pixels within that block. (The direction range is then considered.) The image is divided into 8 bins, and the phase consistency amplitude within each bin is calculated to form a directional phase consistency histogram. L2 norm normalization is performed on the directional phase consistency histogram of each bin. The histograms of all bins are concatenated into a high-dimensional vector. The sum of squared Euclidean distances between these vectors in two images is calculated as the standard for measuring matching similarity. Template matching is used to achieve cross-modal corresponding point recognition. Because phase consistency is invariant to illumination and contrast, it is suitable for multimodal image matching tasks.

[0044] Step 3: Interpolate the AIS signal to the compensated remote sensing image timestamp.

[0045] Step 4: Detect ships and wakes from remote sensing images using deep learning convolutional neural networks and multi-task loss functions.

[0046] ResNet50 with embedded deformable convolutional layers is used to detect ships and wakes. The deformable convolutional layers employ a 9-point sampling strategy, generating geometrically constrained deformable convolutional kernels based on the predicted bounding boxes to achieve sub-pixel-level feature space calibration.

[0047] In this embodiment, the training data for ships and wakes comes from the SWIM (Ship Wake Image) dataset, consisting of approximately 14,000 labeled images. After training, ships and wakes are detected in the area of ​​interest, and the ship's coordinates, size, and heading information are extracted. A slice containing the ship and the complete wake is also obtained to facilitate subsequent speed estimation.

[0048] Step 5: Input the ship and wake slices into the speed estimation module, and output the ship speed end-to-end:

[0049] In this embodiment, the speed estimation module adopts the Faster R-CNN model, which uses real AIS data as labels and is trained on remote sensing images of multiple scenes and multiple modalities. The remote sensing image slices containing ships and wakes obtained in step 4 are input, and the model weights are adjusted by iteratively passing the images and minimizing the loss function to achieve end-to-end ship speed estimation.

[0050] Step 6: Construct the feature description vectors of ship coordinates, heading, size and speed extracted from remote sensing images. Find the AIS feature description vector with the smallest Euclidean distance and calculate whether the Euclidean distance between the two is less than a threshold to obtain the final matching result.

[0051] The scope of protection of this invention is not limited to the technical solutions described in the above specific embodiments. All reasonable changes and optimizations made within the technical principle framework of this invention are within the scope of protection of this patent. Any adaptive modifications made by those skilled in the art based on the disclosure of this invention, as long as they do not depart from the essence of the technical solution defined in the claims of this invention, should be considered to fall within the patent protection scope of this invention.

Claims

1. A method for ship identification based on AIS and multi-source remote sensing collaboration, characterized in that, Includes the following steps: Spatiotemporal alignment of the collected AIS and remote sensing image data for the target sea area is performed. This includes extracting the geographic boundary of the remote sensing image and expanding the geographic boundary by 10km to generate an expanded boundary polygon. AIS signals located within the expanded boundary polygon are selected as AIS data spatially aligned with the remote sensing image. The AIS signals are interpolated to the imaging timestamp of the remote sensing image. The distance difference between the interpolation point and the ship's position in the remote sensing image is calculated. Combined with the speed information in the AIS signal, the imaging time of the remote sensing image is compensated to achieve time alignment. Based on spatiotemporally aligned AIS data and remote sensing imagery, land-sea segmentation is applied to shield against land interference; Interpolate the AIS signal to the compensated remote sensing image timestamp; Deep learning convolutional neural networks and multi-task loss functions are used to detect ships and wakes from remote sensing images; Based on the ship and wake detected by the convolutional neural network, including ship coordinates, heading and size information, the image block is cropped according to the preset scale with the ship and wake detection box as the center and input into the speed estimation module to obtain the ship speed information; We construct feature description vectors for ship coordinates, heading, size, and speed extracted from remote sensing images. We then find the AIS feature description vector with the smallest Euclidean distance and calculate whether the Euclidean distance between the two is less than a threshold to obtain the final matching result.

2. The ship identification method based on AIS and multi-source remote sensing collaboration as described in claim 1, characterized in that, The remote sensing images that are spatiotemporally aligned with AIS include optical images and SAR images, which have medium to high resolution and can present the characteristics of the moving vessel itself and its wake.

3. The ship identification method based on AIS and multi-source remote sensing collaboration as described in claim 1, characterized in that, The land-sea segmentation based on spatiotemporally aligned AIS data and remote sensing imagery specifically includes the following steps: acquiring seabed depth sounding data of the target sea area, filtering data in the AIS signal that are above the sea level threshold to achieve land masking of the AIS signal; acquiring land polygon data of the target sea area, and performing georeferencing unification of the land polygon data with the remote sensing imagery; performing image registration of the georeferencing unified land polygon data with the remote sensing imagery; and performing land masking processing on the remote sensing imagery based on the registered land polygon data.

4. The method for registering land polygon data with remote sensing imagery as described in claim 3, characterized in that, Orientation phase consistency histograms are constructed for land polygon data and remote sensing images respectively. The sum of squared Euclidean distance differences between the high-dimensional vectors formed by stitching the orientation phase consistency histograms of the two images is calculated as the standard for matching similarity measurement. Template matching method is used to realize cross-modal corresponding point identification.

5. The ship identification method based on AIS and multi-source remote sensing collaboration as described in claim 1, characterized in that, The deep learning convolutional neural network uses ResNet50 with embedded deformable convolutional layers.

6. The ship identification method based on AIS and multi-source remote sensing collaboration as described in claim 1, characterized in that, The speed estimation module specifically employs the Faster R-CNN model, using real AIS data as labels, and trains on multi-scene, multi-modal remote sensing images. The input consists of remote sensing image slices containing ships and wakes. By iteratively passing images and adjusting model weights, the loss function is minimized to achieve end-to-end ship speed estimation.

Images