Systems and methods for vessel classification from SAR imagery

Abstract

Provided herein are systems and methods for vessel classification of SAR images using a Deep Learning model, the method includes training a Deep Learning model by generating training chips from previously acquired SAR images, generating training, validation, and test manifest files of generated training chips, training the Deep Learning model using a training manifest file, validating the Deep Learning model using a validation manifest file, and testing the Deep Learning model using a test manifest file, classifying vessels of interest (VOI) by acquiring a SAR image, detecting VOI within the SAR image, preparing image chips for each detected vessel in the SAR image, wherein each training chip includes image data and image metadata for one VOI, and predicting a vessel class of the image chips by the Deep Learning model, and outputting, by the Deep Learning model, an output file which includes a vessel class prediction for each VOI.

Description

SYSTEMS AND METHODS FOR VESSEL CLASSIFICATION FROM SAR IMAGERYTechnical Field

[0001] The following relates generally to maritime observation and satellite imaging, and more particularly to classification of maritime vessels from satellite-acquired imagery.Introduction

[0002] Satellites provide a great deal of the remote sensing imagery commonly used today. Satellites have several unique characteristics which make them particularly useful for remote sensing of the Earth's surface. One particular domain in which satellite imaging is used is maritime observation.

[0003] Vessel (or ship) detection, including the ability to detect non-transmitting dark ships and identify illegal fishing activity, can also be accomplished using satellite imaging. For example, synthetic aperture radar (SAR) satellite imagery is used to detect and monitor vessel traffic over large areas of the ocean. Detected vessels may be correlated with other sources of vessel information, including an automatic identification system (AIS). When other information cannot be found or a vessel is otherwise considered suspicious, there may be a need to infer further information from the SAR imagery, including a vessel type.

[0004] However, the classification of a vessel from a SAR image is very challenging, even for a human, with manual classification by humans having an accuracy between 50-60%. Difficulties in classification are due to low resolution compared to an optical image, defocussing, and noise, as well as surrounding ocean clutter. Additionally, vessels of different classes can appear very similar.

[0005] Classification by computers, for example through machine learning models, do not fair better to date. A machine learning model takes an image chip of a fixed size as an input which may include other objects or vessels in the chip which are not of interest, preventing accurate classification.

[0006] Accordingly, there is a need for systems and methods for classification of vessels through SAR imagery, that overcome at least some of the disadvantages of existing systems and methods.Summary

[0007] Provided herein is a method for vessel classification of SAR images using a Deep Learning model, the method including training a Deep Learning model by generating training chips from previously acquired SAR images, wherein each training chip includes image data, image metadata, and true label data for one vessel, wherein the true label data includes a true identification of a vessel class for the one vessel, generating a training manifest file, a validation manifest file, and a test manifest file each representing one of three groups of the generated training chips, training the Deep Learning model using the training manifest file, validating the Deep Learning model using the validation manifest file, and testing the Deep Learning model using the test manifest file; classifying vessels of interest (VOI) by acquiring at least one SAR image, detecting VOI within the at least one SAR image, preparing image chips for each detected vessel in the at least one SAR image, wherein each training chip includes image data and image metadata for one VOI, and predicting a vessel class of the image chips by the Deep Learning model; and outputting, by the Deep Learning model, an output file which includes a vessel class prediction for each VOI.

[0008] Detecting VOI within the at least one SAR image further may include outputting coordinates of the vessels of interest.

[0009] Detecting VOI within the at least one SAR image may include identifying transmitted AIS signals of vessels within the at least one SAR image.

[0010] Image chips for vessels without AIS signals may be input into the Deep Learning model for vessel class prediction, wherein image chips for vessels with AIS signals are used for further model training.

[0011] Preparing the image chips, predicting vessel classes, and outputting the output file may be performed by an inference script employing the Deep Learning model.

[0012] Chip preparation may further include converting latitude and longitude information from the at least one SAR image to a line-pixel location for each detected VOI, using the line-pixel location of each detected VOI to extract an image chip and write the image chip to a file, and creating an entry in a manifest file for each detected VOI, including the extracted image chip, detection information, and image metadata.

[0013] Converting the latitude and longitude information to the line-pixel location may include interpolating a geoid height using the latitude and longitude of the detection from a gridded geoid height file, and evaluating a rational function with the geoid height, latitude, and longitude to obtain the line and pixel.

[0014] The output file may include a table comprising a vessel class prediction for each VOI.

[0015] The output file may further include a classification probability for each vessel class in the Deep Learning model.

[0016] The at least one SAR image may be preprocessed from raw SAR data to a multi-look image for detecting VOI.

[0017] The least one SAR image may be preprocessed from raw SAR data to a single-look image for preparing image chips.

[0018] The training chips may be normalized by a mean and standard deviation of all vessel pixels.

[0019] The mean and standard deviation may be measured on the training chips and used for both training the Deep Learning model and for vessel class prediction by the Deep Learning model.

[0020] A vessel mask may be computed using the mean noise around an edge of an image chip as a threshold and setting a mask value to 1 when the image is above the threshold and to 0 when the image is at or below the threshold.

[0021] Training the Deep Learning model may include ingesting the training manifest file by a training sequence generator and ingesting the validation manifest file by a validation sequence generator, wherein both the training sequence generator andthe validation sequence generator ingest a training statistics file for preprocessing and normalizing training chips.

[0022] The training manifest file may include true label data and the true label data is used to optimize model weights during model training.

[0023] The validation manifest file may include true label data and the true label data is used to determine accuracy during model validation.

[0024] Model parameters comprising a pre-set network structure and parameters set by a designer may be used for model training.

[0025] Augmentation may be performed on the training chips, during model training, to increase the number of training chips, wherein augmentation may include at least one of noise addition, translation, azimuth scale, azimuth flip, and quantization.

[0026] The Deep Learning model may have a structure based on a ResNet network.

[0027] Provided herein is a computer system comprising at least one processor configured to execute the methods described above and a non-transitory computer readable medium storing processor-executable instructions which, when executed by at least one processor, cause the processor to perform the method as described above.

[0028] Provided herein is a computer system for vessel classification of SAR images using a Deep Learning model, the system including at least one data storage device for storing a plurality of SAR images, a plurality of training chips, a plurality of manifest files, and at least one Deep Learning model, and at least one processor configured to execute processor-executable instructions stored on the at least one data storage device, the instructions configured for training a Deep Learning model by generating training chips from previously acquired SAR images, wherein each training chip includes image data, image metadata, and true label data for one vessel, wherein the true label data includes a true identification of a vessel class for the one vessel, generating a training manifest file, a validation manifest file, and a test manifest file each representing one of three groups of the generated training chips, training the Deep Learning model using the training manifest file, validating the Deep Learning model usingthe validation manifest file, and testing the Deep Learning model using the test manifest file; classifying vessels of interest (VOI) by acquiring at least one SAR image, detecting VOI within the at least one SAR image, preparing image chips for each detected vessel in the at least one SAR image, wherein each training chip includes image data and image metadata for one VOI, and predicting a vessel class of the image chips by the Deep Learning model; and outputting, by the Deep Learning model, an output file which includes a vessel class prediction for each VOI.

[0029] Detecting VOI within the at least one SAR image further may include outputting coordinates of the vessels of interest.

[0030] Detecting VOI within the at least one SAR image may include identifying transmitted AIS signals of vessels within the at least one SAR image.

[0031] Image chips for vessels without AIS signals may be input into the Deep Learning model for vessel class prediction, wherein image chips for vessels with AIS signals are used for further model training.

[0032] Preparing the image chips, predicting vessel classes, and outputting the output file may be performed by an inference script employing the Deep Learning model.

[0033] Chip preparation may further include converting latitude and longitude information from the at least one SAR image to a line-pixel location for each detected VOI, using the line-pixel location of each detected VOI to extract an image chip and write the image chip to a file, and creating an entry in a manifest file for each detected VOI, including the extracted image chip, detection information, and image metadata.

[0034] Converting the latitude and longitude information to the line-pixel location may include interpolating a geoid height using the latitude and longitude of the detection from a gridded geoid height file, and evaluating a rational function with the geoid height, latitude, and longitude to obtain the line and pixel.

[0035] The output file may include a table comprising a vessel class prediction for each VOI.

[0036] The output file may further include a classification probability for each vessel class in the Deep Learning model.

[0037] The at least one SAR image may be preprocessed from raw SAR data to a multi-look image for detecting VOL

[0038] The least one SAR image may be preprocessed from raw SAR data to a single-look image for preparing image chips.

[0039] The training chips may be normalized by a mean and standard deviation of all vessel pixels.

[0040] The mean and standard deviation may be measured on the training chips and used for both training the Deep Learning model and for vessel class prediction by the Deep Learning model.

[0041] A vessel mask may be computed using the mean noise around an edge of an image chip as a threshold and setting a mask value to 1 when the image is above the threshold and to 0 when the image is at or below the threshold.

[0042] Training the Deep Learning model may include ingesting the training manifest file by a training sequence generator and ingesting the validation manifest file by a validation sequence generator, wherein both the training sequence generator and the validation sequence generator ingest a training statistics file for preprocessing and normalizing training chips.

[0043] The training manifest file may include true label data and the true label data is used to optimize model weights during model training.

[0044] The validation manifest file may include true label data and the true label data is used to determine accuracy during model validation.

[0045] Model parameters comprising a pre-set network structure and parameters set by a designer may be used for model training.

[0046] Augmentation may be performed on the training chips, during model training, to increase the number of training chips, wherein augmentation may include at least one of noise addition, translation, azimuth scale, azimuth flip, and quantization.

[0047] The Deep Learning model may have a structure based on a ResNet network.

[0048] Other aspects and features will become apparent to those ordinarily skilled in the art, upon review of the following description of some exemplary embodiments.Brief Description of the Drawings

[0049] The drawings included herewith are for illustrating various examples of articles, methods, and apparatuses of the present specification. In the drawings:

[0050] Figure 1 is a block diagram of an operational workflow of a vessel classification system, according to an embodiment;

[0051] Figure 2 are examples of image chips of vessels generated or processed by the systems and methods of the present disclosure, according to an embodiment;

[0052] Figure 3 is a block diagram of a workflow of chip extraction for classification model training or for input into a trained model, according to an embodiment;

[0053] Figure 4A is a block diagram of a processing workflow for raw SAR imagery data during vessel detection and classification, according to another embodiment;

[0054] Figure 4B is an example of a multi-look SAR image, according to an embodiment;

[0055] Figure 4C is an example of a single-look SAR image, according to an embodiment;

[0056] Figure 5 is an example of a masking process for preprocessing image chips for input into the classification model, according to an embodiment;

[0057] Figure 6 is a block diagram of a processing workflow of training a classification model, according to an embodiment;

[0058] Figure 7 is an example of a ResNet based structure of a Deep Learning model network used for vessel classification, according to an embodiment;

[0059] Figure 8 is an example of a single ResNet Layer of a Deep Learning model network, according to an embodiment; and

[0060] Figure 9 is a block diagram of a workflow of vessel classification prediction by a trained model, according to an embodiment.Detailed Description

[0061] Various apparatuses or processes will be described below to provide an example of each claimed embodiment. No embodiment described below limits any claimed embodiment and any claimed embodiment may cover processes or apparatuses that differ from those described below. The claimed embodiments are not limited to apparatuses or processes having all of the features of any one apparatus or process described below or to features common to multiple or all of the apparatuses described below.

[0062] One or more systems described herein may be implemented in computer programs executing on programmable computers, each comprising at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, and without limitation, the programmable computer may be a programmable logic unit, a mainframe computer, server, and personal computer, cloud-based program or system, laptop, personal data assistance, cellular telephone, smartphone, or tablet device.

[0063] Each program is preferably implemented in a high-level procedural or object oriented programming and / or scripting language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a storage media or a device readable by a general or special purpose programmable computer for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein.

[0064] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary, a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0065] Further, although process steps, method steps, algorithms or the like may be described (in the disclosure and / or in the claims) in a sequential order, such processes, methods and algorithms may be configured to work in alternate orders. In other words,any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any order that is practical. Further, some steps may be performed simultaneously.

[0066] When a single device or article is described herein, it will be readily apparent that more than one device / article (whether or not they cooperate) may be used in place of a single device / article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device / article may be used in place of the more than one device or article.

[0067] The following relates generally to maritime observation and satellite imaging, and more particularly to classification of maritime vessels from satellite-acquired imagery.

[0068] The vessel classification system of the present disclosure may significantly improve the ability to classify or (otherwise provide information about) maritime vessels of interest (“VOIs”). The vessels may be “dark targets” that are not transmitting automatic identification system (AIS) signals or vessels that exhibit other suspicious behaviour. AIS is an automatic tracking system that uses transceivers on ships and is used by vessel traffic services. Satellites may be used to detect AIS signatures, in which case the term Satellite-AIS (S-AIS) may be used. S-AIS may be used for collision avoidance, identification, and location information, as well as for maritime domain awareness, search and rescue, environmental monitoring, and maritime intelligence applications. Generally, dark ship detection includes identifying vessels that are not transmitting AIS or are spoofing AIS information and hotspots where potential illegal activity is occurring.

[0069] The vessel classification system includes a Deep Learning model which has been trained with on the order of thousands of image chips (“training chips”) of detected and classified vessels. The trained model can predict a class of vessel from a preset group of possibilities, for example tanker, cargo, or fishing vessel. The taxonomy for classification depends on the training data available for each class, with the taxonomy being expandable where there is sufficient training data for a given class. That is, only classes with sufficient training data are included in the taxonomy.

[0070] Referring now to Figure 1 , shown therein is a block diagram of an operational workflow 100 of a vessel classification system, according to an embodiment.

[0071] Each block in figure 1 represents an input or process within the workflow of the vessel classification system.

[0072] Blocks above the dash-dot-dot line of Figure 1 represent the training process for the Deep Learning model (represented by dashed line arrows).

[0073] Blocks below the dash-dot-dot line represent the inference process of the vessel classification system wherein a class or type of a vessel of interest (VOI) is predicted (represented by straight line arrows).

[0074] The insertion of VOI image chips and data into the training process is shown by dotted arrows.

[0075] Broadly, images of maritime areas of interest are acquired during the process of image acquisition 110. Since vessel classification requires that a vessel first be detected in an image, images or sets of images which can be used first for vessel detection and subsequently for vessel classification are required. In practice, the requirements for vessel detection versus vessel classification may not be the same. For example, vessel detection often involves a trade-off between a larger swath width to search a larger area and a resolution which enables detectability. Classification, as well, is improved with better resolution. Additionally, the visibility of vessel features for vessel classification depends on the incidence angle at acquisition. The processing of acquired images for detection and classification is shown and discussed at Figure 4A.

[0076] Vessels are detected by vessel detection software during the process of vessel detection 120. The vessel detection software may detect vessels from a large SAR image (e.g., large imaging swath), for example covering hundreds of kilometres and acquires coordinates for possible vessels. The output of the vessel detection software includes a list of locations as coordinates for vessels of interest.

[0077] In other embodiments, other sensors may be used to acquire images of an area of interest. Different sensors will require different Deep Learning models trained on data acquired by the respective sensor for vessel classification. That is, a model trainedon images from one sensor type will not be able to predict vessel classifications for images acquired by a different sensor type. In some cases, a different sensor may be used to detect the vessel (in order to obtain the vessel’s location). Classification is applied to the same type of sensor data as the classification model was trained on.

[0078] Vessel detection may further include vessel identification wherein AIS software is used to identify a transmitted AIS signal of a vessel of interest. Further classification of a vessel of interest may be desired when there is no AIS signal, i.e. , the vessel is “dark” or when there are inconsistencies with the AIS signal (e.g., the movement of the ship or the type of ship does not appear to match the AIS information).

[0079] When a vessel is detected, and the class or type of the detected vessel is desired to be known, the acquired images are input into an inference script which employs the Deep Learning model. The inference script includes two main stages: chip preparation and vessel prediction. The script calls the lower level functions for data preparation, loading and running of the trained Deep Learning model, prediction of the classes of the vessels, and the generation of outputs.

[0080] For detected vessels of interest, an image chip is prepared by an image chip preparation operation 130. The image chips are of a fixed size. For example, the fixed size may be 300 x 300 pixels or 500 x 500 pixels.

[0081] A single chip is extracted from the acquired images for each detected vessel of interest. A manifest file of a set of image chips is created. Detection information of an image includes latitude and longitude information. The latitude and longitude information is converted to a line-pixel location in the image. This may be done using rational function coefficients of the image product.

[0082] The line-pixel location is used to extract the image chip and write the image chip to a file. This file is used along with the detection information and metadata to create an entry in the manifest file. The image chips may undergo preprocessing described further below.

[0083] The inference script runs an ingest function to read the manifest file and training statistics and inputs the image chips into the Deep Learning model 140. Themodel 140 generates predictions and the predictions are written to an output file. The output file may represent output 150 or the output file may be translated into a report or other form as output 150.

[0084] An example output may include a table comprising a classification result for each detected vessel, i.e. , each image chip, as well as the classification probabilities for each vessel class in the model 140. For example, if the model had four vessel classes, cargo, tanker, fishing, and other, each image chip would be classified as one of the four classes when the classification probability if over a pre-determined threshold for a class or would be classified as unknown when none of the classification probabilities is over the threshold.

[0085] Model training 160 of the model 140 includes inputting image data, image metadata, and label data (i.e., true identification data) for previously acquired images 162 which have been extracted as training chips 164 and providing as manifest files. The previously acquired image data is split into three groups, training, validation, and test, wherein each group is input into model training 160, in order, as a separate manifest file. The training and validation manifest files include label data while the test manifest file does not.

[0086] Verification 170 of the predicted class for each image chip in output 150 may be performed by comparing to AIS data or other means of vessel identification. Verification of the predicted class of a vessel of interest is used to determine if the image chip of the vessel of interest is correct and could be added, along with associated data, to the training chips 164 for continuous training and improvement of the Deep Learning model 140.

[0087] Referring now to Figure 2, shown therein are example vessel image chips 200, according to an embodiment.

[0088] The top left image is of a tanker vessel. The top middle and top right images are of cargo vessels. The bottom left and middle images are of fishing vessels. The bottom right image shows multiple vessels.

[0089] Specifics of image data, image metadata, and label data are discussed below.

[0090] Image Data

[0091] Vessel classification requires a ship to first be detected in an image. There may be differences between image requirements for detection of ships and their classification.

[0092] To avoid false alarms due to noise during ship detection, the images input for detection may be smoothed by applying multi-looking during image formation.

[0093] Also, since the whole image is loaded for detection, it may be desirable to reduce the data size by coarser sampling. Ship classification, on the other hand, only looks at a small chip of the image at a time, and requires the best resolution possible. Therefore, ship classification is performed on single-look, oversampled data.

[0094] Image Meta-Data

[0095] Acquired images include associated meta-data. For example, a product.xml file for a SAR image may contain parameters about the acquisition and processing.

[0096] The meta-data may include the start-time of the acquisition, the rational function coefficients of the image, and the pass-direction and look-direction of the acquisitions.

[0097] The start-time is used to identify the acquisition frame that corresponds to a detection.

[0098] The rational functions map the latitude and longitude of a detection to a linepixel value in a single-look (SGX or SLC) image. The line-pixel location in the single look (SGX or SLC) image is used to extract the chip for input to the Deep Learning classifier model.

[0099] The pass direction (ascending or descending), and the look direction (right or left), are used to flip the image in range, if necessary, to obtain a consistent radar look direction in the image.

[0100] Label Data

[0101] The label data are the true classes of the vessels.

[0102] The Deep Learning model inputs the label data during training, for computing a loss function for the optimization of model weights.

[0103] As an example, the main classes may be tanker, cargo and fishing, while there may be a number of other classes which have a relatively small number of instances in the data, such as pleasure craft, tug, coast guard vessel, etc., which may be grouped together in an “other” class.

[0104] In an embodiment, the true classes for the classification taxonomy may be obtained by determining an MMSI number from the AIS correlation, and looking up the MMSI number in a database (e.g., HIS) to get a StatCode5 value.

[0105] If a vessel does not have AIS information associated with it, it is indicated as ‘unlabeled’. Unlabeled data may or may not be included in training data.

[0106] Data Preparation

[0107] A first step for generating image chips and manifest files is to gather the data for each detection produced by detection software.

[0108] For inference, a single image chip is used and the chip matches one entry in the detection file.

[0109] For training, a corresponding image for each detection file is found by matching the start time of the acquisition in metadata in detection and product (image) folders.

[0110] Detection attributes for each detection file and corresponding image product as well as detection attributes for chip extraction are shown in Tables 1 and 2.

[0111] Table 1

[0112] Tabe2

[0113] Referring now to Figure 3, shown therein is a block diagram of a processing workflow 300 of preparing input data for classification model training or for input into a trained model, according to an embodiment.

[0114] As described above, the input for the Deep Learning model ingestion, for either training or inference (i.e., prediction), may be a manifest file which lists all of the chips and associated data and properties. The manifest files may be son files. The properties of the image chips may include the chip file path, acquisition parameters, and detection parameters. When the manifest file is a training or validation manifest file, the properties also include the true labels of each vessel.

[0115] Figure 3 shows a workflow for generating the image chips and manifest files for the input data.

[0116] As described above, only images which include a detected vessel are used for training or are of interest for vessel classification. Detection data 310 from images with detected vessels includes target identifications (ID), time of acquisition, location of vessel in longitude and latitude, and a vessel type (if the data will be used for training).

[0117] The detection data 310 is used for both generation of image chips 330 through chip extraction 320, and generation of manifest files 350 through chip manifest entry 340. The extraction process requires the image data, image metadata, and the rational function coefficients.

[0118] Image chips are extracted from acquired images as described above. Detection information includes a latitude and longitude, which is converted to a line-pixel location in the image using the image product’s rational function coefficients. The linepixel location is then used to extract the chip.

[0119] In an embodiment, specifically, the line and pixel in the detected vessel image is calculated by: interpolating the geoid height using the latitude and longitude of the detection, from the gridded geoid height file, and evaluating the rational function with the geoid height, latitude, and longitude to obtain the line and pixel in the single-look (SLC or SGX) image, wherein if the line or pixel are too close to the edge of the image, the detection is skipped.

[0120] The above assumes that the terrain height used to determine the latitude and longitude in the product used for detection, is the same as the terrain height used to determine the line and pixel in the image for classification.

[0121] An SGX product, as described above, will already be at a desired pixel spacing and, therefore, the extraction of the image chip includes reading the part of the image with the correct offset and size.

[0122] An SLC product is complex-valued, and is at the native radar resolution. Therefore, the image pixel spacing and the incidence angle of the detection are used to calculate the interpolation points for resampling the data to the desired pixel spacing. To accommodate the resampling, a larger chip around the detection is read, and then the desired sized chip is extracted from the resampled data and magnitude detected. The pixel spacing of the resampled complex data is chosen to be the same as that of the SGX product, which provides enough upsampling to allow magnitude detection of the chip without significant aliasing.

[0123] SAR images may be flipped in range and azimuth to obtain an approximately North-up, East-right orientation. However, to preserve the radardependent scattering properties that may provide important features used by the machine-learning algorithm, the image chips should have consistent radar look direction. Therefore, some of the images may be flipped.

[0124] A first output of the chip extraction process 320 is image chips 330.

[0125] A second output of the chip extraction process is to write the image chip information into a manifest file as a single manifest entry for each image chip. The manifest file entry includes the image chip data as well as any detection data associated with the image chip, metadata, true labels (for training), and a chip file location.

[0126] At least one manifest file 350 ( son file) is output from manifest file generation 340. As described above, for training, there are three different manifest files: training, validation, and test, which only training and validation including true labels of the image chips. For classification, the manifest file includes the detected vessel image chips but there is no true label data.

[0127] Referring now to Figure 4A, shown therein is a block diagram of a processing workflow 400 for raw SAR imagery data during vessel detection and classification. Figure 4B is an example 425 of a multi-look image, specifically a SAR Georeferenced Fine (SGF) 2x2 multi-looking SAR image, according to an embodiment. Figure 40 is an example of a single-look image, specifically a SAR Georeferenced Extra (SGX) of Single Look Complex (SLC) 1 -look SAR image 455, according to an embodiment.

[0128] Figures 4A-C show the different processing steps which are applied to an image for vessel detection versus vessel classification.

[0129] For example, for vessel detection, raw SAR data 410 may be multi-looked by a factor of four, two in azimuth and two in range, and SGF processed to a 6.25 m pixel spacing to produce multi-look 420.

[0130] Image 420 is input into vessel detection 430. Vessel detection 430 can detect a vessel as well as assigning a target ID, time, and location of the vessel. If a vessel id detected in image 420, vessel identification is searched for in AIS 440.

[0131] The AIS system 440 provided a label for the vessel with a target ID and vessel type, if the AIS information is available.

[0132] For vessel classification, raw SAR data 410 is not multi-looked but is processed as a single-look image, for example, to an SGX product with oversampling to a pixel spacing (e.g., 2m) or to an SLC product at the resolution of complex-value radar data. The trained Deep Learning model may be trained for either an SGX or SLC image.

[0133] The above scenarios apply to both training image and vessel of interest images. However, if a vessel of interest is identified by AIS 440 then input of the vessel of interest into a trained model for classification may not be necessary unless there is a discrepancy in the AIS data.

[0134] The single-look image 450, vessel detection information 430, and optionally the AIS data 440, are used in the chip manifest generation and chip detection processes 460, as described in Figure 3.

[0135] Figure 4B is an example of a SGF 2x2 multi-look image 425 while Figure 40 is the corresponding 1 -look SGX image 455 of the same vessel.

[0136] Referring now to Figure 5, shown therein is an example of a masking process 500 for processing image chips for input into the classification model, according to an embodiment.

[0137] The training of a Deep Learning model occurs more quickly when the inputs are normalized to a zero mean and a unit standard deviation. Therefore, when image chips are input into the trained model, the chips are normalized by mean and standard deviation of all vessel pixels. For the vessel classification system, the statistics for normalization are measured on the training data, and then are input to the model ingest during both training and inference.

[0138] A SAR image is black and white and has a high dynamic range. Therefore, an extracted image chip consists of some bright pixels corresponding to the ships of different sizes, on a relatively dark background. Thus, it is important to measure the mean and standard deviation only over vessel pixels and not the entire chip.

[0139] To accomplish this, the data preparation includes the computation of a “vessel mask”. The vessel mask is stored in a file which is used to minimize the dynamic range. The mask is determined by first using the mean noise around the edge of an image chip as a threshold, and then setting the mask value to 1 if the image is above the threshold, and 0 otherwise.

[0140] The masks of chips in the training data set are used to compute the training statistics for use in normalization. The training statistics are stored in a son file in the same folder as the trained model weights.

[0141] In Figure 5, a mask is created for original image chip 510. The noise around the edge of the chip is measured, as shown in image chip 520, and the mean noise is taken as a threshold value of the chip.

[0142] A mask 530 is created from the mean noise threshold value and stored as a vessel mask file.

[0143] The vessel mask file is applied to the image chip to generate image chip 540.

[0144] Deep Learning Model Training

[0145] For training the model, a very large number of image frames and corresponding detection files are prepared. Detection files are those acguired image files which have undergone vessel detection. Once the image chips are prepared, a training script instantiates classes for ingest, optimization, and storage of the trained model.

[0146] For each of the detection files, a corresponding image product is found by matching the frame start times in the metadata in the detection file and the product folders (image folders).

[0147] As discussed above, the detection information includes a latitude and longitude, which is converted to a line-pixel location in the image using the image product’s rational function coefficients.

[0148] The line-pixel location is to locate the detection in the image and extract the chip, which is then written to a file.

[0149] The file path, detection information and product meta-data are used to create an entry in a manifest file, as illustrated in Figure 3.

[0150] The manifest file may be a son file and lists all of the image chips and associated properties and parameters.

[0151] At the first stage of training, wherein image chips are processed, all of the chips may be added to a single manifest file.

[0152] In some circumstance images of vessels may be severely defocussed, or contain multiple ships. Therefore, the dataset may require manual screening to remove the problem chips.

[0153] Not all of the chips available for training may include AIS information and therefore true label data. Therefore, the image chips in the manifest file may be split into “labelled” and “unlabelled” groups on two separate manifest files. In general, the labelled chip manifest file is used for training and may be split further into three manifest files fortraining, validation, and testing. In some embodiments, the unlabeled data may be used as a test manifest file.

[0154] The training data is used for the optimization of the model weights, and the validation data is used to measure performance as various hyper-parameters that configure the model are varied. The test data is reserved for final verification of performance on data that the model has not yet seen.

[0155] In some embodiments, if the unlabeled data can be classified and verified, the unlabeled image chips may be added to the training data as labeled image chips for continuing training of the model.

[0156] As discussed above, the training of the Deep Learning model converges more quickly when the inputs are normalized to a zero mean and a unit standard deviation. The statistics for normalization are measured beforehand on just the training data, and then are input to the model ingest during both training and inference.

[0157] Referring now to Figure 6, shown therein is a block diagram of a workflow 600 of training a classification model, according to an embodiment. Figure 6 represents a high level description of model ingest and training.

[0158] Data is ingested by two sequence generators, training sequence generator 610 and validation sequence generator 620.

[0159] The training sequence generator 610 ingests a training manifest file 612.

[0160] The validation sequence generator 620 ingests a validation manifest file 622.

[0161] Both sequence generators 610 and 620 read the respective manifest file containing the list of chips and chip data, create a Python generator to read a chip, pre- processes the chip, and feeds the chip to the model training 640. Each pass through all of the chips in the manifest file by a sequence generator is called an “epoch”, and many epochs are required before the Deep Learning model converges.

[0162] Both sequence generators also ingest a training stats file 630 which includes the training statistics used for normalization. The training statistics for normalization are measured on just the training data but are input for both training andinference. The sequence generators 610 and 620 pre-process the chips through transformation of pixel values and normalization based on training stats.

[0163] Both the training manifest file 612 and the validation manifest file 622 contain true label data of each image chip. During training the true label data is used to optimize the model weights. During validation the true label data is used to determine the accuracy of the model during training.

[0164] Model parameters 642 comprise the pre-set network structure and parameters set by a designer of the model, which specify the variables for model weights and the interconnection between layers. Model parameters 642 are used for model training 640.

[0165] During model training 640 the model weights are optimized and the validation metrics are computed.

[0166] The output of the training and validation process is trained model 650.

[0167] As described above, when model 650 is trained, it may be used to make predictions on unlabeled data (in an unlabeled data manifest file). Any predictions with a high probability may be selected and have the predicted classification applied as true label data. These selected instances can then be added to the training data, and the model trained further on the larger training data set.

[0168] Deep Learning Algorithm

[0169] The ingest function of the Deep Learning Model for both training and inference reads the image chips in the manifest file and performs normalization, as discussed above. The ingest function also performs additional functions described below.

[0170] The ingest function includes a transformation of the input data as a square root of the pixel values, to make the distribution of pixel values more compact,

[0171] The ingest function performs augmentation of the data by slightly perturbing each chip when each subsequent epoch is used as an input for training. Augmentation effectively increases the number of “different” chips which can be used for training the Deep Learning model.

[0172] Types of augmentation include:• Noise addition: a random value for standard deviation is used to generate an array of noise values, which is added to the chip.• Translation: to change the position of the ship, a chip size of 240 by 240 pixels is randomly cropped from the input 300 by 300 pixel chip.• Azimuth scale: the azimuth direction is stretched a random factor.• Azimuth flip: the chip is randomly flipped in the azimuth direction.• Quantization: to simulate quantization noise, the pixel values are multiplied by a factor, cast to integer, and then divided by the same factor.

[0173] In an embodiment, the Deep Learning model may have a structure based on a ResNet network, such as network 700 shown in Figure 7.

[0174] In network 700, Conv2D represents a convolution layer, BN refers to batch normalization, and ReLU is the nonlinear activation at the output of the layers.

[0175] A ResNet layer 800, which has the “skip connections” which are the distinguishing feature of the ResNet network, is illustrated in Figure 8.

[0176] The ResNet examples shown in Figures 7 and 8 show a case wherein the number of classes are four. In other embodiments, the number of classes may be less than or more than four.

[0177] The dimensions of the data at the output of each block of the ResNet network are shown below each block. After flattening, the data is a single vector, which goes through two fully connected layers until it is vector of four numbers, often called “logits.” There is one number per class and the number is related to the probability that a detected vessel is of that class. The prediction takes the class with the maximum logit value as the predicted class of the detected vessel.

[0178] The model is trained by optimizing the weights to minimize a loss function. The loss function contains a term for the Cross Entropy loss. In addition to the usual Cross Entropy loss, extra loss terms may be computed to try group logits of the same class together, and to separate the logits of different classes. Given the ambiguous nature ofthe input data, encouraging a separation of the classes would improve classification performance. The calculation of the loss terms, Li and Lm, is illustrated in Figure 8.

[0179] The first loss term, Li, increases with distance between output logits of the same class, so as to penalize dispersion and encourage clustering of same-class outputs. The loss is computed over a batch of input image chips, as the sum distances between each logit vector and the mean of logits of the same class.

[0180] The second loss term, Lm, encourages a separation of the class means in the batch. That is, the sum of distances of class means from the overall mean of the logit vectors in the batch is calculated and subtracted from the loss.

[0181] Finally, the output vector of four logits values is used to compute a measure of probability per class by a softmax operation. If there is a class probability above a threshold then the class with the largest probability is selected as the prediction for that chip. If there no probability above the threshold, the prediction for the chip is “unknown.”

[0182] The loss function includes: cross entropy; regularization, wherein the sum of the squared weights is added to the loss function for optimization, as a way to try to reduce overfitting to the training data; dropout, wherein in the last couple of layers of the model, a random set of node outputs are set to zero during the computation of the network prediction to reduce overfitting of the model to the training data; and additional terms in the loss function for optimization, based on the distance between features vectors of vessels of the same classes, and the separation between classes.

[0183] Inference

[0184] Referring now to Figure 9, shown therein is a block diagram of a processing workflow 900 of vessel classification prediction by a trained model, according to an embodiment.

[0185] The workflow 900 for inference of a classification of a detected vessel is similar to the workflow for training and validating a Deep Learning model as illustrated in Figure 6.

[0186] A manifest file and training stats 910 for image chips of any detected vessels of interest is input into a classification sequence generator 920.

[0187] The classification sequence generator 920 pre-processes the image chips through transformation of pixel values and normalization based on training stats of the training data for trained model 930.

[0188] The sequence generator 1020 inputs pre-processed, normalized image chips into the trained model 1030.

[0189] The trained model 1030 outputs classification data 1040 which includes a logit for each supported class of the trained model 1030 which represents a classification probability for each class for the detected vessel. A classification result for each image chip (based on whether one of the probabilities reaches a threshold) and classification probabilities for every vessel class supported by the trained model 1030 may be included in a report.

[0190] While the above description provides examples of one or more apparatus, methods, or systems, it will be appreciated that other apparatus, methods, or systems may be within the scope of the claims as interpreted by one of skill in the art.

Claims

Claims:1 . A method for vessel classification of SAR images using a Deep Learning model, the method comprising: training a Deep Learning model by: generating training chips from previously acquired SAR images, wherein each training chip includes image data, image metadata, and true label data for one vessel, wherein the true label data includes a true identification of a vessel class for the one vessel; generating a training manifest file, a validation manifest file, and a test manifest file each representing one of three groups of the generated training chips; training the Deep Learning model using the training manifest file; validating the Deep Learning model using the validation manifest file; and testing the Deep Learning model using the test manifest file; classifying vessels of interest (VOI) by: acquiring at least one SAR image; detecting VOI within the at least one SAR image; preparing image chips for each detected vessel in the at least one SAR image, wherein each training chip includes image data and image metadata for one VOI;predicting a vessel class of the image chips by the Deep Learning model; and outputting, by the Deep Learning model, an output file which includes a vessel class prediction for each VOL2. The method of claim 1 , wherein detecting VOI within the at least one SAR image further includes outputting coordinates of the vessels of interest.

3. The method of claim 1 , wherein detecting VOI within the at least one SAR image includes identifying transmitted AIS signals of vessels within the at least one SAR image.

4. The method of claim 3, wherein image chips for vessels without AIS signals are input into the Deep Learning model for vessel class prediction, and wherein image chips for vessels with AIS signals are used for further model training.

5. The method of claim 1 , wherein preparing the image chips, predicting vessel classes, and outputting the output file is performed by an inference script employing the Deep Learning model.

6. The method of claim 1 , wherein chip preparation includes: converting latitude and longitude information from the at least one SAR image to a line-pixel location for each detected VOI; using the line-pixel location of each detected VOI to extract an image chip and write the image chip to a file; and creating an entry in a manifest file for each detected VOI, including the extracted image chip, detection information, and image metadata.

7. The method of claim 6, wherein converting the latitude and longitude information to the line-pixel location includes interpolating a geoid height using the latitude and longitude of the detection from a gridded geoid height file, and evaluating a rational function with the geoid height, latitude, and longitude to obtain the line and pixel.

8. The method of claim 1 , wherein the output file includes a table comprising a vessel class prediction for each VOL9. The method of claim 7, wherein the output file further includes a classification probability for each vessel class in the Deep Learning model.

10. The method of claim 1 , wherein the at least one SAR image is preprocessed from raw SAR data to a multi-look image for detecting VOL11 . The method of claim 1 , wherein the at least one SAR image is preprocessed from raw SAR data to a single-look image for preparing image chips.

12. The method of claim 1 , wherein the training chips are normalized by a mean and standard deviation of all vessel pixels.

13. The method of claim 12, wherein the mean and standard deviation are measured on the training chips and used for both training the Deep Learning model and for vessel class prediction by the Deep Learning model.

14. The method of claim 12, wherein a vessel mask is computed using the mean noise around an edge of an image chip as a threshold and setting a mask value to 1 when the image is above the threshold and to 0 when the image is at or below the threshold.

15. The method of claim 1 , wherein training the Deep Learning model includes ingesting the training manifest file by a training sequence generator and ingesting the validation manifest file by a validation sequence generator, wherein both thetraining sequence generator and the validation sequence generator ingest a training statistics file for preprocessing and normalizing training chips.

16. The method of claim 15, wherein the training manifest file includes true label data and the true label data is used to optimize model weights during model training.

17. The method of claim 15, wherein the validation manifest file includes true label data and the true label data is used to determine accuracy during model validation.

18. The method of claim 1 , wherein model parameters comprising a pre-set network structure and parameters set by a designer are used for model training.

19. The method of claim 1 , wherein during model training augmentation is performed on the training chips to increase the number of training chips, wherein augmentation includes at least one of noise addition, translation, azimuth scale, azimuth flip, and quantization.

20. The method of claim 1 , wherein the Deep Learning model has a structure based on a ResNet network.

21. A computer system comprising at least one processor configured to execute the method of any one of claims 1 -20.

22. A non-transitory computer readable medium storing processor-executable instructions which, when executed by at least one processor, cause the processor to perform the method of any one of claims 1 -20.

23. A computer system for vessel classification of SAR images using a Deep Learning model, the system comprising: at least one data storage device for storing a plurality of SAR images, a plurality of training chips, a plurality of manifest files, and at least one Deep Learning model;at least one processor configured to execute processor-executable instructions stored on the at least one data storage device, the instructions configured for: training a Deep Learning model by: generating training chips from previously acquired SAR images, wherein each training chip includes image data, image metadata, and true label data for one vessel, wherein the true label data includes a true identification of a vessel class for the one vessel; generating a training manifest file, a validation manifest file, and a test manifest file each representing one of three groups of the generated training chips; training the Deep Learning model using the training manifest file; validating the Deep Learning model using the validation manifest file; and testing the Deep Learning model using the test manifest file; classifying vessels of interest (VOI) by: acquiring at least one SAR image; detecting VOI within the at least one SAR image; preparing image chips for each detected vessel in the at least one SAR image, wherein each training chip includes image data and image metadata for one VOI; predicting a vessel class of the image chips by the Deep Learning model; andoutputting, by the Deep Learning model, an output file which includes a vessel class prediction for each VOL24. The system of claim 23, wherein detecting VOI within the at least one SAR image further includes outputting coordinates of the vessels of interest.

25. The system of claim 23, wherein detecting VOI within the at least one SAR image includes identifying transmitted AIS signals of vessels within the at least one SAR image.

26. The system of claim 25, wherein image chips for vessels without AIS signals are input into the Deep Learning model for vessel class prediction, and wherein image chips for vessels with AIS signals are used for further model training.

27. The system of claim 23, wherein preparing the image chips, predicting vessel classes, and outputting the output file is performed by an inference script employing the Deep Learning model.

28. The system of claim 23, wherein chip preparation includes: converting latitude and longitude information from the at least one SAR image to a line-pixel location for each detected VOI; using the line-pixel location of each detected VOI to extract an image chip and write the image chip to a file; and creating an entry in a manifest file for each detected VOI, including the extracted image chip, detection information, and image metadata.

29. The system of claim 28, wherein converting the latitude and longitude information to the line-pixel location includes interpolating a geoid height using the latitude andlongitude of the detection from a gridded geoid height file, and evaluating a rational function with the geoid height, latitude, and longitude to obtain the line and pixel.

30. The system of claim 23, wherein the output file includes a table comprising a vessel class prediction for each VOL31 . The system of claim 29, wherein the output file further includes a classification probability for each vessel class in the Deep Learning model.

32. The system of claim 23, wherein the at least one SAR image is preprocessed from raw SAR data to a multi-look image for detecting VOL33. The system of claim 23, wherein the at least one SAR image is preprocessed from raw SAR data to a single-look image for preparing image chips.

34. The system of claim 23, wherein the training chips are normalized by a mean and standard deviation of all vessel pixels.

35. The system of claim 34, wherein the mean and standard deviation are measured on the training chips and used for both training the Deep Learning model and for vessel class prediction by the Deep Learning model.

36. The system of claim 34, wherein a vessel mask is computed using the mean noise around an edge of an image chip as a threshold and setting a mask value to 1 when the image is above the threshold and to 0 when the image is at or below the threshold.

37. The system of claim 23, wherein training the Deep Learning model includes ingesting the training manifest file by a training sequence generator and ingesting the validation manifest file by a validation sequence generator, wherein both the training sequence generator and the validation sequence generator ingest a training statistics file for preprocessing and normalizing training chips.

38. The system of claim 37, wherein the training manifest file includes true label data and the true label data is used to optimize model weights during model training.

39. The system of claim 37, wherein the validation manifest file includes true label data and the true label data is used to determine accuracy during model validation.

40. The system of claim 23, wherein model parameters comprising a pre-set network structure and parameters set by a designer are used for model training.41 . The system of claim 23, wherein during model training augmentation is performed on the training chips to increase the number of training chips, wherein augmentation includes at least one of noise addition, translation, azimuth scale, azimuth flip, and quantization.

42. The system of claim 23, wherein the Deep Learning model has a structure based on a ResNet network.

Images