Computer system and method for fusing multiple sources of maritime domain surveillance data

Abstract

A system and method for fusing multiple sources of earth observation data is provided. The system includes a signal-level multi-source data hub configured to ingest and archive signal-level data, the signal-level data being earth observation data; a signal-level data fusion module configured to transform the signal-level data into object-level data comprising a plurality of objects; an object-level multi-source data hub configured to archive the object-level data and auxiliary data; an object-level data fusion module configured to identify inter-object relationship data in the object-level data using the auxiliary data, the inter-object relationship data comprising one or more relationships between the series of objects; and a third data fusion module configured to process the inter-object relationship data to determine sensor placement and acquisition planning for subsequent signal-level data collection.

Description

COMPUTER SYSTEM AND METHOD FOR FUSING MULTIPLE SOURCES OF MARITIME DOMAIN SURVEILLANCE DATATechnical Field

[0001] The following relates generally to geointelligence systems and maritime surveillance, and more particularly to systems and methods for fusing multiple sources of maritime domain surveillance data.Introduction

[0002] Earth observation data is becoming increasingly ubiquitous over time as the number and types of data sources increase and collection techniques evolve. The ability to collect, analyze, and derive insights from such data to provide intelligence to users is desired. However, systems and methods for effectively and efficiently correlating and fusing disparate sources of earth observation data are not readily available. This is particularly important in the maritime surveillance domain, where large areas need to be serviced in order to provide useful intelligence.

[0003] Accordingly, there is a need for an improved system and method for fusing multiple sources of maritime domain (or other earth observation) surveillance data that overcomes at least some of the disadvantages of existing systems and methods.Summary

[0004] A system and method for fusing multiple sources of earth observation data is provided. The system includes: a signal-level multi-source data hub configured to ingest and archive signal-level data, the signal-level data being earth observation data; a signallevel data fusion module configured to transform the signal-level data into object-level data comprising a plurality of objects; an object-level multi-source data hub configured to archive the object-level data and auxiliary data; an object-level data fusion module configured to identify inter-object relationship data in the object-level data using the auxiliary data, the inter-object relationship data comprising one or more relationships between the series of objects; and a third data fusion module configured to process the inter-object relationship data to determine sensor placement and acquisition planning for subsequent signal-level data collection.

[0005] In some embodiments, the system further includes a tip and cross cue service configured to generate a cross-cue signal level data acquisition task based on the inter-relationship data.

[0006] In some embodiments, the system is further configured to communicate the cross-cue signal level data acquisition task to a data collector.

[0007] In some embodiments, the third data fusion module or the tip and cross-cue service process the inter-object relationship data to determine areas of interest (AOIs) intersecting predicted vessel tracks from the inter-object relationship data.

[0008] In some embodiments, the signal-level data is satellite-based earth observation data.

[0009] In some embodiments, the satellite-based earth observation data is maritime surveillance data.

[0010] In some embodiments, the signal-level data fusion module uses one or more image exploitation tools or machine learning models to extract the object-level data.

[0011] In some embodiments, the object-level data includes SAR and optical image chips of vessels, image chips sets capturing a single vessel at different times and locations, ship tracks extrapolated from image chips sets, or Radio Frequency emissions captured from vessels.

[0012] In some embodiments, the signal-level data fusion module is configured to identify and generate labels for vessel characteristics, including vessel class and velocity estimates derived from the analysis of vessels and vessel wakes captured in image chips.

[0013] In some embodiments, the auxiliary data includes Automatic Identification System (AIS) signatures; data from National Vessel Monitoring System; a vessel acoustics signature database, or geospatial data.

[0014] In some embodiments, the inter-object relationship data includes correlated sets of Image chips, ship tracks, AIS, RF emissions, or Ship identification information from VMS databases.

[0015] Other aspects and features will become apparent, to those ordinarily skilled in the art, upon review of the following description of some exemplary embodiments.Brief Description of the Drawings

[0016] The drawings included herewith are for illustrating various examples of articles, methods, and apparatuses of the present specification. In the drawings:

[0017] Figure 1 is a schematic diagram of a system for fusing multiple sources of maritime domain surveillance data, according to an embodiment;

[0018] Figure 2 is a block diagram of a computer system for fusing multiple sources of maritime domain surveillance data, according to an embodiment;

[0019] Figure 3 is a flow diagram of a method of performing maritime surveillance using the systems of Figures 1 -2, according to an embodiment; and

[0020] Figure 4 is a flow diagram of a method of performing maritime surveillance using the systems of Figures 1 -2, according to an embodiment.Detailed Description

[0021] Various apparatuses or processes will be described below to provide an example of each claimed embodiment. No embodiment described below limits any claimed embodiment and any claimed embodiment may cover processes or apparatuses that differ from those described below. The claimed embodiments are not limited to apparatuses or processes having all of the features of any one apparatus or process described below or to features common to multiple or all of the apparatuses described below.

[0022] One or more systems described herein may be implemented in computer programs executing on programmable computers, each comprising at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, and without limitation, the programmable computer may be a programmable logic unit, a mainframe computer, server, and personal computer, cloud-based program or system, laptop, personal data assistance, cellular telephone, smartphone, or tablet device.

[0023] Each program is preferably implemented in a high-level procedural or object-oriented programming and / or scripting language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a storage media or a device readable by a general or special purpose programmable computer for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein.

[0024] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary, a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0025] Further, although process steps, method steps, algorithms or the like may be described (in the disclosure and I or in the claims) in a sequential order, such processes, methods and algorithms may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any order that is practical. Further, some steps may be performed simultaneously.

[0026] When a single device or article is described herein, it will be readily apparent that more than one device I article (whether or not they cooperate) may be used in place of a single device I article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device I article may be used in place of the more than one device or article.

[0027] The following relates generally to geointelligence systems and maritime surveillance, and more particularly to systems and methods for fusing multiple sources of maritime domain surveillance data.

[0028] The present disclosure provides a software platform, that may be cloudbased, that fuses multiple sources of maritime domain surveillance data, databases, and analytics to create a persistent picture of a maritime situation. This persistent picture, orinsights, may be used by, for example, defense and security intelligence, fisheries intelligence, and commercial intelligence users.

[0029] In an aspect, the present disclosure provides systems and methods for using multiple sensing modalities or multiple sources of detection data (e.g., optical, SAR, RF, IR, and AIS) to generate and provide to a user a complete maritime domain awareness picture. This includes detecting ships that want to be seen (i.e. , self-reporting through AIS) and ships that do not want to be seen (i.e., that are not self-reporting through AIS). For ships that are not self-reporting, also referred to as “dark ships”, in some embodiments, the systems and methods described herein may track the dark ships and, in some cases, even identify the dark ships.

[0030] Systems and methods of the present disclosure may be implemented at one or more computer devices. For example, the system may include a plurality of computer devices in communication via a network connection. Further, components of the system (e.g., modules, data hubs, etc.) may be implemented at a single computer device, or across a plurality of computer devices. In some embodiments, the system includes at least one user computing device and at least one server computing device in communication via a network connection. The user device may execute an application that can interact with server-side software components (“services”) hosted by the server computing device. For example, the computer system 300 may execute a network-based software application that executes partially at the server computing device (via serverside software components) and partially at the user device (via client-side software components). In an embodiment, the client-side software components include a user interface (e.g., web-based user interface).

[0031] Referring now to Figure 1 , shown therein is a system 100 for fusing multiple sources of maritime domain surveillance data, according to an embodiment.

[0032] The system 100 includes a space segment 102 and a ground segment 104.

[0033] The space segment 102 includes a plurality of earth observation data sources 106 including satellite-based sensor data collectors 108-1 , 108-2, 108-3. While three satellite-based sensor data collectors 108 are shown in Figure 1 , the number of satellite-based sensor data collectors 108 may vary and is not particularly limited. Further,the satellite-based sensor data collectors 108 may collect one or more types of sensorbased earth observation data. Accordingly, each satellite-based sensor data collector 108-1 , 108-2, 108-3 is equipped with at least one sensor type for collecting sensor data according to that sensor type. Example sensor types include, without limitation, optical sensors (e.g., high resolution optical) and SAR sensors. The sensor data may be signallevel maritime domain surveillance data.

[0034] The ground segment 104 also includes one or more non-space collectors 110 that collect sensor data from locations that are not in space (i.e., other than from satellites). The number of non-space collectors 110 is not particularly limited. The sensor data may be signal-level maritime domain surveillance data.

[0035] The plurality of satellite data sources 106 communicate with a ground terminal 112 (or ground station) via an uplink 114 and downlink 116. The manner of communication is generally known. In other embodiments, there may be a plurality of ground terminals 112 and the number of ground terminals 112 is not particularly limited. The ground terminals may be located in multiple geographic locations.

[0036] The ground terminal 112 includes an antenna system 118 and a data processing device 120. The ground terminal 112 communicates with the data sources 106 via the antenna system 118. The ground terminal 112 also communicates with the non-space collectors. The ground terminal 112 receives signal-level data from the data sources 106, 110.

[0037] The data processing device 120 of the ground terminal 112 processes data to be sent to the data sources 106 and processes data received from the data sources 106.

[0038] The ground segment 104 further includes a cloud data processing and storage platform 122 including a server 124 that is in communication with a data storage 126. In other embodiments, the cloud data processing and storage platform 122 may be implemented as a non-cloud platform. While a single server 124 and single data storage 126 are shown in Figure 1 , the number of servers 124 and data storage devices 126 may vary (e.g., multiple) and the number is not particularly limited.

[0039] The server 124 is configured to fuse multiple sources of earth observation data (e.g., one or more sources from sources 106, 110) to generate insights. Insights include situational ly meaningful data generated by the processing and analysis of signallevel data by the platform 122.

[0040] The cloud platform 122 may be configured to detect vessels, characterize the vessels, identify the vessels, and track the vessels through time. The cloud platform 122 may be configured to assess a threat level of the vessels. This may enable users to prioritize which vessels are worth further action.

[0041] The cloud platform 122 may be configured to detect vessels in the maritime domain regardless of their self-reporting status. This may enable users to find vessels even if the vessels are intentionally trying to hide their location and behaviors.

[0042] The cloud platform 122 may be configured to extract vessel detections from raw imagery. Raw imagery may include optical imagery and SAR imagery. The cloud platform 122 may execute a multi-step process to extract vessels from SAR imagery and characterize them by extracting vessel velocity and class. The cloud platform 122 may extract vessels from raw high resolution optical imagery from various satellite data (e.g., Maxar, Blacksky and Satellogic) using Machine Learning techniques.

[0043] In an embodiment, the cloud platform 122 executes a ship detection algorithm that uses a single-stage customized neural network architecture (e.g., customized RetinaNet architecture) with anchor box generation parameters that have been optimized for each input image resolution, and an EfficientNet backbone for image feature extraction. This is one example and other techniques and neural network architecture may be used in other embodiments. Ships detected in each image chip may then be merged to remove duplicate detections in overlapping regions, transformed from pixel coordinates back into world coordinates, and output (e.g., in a geojson file). A preview image for each detection may be extracted from the source image. Each of the detected objects may subsequently classified into one of a plurality of ship classes. The results are stored in memory. In an embodiment, the results are stored as a compressed file format (e.g., zip file) that is packaged for ingestion into another software application and includes a separate metadata file containing all relevant metadata (e.g., json file

[0044] The cloud platform 122 may be configured to monitor or track vessels. The cloud platform 122 may fuse sources of vessel detections (e.g., AIS, VMS, RF, SAR, Optical, VIIRS). This may enable users of the system to create vessel tracks and follow vessels.

[0045] The cloud platform 122 may be configured to track dark vessels. The cloud platform 122 may fuse detections from completely dark vessels (non-self-reporting) and create tracks from these dark detections.

[0046] The cloud platform 122 may be configured to perform behaviour analytics using AIS or VMS data. The cloud platform 122 may extract behavior segments from AIS and / or VMS data using rule-based algorithms and / or ML techniques. These behaviors may provide insights into past and current vessel activity patterns. In an embodiment, a machine learning algorithm based on a convolutional neural network (CNN) is used. To classify behaviour, all of the layers of the neural network are combined, with upscaling of the upper layers, to produce a set of features at each point. The design of the neural networks may adapt or incorporate aspects or ideas from ResNets and Inception nets, among others, but adapted for a one dimensional (1 D) environment. Such an embodiment may be used, for example, to perform fishing behaviour analytics.

[0047] The cloud platform 122 may be configured to perform or support vessel identification. Vessel identification may enable users to leverage SAR and optical detections to downselect vessel identity candidates to help intel users to determine the probable identity of vessels. The cloud platform 122 may achieve this by executing a multi-step process of feasibility assessment, image matching, and Bayesian reasoning.

[0048] In some embodiments, the cloud server 124 is configured to perform general ship tracking. General ship tracking includes fusing vessel contacts together to create vessel tracks. This may be done regardless of a vessel’s self-reporting status.

[0049] In some embodiments, the cloud server 124 is configured to perform vessel identification. The cloud server 124 outputs a candidate list of potential vessel identifies from optical (e.g., high resolution) and / or SAR imagery.

[0050] In some embodiments, the cloud server 124 is configured to perform vessel behaviour mapping. The cloud server 124 outputs vessel behaviour segmentation from AIS and / or VMS source data (e.g., non-space data 110, or Aux data sources 206 of Figure 2). This separates vessel behaviours into categories. Categories may include, for example, fishing, steaming, anchor, etc.

[0051] In some embodiments, the cloud platform 122 implements a vessel signature database. The vessel signature database includes a pre-populated vessel signature database containing observations of a vessel from multiple sources (e.g., SAR, optical, VI IRS, etc.). In some cases, this may be every source available. The database may be used to enhance downstream analytics performed by the cloud platform 122.

[0052] The data storage 126 stores data processed by the server 124. The data storage 126 stores signal-level data collected by the data sources 106 and non-space collectors 110. The signal-level data is processed by the server 124.

[0053] The ground segment 104 further includes a user device 128. The user device 128 is configured to receive input from a user and display data generated by the cloud platform 122. The input data received from a user may be used to request certain data generated and stored by the cloud platform 122. The user device 128 is configured to display a graphical user interface that allows a user to interact with the cloud platform 122. The user interface may include a series of user interface screens for receiving user input and display output data generated by the server 124.

[0054] The user device 128 and the cloud platform 122 communicate via a network 130. The network 130 may be a wide area network, such as the Internet. The data processing device 120 of the ground terminal 112 communicates with the cloud platform 122 and, in some cases, the user device 128 via the network 130. Communication in this context may include sending and receiving data. In some cases, non-space collectors 110 may communicate with cloud platform 122 via network 130.

[0055] The operation of system 100, in an example, will now be described.

[0056] Signal level data is collected by the satellite-based data collectors 106 and the non-space data collectors 110 via respective sensors. The signal-level data mayinclude data from multiple types of sensors. Collection of data by the collectors 106, 110 may be based on data received from the ground terminal 112 (e.g., via an imaging task sent from the ground terminal 112 to the collector).

[0057] Signal-level data is downlinked from the satellite-based collectors 106 to the ground terminal 112 via downlink 116. Signal-level data is transmitted from the non-space collectors 110 to the ground terminal 112.

[0058] The received signal-level data is sent from the processing device 120 to the cloud platform 122 via network 130. Processing device 120 may process the signal-level data prior to sending. The signal-level data may be sent automatically or may be sent in response to a request from cloud platform 122.

[0059] Signal-level data is processed by the server 124 and stored in data storage 126. The processed signal-level data includes space segment signal-level data and ground segment signal-level data.

[0060] The cloud server 124 processes the signal-level data to obtain object-level data, as further described herein. The object-level data is stored in the data storage 126. Non-space and auxiliary data is also stored in data storage 126.

[0061] The user device 128 requests insights via the user interface. The request is sent from the user device 128 to the cloud platform 122 via the network 130.

[0062] The cloud platform 122 generates the insights by having the cloud server 124 process the object-level data (and, in some cases, auxiliary data as described herein) to identify inter-object relationships. The inter-object relationship data is stored in data storage 126. The inter-object relationship data is sent to the user device 128 via network 130 for display in the user interface.

[0063] In some cases, a subsequent signal-level data collection task may be requested and executed in response to the received insights (inter-object relationship data). For example, a request may be input at the user device 128 and the request sent to the cloud platform 122. The cloud platform 122 may then generate a collection task and send the collection task to the ground terminal 112. The ground terminal 112 processes the collection task and sends the collection task to the signal-level datacollectors 106, 110 for collection of further signal-level data and the process repeats. In an example, the server 124 may be configured to identify anomalous behaviour among the inter-object relationship data and generate a data collection task based on such insight. The server 124 sends the task to the ground terminal 112, which sends the task to data collector 106, 110. In some cases, the cloud server 124 may generate the insight and initiate the subsequent data collection task automatically and without intermediary user requests.

[0064] Referring now to Figure 2, shown therein is a computer system 200 for fusing multiple sources of maritime domain surveillance data, according to an embodiment. The computer system stores and processes data using physical components, such as a processor and data storage device (e.g., memory).

[0065] The system 200 includes earth observation data sources 204 and auxiliary data sources 206. The earth observation (EO) data sources 204 and auxiliary data sources 206 are stored in memory.

[0066] The EO data sources 204 include signal-level maritime domain surveillance data. The EO data sources 204 may include data collected from a plurality of satellites. The EO data source 204 may include multiple types of sensor data.

[0067] The computer system 202 includes a signal-level multi-source data hub 208, a first, signal-level data fusion module 210, an object-level multi-source data hub 212, a second, object-level data fusion module 214, a third data fusion module 216, and a tip and cross cue service 218. The foregoing components 208, 210, 212, 214, 216, 218 may be implemented using one or more physical components of the computer system 202, such as a processor and a memory.

[0068] The signal-level multi-source data hub 208 is configured to ingest and archive the signal-level data maritime domain surveillance data. The archived signal-level data 220 is stored in memory. The archived signal-level data 220 may be stored in a database. In an embodiment, the archived signal-level data 220 may be stored in a NoSQL document driven database.

[0069] In an embodiment, signal-level data processing is performed by external data services (e.g., VIIRS, SDS, GSI / SAR, Vendor systems), where each service implements its own data archiving solution.

[0070] The archived signal-level data 220 is provided as input to the first, signallevel data fusion module 210. The first, signal-level data fusion module 210 is configured to transform signal-level data 220 into objects or object-level data 222. The first, signallevel data fusion module 210 may also be referred to as a signal-to-object transform module.

[0071] The first, signal-level data fusion module 210 extracts a series of objects 222 from the signal-level data 220. The first, signal-level data fusion module 210 may use one or more image exploitation tools and machine learning models to extract the objectlevel data 222. The object-level data 222 is stored in memory.

[0072] Object-level data 222 (also referred to as objects) may include SAR and optical image chips of vessels, image chips sets capturing a single vessel at different times and locations, ship tracks extrapolated from image chips sets, and Radio Frequency emissions captured from vessels. The module 210 may be configured to identify and generate labels for vessel characteristics, including vessel class and velocity estimates derived from the analysis of vessels and vessel wakes captured in the image chips. Such data may be considered objects 222.

[0073] The object-level multi-source data hub 212 archives the object-level data 222 and auxiliary data sources 206. The archived object-level and auxiliary data 224 is stored in memory. The archived object-level and auxiliary data 224 may be stored in one or more databases. In an embodiment, the one or more databases are NoSQL document driven databases. In a particular embodiment, a nosql DB is used with a special (h3) indexing method on bucket pattern to improve query performance in an optimized aggregated geometry. The auxiliary data sources 206 may include navigation information for example, Automatic Identification System (AIS) signatures, data from National Vessel Monitoring System, a vessel acoustics signature DB (e.g., NW-MILO Acoustic Data Collection, The MARS (Marine acoustic research station) Project), and geospatial data(e.g., Map data and MapLayers). Geospatial data includes data or information recorded with a geographic indicator.

[0074] While AIS data is characterized as being an auxiliary data source, it should be noted that in the context of the operation of the system 200, AIS is a primary data source (i.e., along with the EO data sources 204). Other auxiliary data sources may be considered secondary data sources and may include, for example, IHS vessel registry data, IUU (illegal unreported and unregulated fishing) lists, RFMO (regional fisheries management organizations) lists, and environmental data (e.g., wind speeds, sea state, ocean temperature, etc.), as well as open source data sources such as EEZ (Exclusive Economic Zone), FAO (Food and Agriculture Organization), and Major Fishing Areas Data.

[0075] The second, object-level data fusion module 214 is configured to identify inter-object relationships 226 in the object-level data 224. The second, object-level data fusion module 214 performs a situational awareness function and generates data that provides domain situational awareness. As such, the second, object-level data fusion module 214 may be referred to as a situational awareness module.

[0076] Inter-object relationship data 226 may include, for example, correlated sets of Image chips, Ship Tracks, AIS, RF emissions, and Ship identification information from VMS databases.

[0077] The second, object-level data fusion module 214 may identify the interobject relationships 226 using one or more tools.

[0078] The second, object-level data fusion module 214 may include one or more time-series analysis tools configured to perform re-recognition of vessels across large spans of time and location.

[0079] The second, object-level data fusion module 214 may include a statistical fusion framework tool configured to analyze vessel behaviour.

[0080] The second, object-level data fusion module 214 may include one or more tools configured to detect concerning or anomalous vessel behavior patterns.

[0081] The second, object-level data fusion module 214 may include one or more tools configured to predict vessel tracks and contacts between vessels.

[0082] Inter-object relationships detected by the second, object-level data fusion module 214 may include, for example: comparisons of vessel information from navigation information versus vessel information derived from imagery; comparisons of vessel information from navigation information versus vessel information derived from vessel RF emissions detected from space; time series analysis, including historical location information, ports visited by vessel, AIS broadcast interruption; behavior classification including, but not limited to, dark ship identification, fishing behavior, and rendezvous detection.

[0083] The second, object-level data fusion module 214 may be configured to execute further tools, such as: Global tracks (from time-series analysis); Optical characterization; Advanced filters; RF identification; Advance track prediction; Rendezvous Detection; dark rendezvous detection; AIS interruption; Fishing detection; Dark ship identification; Track anomaly detection; Port visit monitoring; Advance track correlation; VIIRS ML detection; AIS anomaly detection; and Loitering detection.

[0084] The inter-object relationships 226 may be sent to a user device for display in a user interface or be used in a subsequent processing operation.

[0085] The third data fusion module 216 is configured to execute sensor placement planning. The third data fusion module 216 receives inter-object relationship data 226 and processes the data 226 to determine sensor placement and acquisition planning for subsequent signal-level data collection (“sensor placement data”). The third data fusion module 216 may be referred to as a sensor placement and planning module (as it performs those functions).

[0086] The sensor placement data may be communicated to the signal-level data collectors, and subsequent signal-level data collected and stored as EO data sources 204. The use of the sensor placement data to collect further signal-level data is represented by the arrow going from the third data fusion module 216 to the EO data sources 204.

[0087] The third data fusion module 216 includes a tip and cross cue service 218. The tip and cross cue service 218 generates a cross-cue signal level data acquisition task 228 based on the inter-object relationship data 226. The cross-cue signal level data acquisition task 228 is then communicated to the data collectors. The data collectors collect signal-level data according to the task 228.

[0088] The tip and cross cue service 218 may plan the acquisition of additional maritime domain surveillance data to enable on-going monitoring of vessels exhibiting concerning or anomalous Behavior, where such behaviour has been identified in the interobject relationships 226. The tip and cross cue service 218 includes one or more tools configured to plan image collection over regional Area of Interest (AOIs) or areas intersecting predicted vessel tracks.

[0089] Accordingly, the third data fusion module 216 and / or the cross-cue service 218 may further process and analyze inter-object relationship data 226 to determine AOIs or areas intersecting predicted vessel tracks from the inter-object relationship data 226.

[0090] As can be seen, the computer system 200 can implement a sort of feedback loop in which signal-level data is processed to generate insights, those insights are used to generate subsequent collection tasks, further signal-level data is collected, and the new signal-level data processed to generate further insights (sometimes using the previously collected signal-level data or objects / inter-object relationships derived therefrom).

[0091] An example of the computer system 200 of Figure 2 in operation will now be described, according to an embodiment.

[0092] The operational sequence begins with the computer system 200 receiving signal level optical data from an optical data provider / vendor (EO data source 204). The received signal level optical data may be considered raw input data.

[0093] The raw input data (optical image) is archived in the signal-level multisource data hub 208. In an embodiment, the image is archived in cloud buckets.

[0094] The signal-level optical data is provided to the signal-level data fusion module 210. The signal-level data fusion module 210 executes a machine learning-basedvessel detection service. The ML-based vessel detection service processes the optical data using a vessel detection algorithm, such as described herein, and output vessel detection results.

[0095] The vessel detection results are saved as a vessel detection report. The vessel detection report is object-level data 222 that is stored by the object-level multisource data hub 212. The vessel detection report includes metadata and image chips.

[0096] At this stage, an operator user (e.g., a quality engineer or QE team) may perform a quality assurance process on the vessel detection report (optional / manual). The vessel detection report may be displayed and reviewed in a object-level data viewing user interface executing on user device (e.g., device 128 of Figure 1 ). This review process generates a second (QE) version of the vessel detection report. The QE version of the vessel detection report is pushed to a data delivery bucket for the object-level data fusion module 214.

[0097] The vessel detection reports are then ingested by the object-level data fusion module and fused (correlated) with all other data types. This outputs correlated and uncorrelated vessel detections, each with associated metadata. The outputs of the fusion process (correlated vessel detections, uncorrelated vessel detections, correlation determinations) are inter-object relationships 226.

[0098] A user interface for viewing inter-object relationships 226 is generated and displayed on a user device (e.g., device 128 of Figure 1 ). The user interface includes a map visualization with the correlated and uncorrelated detections overlayed or plotted on the map. Correlated and uncorrelated detections may be differentially identified by visual indicator in the user interface (e.g., text, shape, colour, etc.). The user interface further includes the available metadata information associated with the displayed detections.

[0099] Then, if applicable, the ingested data triggers behaviour analytics and notification. The system may be configured to generate alerts or notifications upon satisfaction of certain conditions. The conditions for notification may be encoded as rules that indicate that a notification is to be generated and displayed in the user interface upon satisfaction of the condition (notification triggering condition). The conditions for notification may be pre-configured or may be configured by a user through the userinterface. Examples may include Dark Vessel, Tripwire, Heading to Area. Such alerts may be triggered by rules that are set up or configured for the notification and the vessel's actual behaviour. As an example, for a tripwire notification, a user may draw a tripwire on the map via the user interface. The tripwire may be any line. If a vessel crosses the line, or tripwire, the system triggers a notification or alert. The notification indicates that a vessel has crossed the user-defined tripwire. The notification is displayed in the user interface.

[0100] Based on the output of the behaviour analytics events and notification, the user creates a new data collection request through a user interface for submitting a data collection request executing at the user device. The user interface may be generated by the third data fusion module 216. The data collection request is or may be transformed into a cross-cue signal-level acquisition task 228. The request may be for any type of signal-level data (e.g., optical, SAR, RF). In some cases, the data collection request is received by the third data fusion module 216 via the user interface and provided to the tip and cross cue service 218. The tip and cross cue service 218 processes the data collection request based on available EO data source information and outputs a crosscue signal-level data acquisition task 228.

[0101] The new data collection request or cross-cue signal-level data acquisition task 228 is then communicated to one or more EO data sources 204 to perform data collection based on the information contained in the request.

[0102] Referring now to Figure 3, shown therein is a method 300 of maritime surveillance, according to an embodiment.

[0103] The method 300 may be encoded as computer-executable instructions and executed by one or more computing devices comprising one or more processors. In an embodiment, the method 300 may be executed by the server 124 of Figure 1 or the computer system 200 of Figure 2.

[0104] At 302, the method 300 includes receiving and storing signal level maritime surveillance data.

[0105] At 304, the method 300 includes detecting all vessels in the signal level maritime surveillance data.

[0106] Different sensing modalities may use different techniques to detect vessels in the respective sensor data.

[0107] For SAR, SAR-based ship detection may be performed directly on the SAR imagery. At a high level, the SAR image is broken down into smaller tiles, and the ocean clutter (or simply, what is considered ocean) is modeled by a statistical distribution (e.g., either the k-distribution or the trimodal distribution). A detection threshold is then derived from this statistical distribution based on a CFAR (Constant False Alarm Rate). Any pixels that are brighter than the calculated detection threshold are then considered as detected, or ship pixels. Groups of adjacent detected pixels are then clustered into single ship objects on which one can perform vessel characterization.

[0108] For optical, in an embodiment, an optical vessel detection algorithm is used that uses a single-stage customized RetinaNet architecture, with anchor box generation parameters that have been optimized for each input image resolution, and an EfficientNet backbone for image feature extraction. The ships detected in each chip are then merged to remove duplicate detections in overlapping regions, transformed from pixel coordinates back into world coordinates, and output in a geojson file.

[0109] For Infrared (IR, or VIIRS), in an embodiment, detecting vessels in VIIRS imagery uses a spike detection algorithm that generates a list of candidate vessel detections. A second algorithm measures the height of the spikes for the discard of ionospheric energetic particle detections and to rate boat detections as either strong or weak. A sharpness index is used to label boat detections that appear blurry due to the scattering of light by clouds. The candidate spikes are then filtered to remove features on land and gas flares.

[0110] At 306 the method 300 includes characterizing the detected vessels.

[0111] In an embodiment, vessel characterization includes measuring basic attributes about the vessel. Attributes may include the length, width, heading, and sometimes speed of the vessel. Note that this can only be done in SAR and OpticalImagery. In IR imagery, the characterization is at a much higher level because the detections are typically only single pixel detections. The taxonomy for IR vessel characterization may be one of: strong detection, weak detection, blurry detection, ionospheric particle, or gass flair. RF-based vessel characterization may include simply providing the central frequency of the detected signal, and based on that determining what frequency band the detection is in (i.e., VHF, UHF, X-band L-band, S-band, etc.).

[0112] At 308, the method 300 includes identifying the vessels using the vessel characterization at 306.

[0113] Vessel identification may only be done on higher resolution optical ship detections and, in some cases, on SAR detection. Vessel characterization may include comparing the image of the detected dark vessel to images of known vessels. This may be achieved through a machine learning technique, such as a Siamese Twin Neural Network.

[0114] At 310, the method 300 includes tracking the identified vessel through time in signal level maritime surveillance data.

[0115] In an embodiment, at a very high level, tracking a dark vessel through time fundamentally involves the following steps. Once two detections are received, the system tries to determine if the two detections can belong to the same vessel track. This is achieved by using the vessel characterization information (speed, length, heading, etc.) to project the first detection to the timestamp of the second detection, and determine if the two are in close enough proximity. If they are, then a vessel track is formed. The process is repeated once new detections are presented to the system.

[0116] Tracking a self-reporting vessel through time is much simpler. The system simply stores all the ship’s position reports in a time-sorted array, thereby producing a vessel track.

[0117] At 312, the method 300 includes assessing a threat level of the vessels based on the tracking information from 310.

[0118] In an embodiment, a threat level of the vessel may be assessed based on a trajectory of the vessel, previous port visits, or other indicators such as if a vessel is on a list (e.g., sanctions list, illegal, unreported and unregulated fishing (IUU) list, etc.).

[0119] At 314, the method 300 includes prioritizing response action based on the threat level assessment at 312.

[0120] Referring now to Figure 4, shown therein is a method 400 of maritime surveillance, according to an embodiment.

[0121] The method 400 may be encoded as computer-executable instructions and executed by one or more computing devices comprising one or more processors. In an embodiment, the method 400 may be executed by the server 124 of Figure 1 or the computer system 200 of Figure 2.

[0122] At 402, the method 400 includes acquiring signal-level earth observation data with a plurality of sensor data sources. The acquired signal-level earth observation data may be considered a first set of signal-level earth observation data.

[0123] At 404, the method 400 includes receiving and storing the signal-level earth observation data in a first, signal-level data hub.

[0124] At 406, the method 400 includes transforming the signal-level earth observation data into object-level data. The object-level data may be referred to as “objects” or “extracted objects”. The signal-level earth observation data is transformed using a set of image exploitation tools and machine learning models which extract a series of objects from the signal-level earth observation data.

[0125] At 408, the method 400 includes storing the object-level data in a second, object-level data hub with auxiliary data.

[0126] At 410, the method 400 includes identifying inter-object relationships using the object-level data and the auxiliary data.

[0127] At 412, the method 400 includes generating a tip and cross cue task based on at least one of the inter-object relationships.

[0128] At 414, the method 400 includes sending the tip and cross cue task to at least one data source (data acquisition).

[0129] At 416, the method 400 includes acquiring tip and cross cue directed signallevel earth observation data based on the tip and cross cue task using the at least one data source (“tip and cross cue directed signal-level earth observation data”). In the context of method 400, the tip and cross cue directed signal-level earth observation data may be considered a second set of signal-level earth observation data.

[0130] After the tip and cross cue directed signal-level earth observation data has been acquired, the method 400 returns to 404, where the tip and cross cue directed signal level earth observation data is received and stored in the first, signal-level data hub. The method 400 then proceeds through 406-416.

[0131] While the above description provides examples of one or more apparatus, methods, or systems, it will be appreciated that other apparatus, methods, or systems may be within the scope of the claims as interpreted by one of skill in the art.

Claims

Claims:

1. A computer system for fusing multiple sources of earth observation data, the system comprising: a signal-level multi-source data hub configured to ingest and archive signal-level data, the signal-level data being earth observation data; a signal-level data fusion module configured to transform the signal-level data into object-level data comprising a plurality of objects; an object-level multi-source data hub configured to archive the object-level data and auxiliary data; an object-level data fusion module configured to identify inter-object relationship data in the object-level data using the auxiliary data, the inter-object relationship data comprising one or more relationships between the series of objects; and a third data fusion module configured to process the inter-object relationship data to determine sensor placement and acquisition planning for subsequent signallevel data collection.

2. The system of claim 1 , further comprising a tip and cross cue service configured to generate a cross-cue signal level data acquisition task based on the interrelationship data.

3. The system of claim 2, further configured to communicate the cross-cue signal level data acquisition task to a data collector.

4. The system of claim 2, wherein the third data fusion module or the tip and crosscue service process the inter-object relationship data to determine areas of interest (AOIs) intersecting predicted vessel tracks from the inter-object relationship data.

5. The system of claim 1 , wherein the signal-level data is satellite-based earth observation data.

6. The system of claim 2, wherein the satellite-based earth observation data is maritime surveillance data.

7. The system of claim 1 , wherein the signal-level data fusion module uses one or more image exploitation tools or machine learning models to extract the objectlevel data.

8. The system of claim 1 , wherein the object-level data includes SAR and optical image chips of vessels, image chips sets capturing a single vessel at different times and locations, ship tracks extrapolated from image chips sets, or Radio Frequency emissions captured from vessels.

9. The system of claim 1 , wherein the signal-level data fusion module is configured to identify and generate labels for vessel characteristics, including vessel class and velocity estimates derived from the analysis of vessels and vessel wakes captured in image chips.

10. The system of claim 1 , wherein the auxiliary data includes Automatic Identification System (AIS) signatures; data from National Vessel Monitoring System; a vessel acoustics signature database, or geospatial data.

11. The system of claim 1 , wherein the inter-object relationship data includes correlated sets of Image chips, ship tracks, AIS, RF emissions, or Ship identification information from VMS databases.

Images