System and method for advanced associators for tracking marine vessels

Abstract

A system and method for using an advanced associator for tracking marine vessels. The system receives a list of all the observed ships and candidate ships and their known properties which is provided to the advanced associator module which will output a report that lists the associations between the observed ships and the candidate ships, including uncertainties about these associations. The advanced associator collects any information about the properties of the observed ships and of the candidate ships. Then it compares these ship properties and quantifies the overall quality of the comparisons. Finally, it makes a global decision about which observed ship is which candidate ship and provides probabilities that quantify the certainty of its decision.

Description

SYSTEM AND METHOD FOR ADVANCED ASSOCIATORS FOR TRACKING MARINE VESSELSTechnical Field

[0001] The following relates generally to tracking technologies, and more particularly to systems and methods for tracking marine vessels.Background

[0002] Existing approaches to marine vessel tracking, including associating synthetic aperture radar (SAR) and automatic identification system (AIS) data, can perform poorly in many challenging situations. Such approaches do not take all available ship properties, such as ship length, into account.

[0003] Such techniques may not be usable in or suitable for the context of ship identification, where no kinematic information is available.

[0004] In marine surveillance applications, it may be desirable to track ships, including self-reporting or dark ships (or vessels). It may be desirable to provide a single association solution for both ship association and maritime awareness.

[0005] Accordingly, there is a need for an improved system and method for tracking marine vessels that overcomes at least some of the disadvantages of existing systems and methods.Summary

[0006] Provided herein is an advanced associator system for tracking marine vessels, including a computer memory and a processor in communication with the memory and configured to execute instructions, a communication interface for receiving observed ship data including a list of observed ships and their known properties (“observed ship properties”) and candidate ship data including a list of candidate ships and their known properties (“candidate ship properties”), the memory for storing the observed ship data and the candidate ship data, the processor configured to execute an advanced associator module comprising an evidence collector module, an analytics matcher module, a ship image matcher module, a Bayesian Reasoner module, and anadvanced associator controller, wherein the advanced associator module is configured to output a report that lists associations between the observed ships and the candidate ships, including uncertainties about the associations.

[0007] The list of observed ships may include uncertainties about ship properties.

[0008] The list of candidate ships may include uncertainties about ship properties.

[0009] The ship image matcher module may be configured to support machine learning (ML) algorithms.

[0010] The report may be a dark ship association report.

[0011] The advanced associator controller may be configured to coordinate the other components in the advanced associator module.

[0012] The evidence collector module may be configured to assemble the ship matching evidence that the Bayesian reasoner requires to make an informed association decision.

[0013] The analytical matchers module may be configured to provide partial evidence to the evidence collector by deciding if ship features including as ship type and length match.

[0014] The ship image matcher module may be configured to provide partial evidence to the evidence collector by deciding if two images show the same ship.

[0015] The Bayesian reasoner module may be configured to fuse the evidence together and makes an association decision based on all the evidence provided.

[0016] Provided herein is a computer-implemented method of tracking marine vessels using an advanced associator system including receiving, via a communication interface, observed ships data including list of observed ships and their known properties and candidate ships data including a list of candidate ships and their known properties, providing the observed ships data and candidate ships data to an advanced associator module configured to execute instruction to: generate and compute associations between observed ships in the observed ships data and candidate ships in the candidate ships data, including uncertainties about the associations, wherein the computation utilizes aBayesian reasoner module configured to run a Bayesian reasoning framework, and output a report including the associations.

[0017] The list of observed ships may further comprise uncertainties about the known properties.

[0018] The list of all the candidate ships may further comprise uncertainties about the known properties.

[0019] The report may be a dark ship association report.

[0020] The advanced associator module may further comprise a ship image matcher module configured to support machine learning (ML) algorithms.

[0021] The advanced associator module may further comprise an advanced associator controller configured to coordinate the other components in the advanced associator module.

[0022] The advanced associator module may further comprise an evidence collector module is configured to assemble the ship matching evidence that the Bayesian reasoner requires to make an informed association decision.

[0023] The advanced associator module may further comprise an analytical matchers module configured to provide partial evidence to the evidence collector module by deciding if ship features including ship type and length match.

[0024] The advanced associator module may further comprise an image matchers module configured to provide partial evidence to the evidence collector module by deciding if two images show the same ship.

[0025] The Bayesian reasoner module may be configured to fuse the evidence together and make an association decision based on all the evidence provided.

[0026] The global identification and uncertainty problem may be simplified into a local computational mini-problem to quickly compute the conditional probability and association of a small set of ships of interest.

[0027] The uncertainties about the associations may be objective and derived from probabilities.

[0028] The output report may further comprise an explanation of the association decision.

[0029] Other aspects and features will become apparent to those ordinarily skilled in the art, upon review of the following description of some exemplary embodiments.Brief Description of the Drawings

[0030] The drawings included herewith are for illustrating various examples of articles, methods, and apparatuses of the present specification. In the drawings:

[0031] Figure 1 is a block diagram of an advanced associator, according to an embodiment;

[0032] Figure 2 is an illustration of a ship association problem address by the present disclosure, according to an embodiment;

[0033] Figure 3 are schematic diagrams of an ability of an advanced associator of the present disclosure to match various uncertain ship properties and compute correct probabilities for the matches, according to an embodiment;

[0034] Figure 4 are schematic diagrams illustrating an ability of an advanced associator of the present disclosure to take into consideration competing solutions that impact the match probabilities, according to an embodiment;

[0035] Figure 5 is a schematic diagram illustrating an ability of an advanced associator of the present disclosure to de-conflict competing solutions and compute a globally optimal solution, according to an embodiment;

[0036] Figure 6 is a schematic diagram illustrating Bayesian network dependencies between random variables, according to an embodiment; and

[0037] Figure 7 is a schematic diagram of a satellite imaging system including a broad area imaging satellite and a higher resolution imaging satellite, according to an embodiment.Detailed Description

[0038] Various apparatuses or processes will be described below to provide an example of each claimed embodiment. No embodiment described below limits any claimed embodiment and any claimed embodiment may cover processes or apparatuses that differ from those described below. The claimed embodiments are not limited to apparatuses or processes having all of the features of any one apparatus or process described below or to features common to multiple or all of the apparatuses described below.

[0039] One or more systems described herein may be implemented in computer programs executing on programmable computers, each comprising at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, and without limitation, the programmable computer may be a programmable logic unit, a mainframe computer, server, and personal computer, cloud-based program or system, laptop, personal data assistance, cellular telephone, smartphone, or tablet device.

[0040] Each program is preferably implemented in a high-level procedural or object-oriented programming and / or scripting language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a storage media or a device readable by a general or special purpose programmable computer for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein.

[0041] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary, a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0042] Further, although process steps, method steps, algorithms or the like may be described (in the disclosure and / or in the claims) in a sequential order, such processes, methods and algorithms may be configured to work in alternate orders. In other words,any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any order that is practical. Further, some steps may be performed simultaneously.

[0043] When a single device or article is described herein, it will be readily apparent that more than one device / article (whether or not they cooperate) may be used in place of a single device / article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device / article may be used in place of the more than one device or article.

[0044] Referring now to Figure 1 , shown therein is a vessel tracking and identification system including an advanced associator 100 for tracking marine vessels, according to an embodiment. The advanced associator 100 may be encoded in computerexecutable instructions which, when executed by a processor, cause the processor to perform the functions and provide the functionalities described herein.

[0045] The advanced associator 100 is implemented on a computer system including one or more computing devices (in the case of multiple computing devices, the devices may be connected via a computer network). The computer system includes at least one data storage device (e.g., a computer memory) and at least one processor. The processor processes data and the memory stores data processed by the processor. The computer system also includes a communication interface for receiving data from and sending data to external sources. The communication interface may include a network interface for communicating with one or more other computer systems via a network, such as a wide area network (e.g., Internet). The computer system may also include a display device for displaying an output generated by the advanced associator (or some product generated using the output). The computer system may be configured to execute a user interface for enabling a user to interact with the system, including with the advanced associator, which may include receiving a user input (e.g., via an input device) and displaying the user interface via the display device.

[0046] The advanced associator 100 includes an evidence collector module 104, an analytic matchers module 106, a machine learning-based ship image matchersmodule 108, a Bayesian reasoner module 110, and an advanced associator controller 112. Arrows represent the exchange of data between the components of the advanced associator 100.

[0047] Figure 1 also includes a maritime domain surveillance data fusion platform 102 which provides information to the AA 100.

[0048] A function of the Advanced Associator (“AA”) 100 is to track ships, both selfreporting and dark. The advanced associator 100 also serves to discover the identity of dark ships. These are key capabilities for maritime awareness. The advanced associator 100 may also be referred to as a Bayesian Reasoner (BR) since it employs a Bayesian reasoning framework within the Bayesian reasoner module 110.

[0049] Given a set of observed ships and a set of candidate ships, the AA 100 determines which candidate ships are closest to which observed ship. For example, candidate ships may be observed in a satellite image. A set of candidate ship locations may then be generated by extrapolating existing, known ship tracks to the image acquisition time if they fall within the image acquisition footprint.

[0050] The AA 100 may first collect any information about the properties of the observed ships and the candidate ships. The AA 100 then compares the ship properties and quantifies an overall quality of the comparisons. The AA 100 makes a global decision about which observed ship is a given candidate ship and provides probabilities that quantify the certainty of the global decision.

[0051] The AA 100 may be used in or applied to various applications directed to deriving insights from maritime domain surveillance data (“maritime insights”). In one embodiment, the AA 100 may be used to perform association of a ship observed by satellites (or by other platforms) with ships that are self-reporting on AIS or by other selfreporting methods (herein “example application 1”).

[0052] In an embodiment, the AA 100 may be used to perform association of a newly observed dark ship with an existing dark track (herein “example application 2”).

[0053] In an embodiment, the AA 100 may be used to perform identification of dark ships by inferring their maritime mobile service identity (MMSI) numbers (herein“example application 3”). This is achieved by comparing the dark ships with archived candidate ships.

[0054] In an embodiment, the AA 100 may be used to perform joining of existing dark tracks with each other because they belong to the same ship (herein, “example application 4”. In another embodiment, the AA 100 may be used to perform splitting of an existing dark track into two tracks because the existing track belongs to more than one ship.

[0055] In an embodiment, the AA 100 may be used to perform splitting of an existing dark track into two tracks because the existing track does belong to more than one ship (herein, “example application 5”).

[0056] In an embodiment, the AA 100 may be used to perform refining knowledge of dark ship features by fusing information from multiple contacts of a dark track (herein, “example application 6”). In some cases, some of the foregoing functionalities may be provided, in part, by a maritime domain surveillance data fusion platform 102. The platform 102 is configured to communicate with the advanced associator 100.

[0057] The platform 102 may be configured to perform any one or more of the following: detecting vessels, extracting vessel detections from raw imagery (optical and SAR), monitoring / tracking vessels, tracking dark vessels, behavior analytics (AlS / vessel monitoring system (VMS)), and vessel identification. The ability of the platform 102 to perform such functions may be dependent upon data generated by the advanced associator 100.

[0058] In some embodiments, the AA 100 may be a service that serves some or all of the applications noted above. The differences between these applications may be transparent to the AA 100 service.

[0059] For example, an important difference between association of a ship observed by satellites (or other platforms) with ships that are self-reporting on AIS or by other self-reporting methods (example application 1 ) and identification of dark ships by inferring their MMSI numbers (comparing darks ships with archived candidate ships) (example application 3), is that detailed kinematic information, such as location orheading, cannot be used for comparing ships in example application (3). However, in some embodiments, the AA service 100 works even in the absence of kinematic information, although less accurately.

[0060] The AA 100 may improve on other approaches, for example by expanding the type of evidence considered beyond position (and orientation).

[0061] The AA 100 is based on the well-established Bayesian theory of reasoning. It provides a single rigorous theoretical framework for all ship association needs and generates an objectively optimal solution given the available evidence.

[0062] Current, alternative fusion methods use ad-hoc rules and parameters that are poorly motivated, leading to poorer results overall. These non-probability-based fusion solutions are more readily available, but lack scientific rigor and user interpretability.

[0063] There are two main difficulties with a rigorous Bayesian approach: (1 ) it is not obvious how the problem should be formulated, and (2) there are serious numerical challenges for practically computing a solution using this approach.

[0064] The AA 100 overcomes both of these difficulties. The complete Bayesian formulation of the ship-matching problem is not trivial. Moreover, achieving practical computability for this large matching problem is a significant challenge.

[0065] At the core of the example applications 1-6 listed above is a rigorous statistical framework that allows multiple observations of ship features to form a common picture of all our available knowledge of dark ships.

[0066] The Bayesian Reasoner has the task of combining all measurements and all a-priori information about the equality of two ships. The desired result of this fusion of information is to determine which observed ship is the same as a given candidate ship.

[0067] There are many fusion techniques which provide some kind of matching score. The proposed Bayesian Reasoner employs classic Bayesian reasoning to provide an objective statistical measure for ship equality rather than using ad-hoc schemes to combine multiple observations.

[0068] This approach has multiple advantages: (i) it provides optimal fusion results that exploit all available information to the maximum possible, thus guaranteeing the optimal ship re-identification results achievable with the information given; (ii) it includes no made-up, ad-hoc parameters, only well-motivated ones; (iii) it follows well-established statistical principles, and therefore is easily defendable from a theoretical point of view; (iv) it provides a universally acceptable and intuitive measure of association quality that is easily to understand by human operators; and (v) is flexible and can easily be extended to include additional information if required in the future.

[0069] A basic problem that the AA 100 solves is as follows: given a set of observed ships and a set of candidate ships, determine which candidate ship is identical to which observed ship.

[0070] The AA 100 does this by comparing the properties of observed ships with those of the candidate ships. In an embodiment, the following properties are compared: (i) Kinematic properties (e.g., Position, Orientation, Speed); (ii) tombstone properties (e.g., Length, Width, Type); (iii) fingerprint properties (e.g., Optical Images, SAR Images, RF Waveform number); and (iv) identity properties (e.g., MMSI number from Channel 70 RF communication).

[0071] For all practical cases, only a small fraction of these properties may be available at any given time. Thus, the AA 100 is configured to self-configure in order to handle a widely varying composition of evidence types and a-priori knowledge.

[0072] In an example, a satellite takes an image over a given acquisition footprint. A set of ships is observed in this image, all of which are imaged at (almost) the same time. A set of candidate ship locations is then generated by extrapolating the existing candidate ship tracks to the image acquisition time, and keeping only those ships that fall within the image acquisition footprint.

[0073] The AA 100 then collects any information about the properties of the observed ships and of the candidate ships. The AA 100 then compares individual properties and quantifies the overall quality of the comparisons. Finally, the AA 100 makes a decision about which observed ship is which candidate ship and providesprobabilities that quantify the correctness of its decisions. The problem is illustrated in Figure 2.

[0074] Figure 2 is an illustration of the problem statement. According to Figure 2A, a satellite image was taken in which some ships were observed (shown as observed ships 221 -225). The locations of candidate ships are then interpolated from existing AIS tracks to the acquisition time of the image. These inferred candidate ships 231 -234 are then used for matching with the observed ships 221 -225. Matched observed and candidate ships are shown with a connecting line.

[0075] In particular, in Figure 2 there are five observed ships 221 -225 which were simultaneously detected at time, t, and four candidate ships 231 -234 for which properties were predicted for the time, t. Three associations between observed ships and candidate ships were found 222 with 231 , 223 with 232, and 225 with 233, with two unmatched observed ships 221 , and 224, and one unmatched candidate ship 234.

[0076] According to the present disclosure, multiple “vignettes” are shown in Figure 3, that illustrate some of the characteristics for the AA.

[0077] Referring now to Figure 32B, shown therein are diagrams that illustrate the ability of the AA 100 to match various uncertain ship properties and compute the correct probabilities for the matches, according to an embodiment.

[0078] According to Figure 3, the left image 301 illustrates a comparison of physical distance versus error distance. An observed ship 320 (or vessel) may be encompassed in an uncertainty ellipse 340. According to Figure 3, a candidate ship 331 may be matched although it is further away than another candidate ship 332. Put another way, closer candidate ship 332 may not be matched to observed ship 320 although it is physically closer than the candidate ship 331 .

[0079] Image 302 of Figure 3 illustrates matching of ships with various properties where an observed ship and a candidate ship having different length, width, orientation, position, image, RF properties and associated uncertainties. The AA 100 is able to match all of the various uncertain ship properties and compute the correct combined matched probability.

[0080] According to Figure 3, the right image 303 illustrates matching with type uncertainty where both the observed and the candidate ship may have type errors. As with uncertain properties, the AA 100 computes the correct combined match probability with other features.

[0081] Referring now to Figure 4, shown therein are diagrams that illustrate an ability of the AA 100 to take into consideration competing solutions that impact the match probabilities, according to an embodiment.

[0082] According to Figure 4, the left image 401 illustrates similar competing candidate ships. An uncertainty ellipse is shown encircling an observed ship with candidate ship A and candidate ship B which are parallel to the observed ship. The association probability of candidate A is much reduced because of similarity of candidate B.

[0083] According to Figure 4, the central image 402 illustrates dissimilar competing candidate ships where an uncertainty ellipse is also shown encircling the observed ship, candidate ship A and candidate ship B. Candidate A is parallel to the observed ship, but Candidate B is substantially orthogonal. Association probability of Candidate A is less reduced because of dissimilarity of the heading of candidate ship B.

[0084] According to Figure 4, the right image 403 illustrates similar competing observed ships where an uncertainty ellipse is also shown encircling the observed ship, and candidate ship A and candidate ship B are all parallel. An association probability of candidate ship A is much reduced due to the similarity of candidate ship B.

[0085] Referring now to Figure 5, shown therein is a diagram 501 that illustrates an ability of the AA 100 to de-conflict competing solutions and compute the globally optimal solution, according to an embodiment. According to Figure 5, an example of global solution resolution using interlocking assignment conflicts is shown. According to Figure 5, observed ship A and observed ship B are shown with candidate ship A, candidate ship B and candidate ship C. Figure 5 selects the best assignment overall wherein the solution computes the correct uncertainty and the best local solution is not necessarily selected.

[0086] In some embodiments of the AA 100, a comprehensive statistical approach is applied to the aforementioned problem: (i) every piece of information is modeled as a random variable; (ii) measurements are modeled as sampling from random distributions; (iii) we form the joint distribution function of all random variables.

[0087] Furthermore, Bayesian reasoning to the joint distribution function is applied to obtain the desired ship association decisions and confidence values. The system computes the globally optimal associations. The system computes the probability of each individual association. The system generates providence of the results that allows users to examine the reasons that led to the decision.

[0088] The advantage of this approach is that it provides a rigorous, objective, general fusion framework. The main disadvantage is that it could easily cause combinatorial explosions which make it practically incomputable. By skillfully exploiting the high sparsity of our problem and allowing some reasonable approximations, this issue can be avoided.

[0089] According to the present disclosure, a joint probability function of all random variables may be used to describe the problem:

[0090] All the random variables, which constitute the arguments of this function, were introduced earlier in the present disclosure. This function captures all the known information about our problem. All quantities of interest can be derived from it, including the desired solution of our problem.

[0091] However, the joint probability function is too complex to determine directly.We therefore must decompose it into a product of partial terms:P(.ZI-.L,I-.K) (Eq.1 )

[0092] Equation 1 (Eq.1 ) can be simplified by considering the independence of some of the variables:P (Z1:L,1:K) cldentity Prior> (Eq.2)

[0093] According to the disclosure, Equation 2 (Eq.2) is just an approximation of Equation 1 (Eq.1 ), because the independence of some of the variables may not be perfect.

[0094] Bayesian network features will now be described. The dependencies between the random variables that make up the joint probability distribution function described above can be visualized as a Bayesian Network. For illustration purposes, this is shown in Figure 6. Figure 6 is a diagram illustrating Bayesian network dependencies between random variables.

[0095] According to Figure 6, a network of dependencies of random variables are shown to assist with identifying ships. According to Figure 6, measured type equality 641 depends on true type equality 651 , measured length equality 642 depends on true length equality 652, measured width equality 643 depends on true width equality 653, measured orientation equality 644 depends on true orientation equality 654, and measured speed equality 645 depends on true speed equality 655. All these dependencies help assist theidentification of the ship. Furthermore, measured position equality 646, measure image equality 647, and measured RF equality 648 all feed directly to identification of ship identities 660.

[0096] It is worth noting that the feature measurements depend only indirectly on the ship identities. This stands in contrast to position, image and RF measurements, which are directly conditioned by the ship identities.

[0097] As a consequence, even perfect, error-free feature matches may not determine identity perfectly. For example, even if the observed ship and the candidate ship are of the same type, we cannot infer that they are necessarily the same ship unless none of the other ships are of this particular type. This is different from, say, the position matches. If two ships are known to be at exactly the same location, then we are allowed to conclude that they are the same ship.

[0098] An optimal solution will now be described. The main output of the AA 100 is the best combination of matches of the observed ships with the candidate ships. Let’s call it It can be found by conducting an exhaustive search over all applicable statesZI-.L,I-.K , and finding the one that maximizes the joint probability function. This constitutes the global optimum for associating observed ships with candidate ships. In other words:Z°PL,I-.K = argmaxVZ1:L,1:7<

[0099] Only the z variables need to be searched over. The variables y° are the fixed measurements.

[0100] The joint probability function over which the search is conducted is not the original full joint probability function as it was introduced earlier. Rather, it is a reduced function that does not depend on the intermediate feature variables. The reduced joint probability function can be obtained by removing these intermediate variables from the original joint probability function by applying a marginalization operation:Probability of the Optimal State

[0101] Besides the optimal global solution, we are also interested to know how probable this solution is. The probability of the optimal state given the measurements is:

[0102] where the denominator can be obtained via marginalization over all applicable states zUil.Kas:

[0103] Note thatis the conditional probability of the best associations between all observed ships with all candidate ships that make up the optimal global state. This means that there is just a single uncertainty number for this global solution.

[0104] However, this is not what a user expects as an uncertainty measure. Rather, a user wants an uncertainty measure that indicates for a particular observed ship how well this particular ship is associated with the indicated track.

[0105] To provide an improved association, a novel approach is required. For example, we may create “local mini problems”. Given the optimal global solution and a particular, observed ship of interest, we select the top-N observed ships and the top-M candidate ships, which most influence the association of the observed ship of interest, as determined by the optimal global solution. We use these sub-selected top-N and top-M ships to form a mini-problem for which we compute the conditional probability. We then declare this conditional probability of the mini-problem to be the (approximate) uncertainty measure of the association for the observed ship of interest.

[0106] It is desirable to provide a human analyst with explanations that makes the output of the AA 100 more understandable. This increases the trust and acceptance of the solution. It also can be used as starting point for a human analyst to conduct manual quality assurance on results that are in question.

[0107] Several types of evidence can be used to interact with a human analyst. As an example, below is an embodiment of an example explanation report:Global Solution: o There were 75 observed ships and 79 candidate ships o The optimal global solution has 70 associations o After local prob, thresholding, only 65 valid associations were keptValid Association # 59:Observed Ship ID: 346778 with Candidate Ship ID: 567888, Probability: 0.53Ship property matches: o Distance is less than 0.8 standard deviations o Lengths are within 0.4 standard deviations■ There are 5 candidate ships with the same length o Type match probability is better than 0.8■ There are 50 candidate ships with the same type o Optical Image Match probability is better than 0.7Competing solutions (that reduce association probability): o Closest alternative candidate ship is ID 89996, Probability 0.2 o Closest alternative observed ship is ID 78865, Probability 0.4

[0108] Referring again to Figure 1 , the connections between these five components of the AA 100 are provided.

[0109] In an embodiment, the AA 100 takes as input: (i) a list of all the observed ships and their known properties, including the uncertainties about said properties; (ii) a list of all the candidate ships and their known properties, including the uncertainties about said properties.

[0110] The input may be received from an external data source, such as maritime domain surveillance data fusion platform 101. The platform 101 may be a platform and a service. The platform 101 may be a cloud-based platform that fuses multiple sources of maritime domain surveillance data, databases, and analytics to create a persistent picture of the maritime situation for use by, for example, defense and security intelligence, fisheries intelligence and commercial intelligence users.

[0111] The platform 101 may be configured to detect vessels, characterize the detected vessels, identify the vessels, and track the vessels through time. The platform 102 may assess or determine a threat level of the vessels. The threat level may enable users to prioritize which vessels are worth further action.

[0112] The platform 101 collects and generates certain data, including observed ship properties and candidate ship properties, that is provided as input to the advanced associator 100.

[0113] The AA 100 includes advanced associator controller 112. The controller 112 coordinates the other components or modules 104, 106, 108, 110 of the AA 100.

[0114] The coordination includes receiving and parsing the AA input, calling other internal components in the correct sequence and with the correct parameters, handling exceptions, and preparing and formatting the output.

[0115] The AA 100 includes evidence collector module 104. The evidence collector module 104 assembles the ship matching evidence that the Bayesian Reasoner 110 requires to make an informed association decision.

[0116] The evidence collector 104 creates all possible pairs between an observed ship and a candidate ship. For any given pair, the evidence collector 104 compares all ship properties by calling the correct matchers that are responsible for the particular properties. If some information is missing, the evidence collector 104 handles these cases in a statistically correct manner. The evidence collector 104 then assembles the matcher results into a common structure that represents the overall evidence.

[0117] The AA 100 includes Analytical Matchers module 106. The analytic matchers module 106 provides partial evidence to the Evidence Collector 104 by deciding if ship features such as ship type and length match. In an embodiment, the analytical matchers module 106 receives input data from the evidence collector 104 including an observed ship property and a candidate ship property and outputs data to the evidence collector 104 including a score class and a conditional probability.

[0118] The analytical matcher module 106 includes a set of individual property matchers. Each property matcher compares only one particular property. For example,the ship position matcher compares the position of an observed ship with the position of a candidate ship. The analytic matchers modules 106 uses a mathematical expression that incorporates the distance between the ships and the positional uncertainties of the ships. The output of this is a proximity score and its corresponding uncertainty measure.

[0119] The AA 100 includes ML-based ship image matchers module 108. The ship image matchers 108 provide partial evidence to the Evidence Collector 104 by deciding if two images show the same ship. In an embodiment, the ship image matchers module 108 may receive an observed ship image chip and a candidate ship image chip as input from the evidence collector 104 and output a score class and joint probability to the evidence collector 104.

[0120] The ship image matchers module 108 includes a set of specialized image matchers. Each individual image matcher compares only one particular combination of image types. For example, an optical nadir-oblique image matcher compares an optical image that was taken from a nadir viewing geometry with an optical image that was taken from an oblique viewing geometry. The comparison is performed using a Siamese neural network that is trained to recognize if two images with these different viewing geometries show the same ship. The output of the ship image matcher is a similarity score and a corresponding uncertainty measure. The uncertainty measure is calibrated to objective probabilities.

[0121] In some embodiments, the image matchers module 108 may use evidence from other sources, such as RF, GPS, or other data sets. Accordingly, the image matchers may also be referred to as empirical matchers. This term may be used, for example, when other, non-image data sources are matched or used by ship image matchers module 108.

[0122] The AA 100 includes Bayesian reasoner 110. The Bayesian reasoner 110 fuses the evidence together and makes an association decision based on all the evidence provided.

[0123] The flow of data through the advance associator 100 is as follows.

[0124] The observed ship properties and candidate ship properties are received from the platform 101 , by the advance associator controller of the advance associator 100.

[0125] The advance associator controller 112 sends the observed ship properties and candidate ship properties to the evidence collector 104 which in turn sends the observed ship properties and candidate ship properties to the analytical matchers 106 and sends observed and candidate ship image chips to the ship image matchers 108 (which may be machine learning based).

[0126] The analytical matchers 106 and the ship image matchers 108 return probabilities of the ship class to the evidence collector 104.

[0127] The evidence collector 104 sends these probabilities to the advanced associator controller 112 as evidence.

[0128] The evidence and the observed and candidate ship properties are provided to the Bayesian reasoner 110 which then returns a probability of associations with candidate ships for each of the observed ships to the advance associator controller 112.

[0129] The platform 101 may then receive an output from the advanced associator 100, for example, a dark ship association report or a matched observed-candidate ship report.

[0130] The output may be incorporated into a further report generated by the platform 101 (which may be presented in a III) or may be used as input to perform further processing to obtain insights about the maritime domain. The platform 101 may provide a user interface for displaying the AA output 100 (or subsequent data derived therefrom) and allowing the user to interact with the platform 102, and potentially the AA 100.

[0131] In an embodiment, the AA 100 generates as output: (i) a report that lists the associations between the observed ships and the candidate ships, including the uncertainties about these associations. The report may be an association report. In an embodiment, the AA outputs an association report (or report) that indicates which observed ship is identical with which candidate ship. The ships may be considered “dark”or immaterial for the functionality of the AA. The AA output report would only be needed if at least one of the two inputs is “dark”.

[0132] In addition to the associations, the output report may also include an explanation of why the AA made these association decisions. This allows a human operator to gain confidence in the association and in some cases override the association decisions. In an embodiment, a further advantage of the AA over other systems (e.g., “SAR-AIS”) is that information about the candidate ships is derived from AIS tracks, as well as dark tracks. Dark tracks refer to tracks where non-self-reporting ships were observed in the past, perhaps multiple times.

[0133] Example satellite imaging systems that may be used to support the AA 100 will now be described in reference to Figure 4. It should be noted that Figure 4 represents one example satellite system and others are contemplated.

[0134] Referring now to Figure 4, shown therein is a satellite imaging system 700, according to an embodiment. The system 700 can be used to perform earth observation tasks using a plurality of imaging satellites. Earth observation tasks may include, for example, any one or more of vessel detection, land intelligence and change detection, asset and infrastructure monitoring, surface deformation monitoring, oil pollution monitoring, humanitarian assistance and disaster relief (HADR) including flood and earthquake monitoring, agriculture monitoring, and forestry monitoring. Vessel detection may include detecting non-transmitting dark ships, illegal fishing activity, and the like.

[0135] The system 700 may be used to perform cross cueing operations, such as described in greater detail herein. The cross cueing may be SAR-to-SAR cross cueing or SAR-to-optical cross cueing.

[0136] The system 700 includes a space segment 702 and a ground segment 704. The ground segment 704 may have a service-oriented cloud architecture. The ground segment 704 includes all elements of the ground including hardware and software (e.g., constellation planning subsystem, ordering, tasking, receiving, image production, archiving, distribution, etc.).

[0137] The space segment 702 includes a constellation 705 of imaging satellites including a broad area imaging satellite 706 and a higher resolution imaging satellite 708. The broad area satellite 706 may be a broad area surveillance satellite. The higher resolution imaging satellite 708 may be a high resolution target monitoring satellite. While the satellite constellation 705 of system 700 is shown having two satellite 706, 708, in other embodiments, the satellite constellation 705 may have additional imaging satellites and the number is not particularly limited. In a particular embodiment, the satellite constellation 705 includes at least two higher resolution imaging satellites 708.

[0138] In the system 700, the broad area imaging satellite 706 may be considered a “leading satellite” and the higher resolution imaging satellite 708 may be considered a “trailing satellite”. The terms “leading” and “trailing” when used herein in reference to satellites is not intended to indicate any particular physical relationship between the leading and trailing satellites but rather refers to the fact that the trailing satellite is used to perform an imaging operation (i.e., acquisition of image data) after and based on an imaging operation performed by the leading satellite. In some cases, the trailing satellite may be considered to physically trail the leading satellite by a period of time (e.g., 1 hour). The period of time may be termed a “pass interval” of the system 700. The pass interval defines a time period between the leading satellite 706 passing over a location (i.e., the location is within the satellite’s imaging range or view) and the trailing satellite 708 passing over the same location (i.e., such that the same location, or approximately the same location, can be imaged by both satellites 706, 708). In such cases, the leading satellite 706 generally acquires image data at a location first and the trailing satellite 708 acquires image data at approximately the same location (depending on whether a subject being image is static or in motion) second. As a result, the system 700 is generally configured to perform certain processing and communication steps, such as those performed by the ground segment 704 described herein, within the pass interval such that the efficiency of the system 700 is maximized.

[0139] The broad area imaging satellite 706 and higher resolution imaging satellite 708 are each in an orbit. The respective orbits are predefined orbits.

[0140] In an embodiment, the higher resolution imaging satellite 708 has the same orbit inclination as the broad area imaging satellite 706. The higher resolution imaging satellite 708 may have the same orbit altitude and inclination as the broad area imaging satellite 706. In such an embodiment, the higher resolution imaging satellite 708 may trail immediately behind the broad area imaging satellite 706 (for example, the latency between broad area satellite and higher resolution satellite cross-cue may be approximately 1 hr).

[0141] Each of the imaging satellites 706, 708 (and more particularly the imaging subsystems thereof) is adapted to acquire imaging data. The imaging data may be optical data. The imaging data may be SAR data. The imaging satellites 706, 708 may be further configured to encrypt and store the acquired image data.

[0142] Each of the imaging satellites 706, 708 is further adapted to transmit the acquired image data to the ground segment 704 and receive instructions and commands from the ground segment 704 via an RF signal of a predetermined signal frequency band.

[0143] The imaging satellites 706, 708 each have an imaging swath (image scene size that is collected). The imaging swath of the broad area imaging satellite 706 is greater than the imaging swath of the high-resolution imaging satellite 708. The term “broad area” when used herein in reference to an imaging satellite or otherwise refers to a relative relationship between the imaging swath of the imaging satellite to which it refers and the imaging swath of another imaging satellite in the system. For example, the broad area imaging satellite 706 of system 700 is “broad area” in relation to the higher resolution imaging satellite 708 as its respective imaging swath covers a greater area than that of the higher resolution imaging satellite 708. Further, the imaging satellites 706, 708 each have an accessible swath (region where the satellite can look to capture an image).

[0144] In an embodiment of the system 700, the number of higher resolution imaging satellites 708 (or “trailing satellites”) used may be based on the imaging swath and accessible swath of the broad area imaging satellite 706 and the respective imaging swaths and accessible swaths of the higher resolution satellites. For example, the system 700 may be configured such that the broad area imaging satellite 706 and the higher resolution imaging satellites 708 have generally overlapping accessible swaths. This maybe achieved, for example, by using a plurality (e.g., two) of trailing higher resolution imaging satellites 708. If the broad area satellite 706 accessible swath is covered / duplicated by the higher resolution imaging satellites 708, the imaging swath of the higher resolution satellites 708 may not need to overlap the broad area satellite 706 imaging swath. In such cases, ground processing, such as performed by the ground terminal 714 described below or other component of the ground segment 704, can determine the coordinates of where to take a second image using the higher resolution satellite 708, which is centered on the target of interest (first image via satellite 706 searches broadly, second image via satellite 708 zooms in).

[0145] The broad area imaging satellite 706 includes an imaging subsystem 710. The imaging subsystem 710 includes an imaging sensor and is configured to perform imaging operations. The imaging subsystem 710 acquires and stores image data. The imaging sensor may be an optical imaging sensor or a SAR imaging sensor.

[0146] The higher resolution imaging satellite 708 includes an imaging subsystem 712. The imaging subsystem 712 may include a SAR imaging sensor and be configured to perform SAR imaging operations. In other embodiments, the imaging subsystem 712 may include an optical imaging sensor instead of or in addition to the SAR imaging sensor. In such cases, the imaging subsystem 712 is configured to perform optical imaging operations.

[0147] The higher resolution imaging satellite 708 is configured to acquire and transmit higher resolution image data. The term higher resolution image data (and the term “higher resolution”, more generally) as used herein is used to refer to a relative resolution of the image data captured by the imaging satellite 708 as compared to the resolution of the image data captured by the broad area imaging satellite 706. That is, the imaging subsystem 712 of the higher resolution imaging satellite 708 is configured to acquire higher resolution images than the imaging subsystem 710 of the broad area imaging satellite 706.

[0148] In an embodiment, the higher resolution imaging satellite 708 is an X-band satellite. In a particular embodiment, the higher resolution imaging satellite 708 is an X- band satellite and the broad area imaging satellite 706 is a C-band satellite.

[0149] The higher resolution imaging satellite 708 may be used for target monitoring (and thus be considered a target monitoring satellite). For example, the higher resolution imaging satellite 708 may acquire and provide high resolution SAR imagery for target monitoring applications (such as described herein) that augment the broad area capability of the broad area imaging satellite 706.

[0150] Generally, a broad area (lower resolution) SAR image is acquired by the broad area satellite 706 which may be used to tell a user “where” to look. This first imaging operation takes advantage of the broad area surveillance capability the broad area imaging satellite 706 to look for targets of interest. The broad area SAR image can be used to determine coordinates for a subsequent higher resolution imaging operation. Once the coordinates of the target are determined, the higher resolution satellite 708 is cross cued to take a closer (i.e. higher resolution) look at the same target of interest. Without the first broad area SAR image, the utility of the higher resolution satellite may be more limited as the accessible swaths are generally much lower. While this may be less of an issue for fixed land-based targets (where to look may already be known), for maritime surveillance applications the first, broad area SAR image can be particularly beneficial, particularly outside port areas when targets are moving.

[0151] In some cases, one or both imaging satellites 706, 708 may include additional payloads. For example, the broad area imaging satellite 706 may include an Automatic Identification System (AIS) for ships, which may be used independently or in conjunction with the imaging subsystem 710. In another embodiment, instead of or in addition to having an AIS receiver on board the broad area satellite 706, the system 700 may utilize a third party AIS data provider feeding AIS data directly into the ground segment 704 (e.g., ground terminal 714). For example, the ground terminal 714 may be communicatively connected to an AIS data feeding computer system via network 728 and receive the AIS data from the AIS data feeding system via the network. Embodiments of the system 700 including the AIS data feeding system may provide advantages. For example, third party AIS data providers may have a global constellation of satellites dedicated to AIS and may be able to provide the ground terminal 714 with ship track historical data that can be used to correlate against SAR data generated by the system 700. In cases where there is no AIS data feeding system and only AIS onboard thesatellite 706, ship tracking information may only be obtained when the satellite 706 is flying overhead.

[0152] Referring now to the ground segment 704 of system 700, the ground segment 704 is used generally to command and monitor the satellites 706, 708 for navigation and imaging, receive satellite telemetry, receive data from the satellites' payloads (e.g. imaging subsystems 710, 712); and manage the data for users. The ground segment 704 may be configured to implement the advanced associator of the present disclosure (e.g., AA 100) at one or more devices in the ground segment (e.g., ground terminal 714, cloud server 726, user terminal 724). The ground segment 704 may similarly implement a maritime domain surveillance data fusion platform, such as platform 101 , that the advanced associator communicates with (e.g., receive input data from and provide output data to).

[0153] The ground segment 704 includes a ground terminal 714. The term ground terminal may be used to refer to a single ground terminal or multiple ground terminals. The ground terminal 714 includes components (e.g., antennas, transmitters, receivers) for transmitting signals to and receiving signals from the imaging satellites 706, 708 and components for processing data (e.g., one or more computing devices, software modules). Processing data includes processing image data received from the satellites 706, 708.

[0154] The ground terminal 714 includes a data receiving station and a data transmitting station. The data receiving and data transmitting stations may each be configured to receive and transmit, respectively, signals of a predetermined signal frequency band (e.g., X-band, S-band). Data receiving stations and data transmitting stations may be specially adapted to communicate with either the broad area imaging satellite 706 or the higher resolution imaging satellite 708 (e.g., a broad area imaging satellite receiving station, a higher resolution imaging satellite receiving station).

[0155] The ground terminal 714 is adapted to receive data from the broad area imaging satellite 706 via downlink 716 and transmit data to the broad area imaging satellite 706 via uplink 718. The data received via downlink 716 includes data acquired by the imaging subsystem 710. The data transmitted via uplink 718 may include imagingtask data instructing the satellite 706 to acquire and return imaging data from a particular location (e.g. coordinates).

[0156] The ground terminal 714 is adapted to receive data from the higher resolution imaging satellite 708 via downlink 720 and transmit data to the higher resolution imaging satellite 708 via uplink 722. The data received via downlink 720 includes image data acquired by the imaging subsystem 712. The data transmitted via uplink 722 may include imaging task data instructing the satellite 708 to acquire and return higher resolution image data from a particular location (e.g., coordinates).

[0157] The ground segment 704 also includes a user terminal 724 and a cloud server 726. The terms user terminal and cloud server may be used to refer to a single user terminal or cloud server or multiple user terminals or cloud servers. The user terminal 724 and cloud server 726 are communicatively connected to the ground terminal 714, and to each other, via network 728. The network 728 may include local area network connections and / or wide area network connections (e.g., the Internet).

[0158] The user terminal 724 is a computing device configured to perform data processing functions and transmit data to and receive data from other computing devices such as ground terminal 714 and cloud server 726 via network connections such as network 728.

[0159] The user terminal 724 may include a client-side software application configured to communicate with a server-side application running on the cloud server 726 or the ground terminal 714. The client-side software application may include a user interface for receiving input data from a user (e.g., requesting an imaging task, other interactions) and outputting data to the user (e.g. displaying processed image data or an output or determination of an image data processing operation performed, such as by the advanced associator 100). The user interface may be a web-based user interface. The user of the user terminal 724 may be an analyst trained to analyze SAR or other image data.

[0160] The cloud server 726 may include a cloud-based software application configured to communicate with a client-side software application running on the user terminal 724 or a software application running on the ground terminal 714. For example,the ground terminal 714 may upload image data to the cloud server 726, which may then store the image data linked to a user account, which can be accessed by the user terminal 724.

[0161] The system 700 may be used to perform maritime cross cueing to detect target vessels. For example, the broad area satellite 706 may collect a broad area SAR image in which a plurality of vessels (e.g., ships) are seen. The ships and ship locations may be correlated by the ground terminal 714 against AIS data from an AIS data feeding system (e.g., AIS data provider who operates its own constellation of AIS satellites). Most ships are likely transmitting an AIS signal broadcasting their respective ID, position, and heading. From this, a determination may be made using the ground terminal 714 identifying the ships in the broad area SAR image that are transmitting AIS. The ships that are not correlated at the ground terminal 714 may be taken to represent leftover dark targets and thus may be considered potential vessels of interest for interdiction of further monitoring. Further monitoring may include tasking the higher resolution satellite 708 to acquire higher resolution SAR image data of the dark targets using coordinates determined from the broad area SAR image.

[0162] The system 700 may be used to perform a cross-cueing operation including dark ship detection. Generally, dark ship detection includes identifying vessels without an automatic identification system (“AIS”) and hotspots where potential illegal activity is occurring. The system 700 may use spaceborne SAR sensors to detect non-transmitting vessels. Vessels without AIS may be considered non-transmitting dark ships. The automatic identification system (AIS) is an automatic tracking system that uses transceivers on ships and is used by vessel traffic services. Satellites can be used to detect AIS signatures, in which case the term Satellite-AIS (S-AIS) may be used. S-AIS may be used for collision avoidance, identification, and location information, as well as for maritime domain awareness, search and rescue, environmental monitoring, and maritime intelligence applications.

[0163] While the above description provides examples of one or more apparatus, methods, or systems, it will be appreciated that other apparatus, methods, or systems may be within the scope of the claims as interpreted by one of skill in the art.

Claims

Claims:1 . An advanced associator system for tracking marine vessels, comprising: a computer memory and a processor in communication with the memory and configured to execute instructions; a communication interface for receiving observed ship data including a list of observed ships and their known properties (“observed ship properties”) and candidate ship data including a list of candidate ships and their known properties (“candidate ship properties”); the memory for storing the observed ship data and the candidate ship data; the processor configured to execute an advanced associator module comprising an evidence collector module, an analytics matcher module, a ship image matcher module, a Bayesian Reasoner module, and an advanced associator controller, wherein the advanced associator module is configured to output a report that lists associations between the observed ships and the candidate ships, including uncertainties about the associations.

2. The system of claim 1 , wherein the list of observed ships includes uncertainties about ship properties, and wherein the list of candidate ships includes uncertainties about ship properties.

3. The system of claim 1 , wherein the ship image matcher module is configured to support machine learning (ML) algorithms.

4. The system of claim 1 , wherein the advanced associator controller is configured to coordinate the other components in the advanced associator module.

5. The system of claim 1 , wherein the evidence collector module is configured to assemble the ship matching evidence that the Bayesian reasoner requires to make an informed association decision.

6. The system of claim 1 , wherein the analytical matchers module is configured to provide partial evidence to the evidence collector by deciding if ship features including as ship type and length match.

7. The system of claim 1 , wherein the ship image matcher module is configured to provide partial evidence to the evidence collector by deciding if two images show the same ship.

8. The system of claim 1 , wherein Bayesian reasoner module is configured to fuse the evidence together and makes an association decision based on all the evidence provided.

9. A computer-implemented method of tracking marine vessels using an advanced associator system, the method comprising: receiving, via a communication interface, observed ships data including list of observed ships and their known properties and candidate ships data including a list of candidate ships and their known properties; providing the observed ships data and candidate ships data to an advanced associator module configured to execute instruction to: generate and compute associations between observed ships in the observed ships data and candidate ships in the candidate ships data, including uncertainties about the associations, wherein the computation utilizes a Bayesian reasoner module configured to run a Bayesian reasoning framework; and output a report including the associations.

10. The method of claim 9, wherein the list of observed ships further comprises uncertainties about the known properties.11 . The method of claim 9, wherein the list of all the candidate ships further comprises uncertainties about the known properties.

12. The method of claim 9, wherein the advanced associator module further comprises a ship image matcher module configured to support machine learning (ML) algorithms.

13. The method of claim 9, wherein the advanced associator module further comprises an advanced associator controller configured to coordinate the other components in the advanced associator module.

14. The method of claim 9, wherein the advanced associator module further comprises an evidence collector module is configured to assemble the ship matching evidence that the Bayesian reasoner requires to make an informed association decision.

15. The method of claim 9, wherein the advanced associator module further comprises an analytical matchers module configured to provide partial evidence to the evidence collector module by deciding if ship features including ship type and length match.

16. The method of claim 9, wherein the advanced associator module further comprises an image matchers module configured to provide partial evidence to the evidence collector module by deciding if two images show the same ship.

17. The method of claim 9, wherein the Bayesian reasoner module is configured to fuse the evidence together and make an association decision based on all the evidence provided.

18. The method of claim 9, where the global identification and uncertainty problem is simplified into a local computational mini-problem to quickly compute the conditional probability and association of a small set of ships of interest.

19. The method of claim 9, wherein the uncertainties about the associations are objective and derived from probabilities.

20. The method of claim 9, wherein the output report further comprises an explanation of the association decision.

Images