Systems, methods, and devices for mixed sensor performance evaluation and optimization

Abstract

A system, method, and server for sensor evaluation are provided, the system including sensors for monitoring an area of interest over a period of interest disposed on sensor platforms and a server for optimizing placement of the sensors, the server pre-calculating environmental statistics for one or more defined time patterns and including a modelling module for generating models of the sensors, platforms, and targets in the AOI over the POI, a definition module for defining a problem in terms of the AOI, POI, sensors, platforms, targets, and optimization criteria and constraints, a discretization module for transforming the problem into a discrete problem within the defined AOI and POI by generating discretization data, an optimization module for optimizing placement of the sensors, according to the discretization data, models, defined time patterns, quality of the one or more sensors, and pre-calculated environmental statistics, and a visualization module for producing a visualization output.

Description

SYSTEMS, METHODS, AND DEVICES FOR MIXED SENSOR PERFORMANCE EVALUATION AND OPTIMIZATIONTechnical Field

[0001] The following relates generally to evaluating and optimizing sensors, and more particularly to evaluating and optimizing sensors in the context of situational awareness.Introduction

[0002] Conventional sensor placement strategies traditionally use tactical approaches to allocate resources and calculate such strategies, e.g., considering how a particular sensor will perform given a specific configuration of sensors, targets, etc. Such discrete conventional sensor placement strategies may be inadequate for planning how environmental factors may vary across a time period (e.g., over a day-night cycle, over annual seasonality).

[0003] Accordingly, there is a need for improved systems, methods, and devices that overcome at least some of the disadvantages of existing systems and methods.Summary

[0004] A system for mixed sensor performance and evaluation is provided. The system includes one or more sensors for monitoring an area of interest (AOI) over a period of interest (POI), the one or more sensors disposed on one or more sensor platforms located within a broader area, the broader area including the AOI and a server for optimizing placement of the one or more sensors, the server pre-calculating environmental statistics for one or more defined time patterns. The server includes a modelling module for generating models of the one or more sensors, of the one or more platforms, and of targets in the AOI, a definition module for defining a problem in terms of the AOI, the POI, the sensors, the platforms, the targets, and optimization criteria and constraints, a discretization module for transforming the problem into a discrete problem within the defined AOI and POI by generating discretization data, an optimization module for optimizing placement of the one or more sensors, according to the discretization data,the models, the defined time patterns, quality of the one or more sensors, and the precalculated environmental statistics, and a visualization module for producing a visualization output provided to a user.

[0005] The one or more defined time patterns may include any one or more of day- night cycles and seasonality.

[0006] The definition module may define the problem by defining the AOI as a polygon having longitude and latitude coordinates as the vertices, defining the POI, defining the target by selecting a target type from a target type list, defining the sensors by selecting one or more sensor types from a sensor type list, and defining the optimization criteria and constraints by selecting the optimization criteria and constraints from an optimization criteria and constraints list.

[0007] The discretization module may perform the discretization by defining a set of locations for the one or more platforms and / or the one or more sensors disposed on the one or more platforms.

[0008] The discretization module may perform the discretization by defining a grid in the air, on land, and / or in the water forming part of the AOI.

[0009] The discretization module may be configured to a degree of medium fidelity to provide sufficient monitoring of the AOI by the one or more sensors without excessively expensive use of the sensors or excessively expensive optimization.

[0010] The discretization module may be configured to a degree of high fidelity to provide optimal or near-optimal monitoring of the AOI by the one or more sensors.

[0011] The discretization module may be configured to a degree of low fidelity to provide non-zero monitoring of the AOI without incurring significant use of the sensors or significantly expensive optimization.

[0012] The discretization module may provide a discrete list of possible locations of the one or more sensors defined according to one or more respective types of the one or more platforms.

[0013] The optimization module may compute a Pareto front including a plurality of sensor placement strategies, each sensor placement strategy being a point on the Pareto front.

[0014] The selected optimization criteria and constraints may include sensor coverage with respect to the target by optimizing dollar cost vs. any one or more of a probability of detecting the target, spatial coverage of the AOI, redundancy coverage, space-time coverage, and a probability of sensing the target.

[0015] The selected optimization criteria and constraints may include sensor coverage by optimizing dollar cost vs. space-time coverage and the probability of detecting the target.

[0016] The optimization module may include a mixed integer optimizer or a mixed programming optimizer.

[0017] The visualization output may include a GUI showing an implementation of the sensor placement strategy implemented in the AOI.

[0018] The GUI may show the implementation on a three-dimensional globe.

[0019] The environmental statistics may include statistics on any one or more of differing temperatures, precipitation, cloud cover, and day-night cycle lengths over the course of the day-night cycle and / or annually.

[0020] A method for mixed sensor performance and evaluation is provided, the method including disposing one or more sensors, for monitoring an area of interest (AOI) over a period of interest (POI), on one or more sensor platforms located within a broader area, the broader area including the AOI, pre-calculating environmental statistics for one or more defined time patterns, generating models of the one or more sensors, of the one or more platforms, and of targets in the AOI over the POI, defining a problem in terms of the AOI, the POI, sensors, the platforms, the targets, and optimization criteria and constraints, transforming the problem into a discrete problem within the defined AOI and POI by generating discretization data, optimizing placement of the one or more sensors, according to the discretization data, the models, the defined time patterns, quality of theone or more sensors, and the pre-calculated environmental statistics, and producing a visualization output to be provided to a user.

[0021] The one or more defined time patterns may include any one or more of day- night cycles and seasonality.

[0022] Defining the problem may include defining the AOI as a polygon having longitude and latitude coordinates as the vertices, defining the POI, defining the target by selecting a target type from a target type list, defining the sensors by selecting one or more sensor types from a sensor type list, and defining the optimization criteria and constraints by selecting the optimization criteria and constraints from an optimization criteria and constraints list.

[0023] Performing the discretization may include defining a set of locations for the one or more platforms and / or the one or more sensors disposed on the one or more platforms.

[0024] Performing the discretization may include defining a grid in the air, on land, and / or in the water forming part of the AOI.

[0025] The discretization may be performed to a degree of medium fidelity to provide sufficient monitoring of the AOI by the one or more sensors without excessively expensive use of the sensors or excessively expensive optimization.

[0026] The discretization may be performed to a degree of high fidelity to provide optimal or near-optimal monitoring of the AOI by the one or more sensors.

[0027] The discretization may be performed to a degree of low fidelity to provide non-zero monitoring of the AOI without incurring significant use of the sensors or significantly expensive optimization.

[0028] Performing the discretization may include providing a discrete list of possible locations of the one or more sensors defined according to one or more respective types of the one or more platforms.

[0029] The optimization may include a Pareto front including a plurality of sensor placement strategies, each sensor placement strategy being a point on the Pareto front.

[0030] The selected optimization criteria and constraints may include optimizing dollar cost vs. a probability of detecting the target, spatial coverage of the AOI, redundancy coverage, space-time coverage, and a probability of sensing the target.

[0031] The selected optimization criteria and constraints may include sensor coverage by optimizing dollar cost vs. space-time coverage and the probability of detecting the target.

[0032] The optimization may be performed, at least in part, with a mixed integer optimizer or a mixed programming optimizer.

[0033] The visualization output may include a GUI showing an implementation of the sensor placement strategy implemented in the AOI.

[0034] The GUI may show the implementation on a three-dimensional globe.

[0035] The environmental statistics may include statistics on any one or more of differing temperatures, precipitation, cloud cover, and day-night cycle lengths over the course of the day-night cycle and / or annually.

[0036] A server for mixed sensor performance and evaluation by optimizing placement of the one or more sensors is provided, the server pre-calculating environmental statistics for one or more defined time patterns, the server including a modelling module for generating models of one or more sensors, for monitoring an area of interest (AOI) within a broader area over a period of interest (POI), disposed on one or more sensor platforms located within the broader area, of the one or more platforms, and of targets in the AOI, a definition module for defining a problem in terms of the AOI, the POI, the sensors, the platforms, the targets, and optimization criteria and constraints, a discretization module for transforming the problem into a discrete problem within the defined AOI and POI by generating discretization data, an optimization module for optimizing placement of the one or more sensors, according to the discretization data, the models, the defined time patterns, quality of the one or more sensors, and the precalculated environmental statistics, and a visualization module for producing a visualization output provided to a user.

[0037] The one or more defined time patterns may include any one or more of day- night cycles and seasonality.

[0038] The definition module may define the problem by defining the AOI as a polygon having longitude and latitude coordinates as the vertices, defining the POI, defining the target by selecting a target type from a target type list, defining the sensors by selecting a sensor type from a sensor type list, and defining the optimization criteria and constraints by selecting the optimization criteria and constraints from an optimization criteria and constraints list.

[0039] The discretization module may perform the discretization by defining a set of locations for the one or more platforms and / or the one or more sensors disposed on the one or more platforms.

[0040] The discretization module may perform the discretization by defining a grid in the air, on land, and / or in the water forming part of the AOI.

[0041] The discretization module may be configured to a degree of medium fidelity to provide sufficient monitoring of the AOI by the one or more sensors without excessively expensive use of the sensors or excessively expensive optimization.

[0042] The discretization module may be configured to a degree of high fidelity to provide optimal or near-optimal monitoring of the AOI by the one or more sensors.

[0043] The discretization module may be configured to a degree of low fidelity to provide non-zero monitoring of the AOI without incurring significant use of the sensors or significantly expensive optimization.

[0044] The discretization module may provide a discrete list of possible locations of the one or more sensors defined according to one or more respective types of the one or more platforms.

[0045] The optimization module may compute a Pareto front including a plurality of sensor placement strategies, each sensor placement strategy being a point on the Pareto front.

[0046] The selected optimization criteria and constraints may include sensor coverage with respect to the target by optimizing dollar cost vs. a probability of detectingthe target, spatial coverage of the AOI, redundancy coverage, space-time coverage, and a probability of sensing the target.

[0047] Selected optimization criteria and constraints may include optimizing dollar cost vs. space-time coverage and the probability of detecting the target.

[0048] The optimization module may include a mixed integer optimizer or a mixed programming optimizer.

[0049] The visualization output may include a GUI showing an implementation of the sensor placement strategy implemented in the AOI.

[0050] The GUI may show the implementation on a three-dimensional globe.

[0051] The environmental statistics may include statistics on any one or more of differing temperatures, precipitation, cloud cover, and day-night cycle lengths over the course of the day-night cycle and / or annually.

[0052] Other aspects and features will become apparent, to those ordinarily skilled in the art, upon review of the following description of some exemplary embodiments.Brief Description of the Drawings

[0053] The drawings included herewith are for illustrating various examples of articles, methods, and apparatuses of the present specification. In the drawings:

[0054] Figure 1 is a block diagram illustrating a system for evaluating and optimizing sensors, in accordance with an embodiment;

[0055] Figure 2 is a block diagram of a computer system for evaluating and optimizing sensors, according to an embodiment;

[0056] Figure 3 is a block diagram of a computer system for sensor performance evaluation and optimization, according to an embodiment;

[0057] Figures 4A-4C are graphs showing environmental statistics;

[0058] Figure 5 is an example user interface screen, according to an embodiment; and

[0059] Figure 6 is a flow diagram of a method for mixed sensor performance and evaluation, according to an embodiment.Detailed Description

[0060] Various apparatuses or processes will be described below to provide an example of each claimed embodiment. No embodiment described below limits any claimed embodiment and any claimed embodiment may cover processes or apparatuses that differ from those described below. The claimed embodiments are not limited to apparatuses or processes having all of the features of any one apparatus or process described below or to features common to multiple or all of the apparatuses described below.

[0061] One or more systems described herein may be implemented in computer programs executing on programmable computers, each comprising at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, and without limitation, the programmable computer may be a programmable logic unit, a microcontroller and embedded processors, a mainframe computer, server, and personal computer, cloud-based program or system, laptop, personal data assistance, cellular telephone, smartphone, or tablet device.

[0062] Each program is preferably implemented in a high-level procedural or object-oriented programming and / or scripting language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a storage media or a device readable by a general or special purpose programmable computer for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein.

[0063] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary, avariety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0064] Further, although process steps, method steps, algorithms or the like may be described (in the disclosure and I or in the claims) in a sequential order, such processes, methods and algorithms may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any order that is practical. Further, some steps may be performed simultaneously.

[0065] When a single device or article is described herein, it will be readily apparent that more than one device I article (whether or not they cooperate) may be used in place of a single device I article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device I article may be used in place of the more than one device or article.

[0066] The following relates generally to evaluating and optimizing sensors and tools for same, and more particularly to evaluating and optimizing sensors and tools for same in the context of situational awareness.

[0067] Such situational awareness may be particularly relevant in arctic regions of Canada where it may be particularly desirable to monitor for approaching threats (e.g., missiles, aircraft, ships). The vastness of the arctic region may make the arctic region particularly difficult to monitor in a cost-effective fashion. Accordingly, the present disclosure includes a decision support tool for wide-area surveillance, such as a mixed sensor performance and evaluation tool. The effectiveness and efficacy of sensor mixes (e.g., onboard an aircraft, on a stationary platform) may be analyzed in all domains (e.g., surface, subsurface, air) in order to find an optimal mixed sensing strategy for situational awareness. Such optimal mixed sensing strategy may be provided or determined in respect of surveillance scenarios. Such surveillance scenarios may be user-defined (e.g., received as input from a user). Such surveillance scenarios may be provided on or through a graphical user interface (GUI) to explore or visualize surveillance solutions.Such sensing strategy may include existing placed sensors (e.g., assuming one or more sensors remains where it is) and deciding where to place further sensors.

[0068] Where such surveillance is to be performed in respect of vast regions such as the arctic, determining optimal sensing strategies, particularly optimal mixed sensing strategies, may be costly, both from a computational and dollar standpoint. Accordingly, the degree of optimality in the foregoing strategies may itself be user-defined so that users may select a level of optimization to be applied in providing or determining a sensing strategy in a surveillance scenario.

[0069] In an example, the sensors include a satellite with a given resolution, swathe, overage, and revisit rate. The user may provide inputs or make queries such as ‘What coverage does the satellite have?’, ‘Given a particular target in a region (e.g., a boat, a missile), what is the probability that the satellite, when passing by or through the region, will see the target?’. Such analysis may be performed on a statistical rather than a per-inference basis.

[0070] The present disclosure advantageously solves a strategic sensor placement problem by representing how a set of sensors will perform statistically across a time period, thus providing an effective planning tool.

[0071] In an embodiment, provided are medium-fidelity models optimized at interactive speeds, thereby allowing for an intuitive and interactive sensor planning experience that enables rapid integration. Throughout the systems, methods, and devices discussed herein, components may advantageously be generic and customizable, resulting in a versatile and interactive environment. Characteristics of the sensors and platforms discussed herein may advantageously be editable such that a domain expert may define their own (classified) sensor models without disclosing sensitive sensor details.

[0072] Referring now to Figure 1 , shown therein is a schematic diagram illustrating a system 10 for evaluating and optimizing a sensing problem, in accordance with an embodiment.

[0073] The system 10 includes a plurality of sensors 14 (for convenience, represented by a single sensor 14 in Figure 1 ). Each of the plurality of sensors 14 may be disposed on satellites (such as satellite 12 of Figure 1 ), on a stationary radar, on a moving platform (e.g., a search pattern aircraft), or otherwise. The sensors 14 may accordingly be static or dynamic. The sensors 14 may include a sonar array. It will be understood that other sensors are expressly contemplated and included in the present disclosure.

[0074] Each of the sensors 14 may be already active and available to the system 10 or may be available for the system 10 to acquire and place if desirable for evaluating and optimizing the sensing problem. It will be understood that the sensors 14 may represent all the sensors that are available or that may be available to the system 10 and that the system 10 and the optimization performed thereby do not depend on any particular sensor 14. The optimization performed by the system 10 may be run before the sensors 14 exist for the system 10. The optimization performed by the system 10 may include existing sensors 14.

[0075] A platform includes a vehicle (e.g., boats, aircraft) that carries one or more sensors 14 (e.g., the satellite 12).

[0076] The system 10 further includes a server 16 which communicates with the plurality of sensors 14 and one or more command devices 18 via a network 20. The server 16 may be a purpose-built machine designed specifically for evaluating and optimizing the sensors 14 and providing information thereon to the command devices 18.

[0077] The sensors 14 sense, measure, perceive, monitor, or otherwise observe a specific area or “area of interest” (“AOI”) 22 defined for the sensing problem.

[0078] For convenience, throughout the present disclosure, the meanings of each of “sensing”, “measuring”, “perceiving”, “monitoring”, “observing”, and like terms shall be considered to include the meanings of the other terms except where context indicates otherwise. The AOI 22 is located in a broader area 24 that the sensors 14 are configured to sense but do not necessarily sense without commands transmitted by the command devices 18 and received by the server 16 and / or by the sensors 14. The sensors arelocated in the broader area 24 and / or disposed on platforms that are located in or move through the broader area 24.

[0079] The system 10 further includes a target 28 that, according to commands provided by the device 18, it is desirable to monitor. The target 28 may be a general class of targets (e.g., monitoring the presence of any ship along a coastline in the AOI 22).

[0080] The server 16 is configured to enable evaluating and optimizing the sensors 14 with respect to the AOI 22, that is, evaluating and optimizing which sensors 14 to use in a solution to the sensing problem and to determine placement thereof.

[0081] The server 16 is configured to perform discretization with respect to the broader area 24 in order to determine the AOI 22 and further with respect to the AOI 22 in order to efficiently monitor the AOI 22.

[0082] Such discretization includes defining a set of feasible sensor locations (e.g., a location of a stationary sensor 14, a desired orbit of a satellite 12 carrying the sensor 14).

[0083] Such discretization may include gridding or defining a grid in the air, on land, and / or on water forming part of the AOI 22 or the broader area 24. Such gridding may include consideration of the coast or shoreline adjacent or included in the AOI 22 and / or the broader area 24.

[0084] The server 16 computes and stores sensor placement strategies 30.

[0085] The sensor placement strategies 30 may include one or more of a sensor placement strategy. Each such sensor placement strategy may include a list of sensors 14 to be used and how such sensors 14 are to be used (e.g., which pattern an aircraft is to fly, where to place a radar). The sensor placement strategies 30 may be a list of each Pareto-optimal sensor placement strategy found.

[0086] The sensor placement strategies 30 may include differential coverage or placement of sensors 14 during daylight vs. at night. The sensor placement strategies 30 may optimize sensor coverage for daylight, for the night, or for both.

[0087] Where one or more sensors 14 are disposed on a search pattern aircraft, the discretization may include selecting a number of patterns (e.g., 10-20) that the searchpattern aircraft is to navigate or travel over the broader area 24 in order to optimize coverage of the AOI 22 by the one or more sensors 14 disposed on the search pattern aircraft.

[0088] Where one or more sensors 14 are disposed on the satellite 12, the discretization may include selecting a percentage of available coverage of the AOI 22 to purchase or use.

[0089] The discretization may be performed to a degree of medium fidelity, i.e. , to ensure sufficient monitoring by the sensors 14 of the AOI 22 without excessively expensive use of the sensors 14 or excessively expensive optimization calculations.

[0090] The discretization may be performed to a degree of high fidelity, i.e., to ensure optimal or near-optimal monitoring by the sensors 14 of the AOI 22.

[0091] The discretization may be performed to a degree of low fidelity, i.e., to ensure non-zero monitoring by the sensors 14 of the AOI 22 without incurring significant use of the sensors 14 or significantly expensive optimization calculations.

[0092] The foregoing optimization calculations performed by the server 16 pertain to the placement strategies associated with the sensors 14, i.e., where the sensors 14 are to be placed relative to the AOI 22 and / or the broader area 24.

[0093] The server 16 further includes a processor 32 for receiving queries from the device 18 and providing, as an output of the optimization, a Pareto front (i.e., a set of solutions to the query that each demonstrates Pareto optimality).

[0094] For example, based on the query received from the device 18, the Pareto processor 32 may output an optimization (i.e., one or more sensor placement strategies 30) for sensor coverage of the AOI 22 with respect to the target 28 that optimizes according to optimization criteria and constraints, e.g., dollar cost vs. one or more of the probability of detecting the target 28, spatial coverage of the AOI 22, redundancy coverage (in case one or more satellites 12 or sensors 14 is damaged or non-functional), space-time coverage (capturing the revisit time coverage of a sensor), and probability of sensing the target 28 (according to the probability of detecting the target multiplied by theprobability of the target being present; therefore target density maps may advantageously be used to focus an optimization effort).

[0095] The dollar cost may include the cost of the platform and the sensors 14, including purchase and operational costs.

[0096] In an embodiment, the optimization criteria and constraints are dollar cost versus space-time coverage and probability of detecting the target 28.

[0097] The Pareto processor 32 combines linear and non-linear Mixed Integer Programming (MIP) optimizers to optimize in an efficient manner.

[0098] The Pareto processor 32 caches results of time-intensive intermediate calculations such that calculating new sensor placement combinations may advantageously be computed quickly. In an embodiment, the Pareto processor 32 optimizes a moderately sized problem in minutes.

[0099] The query may be a precise statement of the AOI 22 and a dollar limit. The query may be conversational (e.g., “how much money does it cost to detect a certain class of target at 70% accuracy?”, “how do I place the sensors to detect a certain class of target at 70% accuracy for a given amount of money?”).

[0100] The system 10 further includes data storage 26 for storing inputs to the system 10 (e.g., (statistical) weather data to assist the server 16 in sensor placement strategies) and / or outputs of the system (e.g., the optimized sensor placement strategies, data sensed by the sensors 14). The data storage 26 may be cloud storage.

[0101] The system 10 may include or receive as input a corresponding period of interest, e.g., between January 1 and February 15. Each period of interest may have corresponding sensor placement strategies 30 stored in the data storage 26. The sensor placement strategies 30 corresponding to different periods of interest may differ according to seasonality of the broader area 24 (e.g., differing temperatures, precipitation, cloud cover, day-night cycle lengths over the course of the year).

[0102] The weather data may include environmental statistics (e.g., temperature, precipitations) as worldwide maps. Each relevant location on such worldwide map may have statistics that characterize the location. The environmental statistics may be definedfor a time pattern (e.g., seasons of the year, day-night cycle). Given a period of interest (POI), the system 10 is configured to identify the applicable time patterns and the applicable statistics to draw from. The sensors 14 may not be influenced by all environmental or weather conditions. Advantageously, the performance of a sensor 14 is calculated using only relevant factors.

[0103] The sensors 14 may be any one or more of cameras, imaging devices, infrared sensors, lidar devices, accelerometers, acoustic sensors, passive and active radar sensors, and radio-wave detectors.

[0104] The command device 18 is configured to receive input from a human user or operator with respect to the sensors 14 and / or the satellites 12 (e.g., selection of a solution to the sensing problem of query. The command device 18 may be configured to receive input and / or provide output from and / or to other computers, servers, or devices, for example a series of computers intermediating and facilitating communication between the sensors 14 and the command device 18.

[0105] The sensors 14, the server 16, the command devices 18, and / or the data storage 26 may be a server computer, desktop computer, notebook computer, tablet, PDA, smartphone, or another computing device. The sensors 14, server 16, the command device 18, and / or the data storage 26 may include a connection with the network 20 such as a wired or wireless connection to the Internet. In some cases, the network 20 may include other types of computer or telecommunication networks. The sensors 14, the server 16, the command device 18, and / or the data storage 26 may include one or more of a memory, a secondary storage device, a processor, an input device, a display device, and an output device. Memory may include random access memory (RAM) or similar types of memory. Also, memory may store one or more applications for execution by processor. Applications may correspond with software modules comprising computer-executable instructions to perform processing for the functions described below. The secondary storage device may include a hard disk drive, floppy disk drive, CD drive, DVD drive, Blu-ray drive, or other types of non-volatile data storage. The processor may execute applications, computer-readable instructions, or programs. The applications, computer-readable instructions or programs may be storedin memory or in secondary storage or may be received from the Internet or other network 20. The input device may include any device for entering information into the server 16 and / or the command devices 18. For example, the input device may be a keyboard, keypad, cursor-control device, touchscreen, camera, or microphone. The display device may include any type of device for presenting visual information. For example, the display device may be a computer monitor, a flat-screen display, a projector or a display panel. The output device may include any type of device for presenting a hard copy of information, such as a printer for example. The output device may also include other types of output devices such as speakers, for example. In some cases, the server 16 and the sensors 12, the device 18, and / or the data storage 26 may include multiple of any one or more of processors, applications, software modules, secondary storage devices, network connections, input devices, output devices, and display devices.

[0106] Although the server 16, the sensors 14, the device 18, and the data storage 26 are described with various components, one skilled in the art will appreciate that the server 16, the sensors 14, the device 18, and the data storage 26 may in some cases include fewer, additional or different components. In addition, although aspects of an implementation of the server 16, the sensors 14, the device 18, and the data storage 26 may be described as being stored in memory, one skilled in the art will appreciate that these aspects may also be stored on or read from other types of computer program products or computer-readable media, such as secondary storage devices, including hard disks, floppy disks, CDs, or DVDs; a carrier wave from the Internet or other network; or other forms of RAM or ROM. The computer-readable media may include instructions for controlling the server 16, the sensors 14, the device 18, and the data storage 26 and / or processor to perform a particular method.

[0107] In the description that follows, devices such as the server 16, the sensors 14, the device 18, and the data storage 26 are described performing certain acts. It will be appreciated that any one or more of these devices may perform an act automatically or in response to an interaction by a user of that device. That is, the user of the device may manipulate one or more input devices (e.g., a touchscreen, a mouse, or a button) causing the device to perform the described act. In many cases, this aspect may not be described below, but it will be understood.

[0108] As an example, it is described below that the sensors 14, the device 18, and / or the data storage 26 may send information to the server 16. For example, a user using the command device 18 may manipulate one or more input devices (e.g. a mouse and a keyboard) to interact with a user interface displayed on a display of the command device 18, e.g., to select a sensor placement strategy stored at the sensor placement strategies 30. Generally, the command device 18 may receive a user interface from the network 20 (e.g., in the form of a webpage). Alternatively or in addition, a user interface may be stored locally at the command device 18 (e.g., a cache of a webpage or a mobile application).

[0109] The server 16 may be configured to receive a plurality of information, from each of the sensors 14, the command device 18, and the data storage 26.

[0110] In response to receiving information, the server 16 may store the information in a storage database. The storage database may correspond with secondary storage of the server 16 or the data storage 26. Generally, the storage database may be any suitable storage device such as a hard disk drive, a solid-state drive, a memory card, or a disk (e.g., CD, DVD, Blu-ray). Also, the data storage 26 may be locally connected with the server 16. In some cases, the data storage 26 may be located remotely from the server 16 and accessible to the server 16 across a network, for example. In some cases, the data storage 26 may comprise one or more storage devices located at a networked cloud storage provider.

[0111] Each command device 18 may be associated with an operator account. Any suitable mechanism for associating each command device 18 with an account is expressly contemplated. In some cases, a command device 18 may be associated with an account by sending credentials (e.g. a cookie, login, password) to the server 16. The server 16 may verify the credentials (e.g., determine that the received password matches a password associated with the account). If a command device 18 is associated with an account, the server 16 may consider further acts by that command device 18 to be associated with that account.

[0112] In an embodiment, the sensors 14 are internal to the satellite 12, a stationary radar, moving platform or other platform on which the sensors 14 areconsidered to be disposed. In an embodiment, the sensors 14 are external to such platform.

[0113] Referring now to Figure 2, shown therein is a simplified block diagram of a device 200 for evaluating and optimizing sensors, according to an embodiment.

[0114] The device 200 may be for example any of the devices shown in Figure 1 . The device 200 includes a processor 202 that controls the operations of the device 200. Communication functions, including data communications, voice communications, or both may be performed through a communication subsystem 204. The communication subsystem 204 may receive messages from, and send messages to, a wireless network 250. Data received by the device 200 may be decompressed and decrypted by a decoder 206.

[0115] The wireless network 250 may be any type of wireless network, including, but not limited to, data-centric wireless networks, voice-centric wireless networks, and dual-mode networks that support both voice and data communications.

[0116] The device 200 may be a battery-powered device and as shown includes a battery interface 242 for connecting one or more rechargeable batteries 244.

[0117] The processor 202 also interacts with additional subsystems such as a Random Access Memory (RAM) 208, a flash memory 210, a display 212 (e.g., with a touch-sensitive overlay 214 connected to an electronic controller 216 that together comprise a touch-sensitive display 218), an actuator assembly 220, one or more optional force sensors 222, an auxiliary input / output (I / O) subsystem 224, a data port 226, a speaker 228, a microphone 230, short-range communications systems 232 and other device subsystems 234.

[0118] In some embodiments, user interaction with the graphical user interface may be performed through the touch-sensitive overlay 214. The processor 202 may interact with the touch-sensitive overlay 214 via the electronic controller 216. Information, such as text, characters, symbols, images, icons, and other items that may be displayed or rendered on a portable electronic device generated by the processor 202 may be displayed on the touch-sensitive display 218.

[0119] The processor 202 may also interact with an accelerometer 236 as shown in Figure 2. The accelerometer 236 may be utilized for detecting direction of gravitational forces or gravity-induced reaction forces.

[0120] To identify a subscriber for network access according to the present embodiment, the device 200 may use a Subscriber Identity Module or a Removable User Identity Module (SIM / RUIM) card 238 inserted into a SIM / RUIM interface 240 for communication with a network (such as the wireless network 250). Alternatively, user identification information may be programmed into the flash memory 210 or performed using other techniques.

[0121] The device 200 also includes an operating system 246 and software components 248 that are executed by the processor 202 and which may be stored in a persistent data storage device such as the flash memory 210. Additional applications may be loaded onto the device 200 through the wireless network 250, the auxiliary I / O subsystem 224, the data port 226, the short-range communications subsystem 232, or any other suitable device subsystem 234.

[0122] For example, in use, a received signal such as a text message, an e-mail message, web page download, or other data may be processed by the communication subsystem 204 and input to the processor 202. The processor 202 then processes the received signal for output to the display 212 or alternatively to the auxiliary I / O subsystem 224. A subscriber may also compose data items, such as e-mail messages, for example, which may be transmitted over the wireless network 250 through the communication subsystem 204.

[0123] For voice communications, the overall operation of the device 200 may be similar. The speaker 228 may output audible information converted from electrical signals, and the microphone 230 may convert audible information into electrical signals for processing.

[0124] Referring now to Figure 3, shown therein is a block diagram of a computer system 300 for sensor performance evaluation and optimization, according to an embodiment.

[0125] The computer system 300 may be implemented at one or more devices of the system 10 of Figure 1 . For example, some or all of the components of the computer system 300 may be implemented by any one or more of the server 16, the command device 18, and the data storage 26.

[0126] The system 300 includes a processor 302 for executing software models and modules.

[0127] The system 300 further includes a memory 304 for storing data, including output data from the processor 302.

[0128] The system 300 further includes a communication interface 306 for communicating with other devices, such as through receiving and sending data via a network connection (e.g., the network 20 of Figure 1 ).

[0129] The system 300 further includes a display 308 for displaying various data generated by the computer system 300 in human-readable format. For example, the display may be configured to display a visualization output 330. The display 308 may receive queries, commands, or input data 312 from a user.

[0130] The processor 302 includes a modelling module 310 for generating models of sensors, of sensor platforms, of targets, and of other features in an area of interest (AOI) over a period of interest (POI), i.e. , an area to be monitored by sensors (such as the sensors 14 of Figure 1 ). The processor 302 receives the input data 312 (e.g., a command to generate a model of a specific feature in the AOI). The input data 312 is stored in the memory 304. The modelling module 310 generates the selected model as model output data 314 further stored in the memory 304. The input data 312 may include definitions provided by an analyst or expert. The analyst or expert (e.g., a domain expert) may be able to edit characteristics of the model output data 314. The analyst may define, as input data 312, which trends or statistical environmental data are important to the model output data 314.

[0131] The models of sensors and of sensor platforms are described by editable parameter files that a user creates as input to the computer system 300 (e.g., as the inputdata 312). Such editable parameter files are preferably provided before the optimization module 324 proceeds.

[0132] The modelling module 310 defines sensor characteristics (e.g., a first camera has a first resolution and a second camera has a second resolution).

[0133] The modelling module 310 generates the model output data 314 in a statistical manner, e.g., according to captured noise and environmental weather. The statistically generated model output data 314 may be statistically true across seasons, day / night, and regions of the world.

[0134] The modelling module 310 advantageously enables experts to define or modify sensor models (e.g., the model output data 314) based on intuitive or measurable parameters, defining characteristics such as the cost of a sensor, the capabilities of the sensor as to detecting a target, and how environmental factors influence the signal to noise ratio of the sensor. The model output data 314 (and thus the models) may be characterized in text documents and loaded at runtime.

[0135] The model output data 314 may advantageously consider multiple domains, e.g., underwater, on-water, on-land, airborne, in-space. Targets may be defined in any domain (optionally excluding space) and sensors may be disposed (on their respective platforms) in any domain and may detect in any domain.

[0136] Via the modelling module 310, the user may define how a platform is discretized (e.g., on-land vs on-water vs on-the-coast vs a travelling pattern.

[0137] The model output data 314 may advantageously define time patterns (viz., time-of-day and seasonality) under which sensor performance may vary. The model output data 314 may include pre-computed statistics as to how an environmental factor varies per time pattern across the globe. Accordingly, the model output data 314 may be used to estimate how environmental noise may influence the performance of a sensor (on average) across a time period.

[0138] The model output data 314 may include suitably generic models of platforms and sensors such that a platform may be defined so that its performance may be considered with different sensors attached thereto (e.g., a generic model of a ship asa platform to which can be attached models of sonar sensors, camera sensors, and / or radar sensors).

[0139] The functionality of the modelling module 310 may be performed in advance of the functionality of the other modules of the processor 302.

[0140] The processor 302 further includes a definition module for defining a problem in terms of the AOI, the POI, the sensors, the platforms, the targets, and optimization criteria and constraints.

[0141] The definition module 316 defines the problem in terms of the AOI, the POI, the sensors, the platforms, the targets, and optimization criteria and constraints.

[0142] The definition module 316 may define the AOI according to the input data 312, which may further include commands for defining the AOI.

[0143] In an embodiment, a platform and corresponding sensors are provided as a single package, and such platforms are not explicitly defined separately to such sensors. For example, in the case of an unmanned aerial vehicle (UAV), the entire UAV (including the drone itself as a platform and the sensors mounted thereon) is provided as a discrete unit to be selected.

[0144] The definition module 316 may similarly define the target according to the input data 312, which may further include commands for defining the target.

[0145] The definition module 316 may similarly define the sensors according to the input data 312, which may further include commands for defining the sensors. The sensors may be unique.

[0146] A list of potential sensors appears in the display 308. Such sensors have been defined by the modelling module 310. The user selects which sensors they would like to consider for the particular problem they are optimizing. For example, selecting "Unmanned Vessel System", "Coastal Radar", and "Unmanned Aircraft" as sensor systems to consider (assuming these have all been defined).

[0147] The user uses the modelling module 310 to define the sensor characteristics. The user uses the definition module 316 to select which sensors to include in the optimization problem (e.g., considering a first camera but not a secondcamera). The definition module 316 does not adjust the characteristics of the sensor. Upon applying the discretization module 320 thereto (as will be further explained hereinbelow), the sensor behaves uniquely in that the usefulness thereof may depend on the location thereof (e.g., a camera obstructed by a mountain or in a very cloudy location may not be as effective as a camera in a low-noise location).

[0148] Via the definition module 316, the user selects a sensing system that is a package of platform and sensor(s) (e.g., unmanned boat platform + camera sensor + sonar sensor). Such combinations are pre-configured by the modelling module 310 (e.g., a user may not select whether to include the camera sensor and / or sonar sensor on such unmanned boat platform, as such unmanned boat platform has already been modelled as including same; were such flexibility desirable in an aspect of the computer system 300, the user may define, via the modelling module 310, multiple unmanned boats with different combinations of sensors, thereby allowing the definition module 316 to select between them). The definition module 316 may receive or include commands for defining the target.

[0149] The definition module 316 may similarly define optimization criteria and constraints.

[0150] The definition module 316 may be considered to broadly define a problem to be solved.

[0151] The foregoing definition of the problem may be sufficiently generic so as to enable definition of the sensors, targets, and optimization criteria and constraints independently (e.g., including a sensor without optimizing for all feasible targets of such sensor).

[0152] The defined AOI may use location-specific environmental statistics defined from historical data (e.g., statistical cloud cover, rainfall, temperatures defined for the entire globe).

[0153] Optimization criteria and constraints include allowable cost range, limitations on a number of possible sensors, and regions in the AOI where sensors cannot be placed.

[0154] The processor 302 further includes a discretization module 320 for transforming the problem into a discrete problem within the defined AOI and POI by generating discretization data. The discretization data may include a defined set of feasible sensor locations (e.g., a location of a stationary sensor 14, a desired orbit of a satellite 12 carrying the sensor 14). The discretization module 320 may grid or define a grid in the air, on land, and / or on water forming part of the AOI. Such gridding may include consideration of the coast or shoreline adjacent or included in the AOI 22. Outputs generated by the discretization module 320 are stored as discretization data 322 in the memory 304. The user may select between medium, fine, and coarse discretization levels.

[0155] The discretization module 320 may be configured to a degree of medium fidelity, i.e. , to ensure sufficient monitoring of the AOI by the sensors without excessively expensive use of the sensors or excessively expensive optimization calculations.

[0156] The discretization module 320 may be configured to a degree of high fidelity, i.e., to ensure optimal or near-optimal monitoring of the AOI by the sensors, e.g., considering the most possible sensor placements but taking longer to compute.

[0157] The discretization module 320 may be performed to a degree of low fidelity, i.e., to ensure non-zero monitoring of the AOI by the sensors without incurring significant use of the sensors or significantly expensive optimization calculations.

[0158] The discretization data 322 includes a discrete set of sensor locations defined according to platform type (e.g., a land-based platform has locations on land, an airborne platform has a discrete set of search patterns that it may fly). The effectiveness of a sensor varies according to its location (e.g., a sensor that depends on line-of-sight may have its view more or less impeded at different locations). As such effectiveness may vary according to the foregoing and further according to environmental statistics, which may vary, the effectiveness of each sensor is uniquely defined at each possible discrete location, according to the targets defined and the optimization criteria and constraints defined.

[0159] The processor 302 further includes an optimization module 324 for optimizing placement of sensors to monitor the AOI and / or generate the input data 312.The optimization module 324 computes sensor placement strategies 326 to be stored in the memory 304. The sensor placement strategies 326 may include one or more of a sensor placement strategy. The sensor placement strategies 326 may be a list of each Pareto-optimal sensor placement strategy found by the optimization module 324.

[0160] The optimization module 324 considers the interplay of time patterns (e.g., sunlight), quality of sensors (e.g., of the sensors 14), and environmental factors (as shown in Figure 4) to determine the optimized sensor placement strategies 326.

[0161] The sensor placement strategies 326 may be or may include a Pareto front (i.e. , a set of solutions to a query received as input data 312 that each demonstrate Pareto optimality). For example, based on the input data 312, the Pareto front may represent an optimization for sensor coverage of the AOI with respect to a target that optimizes dollar cost vs. one or more of the probability of detecting the target, spatial coverage of the AOI, redundancy coverage (in case one or more sensors is damaged or non-functional), space-time coverage, and probability of sensing the target (according to the probability of detecting the target multiplied by the probability of the target being present; therefore target density maps may advantageously be used to focus an optimization effort). Such optimization may be according to selected optimization criteria and constraints.

[0162] The sensor placement strategies 326 outputs maps of how a particular strategy performs against the above metrics. The Pareto front may be depicted on such maps or otherwise provided therewith. The maps may be viewed via the display 308. The maps may be summarized as a single number with respect to one or more of the above metrics. The maps may be consulted for expected performance across the entire POI, or viewed just by their performance in a specific time pattern (e.g., winter, summer, night). The output maps advantageously allow a user to optimize for a POI of interest yet inspect a strategy’s performance in finer detail (e.g., optimize for year-round performance but inspect how such optimization performs in winter specifically).

[0163] The optimization performed by the optimization module 324 to generate the sensor placement strategies 326 may be mixed integer optimization or mixed programming optimization. Accordingly, the optimization module 324 may be or may include a mixed integer optimizer or mixed programming optimizer.

[0164] The processor 302 further includes a visualization module 328 for producing a visualization output 330, stored at the memory 302. The visualization output 330 is provided to a user via the display 308. The visualization output 330 includes a GUI showing the implementation of the sensor placement strategies 326 (e.g., one such sensor placement strategy) implemented in the AOI (e.g., shown on a three-dimensional globe).

[0165] The visualization module 328 may enable problems to be defined, optimized, visualized, and explored through the user interface. The visualization module 328 may perform the foregoing functionality on a (visually represented) 3D globe such that problems over the pole or across the dateline may be defined and visualized. The foregoing visual representation may advantageously allow the user to visualize different slices of the problem, e.g., selecting a sensor solution (i.e., a sensor placement strategy) on the Pareto front (see Figure 5), examining how the solution performs according to different targets, optimization criteria and constraints, and time patterns, comparing one solution to another. The visual representation provided by the visualization module 328 may advantageously enable the user to edit a solution and experiment with variations thereon.

[0166] The processor 302 pre-calculates environmental statistics for each defined time-pattern (built from historical weather data) before performing any of the foregoing functionality.

[0167] Referring now to Figures 4A-4C, shown therein are graphs 402, 404, and 406, respectively.

[0168] In Figure 4A, the graph 402 shows how sensitivity of sensors (e.g., the sensor 14 of Figure 1 ) varies according to a low cloud fraction.

[0169] In Figure 4B, the graph 404 shows how sensitivity of sensors varies according to the sun elevation angle.

[0170] In Figure 4C, the graph 406 shows how sensitivity of sensors varies according to a precipitation rate.

[0171] The graphs 402, 404, 406 show how the presence of an environmental factor (clouds, sun angle, and precipitation, respectively) influences the sensors.

[0172] Graphs such as those shown in Figures 4A-4C may be provided as input (e.g., input data 312) to the modelling module 310 in order to generate the model output data 314. Graphs such as those shown in Figures 4A-4C may be provided as input to the model output data 314 (which may be, for example, a model). A plurality of each such graph 402, 404, 406 over time may preferably be provided as input.

[0173] A broader function representing how the presence of an environmental factor influences the sensors may be provided as input to the modelling module 310 in order to generate the model output data 314. Such broader function may be provided as input to the model output data 314 (which may be, for example, a model). Each of the graphs 402, 404, 406 may be derivable from such respective broader function.

[0174] An analyst or expert defines settings in the computer system 300, for example the graphs 402, 404, 406, and further defines how various environmental factors influence the sensors to define models in terms of trends considered relevant, e.g., describing how precipitation affects a radar sensor, such description influencing a model (e.g., the model output data 314). The graphs 402, 404, 406 may be adjusted on a per- sensor basis.

[0175] The analyst or expert may define sensors in terms of requirements according to the models such that a specific sensor need not be identified by the analyst or expert, respectively. The analyst or user who defines the sensors (via the modelling module 310) and the analyst or user who runs the optimizations (via the optimization module 324) may be the same or different analysts or users. For example, where the respective analysts or users are different, the former analyst may understand specific sensors, while the latter analyst may consider overall functionality and budget of a surveillance project.

[0176] Advantageously, an end user of the computer system 300 may model or define specific aspects of a proprietary surveillance system and / or of proprietary sensors without providing such aspects or other sensitive information to a provider of the computer system 300.

[0177] Referring now to Figure 5, shown therein is an example user interface screen including the visualization output 330 of Figure 3, according to an embodiment.

[0178] The visualization output 330 includes a graphical depiction of the AOI 22 of Figure 1 within the broader area 24 of Figure 1 . The visualization output 330 further includes sensor placement strategies 326 for placing the sensors 14 shown therein.

[0179] In Figure 5, the visualization output 330 is in the form of a 2D Pareto front as previously discussed. In Figure 5, the visualization output 330 optimizes dollar cost relative to spatial coverage.

[0180] The visualization output 330 further includes a drop-down menu 331 to select the target 28, in this case a small surface vessel. The 2D Pareto front, and thus the visualization output 330, is calculated to provide a set of solutions for optimizing one value (dollar cost) against another (spatial coverage) for the detection and / or identification of the target 28 (small surface vessel).

[0181] Referring now to Figure 6, shown therein is a flow diagram of a method 600 for mixed sensor performance and evaluation, according to an embodiment.

[0182] At 600, the method 600 includes disposing one or more sensors, for monitoring an area of interest (AOI) over a period of interest (POI), on one or more sensor platforms located within a broader area, the broader area including the AOI.

[0183] At 604, the method 600 includes pre-calculating environmental statistics for one or more defined time patterns.

[0184] At 606, the method 600 includes generating models of sensors, of platforms, of targets, and of other features in an area of interest (AOI), i.e. , an area to be monitored by sensors (such as the sensors 14 of Figure 1 ), over a period of interest (POI).

[0185] At 608, the method 600 further includes defining a problem in terms of the AOI, the POI, sensors, platforms, targets, and criteria and constraints. The AOI may be defined by a polygon. The user may draw the polygon on the map. The user may provide a set of coordinates representing the vertices of the polygon.

[0186] At 610, the method 600 further includes transforming the problem into a discrete problem within the defined AOI and POI by generating discretization data.

[0187] The discretization may include defining a set of feasible sensor locations (e.g., a location of a stationary sensor 14, a desired orbit of a satellite 12 carrying the sensor 14) and / or gridding or defining a grid in the air, on land, and / or on water forming part of the AOI. Such gridding may include consideration of the coast or shoreline adjacent or included in the AOI.

[0188] At 612, the method 600 further includes optimizing placement of the one or more sensors, according to the discretization data, the models, the defined time patterns, quality of the one or more sensors, and the pre-calculated environmental statistics. Optimizing the placement of the sensors may include computing a sensor placement strategy.

[0189] At 614, the method 600 further includes producing a visualization output to be provided to a user. The visualization output may include a graphical user interface (GUI) showing the implementation of the sensor placement strategy (e.g., one such sensor placement strategy) implemented in the AOI (e.g., shown on a three-dimensional globe).

[0190] While the above description provides examples of one or more apparatus, methods, or systems, it will be appreciated that other apparatus, methods, or systems may be within the scope of the claims as interpreted by one of skill in the art.

Claims

Claims:1 . A system for mixed sensor performance and evaluation, the system comprising: one or more sensors for monitoring an area of interest (AOI) over a period of interest (POI), the one or more sensors disposed on one or more sensor platforms located within a broader area, the broader area including the AOI; and a server for optimizing placement of the one or more sensors, the server precalculating environmental statistics for one or more defined time patterns, the server comprising: a modelling module for generating models of the one or more sensors, of the one or more platforms, and of targets in the AOI over the POI; a definition module for defining a problem in terms of the AOI, the POI, the sensors, the platforms, the targets, and optimization criteria and constraints; a discretization module for transforming the problem into a discrete problem within the defined AOI and POI by generating discretization data; an optimization module for optimizing placement of the one or more sensors, according to the discretization data, the models, the defined time patterns, quality of the one or more sensors, and the pre-calculated environmental statistics; and a visualization module for producing a visualization output provided to a user.

2. The system of claim 1 , wherein the one or more defined time patterns comprise any one or more of day-night cycles and seasonality.

3. The system of any one of claims 1 -2, wherein the definition module defines the problem by: defining the AOI as a polygon having longitude and latitude coordinates as the vertices; defining the POI; defining the target by selecting a target type from a target type list; defining the sensors by selecting one or more sensor types from a sensor type list; and defining the optimization criteria and constraints by selecting the optimization criteria and constraints from an optimization criteria and constraints list.

4. The system of claim 3, wherein the discretization module performs the discretization by defining a set of locations for the one or more platforms and / or the one or more sensors disposed on the one or more platforms.

5. The system of claim 3, wherein the discretization module performs the discretization by defining a grid in the air, on land, and / or in the water forming part of the AOI.

6. The system of any one of claims 3-4, wherein the discretization module is configured to a degree of medium fidelity to provide sufficient monitoring of the AOI by the one or more sensors without excessively expensive use of the sensors or excessively expensive optimization.

7. The system of any one of claims 3-4, wherein the discretization module is configured to a degree of high fidelity to provide optimal or near-optimal monitoring of the AOI by the one or more sensors.

8. The system of any one of claims 3-4, wherein the discretization module is configured to a degree of low fidelity to provide non-zero monitoring of the AOI without incurring significant use of the sensors or significantly expensive optimization.

9. The system of any one of claims 3-8, wherein the discretization module provides a discrete list of possible locations of the one or more sensors defined according to one or more respective types of the one or more platforms.

10. The system of any one of claims 3-9, wherein the optimization module computes a Pareto front comprising a plurality of sensor placement strategies, each sensor placement strategy being a point on the Pareto front.11 . The system of claim 10, wherein the selected optimization criteria and constraints include sensor coverage with respect to the target by optimizing dollar cost vs. any one or more of a probability of detecting the target, spatial coverage of the AOI, redundancy coverage, space-time coverage, and a probability of sensing the target.

12. The system of claim 11 , wherein the selected optimization criteria and constraints include sensor coverage by optimizing dollar cost vs. space-time coverage and the probability of detecting the target.

13. The system of any one of claims 10-12, wherein the optimization module includes a mixed integer optimizer or a mixed programming optimizer.

14. The system of any one of claims 10-13, wherein the visualization output includes a GUI showing an implementation of the sensor placement strategy implemented in the AOI.

15. The system of claim 14, wherein the GUI shows the implementation on a three- dimensional globe.

16. The system of any one of claims 1 -15, wherein the environmental statistics include statistics on any one or more of differing temperatures, precipitation, cloud cover, and day-night cycle lengths over the course of the day-night cycle and / or annually.

17. A method for mixed sensor performance and evaluation, the method comprising: disposing one or more sensors, for monitoring an area of interest (AOI) over a period of interest (POI), on one or more sensor platforms located within a broader area, the broader area including the AOI; pre-calculating environmental statistics for one or more defined time patterns; generating models of the one or more sensors, of the one or more platforms, and of targets in the AOI over the POI; defining a problem in terms of the AOI, the POI, the sensors, the sensor platforms, the targets, and optimization criteria and constraints; transforming the problem into a discrete problem within the defined AOI and POI by generating discretization data; optimizing placement of the one or more sensors, according to the discretization data, the models, the defined time patterns, quality of the one or more sensors, and the pre-calculated environmental statistics; and producing a visualization output to be provided to a user.

18. The method of claim 18, wherein the one or more defined time patterns comprise any one or more of day-night cycles and seasonality.

19. The method of any one of claims 17-18, wherein defining the problem comprises:defining the AOI as a polygon having longitude and latitude coordinates as the vertices; defining the POI; defining the target by selecting a target type from a target type list; defining the sensors by selecting one or more sensor types from a sensor type list; and defining the optimization criteria and constraints by selecting the optimization criteria and constraints from an optimization criteria and constraints list.

20. The method of any one of claims 18-19, wherein performing the discretization comprises defining a set of locations for the one or more platforms and / or the one or more sensors disposed on the one or more platforms.

21. The method of any one of claims 18-19, wherein performing the discretization comprises defining a grid in the air, on land, and / or in the water forming part of the AOI.

22. The method of any one of claims 18-21 , wherein the discretization is performed to a degree of medium fidelity to provide sufficient monitoring of the AOI by the one or more sensors without excessively expensive use of the sensors or excessively expensive optimization.

23. The method of any one of claims 18-21 , wherein the discretization is performed to a degree of high fidelity to provide optimal or near-optimal monitoring of the AOI by the one or more sensors.

24. The method of any one of claims 18-21 , wherein the discretization is performed to a degree of low fidelity to provide non-zero monitoring of the AOI without incurring significant use of the sensors or significantly expensive optimization.

25. The method of any one of claims 18-24, wherein performing the discretization comprises providing a discrete list of possible locations of the one or more sensors defined according to one or more respective types of the one or more platforms.

26. The method of any one of claims 18-25, wherein the optimization comprises a Pareto front comprising a plurality of sensor placement strategies, each sensor placement strategy being a point on the Pareto front.

27. The method of claim 26, wherein the selected optimization criteria and constraints include optimizing dollar cost vs. a probability of detecting the target, spatial coverage of the AOI, redundancy coverage, space-time coverage, and a probability of sensing the target.

28. The method of claim 27, wherein the selected optimization criteria and constraints include sensor coverage by optimizing dollar cost vs. space-time coverage and the probability of detecting the target.

29. The method of any one of claims 26-28, wherein the optimization is performed, at least in part, with a mixed integer optimizer or a mixed programming optimizer.

30. The method of any one of claims 26-29, wherein the visualization output includes a GUI showing an implementation of the sensor placement strategy implemented in the AOI.

31. The method of claim 30, wherein the GUI shows the implementation on a three- dimensional globe.

32. The method of any one of claims 18-31 , wherein the environmental statistics include statistics on any one or more of differing temperatures, precipitation, cloud cover, and day-night cycle lengths over the course of the day-night cycle and / or annually.

33. A server for mixed sensor performance and evaluation by optimizing placement of the one or more sensors, the server pre-calculating environmental statistics for one or more defined time patterns, the server comprising: a modelling module for generating models of one or more sensors, for monitoring an area of interest (AOI) within a broader area over a period of interest (POI), the sensors disposed on one or more sensor platforms located within the broader area, of the one or more platforms, and of targets in the AOI; a definition module for defining a problem in terms of the AOI, the POI, the sensors, the platforms, the targets, and optimization criteria and constraints; a discretization module for transforming the problem into a discrete problem within the defined AOI and POI by generating discretization data; an optimization module for optimizing placement of the one or more sensors, according to the discretization data, the models, the defined time patterns, quality of the one or more sensors, and the pre-calculated environmental statistics; and a visualization module for producing a visualization output provided to a user.

34. The server of claim 33, wherein the one or more defined time patterns comprise any one or more of day-night cycles and seasonality.

35. The server of any one of claims 34-35, wherein the definition module defines the problem by:defining the AOI as a polygon having longitude and latitude coordinates as the vertices; defining the POI; defining the target by selecting a target type from a target type list; defining the sensors by selecting a sensor type from a sensor type list; and defining the optimization criteria and constraints by selecting the optimization criteria and constraints from an optimization criteria and constraints list.

36. The server of claim 35, wherein the discretization module performs the discretization by defining a set of locations for the one or more platforms and / or the one or more sensors disposed on the one or more platforms.

37. The server of any one of claims 35-36, wherein the discretization module performs the discretization by defining a grid in the air, on land, and / or in the water forming part of the AOI.

38. The server of any one of claims 35-37, wherein the discretization module is configured to a degree of medium fidelity to provide sufficient monitoring of the AOI by the one or more sensors without excessively expensive use of the sensors or excessively expensive optimization.

39. The server of any one of claims 35-37, wherein the discretization module is configured to a degree of high fidelity to provide optimal or near-optimal monitoring of the AOI by the one or more sensors.

40. The server of any one of claims 35-37, wherein the discretization module is configured to a degree of low fidelity to provide non-zero monitoring of the AOIwithout incurring significant use of the sensors or significantly expensive optimization.41 . The server of any one of claims 35-40, wherein the discretization module provides a discrete list of possible locations of the one or more sensors defined according to one or more respective types of the one or more platforms.

42. The server of any one of claims 35-41 , wherein the optimization module computes a Pareto front comprising a plurality of sensor placement strategies, each sensor placement strategy being a point on the Pareto front.

43. The server of claim 42, wherein the selected optimization criteria and constraints include sensor coverage with respect to the target by optimizing dollar cost vs. a probability of detecting the target, spatial coverage of the AOI, redundancy coverage, space-time coverage, and a probability of sensing the target.

44. The server of claim 43, wherein selected optimization criteria and constraints include optimizing dollar cost vs. space-time coverage and the probability of detecting the target.

45. The server of any one of claims 42-44, wherein the optimization module includes a mixed integer optimizer or a mixed programming optimizer.

46. The server of any one of claims 42-45, wherein the visualization output includes a GUI showing an implementation of the sensor placement strategy implemented in the AOI.

47. The server of claim 46, wherein the GUI shows the implementation on a three- dimensional globe.

48. The server of any one of claims 35-47, wherein the environmental statistics include statistics on any one or more of differing temperatures, precipitation, cloud cover, and day-night cycle lengths over the course of the day-night cycle and / or annually.

Images