Fast ordering of targets derived from 2d imagery using space filling curves

Abstract

A method for fast ordering of targets derived from 2D imagery using space filling curves is provided. The method provides the spatial distance between successive targets is minimized allowing the review process to maintain spatial context when compared to traditional row-major ordering. The ordering is also quick to compute, when compared to MLROSE and Greedy algorithms, supporting near real-time reviewing. The method comprises detecting a plurality of targets in a two-dimensional image; scaling a two-dimensional image to cover all of, or a portion of, a space filling curve matrix, such that each pixel in the image is covered by a cell in the matrix; calculating a space filling curve coordinate for each target in the image, based on its image coordinate; and ordering the plurality of targets according to their respective curve coordinate to generate the ordered list.

Description

FAST ORDERING OF TARGETS DERIVED FROM 2D IMAGERY USING SPACE FILLING CURVESTechnical Field

[0001] The following relates generally to automated analysis of satellite imagery, and, in particular, to systems and methods for ordering of targets in two-dimensional imagery.Introduction

[0002] Maritime surveillance software for ship detection is employed for area awareness in maritime transport and navigation. Software such as OceanSuite® provides automated analysis of two-dimensional (2D) satellite imagery to detect the position of ships.

[0003] When reviewing ship detections in OceanSuite, a list of ship detection “targets” is provided to the operator for quality assurance. The operator verifies the automatically derived target information. The list of targets is not explicitly sorted; the order of targets is a function of the way the image is processed by the target detection algorithm (e.g., a CFAR algorithm). An image is broken up into tiles, and these tiles are processed in row-major order. Within each tile, targets are also processed in row-major order. As each tile completes processing, the targets in that tile are added to the list of targets. As a result, the targets are generally added in row-major order. However, as tiles are processed asynchronously, it is possible that the order of completion of the tiles is not the same as the order that they were initiated.

[0004] A problem with cycling through targets in their processing order is that this order is agnostic to the spatial proximity of targets in the image. For example, if a cluster of targets are spread over two tiles in the azimuth direction, they will be separated in the processing order by any targets in tiles processed in-between these two tiles. This can result in the following scenario: 1 ) the software sequences through the targets from the cluster detected in the first tile; 2) the software jumps to targets in tiles further out in range from the first tile, possibly going to targets on the other side of the image; and 3) the software returns to the remaining targets from the cluster detected in the first tile.

[0005] The resulting review order often results in jumping to a target on the far side of the image when there were much closer targets. This causes the operator to lose spatial context. It is desirable to provide an ordering of targets that minimizes jumping around the image to visit nearby targets first before visiting targets further away, and is fast enough to have little or no impact on the near real-time constraints of ship detection.

[0006] The problem of ordering a set of points so that the path travelling through all of them is minimized is known as the “Travelling Salesman Problem”; however, known solutions such as Machine Learning Randomized Optimization and SEarch (MLROSE) algorithms and Greedy (nearest neighbour) algorithms are slow and impractical when there are a large number (e.g., thousands) of targets. Also, quality assurance review of detected targets, does not require a round trip through all detected targets.

[0007] Accordingly, there is a need for new and improved methods for target ordering in two-dimensional satellite imagery that overcomes at least some disadvantages of existing systems and methods.Summary

[0008] Systems and methods for ordering targets derived from 2D imagery such that targets that are close together in the ordering are also close together in the 2D imagery are provided. When the targets are reviewed using this order, the spatial distance between successive targets is minimized allowing the review process to maintain spatial context. The ordering is also quick to compute, supporting near real-time reviewing.

[0009] A method for fast ordering of targets in two-dimensional imagery is provided. The method includes: receiving satellite imagery captured by a satellite, the satellite imagery comprising a plurality of two-dimensional images; detecting a plurality of targets in the two-dimensional images; ordering the plurality of targets by a space filling curve algorithm; and presenting an ordered list of the plurality of targets to an operator.

[0010] In an embodiment, the method further includes providing the two- dimensional image.

[0011] In an embodiment, the method further includes: manual creation of a new target; calculating a space filling curve coordinate for the new target; and inserting thenew target into the ordered list according to the space filling curve coordinate for the new target.

[0012] In an embodiment, the method further includes: splitting a target into a first target and a second target; calculating a space filling curve coordinate for each of the first target and the second target; and inserting the first target and the second target into the ordered list according to the respective space filling curve coordinate for the first target and the second target.

[0013] In an embodiment,, ordering the plurality of targets by a space filling curve algorithm includes: scaling a two-dimensional image to cover all of a space filling curve matrix, such that each pixel in the image is covered by a cell in the matrix; calculating a space filling curve coordinate for each target in the image, based on its image coordinate; and ordering the plurality of targets according to their respective curve coordinate to generate the ordered list.

[0014] In an embodiment, scaling the two-dimensional image to cover all of the space filling curve matrix includes: selecting a Hilbert curve order parameter, N; scaling both line and pixel image coordinate values for each of the plurality of targets to fall within the Hilbert curve matrix interval [0, 2W-1 ]; and calculating the Hilbert curve coordinate for the scaled image coordinate for each of the plurality of targets.

[0015] In an embodiment, scaling the two-dimensional image to cover all of the space filling curve matrix includes: selecting a Peano curve order parameter, N; scaling both line and pixel image coordinate values for each of the plurality of targets to fall within the Peano curve matrix interval [0, 3W-1 ]; and calculating the Peano curve coordinate for the scaled image coordinate for each of the plurality of targets.

[0016] In an embodiment, ordering the plurality of targets by a space filling curve algorithm includes: scaling a two-dimensional image to cover a portion of a space filling curve matrix, such that each pixel in the image is covered by a cell in the matrix; calculating a space filling curve coordinate for each target in the image, based on its image coordinate; and ordering the plurality of targets according to their respective curve coordinate to generate the ordered list.

[0017] In an embodiment, if the two-dimensional image includes substantially more pixels than lines, scaling the two-dimensional image to cover the portion of a space filling curve matrix includes: selecting a Hilbert curve order parameter, N; scaling pixel image coordinate values for each of the plurality of targets to fall within the full Hilbert curve matrix interval [0,2N-1 ] scaling line image coordinate values for each of the plurality of targets to fall within the first half of the Hilbert curve matrix interval [0, 2w-7-1]; and calculating the Hilbert curve coordinate for the scaled image coordinate for each of the plurality of targets.

[0018] In an embodiment, if the two-dimensional image includes substantially more lines than pixels, scaling the two-dimensional image to cover the portion of a space filling curve matrix includes: selecting a Hilbert curve order parameter, N; scaling pixel image coordinate values for each of the plurality of targets to fall within the second half of the Hilbert curve matrix interval [2w-7-1 , 2w-7-1 ]; scaling line image coordinate values for each of the plurality of targets to fall within the full Hilbert curve matrix interval [0,2N-1 ]; calculating the Hilbert curve coordinate for the scaled image coordinate for each of the plurality of targets.

[0019] In an embodiment, if the two-dimensional image includes substantially more pixels than lines, scaling the two-dimensional image to cover the portion of a space filling curve matrix includes: selecting a Peano curve order parameter, N; scaling pixel image coordinate values for each of the plurality of targets to fall within the full Peano curve matrix interval [0,3w- 1 ]; scaling line image coordinate values for each of the plurality of targets to fall within the first third of the Peano curve matrix interval [0, 3w-7-1]; and calculating the Peano curve coordinate for the scaled image coordinate for each of the plurality of targets.

[0020] In an embodiment, if the two-dimensional image includes substantially more lines than pixels, scaling the two-dimensional image to cover the portion of a space filling curve matrix includes: selecting a Peano curve order parameter, N; scaling pixel image coordinate values for each of the plurality of targets to fall within the last third of the Peano curve matrix interval [2(3W’7), 3W-1]; scaling line image coordinate values for each of the plurality of targets to fall within the full Peano curve matrix interval [0,3w- 1 ]; calculatingthe Peano curve coordinate for the scaled image coordinate for each of the plurality of targets.

[0021] Other aspects and features will become apparent, to those ordinarily skilled in the art, upon review of the following description of some exemplary embodiments.Brief Description of the Drawings

[0022] The drawings included herewith are for illustrating various examples of articles, methods, and apparatuses of the present specification. In the drawings:

[0023] FIG. 1 is a schematic diagram of a maritime surveillance system, according to an embodiment;

[0024] FIG. 2 is a block diagram of a maritime surveillance application, according to an embodiment;

[0025] FIG. 3 is diagram of fast ordering of targets detected in 2D imagery, according to an embodiment;

[0026] FIG. 4 is a diagram of the first four orders of a Hilbert curve;

[0027] FIG. 5 is a diagram of the first four orders of a Peano curve;

[0028] FIG. 6A is representative 2D imagery of size 11553 pixels by 12349 lines, showing target detection, according to an embodiment;

[0029] FIG. 6B is a representative target order plot of row-major ordering of the targets in 6A, according to an embodiment;

[0030] FIG. 60 is a representative target order plot for Hilbert curve ordering of the targets in 6A, according to an embodiment; and

[0031] FIG. 6D is a representative target order plot for Peano curve ordering of the targets in 6A, according to an embodiment.Detailed Description

[0032] Various apparatuses or processes will be described below to provide an example of each claimed embodiment. No embodiment described below limits any claimed embodiment and any claimed embodiment may cover processes or apparatusesthat differ from those described below. The claimed embodiments are not limited to apparatuses or processes having all of the features of any one apparatus or process described below or to features common to multiple or all of the apparatuses described below.

[0033] One or more systems described herein may be implemented in computer programs executing on programmable computers, each comprising at least one processor, a data storage system (including volatile and non-volatile memory and / or storage elements), at least one input device, and at least one output device. For example, and without limitation, the programmable computer may be a programmable logic unit, a mainframe computer, server, and personal computer, cloud-based program or system, laptop, personal data assistance, cellular telephone, smartphone, or tablet device.

[0034] Each program is preferably implemented in a high-level procedural or object- oriented programming and / or scripting language to communicate with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. Each such computer program is preferably stored on a storage media or a device readable by a general or special purpose programmable computer for configuring and operating the computer when the storage media or device is read by the computer to perform the procedures described herein.

[0035] A description of an embodiment with several components in communication with each other does not imply that all such components are required. On the contrary a variety of optional components are described to illustrate the wide variety of possible embodiments of the present invention.

[0036] Further, although process steps, method steps, algorithms or the like may be described (in the disclosure and I or in the claims) in a sequential order, such processes, methods and algorithms may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps be performed in that order. The steps of processes described herein may be performed in any order that is practical. Further, some steps may be performed simultaneously.

[0037] When a single device or article is described herein, it will be readily apparent that more than one device I article (whether or not they cooperate) may be used in place of a single device I article. Similarly, where more than one device or article is described herein (whether or not they cooperate), it will be readily apparent that a single device I article may be used in place of the more than one device or article.

[0038] The following relates generally to automated analysis of satellite imagery, and, in particular, to systems and methods for ordering of targets in two-dimensional imagery.

[0039] FIG. 1 is a diagram of a maritime surveillance system 100, according to an embodiment. A satellite 102 orbiting the Earth acquires image data (e.g., synthetic aperture radar imagery) of ships 104 on the open ocean and close to land. The image data is 2D image data. A ground station 106 receives the imagery data from the satellite 102.

[0040] While the present disclosure describes target ordering in the context of marine surveillance imagery and ship detection in 2D satellite imagery, the systems and methods of the present disclosure may be used in any application where a list of targets located in an image needs to be quickly accessed sequentially with minimal jumping around in the image. A specific example, described herein, is for a quality assurance step of a ship detection process where a human operator verifies automatically derived target information. More generally, the systems and methods herein can be applied to any “remote sensing” imagery that measures / maps the physical characteristics of an area of the Earth to allow targets of interest to be detected, not limited to synthetic aperture radar, optical, multispectral, hyperspectral and thermal imagery.

[0041] A server system 108 in communication with the ground station 106 over a communications network 110 runs a maritime surveillance application 120 for ship detection. A user device 112 communicates with the server system 108 over the network 110 and provides a user interface of the maritime surveillance application 120 for the operator to review ship targets automatically identified by the application 120. The user device 112 may be located on a ship 104. According to various embodiments, the maritime surveillance application 120 is hosted by the server system 108, or is installedlocally on the user device 112, or runs on both the server system 108 and the user device 112.

[0042] As shown in FIG. 2, the maritime surveillance application 120 includes a ship detection module 122 and a target sorting / ordering module 124. The modules 122, 124 comprise a set of instructions executable by one or more processors of the server 108 and / or the user device 112.

[0043] Referring to FIGS. 2 and 3, the ship detection module 122 is configured to process 2D satellite image data 200 by one or more target detection algorithms and generate an output of a set targets of interest 202 (e.g., ship detections A, B, C, D, E) within the 2D image.

[0044] The target detection output 202 is provided as input to the target ordering / sorting module 124. The target ordering / sorting module 124 is configured to implement one or more target ordering space algorithms to order the detected targets for review. The target detection algorithms implement a space-filling curve. The space-filling curve may be, for example, a Hilbert curve of order 2 which generates a 4x4 matrix 204 containing the curve to map 2D image coordinates to 1 D coordinates. The image is stretched to overlay the space filling curve matrix 204, which results in each cell of the matrix 204 covering a subregion of the image. The space filling curve cell coordinate 206 that each target falls within is determined. The targets are sorted based on their space filling cell coordinate number to generate a sorted list 208. The sorted list 208 may be presented to an operator 210 (e.g., via user interface on user device 112). Targets that are close to each other in the sorted list 208 are also close to each other in the 2D image 200, which makes it easier for the operator 210 to review.

[0045] It should be noted that the target ordering need not be optimized to absolutely minimize the distance of jumps between successive targets but should be sufficient to provide for fast ordering of targets for near real-time review while improving: a) average edge length; b) standard deviation of edge length; and c) maximum edge length compared to traditional row-major ordering, where an “edge” is defined as the line connecting adjacent targets in the ordering. Accordingly, space filing curves wereproposed for ordering detected targets as opposed to more computationally intensive approaches such as MLROSE and Greedy (nearest neighbour) algorithms.

[0046] The number of operations required to order N detections using a spacefilling curve algorithm is the sum of: (1 ) calculation of curve coordinate for each target, O(N):and (2) cost to sort N targets. On average, this is O( / Vlog / V).

[0047] Additionally, the ordering should support easy insertion of new targets, without requiring the entire set of targets to be re-sorted. There are two scenarios where a new target can be added to the list: (1 ) manual creation of new target, and (2) splitting of a target. The new target(s) must be inserted into the order such that: (a) targets that have already been visited by the operator are not re-visited; (b) targets that have not been visited by the operator will all be visited.

[0048] There are a number of space-filling curves, including the Hilbert curve, Peano curve, Z curve, Gray-coded curve, Onion curve, etc. In an embodiment, the Hilbert curve is used as it is generally considered to have the best spatial locality and has been used in applications such as spatial indexing and video encoding. In another embodiment, the Peano curve is used because the order generated by this curve might be more intuitive to an operator 210 than the Hilbert curve order.

[0049] The Hilbert curve is recursively defined. FIG. 4 shows the first four orders of the curve. The orientation of the curve in this example was set such that the open side is to the left. This results in a curve that starts in the upper left of the image and works its way through the four quadrants in a clockwise order. This may be the most intuitive order for an operator, compared to the other three possible orientations, although any orientation may be used.

[0050] The N-th order Hilbert curve fills a square matrix of 2Nx 2Npixels.

[0051] Hilbert curve ordering of the targets may be implemented as follows: (1 ) Select a value for N; for each target: (2a) scale both line and pixel image coordinate values of the target so they fall in the interval [0, 2W-1 ]; and (2b) calculate the Hilbert curve coordinate corresponding to the scaled image coordinate; and (3) sort the list of targets based on the Hilbert curve coordinate.

[0052] For the Hilbert Curve ordering statistics shown in Tables 1 and 2, a value of N =10 was used, resulting in a 1024 x 1024 pixel Hilbert curve matrix. Increasing the value of N resulted in an insignificant difference in the ordering statistics. Decreasing the value of N also resulted in an insignificant difference until N<6, where the ordering statistics were worse. Conceptually, this process can be viewed as tiling the image with tiles based on the Hilbert curve coordinates at level N. In general, the tiles will be nonsquare because the images are typically not square.

[0053] In the event that an image is rectangular, opposed to square, the line and pixel image coordinate values of the targets may be scaled so that they fall within a portion of the full 2N*2IVgrid. For example, if the image has substantially more pixels than lines then the line coordinates could be mapped to the top half of the Hilbert space [0, 2N’1-1 ], On the other hand, if the image has substantially more lines than pixels then the pixel coordinates could be mapped to the right half of the Hilbert space [2W-1, 2W-1 ], The full range [0, 2W-1 ] for both lines and pixels was used for all examples herein.

[0054] Inserting a new target in a list ordered by the Hilbert curve coordinate is straightforward: (1 ) Compute the Hilbert curve coordinate for the new target; and (2) Insert the target into the sorted list based on the Hilbert curve coordinate.

[0055] FIG. 5 shows the first four orders of the Peano curve. The N-th order Peano curve fills a square matrix of 3N*3Npixels. The orientation of the curve was set so that it was a reverse “S” shape. This may be the most intuitive order for an operator, compared to the other three possible orientations, although any orientation may be used.

[0056] Peano curve ordering of the targets is implemented as follows: (1 ) Select a value for N; for each target: (2a) scale both line and pixel image coordinate values for the target so they fall in the interval [0, 3W-1 ]; and (2b) calculate the Peano curve coordinate corresponding to the scaled image coordinate; and (3) sort the list of targets based on the Peano curve coordinate. For the Peano Curve ordering statistics shown in Tables 1 and 2, a value of N =7 was used resulting in a 2187 x 2187 pixel Peano curve matrix.

[0057] In the event that an image is more rectangular than square, the line and pixel image coordinate values of the targets may be scaled so that they fall within a portionof the full 3N*3Ngrid. For example, if the image has substantially more pixels than lines then the line coordinates could be mapped to the first third of the Peano space [0, 3N’1-1 ]. On the other hand, if the image has substantially more lines than pixels then the pixel coordinates could be mapped to the last third of the Peano space [2(3W-1), 3W-1 ]. The full range [0, 3W-1 ] for both lines and pixels was used for all examples herein.

[0058] Inserting a new target in a list ordered by the Peano curve coordinate is done in the same manner described above for Hilbert curve ordering.

[0059] FIGS. 6B-6D shows representative target order plots for the detections in FIG. 6A, according to an embodiment. The plots were generated by plotting the targets in the order generated by each type of ordering done on the targets in FIG. 6A.

[0060] Row-major ordering of the targets (FIG. 6B) includes many jumps from the right side of the image back to the left side. As shown in Table 1 , the mean edge distance between adjacent targets in the row-major ordering is in excess of 1000 pixels, and the maximum edge distance is almost equal to the width of the image.

[0061] Hilbert curve ordering (FIG. 6C) shows a substantial improvement over the row-major ordering (FIG. 6B), with few long jumps between successive targets. The minimum edge distance is substantially the same as the row-major ordering, however, the maximum edge distance is roughly half. The mean edge distance between adjacent targets as well as the total path length are roughly one fifth of row-major ordering.

[0062] Peano curve ordering (FIG. 6D) also shows a substantial improvement over the row-major ordering (FIG. 6B), with few long jumps between successive targets. The minimum edge distance is substantially the same as the row-major ordering, however, the maximum edge distance is roughly half. The mean edge distance between adjacent targets as well as the total path length are roughly one fifth of row-major ordering.

[0063] Table 1. Ordering Statistics

[0064] Table 2. Ordering Improvement over Row-Major

[0065] While the above description provides examples of one or more apparatus, methods, or systems, it will be appreciated that other apparatus, methods, or systems may be within the scope of the claims as interpreted by one of skill in the art. The target ordering methods and systems described herein can also be applied to any form of 2D imagery and generally be used in any context where an ordered list of points within a 2D measurement space are generated.

Claims

Claims:1 . A method for fast ordering of targets in two-dimensional imagery comprising: receiving satellite imagery captured by a satellite, the satellite imagery comprising a plurality of two-dimensional images; detecting a plurality of targets in the two-dimensional images; ordering the plurality of targets by a space filling curve algorithm; and presenting an ordered list of the plurality of targets to an operator.

2. The method of claim 1 , further comprising: providing the two-dimensional image.

3. The method of claim 1 , further comprising: manually creating a new target; calculating a space filling curve coordinate for the new target; and inserting the new target into the ordered list according to the space filling curve coordinate for the new target.

4. The method of claim 1 , further comprising: splitting a target into a first target and a second target; calculating a space filling curve coordinate for each of the first target and the second target; and inserting the first target and the second target into the ordered list according to the respective space filling curve coordinate for the first target and the second target.

5. The method of claim 1 , wherein ordering the plurality of targets by a space filling curve algorithm comprises: scaling a two-dimensional image to cover all of a space filling curve matrix, such that each pixel in the image is covered by a cell in the matrix; calculating a space filling curve coordinate for each target in the image, based on its image coordinate; and ordering the plurality of targets according to their respective curve coordinate to generate the ordered list.

6. The method of claim 5, wherein scaling the two-dimensional image to cover all of the space filling curve matrix comprises: selecting a Hilbert curve order parameter, N; scaling both line and pixel image coordinate values for each of the plurality of targets to fall within the Hilbert curve matrix interval [0, 2W-1]; and calculating the Hilbert curve coordinate for the scaled image coordinate for each of the plurality of targets.

7. The method of claim 5, wherein scaling the two-dimensional image to cover all of the space filling curve matrix comprises: selecting a Peano curve order parameter, N; scaling both line and pixel image coordinate values for each of the plurality of targets to fall within the Peano curve matrix interval [0, 3W-1]; and calculating the Peano curve coordinate for the scaled image coordinate for each of the plurality of targets.

8. The method of claim 1 , wherein ordering the plurality of targets by a space filling curve algorithm comprises: scaling a two-dimensional image to cover a portion of a space filling curve matrix, such that each pixel in the image is covered by a cell in the matrix; calculating a space filling curve coordinate for each target in the image, based on its image coordinate; and ordering the plurality of targets according to their respective curve coordinate to generate the ordered list.

9. The method of claim 8, wherein if the two-dimensional image includes substantially more pixels than lines, scaling the two-dimensional image to cover the portion of a space filling curve matrix comprises: selecting a Hilbert curve order parameter, N; scaling pixel image coordinate values for each of the plurality of targets to fall within the full Hilbert curve matrix interval [0,2N-1 ]; scaling line image coordinate values for each of the plurality of targets to fall within the first half of the Hilbert curve matrix interval [0, 2W’7-1 ]; and calculating the Hilbert curve coordinate for the scaled image coordinate for each of the plurality of targets.

10. The method of claim 8, wherein if the two-dimensional image includes substantially more lines than pixels, scaling the two-dimensional image to cover the portion of a space filling curve matrix comprises: selecting a Hilbert curve order parameter, N;scaling pixel image coordinate values for each of the plurality of targets to fall within the second half of the Hilbert curve matrix interval [2w-7-1 , 2w-7-1 ]; scaling line image coordinate values for each of the plurality of targets to fall within the full Hilbert curve matrix interval [0,2N-1 ]; and calculating the Hilbert curve coordinate for the scaled image coordinate for each of the plurality of targets.11 . The method of claim 8, wherein if the two-dimensional image includes substantially more pixels than lines, scaling the two-dimensional image to cover the portion of a space filling curve matrix comprises: selecting a Peano curve order parameter, N; scaling pixel image coordinate values for each of the plurality of targets to fall within the full Peano curve matrix interval [0,3w- 1 ]; scaling line image coordinate values for each of the plurality of targets to fall within the first third of the Peano curve matrix interval [0, 3w-7-1 ]; and calculating the Peano curve coordinate for the scaled image coordinate for each of the plurality of targets.

12. The method of claim 8, wherein if the two-dimensional image includes substantially more lines than pixels, scaling the two-dimensional image to cover the portion of a space filling curve matrix comprises: selecting a Peano curve order parameter, N; scaling pixel image coordinate values for each of the plurality of targets to fall within the last third of the Peano curve matrix interval [2(3W-7), 3W-1];scaling line image coordinate values for each of the plurality of targets to fall within the full Peano curve matrix interval [0,3w- 1 ]; and calculating the Peano curve coordinate for the scaled image coordinate for each of the plurality of targets.

Images