Image processing method and system for recognizing features in scene

Abstract

Image processing methods and systems for identifying features within a scene, computing devices, methods, and computer program products for identifying features are provided. In one method, images are captured at different time and spatial points and a reference plane is defined in a coordinate system. The method includes applying an image distortion to map a plurality of reference points of a respective image in a sensor plane to respective reference points of a reference plane, and mapping the plurality of reference points of the respective image to respective reference points of the reference plane based on a positional relationship between the reference plane and a representation of the respective image established by applying the image distortion to the respective image. Pixel coordinates of a plurality of pixels of a reference plane are mapped to respective images. The method further includes interpolating from the pixels of the respective image to the pixels at the pixel coordinates of the reference plane to generate an interpolated representation of the respective image, which is then used to identify a feature within the scene.

Description

Technical Field

[0001] One example relates generally to methods, computing devices, and systems for identifying features within a scene, and more specifically, to methods, computing devices, and systems for identifying features within a scene based on comparison of images of the scene captured at different points in time and space. Background Art

[0002] Various applications depend on the recognition of features that appear in a scene. For example, some applications may need to track objects that are in motion relative to a background. Alternatively, an application may need to identify objects that appear in a scene or disappear from a scene. Although features within a scene can be identified in various ways, image processing techniques can be used to compare image samples of scenes that have been captured at different points in time and space. By eliminating or excluding (discounting) the background of the scene that appears in each image, these image processing techniques attempt to identify objects that change between images, thereby allowing detection and tracking of objects in motion and / or allowing identification of objects that appear in a scene or disappear from a scene. For example, a satellite, spacecraft, aircraft, etc. can carry a sensor configured to capture an image of a portion of the earth. By comparing the images captured by the sensor, objects that are in motion relative to the earth (e.g., aircraft or other objects in flight) can be detected and tracked.

[0003] In order to allow for reliable image processing (e.g., identifying features in a scene), images of a scene captured at different points in time must be registered or aligned with each other. By registering the images, the noise associated with image processing can be reduced, thereby improving the accuracy with which features within the scene can be identified. However, movement of the sensor capturing the image and / or movement of the underlying scene may make it more difficult to register images captured at different points in time. With respect to processing images captured by sensors carried by satellites, spacecraft, aircraft, etc., the rotation of the earth and the movement of satellites, spacecraft, aircraft, sight lines, etc. relative to the earth increase the difficulty associated with correctly registering images, and therefore may increase the noise associated with the resulting image processing, and reduce the accuracy with which features can be identified.

[0004] In some applications, images will be processed in order to identify features in a scene in real time or near real time. While images can be translated to better align the images in real time or near real time, the higher order corrections (e.g., rotations and non-linear stretching) required to better align the images are computationally intensive and generally cannot be performed in real time. As a result, applications that require real time or near real time image processing generally suffer from more noise and are therefore less accurate in identifying features within the scene because the processed images are generally not as well registered as desired. Summary of the invention

[0005] A method, computing device, and system for identifying features within a scene based on images of the scene captured at different points in time and space are provided. The example method, computing device, and system are configured to process the images and identify features within the scene in real-time or near real-time. In addition, by relying on image warping between a reference plane and a sensor plane, the method, computing device, and system are configured to identify features within the scene with reduced noise and improved accuracy by providing not only warping, but also rotation and translation, preferably in real-time or near real-time.

[0006] A method is provided according to an example for identifying features within a scene. The method includes obtaining multiple images captured at different time points and at different spatial points, and defining a reference plane in a coordinate system (e.g., a reference plane perpendicular to the visual axis of the sensor that captured the multiple images and / or a reference plane tangent to the surface of the earth). The method also includes applying image deformation to map multiple reference points of corresponding images in the multiple images in the sensor plane to corresponding reference points of the reference plane, and mapping pixel coordinates of multiple pixels of the reference plane to the corresponding image based on the positional relationship between the reference plane and the representation of the corresponding image established by applying image deformation to the corresponding image. The method also includes interpolating from pixels of the corresponding image to pixels at pixel coordinates of the reference plane to generate an interpolated representation of the corresponding image, and using the interpolated representation of the corresponding image to identify features within the scene.

[0007] An example method applies image deformation by applying at least one of a 1.5 or higher order polynomial, bilinear, or projective transformation to the corresponding image to simultaneously provide rotation, translation, and deformation of the corresponding image. In conjunction with or independently of the above example, an example method interpolates by performing bilinear interpolation from pixels of the corresponding image in the sensor plane to pixel coordinates of a reference plane mapped to the sensor plane.

[0008] In addition to or independently of the above examples, the method of the example defines a reference plane by defining the reference plane based on a mapping of another image in a plurality of images to the reference plane. In this example, the method may use an interpolated representation of the corresponding image to identify features within the scene by determining the difference between the interpolated representation of the corresponding image and another image in a plurality of images mapped to the reference plane to identify features within the scene. In addition to or independently of the above examples, the method of the example uses an interpolated representation of the corresponding image to identify features within the scene by determining the difference between: (i) an interpolated representation of an average of a plurality of images captured during a first time period mapped to the reference plane (including the interpolated representation of the corresponding image), and (ii) an average of a second plurality of images captured during a second time period also mapped to the reference plane.

[0009] In another example, a computing device is provided that is configured to identify features within a scene. The computing device includes a processing circuit that is configured to obtain multiple images captured at different time points and different spatial points, and define a reference plane in a coordinate system (e.g., a reference plane perpendicular to the visual axis of the sensor that captures the multiple images, and / or a reference plane tangent to the surface of the earth). The processing circuit is also configured to apply image deformation to map multiple reference points of a corresponding image in the multiple images in the sensor plane to corresponding reference points of the reference plane. The processing circuit is also configured to map pixel coordinates of multiple pixels of the reference plane to the corresponding image based on a positional relationship between the reference plane and a representation of the corresponding image established by applying image deformation to the corresponding image. The processing circuit is also configured to interpolate from pixels of the corresponding image to pixels at pixel coordinates of the reference plane to generate an interpolated representation of the corresponding image, and use the interpolated representation of the corresponding image to identify features within the scene.

[0010] The processing circuitry of the example is configured to apply image deformation by applying at least one of a 1.5 or higher order polynomial, bilinear, or projective transformation to the corresponding image to simultaneously provide rotation, translation, and deformation of the corresponding image. In addition to or independently of the above example, the processing circuitry of the example is configured to interpolate by performing bilinear interpolation from pixels of the corresponding image in the sensor plane to pixel coordinates of a reference plane mapped to the sensor plane.

[0011] In addition to or independently of the above examples, the processing circuit of the example is configured to define a reference plane by defining the reference plane based on a mapping of another image in a plurality of images to the reference plane. In this example, the processing circuit may be configured to use an interpolated representation of the corresponding image to identify features within the scene by determining a difference between the interpolated representation of the corresponding image and another image in a plurality of images mapped to the reference plane so as to identify features within the scene. In addition to or independently of the above examples, the processing circuit of the example is configured to use an interpolated representation of the corresponding image to identify features within the scene by determining a difference between (i) an interpolated representation of an average of a plurality of images captured during a first time period mapped to the reference plane (including the interpolated representation of the corresponding image), and (ii) an average of a second plurality of images captured during a second time period also mapped to the reference plane.

[0012] According to an example, a system is provided, which is configured to identify features within a scene. The system includes at least one sensor configured to capture multiple images at different time points and spatial points. The system also includes a processing circuit configured to obtain multiple images captured at different time points and different spatial points, and define a reference plane in a coordinate system (e.g., a reference plane perpendicular to the visual axis of the sensor that captures the multiple images and / or a reference plane tangent to the surface of the earth). The processing circuit is also configured to apply image deformation to map multiple reference points of a corresponding image in the multiple images in the sensor plane to corresponding reference points of the reference plane. The processing circuit is also configured to map the pixel coordinates of multiple pixels of the reference plane to the corresponding image based on the positional relationship between the reference plane and the representation of the corresponding image established by applying image deformation to the corresponding image. The processing circuit is also configured to interpolate from the pixels of the corresponding image to the pixels at the pixel coordinates of the reference plane to generate an interpolated representation of the corresponding image, and use the interpolated representation of the corresponding image to identify features within the scene.

[0013] The processing circuit of the example is configured to apply image deformation by applying at least one of a 1.5-order or higher polynomial, bilinear, or projective transformation to the corresponding image to simultaneously provide rotation, translation, and deformation of the corresponding image. In addition to the above examples or independently of the above examples, the processing circuit of the example is configured to interpolate by performing bilinear interpolation from pixels of the corresponding image in the sensor plane to pixel coordinates of a reference plane mapped to the sensor plane. In addition to the above examples or independently of the above examples, the processing circuit of the example is configured to define the reference plane in the following manner: define the reference plane based on a mapping of another image in a plurality of images to the reference plane. In this example, the processing circuit can be configured to use the interpolated representation of the corresponding image to identify features within the scene in the following manner: determine the difference between the interpolated representation of the corresponding image and another image in a plurality of images mapped to the reference plane to identify features within the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Having thus generally described certain examples of the present disclosure, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and in which:

[0015] Figure 1 A sensor is depicted, the sensor being carried by a satellite and configured to capture a plurality of images of a portion of the Earth at different points in time and space;

[0016] Figure 2 is a block diagram of a system configured to identify features within a scene according to an example of the present disclosure;

[0017] Figure 3 According to an example of the present disclosure, for example, Figure 2 A flowchart of the operations performed by the system;

[0018] Figure 4 is a schematic diagram of a reference plane defined according to an example of the present disclosure;

[0019] Figure 5A and Figure 5B The reference plane and the images captured at different time points in the reference plane and the sensor plane are shown, respectively, and Figure 5C shows a reference plane and a stack of images captured at different points in time after performing image warping according to an example of the present disclosure; and

[0020] Figure 6 A first set of images and a second set of images captured during a first time period and a second time period are shown, respectively. DETAILED DESCRIPTION

[0021] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which some but not all aspects are shown. In fact, the present disclosure may be embodied in many different forms and should not be construed as limited to the aspects set forth herein. On the contrary, these aspects are provided so that the present disclosure will satisfy applicable legal requirements. Like numbers refer to like elements throughout.

[0022] According to the present disclosure, a method, computing device, and system are provided to capture and process multiple images of a scene to identify features within the scene. Although described below with respect to processing images of a portion of the earth captured by a sensor carried by a satellite, spacecraft, aircraft, etc. to identify an object moving relative to the earth (e.g., an aircraft or other aerial object), the method, computing device, and system can be configured to identify various features within various scenes and are not limited to identifying objects moving relative to a portion of the earth.

[0023] Additionally, the following describes example methods, computing devices, and systems to identify features within a scene based on multiple images of the scene captured at different points in time and at different points in space and processed in real time or near real time, thereby identifying features within the scene while the features remain within the scene. However, other example methods, computing devices, and systems are configured to process images to identify features within the scene at a later point in time. Therefore, the images processed in these other examples may have been stored to allow subsequent identification of features that appeared in the scene in the past.

[0024] refer to Figure 1 , depicting the capture of multiple images of a scene. These images may then be processed as described below to identify features within the scene, in accordance with examples of the present disclosure. In the illustrated example, a satellite 10 carries a sensor 12 (e.g., an image sensor) configured to capture images of a portion of the Earth 14 around which the satellite orbits. In this regard, the sensor carried by the satellite defines a visual axis 16 (e.g., a line of sight) about which the images captured by the sensor are centered. The sensor is configured to capture multiple images at respective points in time. For example, the sensor may be configured to capture a video comprising multiple consecutive images of a scene, wherein each image of the video is captured at a different location and respective point in time.

[0025] Due to the rotation of the earth 14 and the orbital motion of the satellite 10 carrying the sensor 12 around the earth, images captured at multiple different times are not properly registered or aligned with each other even if the images capture the same scene or portions of the same scene. Therefore, example methods, computing devices, and systems provide processing of images to allow for more accurate registration of the images, thereby reducing noise associated with image analysis and improving the reliability and accuracy with which features within the scene can be identified. As described below, example methods, computing devices, and systems are configured to process images to improve registration of the images in a manner that can be performed in real time or near real time, thereby supporting applications that require identifying features within a scene while the features within the scene are still within the scene.

[0026] Reference now Figure 2 , a block diagram of a system 20 configured to identify features within a scene according to an example of the present disclosure is provided. As shown, the system includes one or more sensors 12, which are configured to capture multiple images at different points in time and at different points in space. The sensor can be an image sensor, and therefore can include multiple sensor elements that define corresponding pixels of the resulting image. The sensor elements can be positioned in a planar arrangement to define a sensor plane that captures the image. In one example, the sensor is carried by an aerial platform (e.g., a satellite 10, a spacecraft, an aircraft, or other aerial vehicle) and is configured to capture images of a portion of the earth 14. However, other example systems may include other types of sensors for capturing images of different types of scenes.

[0027] The system 20 also includes a computing device 22 configured to receive a plurality of images captured by the sensor 12 at different points in time and at different points in space, and then process the images as described below. Additionally, the computing device may be configured to receive other types of information, including, for example, navigation information of an aerial platform carrying the sensor. The navigation information may include, for example, a state vector of the platform, a posture of the platform, a location of the platform when the image was captured (e.g., which may be provided as a global positioning system (GPS) coordinate), a time when the image was captured, etc.

[0028] The computing device 22 may be embodied in a variety of ways, including, for example, as any of a variety of different types of computers (e.g., servers), image processing systems, etc. Thus, no particular hardware design is required, but any of a variety of computing devices may be configured to operate as described herein. Regardless of the type of computing device configured to process images in order to identify features within a scene, such as Figure 3 The illustrated computing devices each include processing circuitry 24 and optionally at least one non-transitory memory device 26 and / or a communications interface 28 .

[0029] In some examples, the processing circuit 24 can communicate with the memory device 26 via a bus for transferring information between components of the computing device 22. The memory device can be non-temporary and can include, for example, one or more volatile and / or non-volatile memories. In other words, for example, the memory device can be an electronic storage device (e.g., a computer-readable storage medium) that includes a door configured to store data (e.g., bits) that can be retrieved by a machine (e.g., a computing device like a processing circuit). The memory device can be configured to store information, data, content, applications, instructions, etc. to enable the computing device to perform various functions according to examples of the present disclosure. For example, the memory device can be configured to store navigation information and / or images captured by a sensor for processing by a processing circuit. Additionally or alternatively, the memory device can be configured to store instructions for execution by a processing circuit.

[0030] The processing circuit 24 can be embodied in a variety of different ways. For example, the processing circuit can be embodied as one or more of various hardware processing devices such as the following: a processor, a coprocessor, a microprocessor, a controller, a digital signal processor (DSP), a processing element with or without an accompanying DSP, or various other processing circuits including integrated circuits (such as, for example, ASIC (application-specific integrated circuit), FPGA (field programmable gate array), microcontroller unit (MCU), hardware accelerator, dedicated computer chip, etc.). Therefore, in some examples, the processing circuit may include one or more processing cores configured to execute independently. A multi-core processor can implement multi-processing within a single physical package. Additionally or alternatively, the processing circuit may include one or more processors configured in series via a bus to implement independent execution, pipelining, and / or multithreading of instructions.

[0031] In one example, the processing circuit 24 may be configured to execute instructions stored in the memory device 26 or otherwise accessible to the processing circuit. Alternatively or additionally, the processing circuit may be configured to perform hard-coded functions. Therefore, whether configured by hardware or software methods, or by a combination thereof, the processing circuit may represent an entity (e.g., physically embodied in a circuit) that is capable of performing operations according to examples of the present disclosure while being configured accordingly. Therefore, for example, when the processing circuit is embodied as an ASIC, FPGA, etc., the processing circuit may be specially configured hardware for performing the operations described herein. Alternatively, as another example, when the processing circuit is embodied as an executor of software instructions, the instructions may specifically configure the processing circuit to perform the algorithms and / or operations described herein when the instructions are executed. However, in some cases, the processing circuit may be a processor of a specific device (e.g., an image processing system) that is configured to further configure the processor by instructions for performing the algorithms and / or operations described herein to adopt examples of the present disclosure. The processing circuit may include, among other things, a clock, an arithmetic logic unit (ALU), and / or one or more logic gates configured to support operation of the processing circuit.

[0032] The example computing device 22 may also include a communication interface 28. The communication interface may be any device (e.g., a device or circuit embodied in hardware or a combination of hardware and software) configured to receive data from the sensor 12 and other electronic devices communicating with the computing device and / or transmit data to the sensor 12 and other electronic devices communicating with the computing device. The communication interface may be configured to communicate according to various wireless protocols, including the Global System for Mobile Communications (GSM), such as but not limited to Long Term Evolution (LTE). In this regard, the communication interface may include, for example, an antenna (or multiple antennas) and supporting hardware and / or software for enabling communication with a wireless communication network. Additionally or alternatively, the communication interface may include circuitry for interacting with the antenna to cause the transmission of a signal via the antenna or the reception of a signal received via the antenna. In some environments, the communication interface may alternatively or also support wired communication.

[0033] Now go to Figure 3 , depicting a method according to an example such as that Figure 2Operations performed by a computing device 22 of the present invention. As shown in block 30, the computing device includes means (e.g., processing circuitry 24, communication interface 28, memory device 26, etc.) for obtaining a plurality of images captured by one or more sensors 12 at different points in time and at different points in space. The computing device may receive the images directly from the sensor or from an intermediate device that stores the images and / or otherwise relays the images. Along with the plurality of images, the computing device typically also receives navigation information associated with an airborne platform, such as a satellite 10 (carrying the sensor when the sensor captures each image). The computing device (e.g., processing circuitry) may be configured to store the plurality of images and associated navigation information (e.g., in a memory device) to facilitate processing of the images as described below.

[0034] As indicated at block 32, computing device 20 also includes means (e.g., processing circuitry 22, etc.) for defining a reference plane in the coordinate system. The reference plane may be defined in a variety of ways. Figure 4 In one example depicted, reference plane 42 is a plane perpendicular or orthogonal to viewing axis 16 (e.g., line of sight) along which sensor 12 captures images within sensor plane 44. In one example environment, the reference plane may be positioned so as to be orthogonal to the viewing axis at a location where the viewing axis intersects the surface of the earth 14. However, in other examples, the reference plane may be orthogonal to the viewing axis at a predefined distance along the viewing axis from where the viewing axis intersects the surface of the earth. The reference plane may also be defined to be tangent to the surface of the oblate earth at the location where the viewing axis intersects the surface of the earth (by projecting the sensor field of view onto the tangent plane). In this example, the reference plane may be defined to be a predefined distance above the surface of the earth at the location where the viewing axis intersects the surface of the earth, or a predefined distance below the surface of the earth at the location where the viewing axis intersects the surface of the earth.

[0035] In order to define a reference plane within the coordinate system 42, the computing device 22 (e.g., processing circuit 24) can be configured to identify the intersection of the periphery or edge of one of the images captured by the sensor 12 with the reference plane. In this example, the periphery or edge of the image intersecting the reference plane defines the reference plane within the coordinate system. For example, with respect to an image having a square or rectangular shape, the computing device (e.g., processing circuit) can be configured to identify the intersection of the corner (corner) of one of the images captured by the sensor with the reference plane. In this example, the corners in the image that intersect the reference plane can be connected by a straight line to define the reference plane within the coordinate system. Although the computing device (e.g., processing circuit) can be configured to use any one of the images captured at different points in time and space to define the reference plane within the coordinate system, the computing device of the example is configured to use the first image of multiple images to define the reference plane within the coordinate system. In this regard, the first image is an image of multiple images captured at the earliest point in time from the multiple images. Although the reference plane can be selected arbitrarily, which may require mapping an image (e.g., the first image) to the reference plane, the reference plane can also be selected from the available images, and in this case, since the image is already in the reference plane, the image from which the reference plane is selected will advantageously not need to be deformed to the reference plane.

[0036] Figure 5A An example of a reference plane 50 in coordinate system 42 is depicted in FIG. As shown, the reference plane depicts feature 51. In this case, the feature is a cloud, although the reference plane may depict any of a variety of features and any number of features. Due to movement of the scene (e.g., Earth 14) relative to sensor 12 and / or movement of the sensor relative to the scene, images of the same scene captured at different points in time and space are typically not aligned in the reference plane, but are misaligned relative to each other. By way of example of such misalignment, Figure 5B A second image 52 captured at a second point in time after the point in time at which the image defining the reference plane was captured is depicted in dashed lines in FIG. The second image depicts the same features (i.e., clouds), albeit in slightly different locations. In this regard, the second image captured at a different point in time than the first image defining the reference plane is misaligned in the coordinate system relative to the reference plane because relative movement between the sensor and the scene occurring between the first time and the second time at which the first and second images were captured causes the sensor plane 44 to be positioned differently, and in one example, the sensor plane 44 is no longer parallel to the reference plane when the second image is captured.

[0037] Due to the misalignment in the reference plane 42, the pixels of the second image 52, which have been captured at a different point in time, are misaligned relative to the pixels of the reference plane 50. While interpolation can be performed from the pixels of the second image to the pixels of the reference plane in order to facilitate registration of images captured at different points in time and space within the coordinate system, such interpolation is computationally intensive and, therefore, time consuming. Therefore, such interpolation in the reference plane cannot typically be performed in this manner in real time or near real time, as required by at least some applications regarding the identification of features within a scene.

[0038] Thus, the example computing device 22 includes means (e.g., processing circuitry 24, etc.) for applying image deformation to map reference points of a corresponding image (hereinafter referred to as second image 52) of the plurality of images in the sensor plane 44 to corresponding reference points of the image defining the reference plane 50. Figure 3 Block 34. Although any type of reference point may be used for image deformation, in the case where the second image is rectangular, one example reference point is a corner of the second image. In this example, the corner of the second image is mapped to a corresponding reference point of the image defining the reference plane, i.e., a point of the image defining the reference plane that depicts or represents the same feature as the corner of the second image. By way of example, Figure 5C Depicted is a portion of a representation of the second image (indicated by a dashed outline) that falls within the boundaries of the reference plane after an image deformation has been applied to the second image.

[0039] The computing device 22 (e.g., processing circuitry 24) may be configured to apply image deformation to map reference points of the second image 52 in the sensor plane 44 to corresponding reference points of the image defining the reference plane 50 in various ways. However, in one example, the computing device (e.g., processing circuitry) applies one of a 1.5-order or higher-order polynomial transform, a bilinear transform, or a projective transform to the second image to map the reference points of the second image to corresponding reference points of the image defining the reference plane. In doing so, the computing device (e.g., processing circuitry) provides not only deformation of the second image, but also rotation, translation, and nonlinear scaling of the second image simultaneously.

[0040] like Figure 3As shown in block 36 of , and based on the image deformation, the computing device 22 also includes means (e.g., processing circuitry 24, etc.) for mapping pixel coordinates of a plurality of pixels of the reference plane 50 to a corresponding image 52 (e.g., a second image). In this regard, based on the image deformation and the resulting positional relationship between the reference plane and the representation of the second image in the reference plane 42 (the resulting positional relationship being established by mapping reference points from the second image to corresponding reference points in the image defining the reference plane), the pixel coordinates of a plurality of pixels of the reference plane are mapped to the corresponding image. In this regard, the pixel coordinates of each pixel of the reference plane, or at least each pixel of the reference plane that falls within the representation of the second image in the reference plane, are mapped by an offset (offset) equal to an offset (offset) in the positional relationship between the reference plane and the representation of the second image in the reference plane. The offset in the positional relationship can be defined in various ways, but in one example, is defined by an angular rotation of the representation of the second image relative to the reference plane and a positional offset (e.g., an x-offset and a y-offset) between a center point of the representation of the second image and a center point of the reference plane. In this example, pixel coordinates of pixels of the reference plane may be mapped to the second image in the sensor plane 44 by modifying the pixel coordinates by an amount defined for the corresponding pixels by an angular rotation and positional offset between the representation of the second image and the reference plane.

[0041] like Figure 3 As shown in block 38 of , the computing device 22 also includes means (e.g., processing circuitry 24, etc.) for interpolating from pixels of the corresponding image 52 (i.e., the second image) to pixels at pixel coordinates of the reference plane 50 to generate an interpolated representation of the second image. In this regard, once the image deformation has been applied to the second image, each pixel of the second image, or at least each pixel of the portion of the second image that is aligned with or covers the reference plane, can be interpolated to the pixel at the pixel coordinates of the reference plane.

[0042] By mapping the pixel coordinates of the pixels of the reference plane 50 to the second image 52 in the sensor plane 44, the subsequent interpolation of the pixels of the second image to the pixel coordinates of the reference plane can be performed in a more efficient computational manner, and can therefore be performed in real time or near real time. By performing the interpolation in a more efficient computational manner, processing resources are conserved, thereby providing technical advantages for the methods, systems 20 and computer program products of the examples of the present disclosure. The computing device 22 (e.g., the processing circuit 24) can be configured to interpolate from the pixels of the second image to the pixel coordinates of the reference plane in a variety of ways, including, for example, by performing bilinear interpolation to generate an interpolated representation of the corresponding image.

[0043] like Figure 3As shown in block 40 of , the computing device 22 also includes a device (e.g., processing circuit 24, etc.) for using an interpolated representation of a corresponding image (e.g., a second image 52) to identify a feature 51 within a scene. When interpolation is performed in the sensor plane 44, the interpolated representation of the corresponding image can first be mapped back to the reference plane 42. In this regard, the position of each pixel mapped to the corresponding image in the reference plane (e.g., pixel (0,0), pixel (0,1), pixel (1,0) can be maintained so that the image in the reference plane can be reconstructed with each pixel in the reference plane having a pixel value equal to the interpolated value of each pixel (as determined in the reference plane). The interpolated representation of the corresponding image in the reference plane can then be used in various ways to identify features within the scene. For example, the computing device (e.g., processing circuit) can be configured to determine the difference between the interpolated representation of the corresponding image (e.g., the second image) in the reference plane and another image in a plurality of images that have been captured and mapped to the reference plane in order to identify features within the scene. In Figure 5C In one depicted example, an interpolated representation of the second image depicted in dashed lines can be compared in a reference plane to a reference image depicted in solid lines to detect movement of features (e.g., clouds) over time between capture of the reference image and the second image.

[0044] Because the interpolated representation of the second image has been mapped to the reference plane 42, the reference image 50 and the representation of the second image 52 are aligned or better aligned, so that the difference between the interpolated representation of the second image in the reference plane and the reference image has less noise, and therefore, provides identification of features 51 within the scene captured by the multiple images with improved reliability and accuracy. In this regard, by determining the difference between the aligned representations of the images, the background or other parts of the scene that are common to the two images are eliminated, so that the resulting difference image includes any features that change from one image to the next. For example, once the background has been eliminated from the image, objects that are moving relative to the earth 14 (e.g., aerial objects) can be easily identified.

[0045] exist Figure 6In another example shown in , a first plurality of images 64 are captured at different points in time and space during a first time period, and a second plurality of images 66 are captured at different points in time and space during a second time period. In this example, the computing device 22 (e.g., the processing circuit 24) is configured to use interpolated representations of the respective images to identify features within a scene (e.g., features in motion relative to a background) by determining a difference between: (i) an average of the first plurality of images captured during the first time period after being mapped to a reference plane 42, and (ii) an average of the second plurality of images captured during the second time period after being mapped to the reference plane. In this example, the first plurality of images or the second plurality of images include interpolated representations of the respective images, such that the determination of the difference between the averages of the first plurality of images and the second plurality of images mapped to the reference plane uses the interpolated representations of the respective images to identify features within the scene. Because the interpolated representations of the corresponding images mapped in the reference plane are used in the manner described above, the interpolated representations of the corresponding images in the reference plane are better aligned with other images also mapped to the reference plane, so that the average or sum of the registered images will result in reduced noise and improved accuracy and reliability with which features within the scene can be identified, for example, in the manner described above.

[0046] As described above, a method, computing device 22, and system 20 are provided for identifying features within a scene based on images of the scene captured at different points in time and space. The example method, computing device, and system may be configured to process the images and identify features within the scene in real time or near real time. In addition, by relying on image deformation between reference plane 42 and sensor plane 44, the method, computing device, and system are configured to identify features within the scene with reduced noise and improved accuracy by providing not only deformation of the image, but also rotation and translation of the image in real time or near real time.

[0047] As mentioned above, Figure 3Flowcharts of computing devices 22, methods, and computer program products configured to process images in order to identify features within a scene (e.g., provide tracking objects) according to an example. It should be understood that each block of the flowchart and the combination of blocks in the flowchart can be implemented by various devices (e.g., hardware, firmware, processing circuit 24, and / or other devices associated with the execution of software including one or more computer program instructions). For example, one or more of the above processes can be embodied by computer program instructions. In this regard, the computer program instructions embodying the above processes can be stored by the memory device 26 of the computing device and executed by the processing circuit, etc. As will be understood, any such computer program instructions can be loaded onto a computer or other programmable device (e.g., hardware) to produce a machine so that the resulting computer or other programmable device implements the functions specified in the flowchart blocks. These computer program instructions can also be stored in a computer-readable memory, which can instruct a computer or other programmable device to operate in a specific manner so that the instructions stored in the computer-readable memory produce an article of manufacture, which executes to implement the functions specified in the flowchart blocks. Computer program instructions may also be loaded onto a computer or other programmable apparatus to cause a series of operations to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions executed on the computer or other programmable apparatus provide operations for implementing the functions specified in the flowchart block.

[0048] Therefore, the blocks of the flowchart support a combination of means for performing the specified functions and a combination of operations for performing the specified functions. It will also be understood that one or more blocks of the flowchart and a combination of blocks in the flowchart can be implemented by a computer system based on dedicated hardware or a combination of dedicated hardware and computer instructions that performs the specified functions.

[0049] In some examples, some of the operations described above may be modified or further amplified. In addition, in some examples, additional optional operations may be included. The modifications, additions, or amplifications to the operations described above may be performed in any order and in any combination.

[0050] With the benefit of the teachings presented in the above description and the associated drawings, those skilled in the art to which these inventions belong will think of many modifications and other examples set forth herein. Therefore, it should be understood that the present invention is not limited to the specific examples disclosed, and modifications are intended to be included within the scope of the appended claims. In addition, although the above description and the associated drawings describe examples in the context of certain example combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions may be provided by alternative implementations without departing from the scope of the appended claims. In this regard, for example, as may be set forth in some of the appended claims, combinations of elements and / or functions different from the combinations of elements and / or functions explicitly described above are also considered. Although specific terms are employed herein, they are used only in a general and descriptive sense, and not for the purpose of limitation.

Claims

1. A method for identifying features (51) within a scene, the method comprising: obtaining (30) a plurality of images (64, 66) captured at different time points and different spatial points; defining (32) a reference plane (50) in the coordinate system (42); applying (34) image deformation to map a plurality of reference points of corresponding ones of the plurality of images in the sensor plane (44) to corresponding reference points of the reference plane (50); mapping (36) pixel coordinates of a plurality of pixels of the reference plane (50) to the corresponding image based on a positional relationship between the reference plane (50) and a representation of the corresponding image established by applying an image deformation to the corresponding image; interpolating (38) from pixels of the corresponding image to pixels at the pixel coordinates of the reference plane (50) to generate an interpolated representation of the corresponding image; and The features (51) within the scene are identified using (40) the interpolated representations of the corresponding images.

2. The method of claim 1, wherein applying (34) an image deformation comprises applying at least one of a 1.5 or higher order polynomial, bilinear, or projective transformation to the respective images so as to simultaneously provide rotation, translation, and deformation of the respective images.

3. The method of claim 1, wherein interpolating (38) comprises performing bilinear interpolation from the pixels of the corresponding image in the sensor plane (44) to the pixel coordinates of the reference plane (50) mapped to the sensor plane (44).

4. The method according to claim 1, wherein: The reference plane (50) is perpendicular to a viewing axis (16) of a sensor (12) that captures the plurality of images (64, 66).

5. The method according to claim 1, wherein: Defining (32) the reference plane (50) includes defining the reference plane (50) based on a mapping of another image of the plurality of images to the reference plane (50).

6. The method according to claim 5, wherein: Using (40) the interpolated representation of the corresponding image to identify the feature (51) within the scene includes determining a difference between the interpolated representation of the corresponding image and the other image of the plurality of images mapped to the reference plane (50) so as to identify the feature (51) within the scene.

7. The method according to claim 1, wherein: Using (40) the interpolated representation of the corresponding image to identify the feature (51) within the scene includes determining the difference between: (i) the interpolated representation of the average of the plurality of images (64) captured during a first time period mapped to the reference plane (50), which includes the interpolated representation of the corresponding image, and (ii) the average of a second plurality of images (66) captured during a second time period also mapped to the reference plane (50).

8. A computing device (22) configured to identify features (51) within a scene, the computing device (23) comprising processing circuitry (24) configured to: Acquiring a plurality of images (64, 66) captured at different time points and different spatial points; defining a reference plane (50) in the coordinate system (42); applying image deformation to map a plurality of reference points of corresponding ones of the plurality of images in the sensor plane (44) to corresponding reference points of the reference plane (50); mapping pixel coordinates of a plurality of pixels of the reference plane (50) to the corresponding image based on a positional relationship between the reference plane (50) and a representation of the corresponding image established by applying an image deformation to the corresponding image; interpolating from pixels of the corresponding image to pixels at the pixel coordinates of the reference plane (50) to generate an interpolated representation of the corresponding image; and The features within the scene are identified using the interpolated representation of the corresponding images (51).

9. The computing device (22) of claim 8, wherein: The processing circuit (24) is configured to apply image deformation by applying at least one of a 1.5 or higher order polynomial, bilinear or projective transformation to the corresponding image to simultaneously provide rotation, translation and deformation of the corresponding image.

10. The computing device (22) of claim 8, wherein: The processing circuit (24) is configured to interpolate by performing bilinear interpolation from the pixels of the corresponding image in the sensor plane (44) to the pixel coordinates mapped to the reference plane (50) of the sensor plane (44).

11. The computing device (22) of claim 8, wherein: The reference plane (50) is perpendicular to a viewing axis (16) of a sensor (12) that captures the plurality of images (64, 66).

12. The computing device (22) of claim 8, wherein the reference plane (50) is tangential to the surface of the earth.

13. The computing device (22) of claim 8, wherein: The processing circuit (24) is configured to define the reference plane (50) by defining the reference plane (50) based on a mapping of another image of the plurality of images to the reference plane (50).

14. The computing device (22) of claim 13, wherein: The processing circuit (24) is configured to use the interpolated representation of the corresponding image to identify the feature (51) within the scene by determining a difference between the interpolated representation of the corresponding image and the other image of the plurality of images mapped to the reference plane (50) so as to identify the feature (51) within the scene.

15. The computing device (22) of claim 8, wherein: The processing circuit (24) is configured to use the interpolated representation of the corresponding image to identify the feature (51) within the scene by determining the difference between: (i) the interpolated representation of an average of the plurality of images (64) captured during a first time period mapped to the reference plane (50), which includes the interpolated representation of the corresponding image, and (ii) the average of a second plurality of images (66) captured during a second time period also mapped to the reference plane (50).

16. A system (20) configured to identify features (51) within a scene, the system comprising: at least one sensor (12) configured to capture a plurality of images (64, 66) at different points in time and at different points in space; and A processing circuit (24) configured to: obtaining a plurality of images captured at the different time points and the different spatial points; defining a reference plane (50) in the coordinate system (42); applying image deformation to map a plurality of reference points of corresponding ones of the plurality of images in the sensor plane (44) to corresponding reference points of the reference plane (50); mapping pixel coordinates of a plurality of pixels of the reference plane (50) to the corresponding image based on a positional relationship between the reference plane (50) and a representation of the corresponding image established by applying an image deformation to the corresponding image; interpolating from pixels of the corresponding image to pixels at the pixel coordinates of the reference plane (50) to generate an interpolated representation of the corresponding image; and The features within the scene are identified using the interpolated representation of the corresponding images (51).

17. The system (20) of claim 15, wherein the processing circuit (24) is configured to apply image deformation by applying at least one of a 1.5-order or higher-order polynomial, bilinear, or projective transformation to the corresponding image so as to simultaneously provide rotation, translation, and deformation of the corresponding image.

18. The system (20) of claim 15, wherein the processing circuit (24) is configured to interpolate by performing bilinear interpolation from the pixels of the corresponding image in the sensor plane (44) to the pixel coordinates of the reference plane (50) mapped to the sensor plane (44).

19. The system (20) of claim 15, wherein the processing circuit (24) is configured to define the reference plane (50) by defining the reference plane (50) based on a mapping of another image of the plurality of images to the reference plane (50).

20. A system (20) according to claim 19, wherein the processing circuit (24) is configured to use the interpolated representation of the corresponding image to identify the feature (51) within the scene by determining a difference between the interpolated representation of the corresponding image and the other image of the multiple images mapped to the reference plane (50) so as to identify the feature (51) within the scene.

Images