Image processing method and system for identifying feature within scene
Patent Information
- Application Number
- JP2024130492
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2023-08-18
- Filing Date
- 2024-08-07
- Publication Date
- 2025-03-11
Smart Images
Figure 2025032059000001_ABST
Abstract
Description
[Technical field]
[0001]
[0001] Embodiments relate generally to methods, computing devices, and systems for identifying features in a scene, and in particular to methods, computing devices, and systems for identifying features in a scene based on a comparison of multiple images of the scene captured at different points in time and space. [Background technology]
[0002]
[0002] Various applications rely on identifying features that appear in a scene. For example, some applications may need to track objects that are moving against a background. Alternatively, applications may need to identify objects that either appear in a scene or disappear from the scene. While features in a scene may be identified in various ways, image processing techniques may be utilized to compare multiple image samples of a scene captured at different points in time and space. By removing or subtracting the background of the scene that appears in each of the multiple images, these image processing techniques attempt to identify objects that change between the multiple images, thereby allowing moving objects to be detected and tracked, and / or objects that appear or disappear from the scene to be identified. For example, satellites, spacecraft, aircraft, etc. may be equipped with sensors configured to capture images of a portion of the Earth. By comparing multiple images captured by the sensors, objects moving relative to the Earth, such as aircraft or other objects in flight, may be detected and tracked.
[0003]
[0003] To enable reliable image processing, such as to identify features within a scene, multiple images of a scene captured at different points in time need to be registered or aligned with one another. By registering multiple images, noise associated with image processing may be reduced, thereby improving the precision with which features may be identified within a scene. However, movement of the sensor capturing the images and / or movement of the underlying scene may make it more difficult to align images captured at different points in time. With respect to processing images captured by sensors onboard satellites, spacecraft, aircraft, etc., the rotation of the Earth, movement of the satellite, spacecraft, aircraft, line of sight, etc. relative to the Earth may increase the difficulty associated with properly aligning the images, thereby increasing the noise associated with the resulting image processing and reducing the precision with which features may be identified.
[0004]
[0004] In some applications, multiple images are processed to identify features in a scene in real time or near real time. To better align the multiple images in real time or near real time, the multiple images can be translated, but higher order corrections such as rotation and nonlinear stretching required to better align the multiple images are computationally intensive and generally cannot be performed in real time. As a result, applications that require image processing in real time or near real time generally suffer from more noise and, as a result, less accuracy in identifying features in a scene. This is because the multiple images processed are typically not as well aligned as desired. Summary of the Invention
[0005]
[0005] A method, computing device, and system are provided for identifying features in a scene based on multiple captured images of the scene at different points in time and space. In one embodiment, the method, computing device, and system are configured to process multiple images and identify features in the scene in a real-time or near real-time manner. In addition, by relying on image warping between a reference plane and a sensor plane, the method, computing device, and system are configured to identify features in the scene, preferably in real-time or near real-time, with reduced noise and improved accuracy by providing not only warping but also rotation and translation.
[0006]
[0006] A method according to an embodiment for identifying features in a scene is provided. The method includes obtaining a plurality of images captured at different points in time and different points in space, and defining a reference plane in a reference frame, such as a reference plane perpendicular to the boresight of a sensor capturing the plurality of images and / or a reference plane tangential to the surface of the Earth. The method also includes applying image warping to map a plurality of reference points of each of the plurality of images in the sensor plane to corresponding reference points of the reference plane, and mapping pixel coordinates of a plurality of pixels of the reference plane to each of the images based on a positional relationship between the reference plane and a representation of each of the images established by applying the image warping to each of the images. The method further includes interpolating from pixels of each of the images to pixels at pixel coordinates of the reference plane to generate an interpolated representation of each of the images, and using the interpolated representation of each of the images to identify features in the scene.
[0007]
[0007] In one embodiment, the method applies image warping by applying at least one of a polynomial transformation of degree 1.5 or higher, a bilinear transformation, or a projective transformation to each image to simultaneously provide rotation, translation, and warping of each image. In combination with or independent of the above embodiment, the method of one embodiment interpolates by performing a bilinear interpolation from pixels of each image in the sensor plane to pixel coordinates of a reference plane mapped to the sensor plane.
[0008]
[0008] In addition to or independently of the above-mentioned embodiments, a method of an embodiment defines a reference plane by defining a reference plane based on mapping another one of the plurality of images to the reference plane. In this embodiment, the method may use the interpolated representation of each image to identify features in the scene by determining a difference between the interpolated representation of each image and another one of the plurality of images mapped to the reference plane to identify features in the scene. In addition to or independently of the above-mentioned embodiments, a method of an embodiment uses the interpolated representation of each image to identify features in the scene by determining a difference between: (i) an interpolated representation of an average of the plurality of images captured during a first period of time, the plurality of images mapped to the reference plane including the interpolated representation of each image, and (ii) an average of the second plurality of images captured during a second period of time, also mapped to the reference plane.
[0009]
[0009] In another embodiment, a computing device configured to identify features in a scene is provided. The computing device includes a processing circuit configured to obtain a plurality of images captured at different points in time and different points in space, and to define a reference plane in a reference frame, such as a reference plane perpendicular to the boresight of a sensor capturing the plurality of images and / or a reference plane tangential to the surface of the Earth. The processing circuit is also configured to apply image warping to map a plurality of reference points of each of the plurality of images in the sensor plane to corresponding reference points of the reference plane. The processing circuit is also configured to map pixel coordinates of a plurality of pixels of the reference plane to the respective images based on a positional relationship between the reference plane and a representation of the respective images established by applying the image warping to the respective images. The processing circuit is further configured to interpolate from pixels of the respective images to pixels at pixel coordinates of the reference plane to generate an interpolated representation of the respective images, and to use the interpolated representation of the respective images to identify features in the scene.
[0010]
[0010] In one embodiment, the processing circuitry is configured to apply image warping by applying at least one of a polynomial transformation of degree 1.5 or higher, a bilinear transformation, or a projective transformation to each image to simultaneously provide rotation, translation, and warping of each image. In addition to or independently of the above embodiment, the processing circuitry of one embodiment is configured to interpolate by performing a bilinear interpolation from pixels of each image in the sensor plane to pixel coordinates of a reference plane mapped to the sensor plane.
[0011]
[0011] In addition to or independently of the above-mentioned embodiments, the processing circuitry of one embodiment is configured to define a reference plane by defining a reference plane based on mapping another one of the plurality of images to the reference plane. In this embodiment, the processing circuitry may be configured to use the interpolated representation of each image to identify a feature in the scene by determining a difference between the interpolated representation of each image and another one of the plurality of images mapped to the reference plane to identify a feature in the scene. In addition to or independently of the above-mentioned embodiments, the processing circuitry of one embodiment is configured to use the interpolated representation of each image to identify a feature in the scene by determining a difference between: (i) an interpolated representation of an average of the plurality of images mapped to the reference plane, the plurality of images including the interpolated representation of each image, captured during a first period of time, and (ii) an average of the second plurality of images also mapped to the reference plane, the second plurality of images captured during a second period of time.
[0012]
[0012] A system according to an embodiment configured to identify features in a scene is provided. The system includes at least one sensor configured to capture a plurality of images at different points in time and space. The system also includes a processing circuit configured to obtain the plurality of images captured at different points in time and different points in space, and to define a reference plane in a reference frame, such as a reference plane perpendicular to the boresight of the sensor capturing the plurality of images and / or a reference plane tangential to the surface of the Earth. The processing circuit is also configured to apply image warping to map a plurality of reference points of each of the plurality of images in the sensor plane to corresponding reference points of the reference plane. The processing circuit is also configured to map pixel coordinates of a plurality of pixels of the reference plane to the respective images based on a positional relationship between the reference plane and a representation of the respective images established by applying the image warping to the respective images. The processing circuit is further configured to interpolate from pixels of the respective images to pixels at pixel coordinates of the reference plane to generate an interpolated representation of the respective images, and to use the interpolated representation of the respective images to identify features in the scene.
[0013]
[0013] The processing circuitry of one embodiment is configured to apply image warping by applying at least one of a polynomial transformation of degree 1.5 or higher, a bilinear transformation, or a projective transformation to each image to simultaneously provide rotation, translation, and warping of each image. In addition to or independently of the above embodiment, the processing circuitry of one embodiment is configured to interpolate by performing a bilinear interpolation from pixels of each image in the sensor plane to pixel coordinates of a reference plane mapped to the sensor plane. In addition to or independently of the above embodiments, the processing circuitry of one embodiment is configured to define a reference plane by defining a reference plane based on mapping another one of the multiple images to the reference plane. In this embodiment, the processing circuitry may be configured to use the interpolated representation of each image to identify a feature in the scene by determining a difference between the interpolated representation of each image and another one of the multiple images mapped to the reference plane to identify a feature in the scene.
[0014]
[0014] Having now described in general terms certain embodiments of the present disclosure, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and in which: [Brief description of the drawings]
[0015] [Figure 1]
[0015] A sensor is shown mounted on a satellite and configured to capture multiple images of a portion of the Earth at different points in time and space. [Diagram 2]
[0016] FIG. 1 is a block diagram of a system configured to identify features in a scene according to one embodiment of the present disclosure. [Diagram 3]
[0017] 3 is a flowchart of operations performed by a system such as that of FIG. 2 according to one embodiment of the present disclosure. [Figure 4]
[0018] 1 is a schematic representation of a reference plane defined in accordance with one embodiment of the present disclosure. [Figure 5A]
[0019] 5A and 5B show a reference plane and images captured at different points in time within the reference plane and the sensor plane, respectively. [Figure 5B] 5A and 5B show a reference plane and images captured at different points in time within the reference plane and the sensor plane, respectively. [Figure 5C] 1 illustrates stacking of a reference plane and images captured at different points in time after performing image warping, according to one embodiment of the present disclosure. [Figure 6]
[0020] 3 shows a first group of images and a second group of images captured during a first period and a second period, respectively. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0016]
[0021] The present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which not all aspects of the disclosure are shown. Indeed, the disclosure may be embodied in many different forms and should not be construed as limited to the aspects set forth herein. Rather, these aspects are provided so that this disclosure will satisfy applicable legal requirements. Like numbers refer to like elements throughout.
[0017]
[0022] Methods, computing devices, and systems according to the present disclosure are provided for capturing and processing multiple images of a scene to identify features within the scene. Although described below with respect to processing captured images of a portion of the Earth by sensors onboard a satellite, spacecraft, aircraft, or the like to identify objects, such as aircraft or other airborne objects moving relative to the Earth, the methods, computing devices, and systems may be configured to identify a wide variety of features within a variety of scenes and are not limited to identifying objects moving relative to a portion of the Earth.
[0018]
[0023] Moreover, an embodiment method, computing device, and system is described below to identify features in a scene based on multiple images of the scene captured at different points in time and different points in space and processed in real time or near real time, thereby identifying features in the scene while they remain in the scene. However, other embodiment methods, computing devices, and systems are configured to process images to identify features in the scene at later points in time. Thus, the images processed in these other embodiments may be stored to enable later identification of features that appeared in the scene in the past.
[0019]
[0024] Referring to FIG. 1, the capture of multiple images of a scene is shown. These images may then be processed as described below in accordance with an embodiment of the present disclosure to identify features within the scene. In one embodiment shown, a satellite 10 carries a sensor 12 (such as an image sensor) configured to capture images of a portion of the Earth 14 around which the satellite orbits. In this regard, the sensor carried by the satellite defines a boresight 16 (e.g., a line of sight). The images captured by the sensor are centered about the boresight 16. The sensor is configured to capture multiple images at respective points in time. For example, the sensor may be configured to capture a video made up of multiple successive images of a scene, with each image of the video captured at a different location and respective point in time.
[0020]
[0025] As a result of the rotation of the Earth 14 and the orbital movement of the satellite 10 carrying the sensor 12 around the Earth, images captured at different times are not properly registered or aligned with one another even if the images capture the same scene or portions of the same scene. Thus, an example method, computing device, and system processes the images to enable more accurate registration of the images, thereby reducing noise associated with image analysis and improving the reliability and accuracy with which features within the scene may be identified. As described below, an example method, computing device, and system are configured to process the images to improve the registration of the images in a manner that may be performed in real time or near real time, thereby supporting applications that require features within a scene to be identified while the features remain within the scene.
[0021]
[0026] Referring now to FIG. 2, a block diagram of a system 20 configured to identify features in a scene according to one embodiment of the present disclosure is provided. As shown, the system includes one or more sensors 12 configured to capture multiple images at different points in time and different points in space. The sensor may be an image sensor and thus may include multiple sensor elements that define respective pixels of the resulting image. The sensor elements may be arranged in a planar configuration, thereby defining a sensor plane in which the image is captured. In one embodiment, the sensor is mounted on an airborne platform, such as a satellite 10, spacecraft, aircraft, or other airborne vehicle, and is configured to capture images of a portion of the Earth 14. However, other embodiment systems may include other types of sensors for capturing images of different types of scenes.
[0022]
[0027] The system 20 also includes a computing device 22 configured to receive multiple images captured by the sensor 12 at different points in time and different points in space, and then process the multiple images as described below. In addition, the computing device may be configured to receive other types of information, including, for example, navigational information for the airborne platform carrying the sensor. This navigational information may include, for example, a state vector of the platform, the attitude of the platform, the position of the platform at the time the image was captured (which may be provided as Global Positioning System (GPS) coordinates), the time the image was captured, etc.
[0023]
[0028] Computing device 22 may be embodied in a variety of ways, such as, for example, as any of a variety of different types of computers, such as a server, an image processing system, etc. Thus, no particular hardware design is required, but any of a variety of computing devices may be configured to operate as described herein. Regardless of the type of computing device configured to process a plurality of images to identify features within a scene, a computing device such as that shown in FIG. 2 includes processing circuitry 24, optionally at least one non-transitory memory device 26, and / or a communications interface 28.
[0024]
[0029] In some embodiments, the processing circuitry 24 may communicate with the memory device 26 via a bus for passing information between components of the computing device 22. The memory device may be non-transitory, e.g., may include one or more volatile and / or non-volatile memories. In other words, for example, the memory device may be an electronic storage device (e.g., a computer-readable storage medium) including gates configured to store data (e.g., bits) that may be retrievable by a machine (e.g., a computing device such as a processing circuit). The memory device may be configured to store information, data, content, applications, instructions, etc. to enable the computing device to perform various functions according to an embodiment of the present disclosure. For example, the memory device may be configured to store navigation information and / or images captured by a sensor for processing by the processing circuitry. Additionally or alternatively, the memory device may be configured to store instructions for execution by the processing circuitry.
[0025]
[0030] The processing circuitry 24 may be embodied in a number of different ways. For example, the processing circuitry may be embodied as one or more of a variety of hardware processing means, such as a processor, a co-processor, a microprocessor, a controller, a digital signal processor (DSP), a processing element with or without a DSP, or a variety of other processing circuits including integrated circuits, such as, for example, an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a microcontroller unit (MCU), a hardware accelerator, a dedicated computer chip, etc. Thus, in some embodiments, the processing circuitry may include one or more processing cores configured to execute independently. A multi-core processor may enable multi-processing within a single physical package. Additionally or alternatively, the processing circuitry may include one or more processors configured in tandem via a bus to enable independent execution, pipelining, and / or multi-threading of instructions.
[0026]
[0031] In one embodiment, the processing circuitry 24 may be configured to execute instructions stored in the memory device 26 or otherwise accessible to the processing circuitry. Alternatively or additionally, the processing circuitry may be configured to execute hard-coded functions. Thus, whether configured by hardware or software methods, or a combination thereof, the processing circuitry may represent an entity (e.g., physically embodied as a circuit) that, while appropriately configured, can perform operations according to an embodiment of the present disclosure. Thus, for example, when the processing circuitry is embodied as an ASIC, FPGA, or the like, the processing circuitry may be hardware that is specifically configured to perform the operations described herein. Alternatively, as another example, when the processing circuitry is embodied as an execution device for software instructions, the instructions may specifically configure the processing circuitry to perform the algorithms and / or operations described herein when the instructions are executed. However, in some cases, the processing circuitry may be a processor of a particular device (e.g., an image processing system) that is configured to employ an embodiment of the present disclosure upon further configuration of the processor by the instructions to perform the algorithms and / or operations described herein. The processing circuitry may include, among other things, a clock, an arithmetic logic unit (ALU), and / or one or more logic gates configured to support operation of the processing circuitry.
[0027]
[0032] The computing device 22 of one embodiment may also include a communications interface 28. The communications interface may be any means, such as a device or circuitry embodied in either hardware or a combination of hardware and software, configured to transmit and receive data to and from the sensors 12 and other electronic devices in communication with the computing device. The communications interface may be configured to communicate according to various wireless protocols, including, but not limited to, Global System for Mobile Communications (GSM), such as Long Term Evolution (LTE). In this regard, the communications interface may include, for example, an antenna (or antennas) and supporting hardware and / or software to enable communication with a wireless communications network. Additionally or alternatively, the communications interface may include circuitry for interacting with the antenna(s) to cause signals to be transmitted via the antenna(s) or to handle reception of signals received via the antenna(s). In some environments, the communications interface may alternatively support or also support wired communications.
[0028]
[0033] Referring now to Figure 3, operations performed according to one embodiment, such as by the computing device 22 of Figure 2, are shown. As illustrated at block 30, the computing device includes means, such as processing circuitry 24, communications interface 28, memory device 26, etc., for acquiring a plurality of images captured by one or more sensors 12 at different points in time and different points in space. The computing device may receive the images directly from the sensor or may receive the images from an intermediate device that stores and / or otherwise relays the images. Along with the plurality of images, the computing device also generally receives navigation information associated with an airborne platform, such as a satellite 10 carrying the sensor, at the time each image was captured by the sensor. A computing device, such as a processing circuitry, may be configured to store the plurality of images, as well as the associated navigation information, such as in a memory device, to facilitate processing of the images as described below.
[0029]
[0034] As illustrated at block 32, the computing device 20 also includes means, such as the processing circuitry 22, for defining a reference plane in the reference frame. The reference plane may be defined in a variety of ways. In one embodiment shown in FIG. 4, the reference plane 42 is a plane perpendicular to or normal to the boresight 16 (e.g., line of sight). The sensor 12 captures images in a sensor plane 44 along the boresight 16. In one exemplary environment, the reference plane may be positioned such that it is normal to the boresight at the location where the boresight intersects the surface of the Earth 14. However, in other embodiments, the reference plane may be normal to the boresight at a predefined distance along the boresight from the location where the boresight intersects the surface of the Earth. The reference plane may also be defined to be tangential to the surface of the flattened Earth at the location where the boresight intersects the surface of the Earth (by projecting the sensor's field of view onto the tangential plane). In this embodiment, the reference plane may be defined to be a predefined distance above the surface of the Earth where the boresight intersects with the surface of the Earth, or a predefined distance below the surface of the Earth where the boresight intersects with the surface of the Earth.
[0030]
[0035] To define a reference plane in the reference frame 42, a computing device 22 such as the processing circuit 24 may be configured to identify an intersection of a perimeter or edge of one of the images captured by the sensor 12 with the reference plane. In this example, the perimeter or edge of the image that intersects with the reference plane defines the reference plane in the reference frame. For example, for a plurality of images having a square or rectangular shape, a computing device such as the processing circuit may be configured to identify an intersection of a corner of one of the images captured by the sensor with the reference plane. In this example, the corners of the image that intersect with the reference plane may be connected by straight lines to define the reference plane in the reference frame. Although a computing device such as the processing circuit may be configured to utilize any of the images captured at different points in time and space to define the reference plane in the reference frame, a computing device in one example is configured to utilize a first image of the plurality of images to define the reference plane in the reference frame. In this regard, the first image is the image of the plurality of images captured at the earliest point in time among the plurality of images. The reference plane may be selected arbitrarily, in which case an image such as the first image may need to be mapped to the reference plane, but the reference plane may also be selected from a number of available images, in which case the image from which the reference plane is selected advantageously does not need to be warped to the reference plane since it is already in the reference plane.
[0031]
[0036] One example of a reference plane 50 in the reference frame 42 is shown in FIG. 5A. As shown, the reference plane shows a feature 51. In this case, the feature is a cloud, but any and any number of a variety of features may be shown by the reference plane. As a result of movement of the scene, such as the Earth 14 relative to the sensor 12, and / or movement of the sensor relative to the scene, multiple images of the same scene captured at different points in time and space are generally not aligned in the reference plane, but are instead misaligned with respect to each other. As an example of this misalignment, a second image 52 captured at a second point in time (after the point in time at which the image defining the reference plane was captured) is shown in dashed lines in FIG. 5B. The second image shows the same feature (i.e., a cloud), albeit in a somewhat different position. In this regard, the second image captured at a different point in time from the first image defining the reference plane is misaligned in the reference frame with respect to the reference plane. This is because relative movement between the sensor and the scene that occurs between the first and second times at which the first and second images are captured results in, in one embodiment, the sensor plane 44 being positioned differently and no longer parallel to the reference plane when the second image is captured.
[0032]
[0037] As a result of the misalignment in the reference plane 42, the pixels of the second image 52 captured at different points in time are misaligned with respect to the pixels of the reference plane 50. To facilitate the alignment of the images captured at different points in time and space within the reference frame, an interpolation from the pixels of the second image to the pixels of the reference plane may be performed, but this interpolation is computationally intensive and, as a result, time consuming. Thus, this interpolation in the reference plane generally cannot be performed in this manner in real time or near real time, as required by at least some applications for identifying features in a scene.
[0033]
[0038] Thus, the computing device 22 of one embodiment includes means, such as processing circuitry 24, for applying image warping to map reference points of each of the multiple images in the sensor plane 44 (hereinafter referred to as second image 52) to corresponding reference points of the image defining the reference plane 50. See block 34 of FIG. 3. While any kind of reference points may be utilized for image warping, the reference points of one embodiment are corners of the second image in one case where the second image is rectangular. In this embodiment, the corners of the second image are mapped to corresponding reference points of the image defining the reference plane, i.e., points of the image defining the reference plane that show or represent the same features as the corners of the second image. As an example, FIG. 5C shows a representation of a portion of the second image (represented by a dashed outline) that falls within the boundary of the reference plane after applying image warping to the second image.
[0034]
[0039] A computing device 22, such as a processing circuit 24, may be configured to apply image warping to map reference points of the second image 52 in the sensor plane 44 to corresponding reference points of the image defining the reference plane 50 in a variety of ways. However, in one embodiment, a computing device, such as a processing circuit, applies any one of a polynomial transformation of degree 1.5 or higher, a bilinear transformation, or a projective transformation to the second image to map the reference points of the second image to corresponding reference points of the image defining the reference plane. In doing so, a computing device, such as a processing circuit, not only provides warping of the second image, but also simultaneously provides rotation, translation, and nonlinear scaling of the second image.
[0035]
[0040] As indicated by block 36 of FIG. 3 and based on image warping, the computing device 22 also includes means, such as the processing circuitry 24, for mapping pixel coordinates of a plurality of pixels of the reference plane 50 to a respective image 52 (e.g., a second image). In this regard, pixel coordinates of a plurality of pixels of the reference plane are mapped to the respective image based on the image warping and the resulting positional relationship in the reference plane 42 between the reference plane and the representation of the second image established by mapping reference points from the second image to corresponding reference points in the image defining the reference plane. In this regard, pixel coordinates of each pixel of the reference plane, or at least each pixel of the reference plane falling within the representation of the second image in the reference plane, are mapped by being offset by an amount equal to an offset in the positional relationship between the reference plane and the representation of the second image in the reference plane. The offset in the positional relationship may be defined in various ways, but in one embodiment, it is defined by an angular rotation of the representation of the second image relative to the reference plane and a positional offset (e.g., an x-offset and a y-offset) between a center point of the representation of the second image and a center point of the reference plane. In this embodiment, pixel coordinates of pixels of the reference plane can be mapped to the second image in the sensor plane 44 by modifying the pixel coordinates by an amount defined for each pixel by the angular rotation and position offset between the representation of the second image and the reference plane.
[0036]
[0041] 3, computing device 22 further includes means, such as processing circuitry 24, for interpolating from pixels of the respective image 52 (i.e., the second image) to pixels at pixel coordinates of the reference plane 50 to generate an interpolated representation of the second image. In this regard, each pixel of the second image, or at least each pixel of that portion of the second image that aligns with or overlaps the reference plane (if image warping has been applied to the second image), may be interpolated to a pixel at pixel coordinates of the reference plane.
[0037]
[0042] By mapping pixel coordinates of pixels of the reference plane 50 to the second image 52 in the sensor plane 44, subsequent interpolation of pixels of the second image to pixel coordinates of the reference plane can be performed in a more computationally efficient manner, and consequently in real-time or near real-time. By performing the interpolation in a more computationally efficient manner, processing resources are conserved, thereby providing technical advantages to the method, system 20, and computer program product of one embodiment of the present disclosure. A computing device 22, such as processing circuitry 24, can be configured to interpolate from pixels of the second image to pixel coordinates of the reference plane in a variety of manners, including, for example, performing bilinear interpolation, to generate an interpolated representation of the respective image.
[0038]
[0043] As shown in block 40 of FIG. 3, the computing device 22 also includes means, such as the processing circuitry 24, for using the interpolated representation of the respective image (e.g., the second image 52) to identify features 51 in the scene. Since the interpolation is performed in the sensor plane 44, the interpolated representation of the respective image may first be mapped back to the reference plane 42. In this regard, the location of each pixel in the reference plane (e.g., pixel (0,0), pixel (0,1), pixel (1,0), etc.) that is mapped to the respective image may be maintained such that the image in the reference plane may be reconstructed with each pixel in the reference plane having a pixel value equal to the interpolated value of each pixel (determined in the reference plane). The interpolated representation of each image in the reference plane may then be used in various ways to identify features in the scene. For example, a computing device, such as a processing circuitry, may be configured to determine a difference between the interpolated representation of the respective image (e.g., the second image) in the reference plane and another one of the multiple images captured and mapped to the reference plane in order to identify features in the scene. In one example shown in FIG. 5C, an interpolated representation of the second image, shown in dashed lines, can be compared to the reference image, shown in solid lines, in the reference plane to detect movement of a feature (e.g., a cloud) in time between the capture of the reference image and the second image.
[0039]
[0044] Since the interpolated representation of the second image has been mapped to the reference plane 42, the representations of the reference image 50 and the second image 52 are aligned or better aligned. Thereby, the difference between the interpolated representation of the second image in the reference plane and the reference image has less noise, thereby providing for the identification of features 51 in the scene captured by the multiple images with improved reliability and accuracy. In this regard, by determining the difference between the aligned representations of the multiple images, the background or other parts of the common scene in both images are removed, whereby the resulting difference image includes any features that change from one image to the next. For example, objects such as airborne objects moving relative to the Earth 14 can be easily identified if the background is removed from the images.
[0040]
[0045] In another embodiment shown in FIG. 6, a first plurality of images 64 are captured at different points in time and space within a first period of time, and a second plurality of images 66 are captured at different points in time and space within a second period of time. In this embodiment, a computing device 22, such as processing circuitry 24, is configured to use the interpolated representations of the respective images to identify features in the scene (such as features moving relative to a background) by determining a difference between: (i) an average of the first plurality of images captured during the first period of time after being mapped to the reference plane 42; and (ii) an average of the second plurality of images captured during the second period of time after being mapped to the reference plane. In this embodiment, either the first plurality of images or the second plurality of images includes an interpolated representation of the respective images. Determining the difference between the average of the first plurality of images and the average of the second plurality of images mapped to the reference plane identifies features in the scene using the interpolated representations of the respective images. As a result of using an interpolated representation of each image mapped into the reference plane in the manner described above, the interpolated representation of each image in the reference plane is better aligned with other images that are also mapped into the reference plane, such that the average or sum of the aligned images results in reduced noise, increased accuracy and reliability of features that may be identified in the scene, such as in the manner described above.
[0041]
[0046] A method, computing device 22, and system 20 are provided for identifying features within a scene based on multiple images captured of the scene at different points in time and space. The method, computing device, and system of an embodiment may be configured to process multiple images and identify features within the scene in a real-time or near real-time manner. Additionally, by relying on image warping between a reference plane 42 and a sensor plane 44, the method, computing device, and system are configured to identify features within the scene, preferably in real-time or near real-time, with reduced noise and improved accuracy by providing not only warping but also rotation and translation.
[0042]
[0047] As mentioned above, FIG. 3 is a flowchart of a computing device 22 configured to process images to identify features in a scene, such as to provide object tracking, according to one embodiment, a method, and a computer program product. It should be understood that each block of the flowchart, and combinations of blocks in the flowchart, may be implemented by various means, such as hardware, firmware, processing circuitry 24, and / or other devices associated with the execution of software, including one or more computer program instructions. For example, one or more of the procedures described above may be embodied by computer program instructions. In this regard, computer program instructions embodying the procedures described above may be stored by memory device 26 of the computing device, executed by processing circuitry, and so on. As will be understood, any such computer program instructions may be loaded into a computer or other programmable apparatus (e.g., hardware) to generate a machine. The resulting computer or other programmable apparatus thereby implements the functions specified in the blocks of the flowchart. These computer program instructions may also be stored in a computer-readable memory, which may direct the computer or other programmable apparatus to function in a particular manner. The instructions stored in the computer readable memory thereby produce an article of manufacture, and execution of the instructions performs the functions specified in the flowcharts. Computer program instructions may also be loaded into a computer or other programmable device and cause the computer or other programmable device to execute a series of operations to generate a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide operations for implementing the functions specified in the flowchart blocks.
[0043]
[0048] Thus, the blocks of the flowcharts support combinations of means for performing the specified functions, and combinations of acts for performing the specified functions. It will also be understood that one or more blocks of the flowcharts, and combinations of blocks within the flowcharts, may be implemented by a dedicated hardware-based computer system that performs the specified functions, or a combination of dedicated hardware and computer instructions.
[0044]
[0049] In some embodiments, certain of the operations described above may be modified or further extended. Additionally, in some embodiments, additional optional operations may be included. Modifications, additions, or extensions of the operations described above may be made in any order and in any combination.
[0045]
[0050] Numerous modifications and other examples of the present disclosure will come to mind to one skilled in the art to which these inventions pertain having the benefit of the teachings presented in the foregoing description and the associated drawings. It is therefore to be understood that the invention is not limited to the specific embodiments disclosed, and that modifications and other embodiments are intended to be included within the scope of the appended claims. Moreover, while the foregoing description and the associated drawings describe exemplary embodiments in the context of certain exemplary combinations of elements and / or functions, it should be recognized that alternative embodiments may provide various combinations of elements and / or functions without departing from the scope of the appended claims. That is, other combinations of elements and / or functions than those expressly described above are contemplated, for example, as may be set forth in some of the appended claims. Although specific terms are employed herein, they are used in a generic and descriptive sense only and not for purposes of limitation.
[0046]
[0051] Furthermore, the present disclosure includes embodiments according to the following clauses:
[0047]
[0052] Article 1. 1. A method for identifying a feature in a scene, comprising: Obtaining a plurality of images captured at different points in time and different points in space; Defining a reference plane within a reference frame; applying image warping to map a plurality of reference points of each image of the plurality of images in a sensor plane to corresponding reference points of the reference plane; mapping pixel coordinates of a plurality of pixels of the reference plane to the respective images based on a positional relationship between the reference plane and a representation of the respective images established by applying the image warping to the respective images; interpolating from pixels of the respective images to pixels at the pixel coordinates of the reference plane to generate an interpolated representation of the respective images; and using the interpolated representation of each of the images to identify the feature within the scene.
[0048]
[0053] Article 2. 2. The method of claim 1, wherein applying the image warping includes applying at least one of a polynomial transformation of degree 1.5 or higher, a bilinear transformation, or a projective transformation to each of the images to simultaneously provide rotation, translation, and warping of the respective images.
[0049] Article 3. 2. The method of claim 1, wherein interpolating comprises performing bilinear interpolation from the pixels of the respective images in the sensor plane to the pixel coordinates of the reference plane mapped to the sensor plane.
[0050] Article 4. 2. The method of claim 1, wherein the reference plane is perpendicular to a boresight of a sensor that captures the multiple images.
[0051] Article 5. 13. The method of claim 1, wherein defining the reference plane includes defining the reference plane based on mapping another one of the plurality of images to the reference plane.
[0052] Article 6. The method of claim 5, wherein using the interpolated representation of the respective image to identify the feature in the scene includes determining a difference between the interpolated representation of the respective image and another one of the plurality of images mapped to the reference plane to identify the feature in the scene.
[0053] Article 7. The method of claim 1, wherein using the interpolated representation of the respective image to identify the feature within the scene includes determining a difference between (i) the interpolated representation of an average of the plurality of images mapped to the reference plane, the plurality of images including the interpolated representation of the respective image, captured during a first period of time, and (ii) an average of a second plurality of images, also mapped to the reference plane, the second plurality of images captured during a second period of time.
[0054] Article 8. 1. A computing device configured to identify features in a scene, comprising: Obtaining a plurality of images captured at different points in time and different points in space; Defining a reference plane within a reference frame; applying image warping to map a plurality of reference points of each image of the plurality of images in a sensor plane to corresponding reference points of the reference plane; mapping pixel coordinates of a plurality of pixels of the reference plane to the respective images based on a positional relationship between the reference plane and a representation of the respective images established by applying the image warping to the respective images; interpolating from pixels of the respective images to pixels at the pixel coordinates of the reference plane to generate an interpolated representation of the respective images; and 9. The computing device of claim 8, further comprising a processing circuit configured to perform the steps of: using the interpolated representation of the respective image to identify the feature within the scene.
[0055] Article 9. The computing device of clause 8, wherein the processing circuitry is configured to apply the image warping by applying at least one of a polynomial transformation of degree 1.5 or higher, a bilinear transformation, or a projective transformation to the respective images to simultaneously provide rotation, translation, and warping of the respective images.
[0056] Article 10. The computing device of clause 8, wherein the processing circuitry is configured to interpolate by performing bilinear interpolation from the pixels of the respective images in the sensor plane to the pixel coordinates of the reference plane mapped to the sensor plane.
[0057] Article 11. 9. The computing device of claim 8, wherein the reference plane is perpendicular to a boresight of a sensor that captures the multiple images.
[0058] Article 12. 9. The computing device of claim 8, wherein the reference plane is tangential to the surface of the Earth.
[0059] Article 13. The computing device of clause 8, wherein the processing circuitry is configured to define the reference plane by defining the reference plane based on mapping another one of the plurality of images to the reference plane.
[0060] Article 14. The computing device of claim 13, wherein the processing circuitry is configured to use the interpolated representation of the respective image to identify the feature within the scene by determining a difference between the interpolated representation of the respective image and another one of the plurality of images mapped to the reference plane to identify the feature within the scene.
[0061] Article 15. The computing device of clause 8, wherein the processing circuitry is configured to use the interpolated representation of the respective images to identify the feature within the scene by determining a difference between (i) the interpolated representation of an average of the plurality of images mapped to the reference plane, the average of the plurality of images captured during a first period of time, the plurality of images including the interpolated representation of the respective images, and (ii) an average of a second plurality of images also mapped to the reference plane, the second plurality of images captured during a second period of time.
[0062] Article 16. 1. A system configured to identify a feature in a scene, comprising: at least one sensor configured to capture a plurality of images at different points in time and different points in space; and A processing circuit is provided, the processing circuit comprising: obtaining a plurality of images captured at different points in time and at different points in space; Defining a reference plane within a reference frame; applying image warping to map a plurality of reference points of each image of the plurality of images in a sensor plane to corresponding reference points of the reference plane; mapping pixel coordinates of a plurality of pixels of the reference plane to the respective images based on a positional relationship between the reference plane and a representation of the respective images established by applying the image warping to the respective images; interpolating from pixels of the respective images to pixels at the pixel coordinates of the reference plane to generate an interpolated representation of the respective images; and using the interpolated representation of each of the images to identify the feature within the scene.
[0063] Article 17. 17. The system of claim 16, wherein the processing circuitry is configured to apply the image warping by applying at least one of a polynomial transformation of degree 1.5 or higher, a bilinear transformation, or a projective transformation to the respective images to simultaneously provide rotation, translation, and warping of the respective images.
[0064] Article 18. 17. The system of claim 16, wherein the processing circuitry is configured to interpolate by performing bilinear interpolation from the pixels of the respective images in the sensor plane to the pixel coordinates of the reference plane mapped to the sensor plane.
[0065] Article 19. The system of claim 16, wherein the processing circuitry is configured to define the reference plane by defining the reference plane based on mapping another one of the plurality of images to the reference plane.
[0066] Article 20. The system of claim 19, wherein the processing circuitry is configured to use the interpolated representation of the respective image to identify the feature within the scene by determining a difference between the interpolated representation of the respective image and another one of the plurality of images mapped to the reference plane to identify the feature within the scene.
Claims
1. A method for identifying a feature (51) in a scene, comprising: Obtaining (30) a plurality of images (64, 66) captured at different points in time and different points in space; Defining (32) a reference plane (50) within a reference frame (42); applying image warping (34) to map a plurality of reference points of each image of the plurality of images in a sensor plane (44) to corresponding reference points of the reference plane (50); Mapping (36) pixel coordinates of a plurality of pixels of the reference surface (50) to the respective images based on a positional relationship between the reference surface (50) and a representation of the respective images established by applying the image warping to the respective images; Interpolating (38) from pixels of the respective images to pixels at the pixel coordinates of the reference surface (50) to generate an interpolated representation of the respective images; and using (40) the interpolated representation of each of the images to identify the feature (51) within the scene.
2. 2. The method of claim 1, wherein applying image warping (34) comprises applying at least one of a polynomial transformation of degree 1.5 or higher, a bilinear transformation, or a projective transformation to the respective images to simultaneously provide rotation, translation, and warping of the respective images.
3. 2. The method of claim 1, wherein interpolating comprises performing a bilinear interpolation from the pixels of the respective images in the sensor plane to the pixel coordinates of the reference plane mapped to the sensor plane.
4. 2. The method of claim 1, wherein using (40) the interpolated representation of the respective image to identify the feature (51) within the scene comprises determining a difference between (i) the interpolated representation of an average of the plurality of images (64) mapped to the reference plane (50) that includes the interpolated representation of the respective image, the plurality of images (64) captured during a first period of time, and (ii) an average of a second plurality of images (66) also mapped to the reference plane (50), the second plurality of images (66) captured during a second period of time.
5. The method of any one of claims 1 to 4, wherein the reference plane (50) is perpendicular to a boresight (16) of a sensor (12) that captures the multiple images (64, 66).
6. 5. The method of claim 1, wherein defining (32) the reference plane (50) comprises defining the reference plane (50) based on mapping another one of the plurality of images onto the reference plane (50).
7. A computing device (22) configured to identify features (51) in a scene, comprising: Obtaining a plurality of images (64, 66) captured at different points in time and different points in space; Defining a reference plane (50) within a reference frame (42); applying image warping to map a plurality of reference points of each image of the plurality of images in a sensor plane (44) to corresponding reference points of the reference plane (50); mapping pixel coordinates of a plurality of pixels of said reference surface (50) to said respective images based on a positional relationship between said reference surface (50) and a representation of said respective images established by applying said image warping to said respective images; interpolating from pixels of the respective images to pixels at the pixel coordinates of the reference surface (50) to generate an interpolated representation of the respective images; and using the interpolated representation of each of the images to identify the features (51) within the scene.
8. 8. The computing device of claim 7, wherein the processing circuitry is configured to apply the image warping by applying at least one of a polynomial transformation of degree 1.5 or higher, a bilinear transformation, or a projective transformation to the respective images to simultaneously provide rotation, translation, and warping of the respective images.
9. 9. The computing device (22) of claim 7 or 8, wherein the processing circuit (24) is configured to interpolate by performing bilinear interpolation from the pixels of the respective images in the sensor plane (44) to the pixel coordinates of the reference plane (50) mapped to the sensor plane (44).
10. A system (20) configured to identify features (51) in a scene, comprising: at least one sensor (12) configured to capture a plurality of images (64, 66) at different points in time and different points in space; A processing circuit (24) is provided, the processing circuit (24) comprising: obtaining a plurality of images captured at different points in time and at different points in space; Defining a reference plane (50) within a reference frame (42); applying image warping to map a plurality of reference points of each image of the plurality of images in a sensor plane (44) to corresponding reference points of the reference plane (50); mapping pixel coordinates of a plurality of pixels of said reference surface (50) to said respective images based on a positional relationship between said reference surface (50) and a representation of said respective images established by applying said image warping to said respective images; interpolating from pixels of the respective images to pixels at the pixel coordinates of the reference surface (50) to generate an interpolated representation of the respective images; and using the interpolated representation of each of the images to identify the feature (51) within the scene.