System and method for generating a colored image of a scene

Abstract

1. A system for colorizing a monochromatic image of a scene, the system comprising: a processing circuit configured to: capture, using a non-visible spectrum sensor capable of capturing a monochromatic image from a viewpoint having a given position and a given orientation, the monochromatic image of the scene; acquire one or more previously captured color images covering at least a portion of the scene, the previously captured color images being captured at a time prior to capturing the monochromatic image; determine an alignment between the monochromatic image and the previously captured color images, the alignment being determined by projecting the previously captured color image onto a plane that is isometric with the viewpoint; render one or more rendered images, the previously captured color images being aligned to the viewpoint using the alignment; and generate a colored image of the scene by, for at least one given pixel of the monochromatic image, modifying values of one or more chroma components of the given pixel according to values of one or more chroma components of a corresponding pixel in the rendered image.

Description

[Technical Field]

[0001] The present invention relates to a system and method for generating a colored image of a scene based on the luma component of a monochrome image of the scene and the chroma component of a previously captured color image of at least a portion of the scene. [Background technology]

[0002] Non-visible spectrum sensors, such as night vision systems, are based on non-visible spectrum sensors, such as infrared (IR) sensors, short-wave infrared (SWIR) sensors, or low-light-level television (LLTV) sensors. These systems provide monochromatic images of the captured scene. Users of such non-visible spectrum sensors are limited in their ability to understand what is happening in the scene due to the lack of color imagery. For example, when aircraft use such non-visible spectrum sensors, they provide the aircraft pilot with monochromatic video of the scene captured from the aircraft's sensors. Situational awareness is a demanding task for already overworked pilots, and the provided monochromatic video is insufficient. Providing color video can dramatically increase the pilot's situation awareness capability.

[0003] In many cases, it is possible to access previously captured color images of at least a portion of the same scene currently being imaged by a non-visible spectrum sensor. Current systems do not utilize these previously captured color images to apply color to the monochrome image and to generate a colored image of the scene. When current systems utilize these previously captured color images, they do not combine those previously captured color images with the real-time information captured by the non-visible spectrum sensor.

[0004] The previously captured color image and the non-visible spectrum sensor image each have advantages and disadvantages. The previously captured color image has better visibility and easier situational awareness support, but lacks current details and element information captured during the current mission. The non-visible spectrum sensor image contains current, updated details captured during the current mission, but has lower visibility and situational awareness. Therefore, it is necessary to generate a fused image, which is a color image containing updated details and element information from the current mission. Summary of the Invention [Problem to be solved by the invention]

[0005] Therefore, there is a need in the art for new methods and systems for generating a colored image of a scene based on the luma component of a monochrome image of the scene and the chroma components of a previously captured color image of at least a portion of the scene. [Means for solving the problem]

[0006] According to a first aspect of the subject matter of the present disclosure, there is provided a system for colorizing a monochromatic image of a scene, the system comprising: a processing circuit configured to: capture, using a non-visible spectrum sensor capable of capturing the monochromatic image from a viewpoint having a given position and a given orientation, the monochromatic image of the scene; acquire one or more previously captured color images covering at least a portion of the scene, the previously captured color images being captured at a time prior to capturing the monochromatic image; determine a registration between the monochromatic image and the previously captured color images, the registration being determined by projecting the previously captured color image onto a plane that is isometric with the viewpoint; render one or more rendered images, the previously captured color images being aligned to the viewpoint using the registration; and generate a colored image of the scene by, for at least one given pixel of the monochromatic image, modifying values of one or more chroma components of the given pixel according to values of one or more chroma components of a corresponding pixel in the rendered image.

[0007] In some cases, the monochrome image includes one or more imaging elements that are not included in one or more previously captured color images, and the generated colorized image includes the imaging elements.

[0008] In some cases, the alignment between the monochromatic image and the previously captured color image is also determined based on a first geo-alignment of the monochromatic image to the Earth coordinate system and a second geo-alignment of the previously captured color image, and generation of the colorized image utilizes the first geo-alignment and the second geo-alignment to register pixels of the monochromatic image with corresponding pixels of the previously captured color image.

[0009] In some cases, the alignment between the monochrome image and the previously captured color image is determined by projecting the previously captured color image onto a plane that is equiangular to the viewpoint.

[0010] In some cases, the non-visible spectrum sensor is a night vision sensor.

[0011] In some cases, the non-visible spectrum sensor is an airborne sensor.

[0012] In some cases, the colored image of the scene is in YCbCr format.

[0013] In some cases, the colored image of the scene is in HSV format.

[0014] According to a second aspect of the subject matter of the present disclosure, there is provided a method for colorizing a monochromatic image of a scene, the method including: capturing, by a processing circuit, a monochromatic image of the scene using a non-visible spectrum sensor capable of capturing monochromatic images from a viewpoint having a given position and a given orientation; acquiring, by the processing circuit, one or more previously captured color images covering at least a portion of the scene, the previously captured color images being captured at a time prior to capturing the monochromatic image; determining, by the processing circuit, an alignment between the monochromatic image and the previously captured color images, the alignment being determined by projecting the previously captured color image onto a plane that is isometric with the viewpoint; rendering one or more rendered images, the previously captured color images aligned to the viewpoint using the alignment; and generating, by the processing circuit, a colored image of the scene by modifying, for at least one given pixel of the monochromatic image, values of one or more chroma components of the given pixel according to values of one or more chroma components of a corresponding pixel in the rendered image.

[0015] In some cases, the monochrome image includes one or more imaging elements that are not included in one or more previously captured color images, and the generated colorized image includes the imaging elements.

[0016] In some cases, the alignment between the monochromatic image and the previously captured color image is also determined based on a first geo-alignment of the monochromatic image to an Earth coordinate system and a second geo-alignment of the previously captured color image, and generation of the colorized image utilizes the first geo-alignment and the second geo-alignment to align pixels of the monochromatic image with corresponding pixels of the previously captured color image.

[0017] In some cases, the alignment between the monochrome image and the previously captured color image is determined by projecting the previously captured color image onto a plane that is equiangular to the viewpoint.

[0018] In some cases, the non-visible spectrum sensor is a night vision sensor.

[0019] In some cases, the non-visible spectrum sensor is an airborne sensor.

[0020] In some cases, the colored image of the scene is in YCbCr format.

[0021] In some cases, the colored image of the scene is in HSV format.

[0022] According to a second aspect of the subject matter of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer-readable program code embodied thereon, the computer-readable program code being executable by at least one processor of a computer to perform a method for detecting movement of at least one non-line-of-sight object, the method comprising: capturing, by a processing circuit, a monochromatic image of a scene utilizing a non-visible spectrum sensor capable of capturing the monochromatic image from a viewpoint having a given position and a given orientation; and obtaining, by the processing circuit, one or more previously captured color images covering at least a portion of the scene, the previously captured color images being captured at a time prior to capturing the monochromatic image. A non-transitory computer-readable storage medium is provided that includes: acquiring a captured monochrome image and a previously captured color image, determining, by a processing circuit, an alignment between the monochrome image and a previously captured color image, where the alignment is determined by projecting the previously captured color image onto a plane that is equiangular to the viewpoint; rendering one or more rendered images, the previously captured color image aligned to the viewpoint by utilizing the alignment; and generating a colored image of the scene by, for at least one given pixel of the monochrome image, modifying, by the processing circuit, values of one or more chroma components of the given pixel according to values of one or more chroma components of a corresponding pixel of the rendered image.

[0023] In order to understand the subject matter of the present disclosure and to see how it may be carried out in practice, the subject matter will now be described, by way of non-limiting example only, with reference to the accompanying drawings, in which: [Brief explanation of the drawings]

[0024] [Figure 1]FIG. 1 is a block diagram that schematically illustrates an example of a system for generating a colored image of a scene based on a luma component of a monochrome image of the scene and a chroma component of a previously captured color image of at least a portion of the scene, in accordance with the subject matter of this disclosure. [Figure 2] 1 is a flowchart illustrating an example sequence of operations performed to generate a colored image of a scene based on a luma component of a monochrome image of the scene and a chroma component of a previously captured color image of at least a portion of the scene, in accordance with the subject matter of this disclosure. [Figure 3] FIG. 1 is a block diagram illustrating an example of generating a colored image of a scene based on a monochrome image of the scene at viewpoint A and chroma components of a color image captured before at least a portion of the scene rendered at viewpoint A, in accordance with the subject matter of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0025] In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the subject matter of the present disclosure. However, it will be understood by those skilled in the art that the subject matter of the present disclosure may be practiced without these specific details. In other instances, well-known methods, procedures, and components have not been described in detail so as not to obscure the subject matter of the present disclosure.

[0026] In the drawings and written description, the same reference numbers indicate components that are common to different embodiments or configurations.

[0027] Unless otherwise specified, as will be apparent from the following description, it will be appreciated that the use of terms such as "generating," "formatting," "determining," "capturing," "performing," "updating," "transmitting," "receiving," and the like throughout the description herein includes computer actions and / or processes that manipulate and / or transform data into other data that is represented as a physical quantity, e.g., an electronic quantity, and / or that represents a physical object. The terms "computer," "processor," "processing resource," "processing circuitry," and "controller" should be interpreted broadly to cover any type of electronic device with data processing capability, including, by way of non-limiting example, personal desktop / laptop computers, servers, computing systems, communications devices, smartphones, tablet computers, smart televisions, processors (e.g., digital signal processors (DSPs), microcontrollers, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), etc.), groups of multiple physical machines that share the performance of various tasks, virtual servers co-resident on a single physical machine, any other electronic computing device, and / or any combination thereof.

[0028] Operations according to the teachings herein may be performed by a specially constructed computer for the desired purpose, or by a general-purpose computer specially configured for the desired purpose by a computer program stored on a non-transitory computer-readable storage medium. The term "non-transitory" is used herein to exclude transitory propagating signals, but to include any volatile or non-volatile computer memory technology, as the case may be, suitable for the application.

[0029] As used herein, the phrases "for example," "such as," "for instance," and variations thereof describe non-limiting examples of the presently disclosed subject matter. Reference herein to "one case," "some cases," "other cases," or variations thereof means that a particular feature, structure, or characteristic described with respect to one or more examples is included in at least one example of the presently disclosed subject matter. Thus, appearances of the phrases "one case," "some cases," "other cases," or variations thereof do not necessarily refer to the same example(s).

[0030] It will be appreciated that, unless otherwise stated, certain features of the presently disclosed subject matter, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the presently disclosed subject matter, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination.

[0031] In embodiments of the presently disclosed subject matter, fewer, more, and / or different steps than those shown in FIG. 2 may be performed. In embodiments of the presently disclosed subject matter, one or more steps shown in FIG. 2 may be performed in a different order and / or one or more groups of steps may be performed simultaneously. FIG. 1 shows a general schematic diagram of a system architecture according to one embodiment of the presently disclosed subject matter. Each module in FIG. 1 may be comprised of any combination of software, hardware, and / or firmware that performs the functions defined and described herein. The modules in FIG. 1 may be centralized in one location or distributed across two or more locations. In other embodiments of the presently disclosed subject matter, a system may include fewer, more, and / or different modules than those shown in FIG. 1.

[0032] References in this specification to methods should apply mutatis mutandis to systems capable of carrying out the methods, and should apply mutatis mutandis to non-transitory computer-readable media storing instructions that, when executed by a computer, result in the performance of the methods.

[0033] References herein to a system should apply mutatis mutandis to methods that may be performed by the system, and should apply mutatis mutandis to non-transitory computer-readable media that store instructions that may be executed by the system.

[0034] References in this specification to non-transitory computer readable media should apply mutatis mutandis to a system capable of executing instructions stored on the non-transitory computer readable media, and should apply mutatis mutandis to a method that can be executed by a computer reading instructions stored on the non-transitory computer readable media.

[0035] With this in mind, attention is directed to FIG. 1 , which is a block diagram that schematically illustrates one example of a system for generating a colored image of a scene based on the luma component of a monochrome image of the scene and the chroma component of a previously captured color image of at least a portion of the scene, in accordance with the subject matter of this disclosure.

[0036] System 200 may be part of a non-visible spectrum sensor, such as an IR sensor, a SWIR sensor, or an LLTV sensor, capable of capturing monochromatic images from a viewpoint having a given position and a given orientation. In some cases, system 200 is installed on a platform that carries the non-visible spectrum sensor (such as an ariel platform). In some cases, system 200 is distributed among several locations. For example, part of system 200 may be installed on the platform, and part of system 200 may be installed on a ground station that is in communication with the platform.

[0037] System 200 may comprise or otherwise be associated with a data repository 210 (e.g., a database, a storage system, memory including read-only memory (ROM), random access memory (RAM), or any other type of memory, etc.) configured to store data including, among other things, a monochromatic image of a scene, a previously captured color image covering at least a portion of the scene, the luma components of pixels of the monochromatic image, the chroma components of pixels of the previously captured color image, the position and orientation of the sensor when capturing the monochromatic image of the scene, etc. In some cases, data repository 210 may be further configured to enable retrieval and / or update and / or deletion of data stored thereon. Note that in some cases, data repository 210 may be local, and in other cases, it may be distributed. Note that in some cases, data repository 210 may be stored on cloud-based storage.

[0038] System 200 may further include a network interface 220 that allows system 200 to connect to a communications network and to send and receive data, such as a previously captured color image covering at least a portion of a scene. In some cases, network interface 220 may be connected to a local area network (LAN), a wide area network (WAN), or the Internet. In some cases, network interface 220 may be connected to a wireless network.

[0039] System 200 further comprises processing circuitry 230. Processing circuitry 230 may be one or more processing circuitry units (e.g., central processing units), microprocessors, microcontrollers (e.g., microcontroller units (MCUs)), including multiple and / or parallel and / or distributed processing circuitry units, or any other computing device or module adapted to process data, independently or cooperatively, to control and enable operations related to associated system 200 resources.

[0040] The processing circuit 230 comprises the following modules: a colored image generation module 240 .

[0041] The colored image generation module 240 may be configured to perform a colored image generation process as further detailed herein, with particular reference to FIG.

[0042] FIG. 2 is a flowchart illustrating an example sequence of operations performed to generate a colored image of a scene based on the luma component of a monochrome image of the scene and the chroma component of a previously captured color image of at least a portion of the scene, in accordance with the subject matter of this disclosure.

[0043] According to some examples of the subject matter of this disclosure, system 200 may be configured, for example, using color image generation module 240, to perform color image generation process 300, which includes activities for generating a color image of a scene by fusing a currently captured monochromatic image of a scene, which includes current and updated details of the scene but is monochromatic, with a previously captured color image of the scene that lacks the current and / or updated details of the scene but may have the visibility and ease of situational awareness associated with the color image. System 200 is thereby a system for colorizing a currently captured monochromatic image based on a previously captured color image. The color image of the scene may be generated based on a monochromatic image of the scene captured using a non-visible spectrum sensor capable of capturing a monochromatic image from a viewpoint having a given position and a given orientation. In some cases, system 200 is airborne. In a non-limiting example, the non-visible spectrum sensor is installed on an aerial platform, and the monochromatic image may be captured during a given flight mission. The position and orientation of the airborne platform and the position and orientation of the non-visible spectrum sensor can be utilized to determine the given position and orientation at which the monochromatic image was captured. System 200 can use a previously captured color image of at least a portion of the scene. The previously captured color image, for example, was captured during a previous mission prior to the given flight mission. In other cases, the previously captured color image was captured at an earlier time relative to the capture time of the monochromatic image. System 200 can have access to the position and orientation at which the previously captured color image was captured.

[0044] Previously captured color images may be stored in a three-dimensional database of color images or in a regional color database (e.g., an orthophoto database). In some cases, these color images are rendered from the same location and orientation of an airborne platform and placed on the terrain. These images may also be generated from a Digital Terrain Elevation Data (DTED) database. This rendering creates a set of color images that are conformal to the monochromatic images captured by the non-visible spectrum sensor. Previously captured color images may be stored on the non-visible spectrum sensor, on the airborne platform, and / or on a ground station and / or any location with which system 200 can communicate. Previously captured color images may be used by system 200 to apply color to the monochromatic images. System 200 may use the previously captured color images to set the chroma components of the pixels of the monochromatic images. In a non-limiting example, system 200 renders a conformal color image of at least a portion of a scene based on the previously captured color images. The system 200 can generate a colored image of a scene by preserving the luma component in some of the pixels of the monochrome image and modifying the chroma components of the pixels of the monochrome image according to the values of one or more chroma components of the corresponding pixels of the rendered conformal color image.

[0045] To this end, system 200 may be configured to capture (block 310) a monochromatic image of a scene using a non-visible spectrum sensor capable of capturing the monochromatic image from a viewpoint having a given position and a given orientation. In some cases, the monochromatic image of the scene is captured by the non-visible spectrum sensor in real time during a current mission. Continuing with a non-limiting example, the non-visible spectrum sensor may be installed on an airborne platform (e.g., an airborne sensor), and the monochromatic image is captured during a given flight mission. In some cases, the non-visible spectrum sensor is a night vision sensor.

[0046] After capturing the monochromatic image, system 200 may be further configured to acquire (block 320) one or more previously captured color images covering at least a portion of the scene, the previously captured color images being captured at a time prior to capturing the monochromatic image. The previously captured color images may, for example, have been captured during a previous mission prior to the given flight mission. In other cases, the previously captured color images were captured at an earlier time relative to the capture time of the monochromatic image. System 200 may have access to the location and orientation at which the previously captured color images were captured.

[0047] In some cases, the monochromatic image includes one or more imaging elements that were not included in one or more previously captured color images (e.g., a tree was captured in the monochromatic image of the scene, but the same tree was not captured in the previously captured color image because, e.g., it was not present when the previously captured color image was captured). Note that in these cases, the generated color image (see block 350 below) based on the monochromatic image and the previously captured color image includes the imaging elements, and thus the reality of the current mission as captured by the non-visible spectrum sensor is shown in the generated color image.

[0048] Once system 200 has obtained one or more previously captured color images, system 200 may be further configured to determine (block 330) an alignment between the monochrome image and the previously captured color image, where the alignment is determined by projecting the previously captured color image onto a plane that is isotropic to the viewpoint.

[0049] In some cases, the alignment between the monochromatic image and the previously captured color image is determined based on a first geo-alignment of the monochromatic image to an Earth coordinate system and a second geo-alignment of the previously captured color image, and the generation of the colorized image utilizes the first geo-alignment and the second geo-alignment to align pixels of the monochromatic image with corresponding pixels of the previously captured color image. As a non-limiting example, the monochromatic image is captured during a current mission by utilizing a non-visible spectrum sensor from a viewpoint having a given location and a given orientation. The given location and the given orientation determine the first geo-alignment of the monochromatic image to an Earth coordinate system. The previously captured colorized image was captured on a previous mission by utilizing a color sensor from a different viewpoint having a different location and a different orientation. The different location and the different orientation determine the second geo-alignment of the previously captured color image. The generation of the colored image utilizes the first geo-alignment and the second geo-alignment to align pixels of the monochromatic image with corresponding pixels of the previously captured color image, for example, by projecting the monochromatic image into Earth coordinates (using the first geo-alignment) and projecting the previously captured color image into Earth coordinates (using the first geo-alignment).

[0050] In some cases, the alignment between the monochromatic image and a previously captured color image is determined by projecting the previously captured color image onto a plane that is conformal to the viewpoint. Continuing with a non-limiting example, the monochromatic image is captured during a current mission by utilizing a non-visible spectrum sensor from a viewpoint having a given position and a given orientation. The given position and orientation can be used to determine a plane that is conformal to the given position and orientation. The previously captured colorized images were captured on a previous mission by utilizing a color sensor from a different viewpoint having a different position and a different orientation. These previously captured colorized images can be projected onto the determined plane, thereby defining the alignment between the monochromatic image and the previously captured colorized image.

[0051] System 200 may further be configured to render (block 340) one or more rendered images that are previously captured color images aligned to the viewpoint by utilizing the alignment. System 200 renders the previously captured color images to the viewpoint of the monochrome image by utilizing the alignment determined in block 330, as further detailed herein, particularly with reference to FIG.

[0052] The system 200 may further be configured to generate (block 350) a colored image of the scene by modifying, for at least one given pixel of the monochrome image, the value of one or more chroma components of the given pixel according to the value of one or more chroma components of a corresponding pixel of the rendered image. Non-limiting examples of algorithms for generating a color image of a scene may include (i) reading a monochrome image in a Red, Green, and Blue (RGB) format, e.g., captured using a non-visible spectrum sensor during a current mission; (ii) reading a previously captured color image in an RGB format, e.g., captured by a color image sensor during a previous mission; (iii) converting the previously captured color image from RGB format to a Hue, Saturation, and Value (HSV) format or to a YCbCr format; (iv) blending the converted previously captured color image by replacing the luma component of each pixel of the converted previously captured color image with the luma component of the corresponding pixel of the monochrome image, resulting in a color image of the scene in HSV format or from a YCbCr format; and (v) converting the color image of the scene from HSV format or from a YCbCr format to an RGB format, resulting in a color image of the scene in RGB format. This algorithm for the generation of colored images achieves preservation of the ability to discern details, as in the case of the previously captured colored image, compared to the monochromatic image, while avoiding the presentation of elements that are only present in the previously captured colored image and not in the monochromatic image (thus avoiding "ghosting" on the generated colored image).

[0053] The system 200 can repeat the above process for a series of monochrome images, in this case creating a color video of the scene.

[0054] The generated color image may be presented to a user of system 200 (e.g., a pilot of an airborne platform). The user receives a color visual image of the scene during the mission without losing the ability to compare it with the monochrome image and discern details in the color image. Furthermore, imaging elements that were present only in the previously captured color image but not in the monochrome image are not shown in the generated color image, thereby preventing "ghost" elements in the generated color image, i.e., imaging elements that were in the scene in the past (at the time of capturing the previously captured color image) from being shown, even though they are not present at the time of capturing the monochrome image during the current mission.

[0055] It should be noted that in some cases, the generated colored image of the scene is in YCbCr format. In these cases, the generation of the colored image is performed by preserving the Y component (luma component) of the pixel of the monochrome image and modifying the Cb and Cr components (chroma components) of the pixel of the monochrome image according to the chroma components of the corresponding pixel of the previously captured color image. In some cases, a hue, saturation, and value (HSV) method is used to generate the colored image of the scene. In these cases, the generation of the colored image is performed by preserving the H component (luma component) of the pixel of the monochrome image and modifying the S and V components (chroma components) of the pixel of the monochrome image according to the chroma components of the corresponding pixel of the previously captured color image.

[0056] With respect to Figure 2, it should be noted that some of the blocks may be combined into a grouped block or broken down into several blocks, and / or other blocks may be added. Furthermore, in some cases, the blocks may be performed in a different order than described herein. Furthermore, it should be noted that some of the blocks are optional. It should also be noted that while the flow charts are described in terms of the system elements that implement them, this is in no way binding, and the blocks may be performed by elements other than those described herein.

[0057] With this in mind, attention is directed to FIG. 3, a block diagram that schematically illustrates one example of generating a colored image of a scene based on a monochromatic image of the scene at viewpoint A and chroma components of a previously captured color image of at least a portion of the scene rendered at viewpoint A, in accordance with the subject matter of this disclosure.

[0058] As detailed above, with respect to FIG. 2 , system 200 can be configured to perform color image generation process 300. This process involves the activities required to generate a color image of a scene by fusing a currently captured monochromatic image of the scene, which includes current and updated details of the scene (but is monochromatic), with a previously captured color image of the scene, which lacks the current and / or updated details of the scene but may have the visibility and ease of situational awareness associated with a color image. To this end, system 200 can be configured to generate a monochromatic image at the viewpoint from which the monochromatic image was captured (label this viewpoint as “Viewpoint A”), where the monochromatic image is colored with the chroma components of the pixels of the previously captured color image rendered to Viewpoint A. System 200 captures a monochromatic image 410 at Viewpoint A. The monochromatic image 410 at Viewpoint A lacks color but has all the details of the real-time current situation. In a non-limiting example, the monochromatic image 410 at viewpoint A may be captured by an airborne non-visible spectrum sensor capturing one or more aerial objects that are part of the real-time scene as part of the monochromatic image 410 at viewpoint A. The system 200 acquires one or more previously captured color images 420 at viewpoint B. The one or more previously captured color images 420 at viewpoint B cover at least a portion of the scene, the previously captured color images being captured at a time prior to capturing the monochromatic image 410 at viewpoint A. Viewpoint B may differ from viewpoint A; therefore, the system 200 renders the previously captured color image 420 at viewpoint B to viewpoint A, thereby generating the previously captured color image 430 rendered at viewpoint A. The rendering may be performed by utilizing the alignment between viewpoint A and viewpoint B, where the alignment is determined by projecting the previously captured color image 420 at viewpoint B onto a plane that is isometric with viewpoint A.In some cases, the alignment between the monochromatic image and the previously captured color image is determined based on a first geo-alignment of the monochromatic image to an Earth coordinate system and a second geo-alignment of the previously captured color image, and the generation of the colorized image utilizes the first geo-alignment and the second geo-alignment to align pixels of the monochromatic image 410 at viewpoint A with corresponding chroma components 440 of pixels of the rendered previously captured color image at viewpoint A. As a non-limiting example, the monochromatic image is captured during a current mission by utilizing a non-visible spectrum sensor from a viewpoint having a given location and a given orientation. The given location and the given orientation determine the first geo-alignment of the monochromatic image to an Earth coordinate system. The previously captured colorized image was captured on a previous mission by utilizing a color sensor from a different viewpoint having a different location and a different orientation. The different location and the different orientation determine the second geo-alignment of the previously captured color image. The generation of the color image utilizes the first geo-registration and / or the second geo-registration to align pixels of the monochromatic image with corresponding pixels of the previously captured color image, for example, by projecting the monochromatic image into Earth coordinates (using the first geo-registration) and projecting the previously captured color image into Earth coordinates (using the first geo-registration). In some cases, the alignment between the monochromatic image and the previously captured color image is determined by projecting the previously captured color image onto a plane that is conformal to the viewpoint. Continuing with a non-limiting example, the monochromatic image is captured during a current mission using a non-visible spectrum sensor from viewpoint A having a given location and a given orientation. The given location and orientation can be used to determine a plane that is conformal to the given location and orientation. The previously captured color image was captured on a previous mission using a color sensor from another viewpoint B having a different location and a different orientation. These previously captured color images are projected onto the determined plane, thereby defining a registration between the monochrome image and the previously captured color image.The alignment may be used by system 200 to render a previously captured color image 430 rendered at viewpoint A. System 200 may then colorize monochromatic image 410 at viewpoint A using chroma components 440 of the pixels of the previously captured color image rendered at viewpoint A. The colorization may be pixel-by-pixel. The result is a monochromatic image 450 at viewpoint A colorized with the chroma components of the pixels of the previously captured color image rendered at viewpoint A. The use of chroma components preserves the objects of the original monochromatic image 410 at viewpoint A. In this example, aerial objects that were captured as part of the real-time scene in monochromatic image 410 at viewpoint A but were not captured in previously captured color image 420 at viewpoint B will not disappear from colorized image 450, although the aerial objects may receive unrealistic color due to the use of only the chroma components of the previously captured color image. Furthermore, past objects captured in the previously captured color image 420 at viewpoint B that are not present in the monochrome image 410 at viewpoint A will not be seen in the colorized image 450 because the luma components of the pixels of the previously captured color image are not used. Continuing with the example, aerial objects captured in the previously captured color image but that are not present in the monochrome image 410 at viewpoint A will also not be present in the resulting colorized image 450.

[0059] The described colored image generation process 300 used by the system 200 has the following advantages:

[0060] Preserving Visibility of Objects in the Live and / or Real-Time Monochrome Image 410: The live and / or real-time monochromatic image 410 is the primary motivation for the colored image generation process 300. The colored image generation process 300 preserves the visibility of all objects and details present in the live and / or real-time monochromatic image 410, even if they are not part of the previously captured color image. In contrast, object recognition and / or artificial intelligence (AI)-based colorization methods may add non-existent objects to the live and / or real-time image.

[0061] Pixel-Based Process. The color image generation process 300 is pixel-by-pixel, where the live and / or real-time monochrome image 410 from the monochrome sensor is not manipulated or used as the primary element. The format of the previously captured color image 420 can be converted to a format such as HSV, YCbCr, etc., which has a parameter for pixel intensity ("Y" in YCbCr format, "V" in HSV) and different parameters for pixel color ("Cb" and "Cr" in YCbCr format, "H" and "S" in HSV). Then, for each pixel, the intensity parameter is replaced with the corresponding intensity parameter in the live and / or real-time monochrome image 410. Because lighting intensity is the most dominant factor, the color image maintains live image visibility, detail, and the ability to distinguish details. As long as large zones are applied with the correct colors, the overall situational awareness in the image is significantly improved. The previously captured color image 420 may not be consistent with the live one. The reasons may be the presence of transient objects, inaccuracies in location and view parameters, or changes in the scene since the previous image was taken. The previously captured colored image 420 may be based on the live image, so such consistency issues do not affect overall image visibility. They may cause some effect of incorrectly colored details, but the goal of correctly understanding the scene is not compromised.

[0062] The color image generation process 300 may be used for real-time situations where a user of the system 200 relies on the color image 450 for a critical mission. In a non-limiting example, the user may be a pilot utilizing the system 200 to improve pilot situational awareness for night flights. The color image generation process 300 is safe and completely reliable because it is deterministic and immune to misleading behavior. In this example, the color image 450 improves the pilot's situational awareness in night flight scenarios for any display, but particularly for see-through displays. The system 200 highlights large zones by adding their corresponding color (sky—blue, forest—green, field—brown, etc.). The applied colors may be determined from a satellite image database, which undergoes real-time ongoing transformation to a conformal view that provides accurate geolocation, height, and point of view. The live and / or real-time monochromatic image 410 is a key element in the color image 450. The colorized image 450 should not remove and / or eliminate any of the details or impair the ability to discern details compared to the live and / or real-time monochromatic image 410. The colorized image 450 should not introduce objects / details that are not present on the live and / or real-time monochromatic image 410. If an object is only present on the previously captured image 420, that object should not appear on the colorized image 450, thereby providing an anti-ghosting mechanism. The system 200 maintains the live and / or real-time monochromatic image 410 as the primary source by preserving the luma component of the pixels of the live and / or real-time monochromatic image 410 and modifying only the chroma.

[0063] It is to be understood that the subject matter of the present disclosure is not limited in its application to the details set forth in the description contained herein or illustrated in the drawings. The subject matter of the present disclosure is capable of other embodiments and of being practiced and carried out in various ways. Accordingly, it is to be understood that the phraseology and terminology employed herein is for the purpose of description and should not be regarded as limiting. Those skilled in the art will thus appreciate that the conception upon which the present disclosure is based may readily be utilized as a basis for the designing of other structures, methods and systems for carrying out some of the purposes of the subject matter of the present disclosure.

[0064] It will also be understood that systems in accordance with the presently disclosed subject matter may be implemented, at least in part, as a suitably programmed computer. Similarly, the presently disclosed subject matter contemplates a computer program readable by a computer to perform the disclosed methods. The presently disclosed subject matter further contemplates a machine-readable memory tangibly embodying a program of instructions executable by a machine to perform the disclosed methods.

Claims

1. A system for colorizing a monochrome image of a scene, wherein the system is The monochrome image of the aforementioned scene is captured using a non-visible spectral sensor capable of capturing a monochrome image from a viewpoint having a given position and orientation. Acquiring one or more previously captured color images that cover at least a portion of the aforementioned scene, wherein the previously captured color images were captured at a time prior to the capture of the monochrome images. The method for determining the alignment between the monochrome image and the previously captured color image is to determine the alignment by projecting the previously captured color image onto a plane equilateral to the viewpoint, and also based on a first geo-alignment of the monochrome image to the Earth coordinate system and a second geo-alignment of the previously captured color image, and the generation of the colored image is to determine that utilizes the first geo-alignment and the second geo-alignment to align the pixels of the monochrome image with the corresponding pixels of the previously captured color image. By utilizing the aforementioned alignment, one or more rendered images, which are previously captured color images adjusted to the aforementioned viewpoint, are rendered. A color image of the scene is generated by changing the value of one or more chroma components of at least one given pixel of the monochrome image according to the value of one or more chroma components of the corresponding pixels of the rendered image. A system comprising a processing circuit configured to perform the following actions.

2. The system according to claim 1, wherein the monochrome image includes one or more imaging elements not included in the one or more previously captured color images, and the generated color image includes the imaging elements.

3. The system according to claim 1, wherein the non-visible spectrum sensor is a night vision sensor.

4. The system according to claim 1, wherein the non-visible spectral sensor is an airborne sensor.

5. The system according to claim 1, wherein the color image of the scene is in YCbCr format.

6. The system according to claim 1, wherein the color image of the scene is in HSV format.

7. A method for colorizing a monochrome image of a scene, wherein the method is The processing circuit captures the monochromatic image of the scene using a non-visible spectral sensor capable of capturing a monochromatic image from a viewpoint having a given position and orientation. The processing circuit acquires one or more previously captured color images covering at least a portion of the scene, wherein the previously captured color images were captured at a time prior to the capture of the monochrome image. The processing circuit determines the alignment between the monochrome image and the previously captured color image, wherein the alignment is determined by projecting the previously captured color image onto a plane equilateral to the viewpoint, and also based on a first geo-alignment of the monochrome image to the Earth coordinate system and a second geo-alignment of the previously captured color image, and the generation of the colored image is determined to utilize the first geo-alignment and the second geo-alignment to align the pixels of the monochrome image with the corresponding pixels of the previously captured color image. The processing circuit renders one or more rendered images, which are previously captured color images adjusted to the viewpoint by utilizing the alignment, The processing circuit generates a color image of the scene by changing the value of one or more chroma components of at least one given pixel of the monochrome image according to the value of one or more chroma components of the corresponding pixels of the rendered image. Methods that include...

8. The method according to claim 7, wherein the monochrome image includes one or more imaging elements not included in the one or more previously captured color images, and the generated color image includes the imaging elements.

9. The method according to claim 7, wherein the non-visible spectrum sensor is a night vision sensor.

10. The method according to claim 7, wherein the invisible spectral sensor is an airborne sensor.

11. The method according to claim 7, wherein the color image of the scene is in YCbCr format.

12. The method according to claim 7, wherein the color image of the scene is in HSV format.

13. A non-temporary computer-readable storage medium that embodies both computer-readable program code, wherein the computer-readable program code is executed by at least one processor of at least one computer to carry out a method for coloring a monochrome image of a scene, and the method is The processing circuit captures the monochromatic image of the scene using a non-visible spectral sensor capable of capturing a monochromatic image from a viewpoint having a given position and orientation. The processing circuit acquires one or more previously captured color images covering at least a portion of the scene, wherein the previously captured color images were captured at a time prior to the capture of the monochrome image. The processing circuit determines the alignment between the monochrome image and the previously captured color image, wherein the alignment is determined by projecting the previously captured color image onto a plane equilateral to the viewpoint, and also based on a first geo-alignment of the monochrome image to the Earth coordinate system and a second geo-alignment of the previously captured color image, and the generation of the colored image is determined to utilize the first geo-alignment and the second geo-alignment to align the pixels of the monochrome image with the corresponding pixels of the previously captured color image. The processing circuit renders one or more rendered images, which are previously captured color images adjusted to the viewpoint by utilizing the alignment, The processing circuit generates a color image of the scene by changing the value of one or more chroma components of at least one given pixel of the monochrome image according to the value of one or more chroma components of the corresponding pixels of the rendered image. Non-temporary computer-readable storage media, including [specific type of storage medium].

Images