Method and apparatus for modifying an image

Abstract

A method for correcting images from a night vision camera is disclosed, the method comprising: receiving at least one image from the night vision camera, correcting the at least one image based on motion data from a tracking system and a shutter time of the night vision camera to remove smear from the image, and outputting the corrected at least one image.

Description

[Background technology]

[0001] Night vision cameras can be used in display systems to display images to a user when light levels are too low for conventional visible light cameras. To maximize the sensitivity of a night vision camera, the shutter time is as long as practical. However, a long shutter time can cause smearing because the user may move while the shutter is open. This reduces image quality, and the user may miss critical details in the scene captured by the night vision camera. [Brief explanation of the drawings]

[0002] [Figure 1] FIG. 1 illustrates a method for modifying an image according to some examples. [Figure 2] FIG. 2 illustrates a method for aligning and averaging two sequential images according to some examples. [Figure 3] FIG. 3 illustrates a method for aligning and averaging images across n sequential images according to some examples. [Figure 4] FIG. 4 illustrates a method for filtering an image according to some examples. [Figure 5a] FIG. 5a illustrates an image captured by a night vision camera. [Figure 5b] FIG. 5b illustrates the image after simulated motion smear and noise has been applied. [Figure 5c] FIG. 5c illustrates the filtered image. [Figure 5d] FIG. 5d illustrates the averaged and filtered image. [Figure 6] FIG. 6 illustrates how images may be aligned, averaged, and filtered according to some examples. [Figure 7]FIG. 7 illustrates a processing means according to some examples. Detailed Description

[0003] Night vision cameras (NVCs) can be used when light levels are too low for conventional visible light cameras to display images to the user with usable detail. To increase the amount of light received by the NVC, the shutter time can be maximized. However, this has the effect of causing smearing in the image when there is motion. Smearing can degrade the image and obscure important features within the image. This can reduce safety or user effectiveness when the NVC is used as part of a display system, especially in high-speed moving systems such as head-mounted or head-up displays in aircraft or vehicles.

[0004] A typical approach to reducing smear may be to shorten the shutter time, however shortening the shutter time may increase the signal-to-noise ratio (SNR) of the image, and integration over several images may be required to compensate for the shortened shutter time.

[0005] In many applications, known tracking systems may be included. For example, a head-mounted display may include a tracking system for tracking the orientation of the display in its environment. Tracking systems may also be used on drones where images are provided to a user. The tracking system may output motion data related to the tracking system's frame of reference and / or the object tracked by the tracking system. For example, in an example with a tracking system used to track a head-mounted display on an aircraft, the tracking system may also track the motion of the aircraft and the HWD within the aircraft. Typically, the object includes an NVC. Because the tracking system can track the movement of the NVC, the motion data may be used to mitigate the effects of smearing.

[0006] A method for reducing smear without increasing the SNR or reducing the shutter time is described in connection with FIG. 1 , where the method is generally designated by the reference numeral 100. A processing means receives at least one image from an NVC 110. Substantially simultaneously, the processing means may receive motion data from a tracking system associated with the NVC. The tracking system may track at least the orientation of the NVC or an object in a known relationship to the NVC. The tracking system may comprise a system similar to that described in PCT Publication No. WO 2017 / 042578, which is incorporated herein by reference. Another suitable tracking system may also be used.

[0007] At 120, the processing means may modify the received image to remove smear from the image, the modification being based on knowledge of the motion of the NVC and the shutter time of the NVC.

[0008] At 130, the modified image is output for display to a user, which may comprise transmitting the modified image to a display device, which may also comprise displaying the modified image to a user.

[0009] In some examples, the images may be processed in substantially real time, such that the images displayed to the user are substantially instantaneous. In applications where images are displayed to the user on a display in real time, the processing should be performed on the display or on the helmet. This avoids additional latency. However, if real-time use is not required, the images may be processed in a location separate from the display.

[0010] 2 illustrates a method 200 according to some examples. At 210, first and second sequential images are received. The images may be received from two consecutive frames of a camera. However, depending on the frame rate of the camera, the frames may not be consecutive. The first and second sequential images are received at a known time interval.

[0011] At 220, the first and second sequential images are aligned based on the known time interval between the images, the shutter speed of the NVC, and motion data provided by the tracking system. The motion data may comprise changes in orientation of the NVC, which allows changes in orientation of the images to be determined.

[0012] At 230, the registered images may be averaged. In some examples, averaging the images may comprise averaging pixel intensities. Pixel averaging is advantageous because it also results in the reduction of other types of noise. For example, pixel averaging may reduce the appearance of speckle in the image.

[0013] The averaged image is output at 240. The image may be output to a processing means, which is configured to provide instructions for displaying the image to a user. In some examples, outputting the image may comprise displaying the image to a user on a display.

[0014] In some examples, the frame rate of the NVC may be equal to the frame rate of a display for displaying the image to a user. In some examples, the frame rate of the NVC may be different from the frame rate of the display.

[0015] In some examples, the known time interval between the first and second sequential images may be less than the shutter speed. In some examples, the known time interval between the first and second sequential images may be different from the shutter speed.

[0016] Figure 3 illustrates a method substantially similar to that described in connection with Figure 2. At 310, n sequential images are received, where n is at least 2. The images may be received from two consecutive frames of a camera. However, depending on the frame rate of the camera, the frames may not be consecutive. The n sequential images are received at known time intervals.

[0017] At 320, the n sequential images are aligned based on the known time interval between images, the shutter speed of the NVC, and motion data provided by the tracking system. The motion data may comprise changes in orientation of the NVC, which allows changes in orientation of the images to be determined.

[0018] The registered n images may be averaged at 330. In some examples, averaging the images may comprise averaging pixel intensities.

[0019] The averaged image is output at 340. The image may be output to a processing means, which is configured to provide instructions for displaying the image to a user. In some examples, outputting the image may comprise displaying the image to a user on a display.

[0020] Figure 4 illustrates a method 400 according to some examples. The method is substantially similar to the method described with reference to Figure 1. The method may be used independently of or in combination with the method described with reference to Figures 2 and 3.

[0021] The method 400 comprises receiving an image, at 410. The image may be recorded using NVC.

[0022] At 420, an estimate of smear is made based on motion data from the tracking system. The motion data may comprise a change in orientation of the NVC during the shutter open time. The estimate of smear may comprise a point spread function (PSF). The PSF is a function that represents the extent to which the image is smeared. The image is filtered at 430 using a filter based on an estimate of smear. The filter may be further based on an estimate of noise in the image. The filter may comprise a Wiener filter. The image may be transformed before filtering. In some examples, the transformation may comprise a discrete Fourier transform. Transforming the image before filtering may allow the image to be filtered more efficiently. A Wiener filter may be preferred over an inverse filter because an inverse filter may amplify noise in the image, degrading the image. A Wiener filter reduces the amount of correction based on the level of noise, and is therefore a compromise between fully correcting the image and not adding too much noise.

[0023] At 440, the filtered image is output. The image may be output to a processing means, which is configured to provide instructions for displaying the image to a user. In some examples, outputting the image may comprise displaying the image to a user on a display. The image may still be output in the transformed domain and an inverse transform may be performed at a later time, or may be transformed back to the original domain using an inverse transform. In some examples, the transform may comprise an inverse Fourier transform or an inverse discrete Fourier transform.

[0024] Images filtered using method 400 are illustrated in Figures 5a-5c. Figure 5a represents an image recorded by a night vision camera without noticeable smear. In Figure 5b, smear and random noise and speckle have been applied to the image to simulate what a user of NVC might observe when viewing a smeared image. The noise and speckle are motion independent, while the smear is motion dependent.

[0025] Figure 5c illustrates an example image when the image is filtered using the method substantially as described with reference to Figure 4. As can be seen from Figure 5c, the smearing is substantially reduced. However, the noise is smeared because the noise is not dependent on the motion of the NVC.

[0026] The presence of noise can reduce the effectiveness of the filtering process. To reduce the effects of noise, the image can be filtered before passing the averaged image through the filter. Figure 5d illustrates an image in which four frames are aligned and averaged before filtering. As can be seen in Figure 5d, the appearance of speckle is reduced compared to Figures 5b and 5c, allowing more image detail to be seen.

[0027] A method for averaging sequential images and filtering the averaged images is illustrated in Figure 6. The method 600 comprises, at 510, receiving at least two sequential images. The images may be received from two consecutive frames of a camera. However, depending on the frame rate of the camera, the frames may not be consecutive. The at least two sequential images are received at a known time interval.

[0028] At 620, at least two sequential images are aligned based on a known time interval between the images, the shutter speed of the NVC, and motion data provided by a tracking system. The motion data may comprise changes in orientation of the NVC, which allows changes in orientation of the images to be determined.

[0029] The registered images may be averaged at 630. In some examples, averaging the images may comprise averaging pixel intensities.

[0030] At 640, an estimate of smearing is made based on motion data from the tracking system. The motion data may comprise a change in orientation. The change may comprise a delta angle value or may comprise an angular velocity. The estimate of smearing may comprise a point spread function (PSF). The PSF is a function that represents the extent to which the image is smeared.

[0031] The image is filtered at 650 using a filter based on an estimate of smear. The filter may be further based on an estimate of noise in the image. The filter may comprise a Wiener filter. The image may be transformed before filtering. In some examples, the transform may comprise a discrete Fourier transform. Transforming the image before filtering may allow the image to be filtered more efficiently.

[0032] At 660, the averaged image is output. The image may be output to a processing means, which is configured to provide instructions for displaying the image to a user. In some examples, outputting the image may comprise displaying the image to a user on a display. The image may still be output in the transformed domain, or may be transformed back to the original domain. In some examples, the transformation may comprise an inverse Fourier transform.

[0033] A Wiener filter can be used to estimate the optimal image as follows:

[0034]

number

[0035] where:

[0036]

number

[0037] is the discrete Fourier transform (DFT) of the captured image,

[0038]

number

[0039] is the DFT of the estimated corrected image,

[0040]

number

[0041] is the DFT of the smearing PSD (power spectral density),

[0042]

number

[0043]

number

[0044] is the power spectrum of the noise,

[0045]

number

[0046] is the power spectrum of the image (without smear or noise).

[0047] K is pre-calculated before the system is used,

[0048]

number

[0049] is estimated based on typical images,

[0050]

number

[0051] is calculated based on the known (estimated) noise performance of the camera. Generally, K can be adjusted to optimize the tradeoff between noise and image sharpness in the final image. K can also be adjusted to account for different noise levels depending on the electronic gain of the camera. That is, at lower light levels where the camera gain is increased, K is increased to increase the emphasis on smoothing.

[0052] The final image is

[0053]

number

[0054] It is calculated by performing an inverse DFT on

[0055] To reduce the amount of spatial ringing due to edge effects in the DFT, the image can be blurred at the edges before performing the DFT.

[0056] 7 illustrates a processing means 700 according to some examples. The processing means 700 comprises a first circuit 710, a second circuit 720, and a third circuit 730.

[0057] The first circuit 710 is configured to receive at least one image from a night vision camera.

[0058] The second circuit 720 is configured to correct at least one image based on the motion data from the tracking system and the shutter time of the night vision camera to remove smear from the image.

[0059] The third circuit 730 is configured to output the modified at least one image.

[0060] The second circuit 720 may be configured to align the at least two sequential images based on motion data from the tracking system, and after aligning the at least two sequential images, average pixel intensities across the aligned at least two sequential images.

[0061] Aligning the at least two sequential images based on motion data from the tracking system may comprise calculating a change in rotation of the night vision camera between the at least two sequential images.

[0062] The second circuit 720 may be configured to apply a filter to the at least one image, the filter coefficients of the filter being based on the motion data from the tracking system.

[0063] The second circuit 720 may be configured to average pixel intensities over at least two sequential images before applying the filter.

[0064] In some examples, the filter may comprise a Wiener filter.

[0065] In some instances, the frame rate of the night vision camera is higher than the frame rate of the display system.

[0066] In some instances, the field of view of the night vision camera is larger than the field of view of the display for displaying the modified image to the user.

[0067] In some examples, the second circuit 720 may be configured to estimate a point spread function associated with at least one image.

[0068] In some examples, the second circuit 720 may be configured to estimate a measure of noise in the image.

[0069] In some examples, the motion data comprises the orientation of a night vision camera.

[0070] While the above techniques have been described with reference to night vision cameras, the methods are applicable to any type of camera where smearing is caused by camera movement while the shutter is open and the speed of the movement is comparable to the camera's shutter time. In some examples, the described methods may be applied to night vision cameras on head-mounted displays on aircraft by leveraging existing head tracking systems. In some examples, the methods may also be applied to cameras on moving platforms such as drones equipped with tracking software.

[0071] It is also understood that a camera may not have a mechanical shutter, so such shutter speeds or times may refer to the time a sensor detects light in a frame as an electronic shutter. These methods are equally applicable to mechanical or electronic shutters. The inventions described in the claims of the present application as originally filed are set forth below. [C1] 1. A method comprising: receiving at least one image from a night vision camera; correcting the at least one image based on motion data from a tracking system and a shutter time of a night vision camera to remove smear from the image; outputting the modified at least one image; A method comprising: [C2] The method of claim 1, wherein the at least one image comprises at least two sequential images, and correcting the at least one image comprises aligning the at least two sequential images based on motion data from the tracking system, and averaging pixel intensities across the aligned at least two sequential images after aligning the at least two sequential images. [C3] The method of C2, wherein aligning the at least two sequential images based on motion data from the tracking system comprises calculating a change in rotation of the night vision camera between the at least two sequential images. [C4] 4. The method of any one of C1 to C3, wherein correcting comprises applying a filter to the at least one image, the filter coefficients of the filter being based on motion data from the tracking system and a shutter time of the night vision camera. [C5] The method of C4 when dependent on C2 or 3, wherein correcting comprises averaging pixel intensities across the at least two sequential images before applying the filter. [C6] 6. The method of any one of C4 or C5, wherein the filter comprises a Wiener filter. [C7] The method of any one of C1 to C6, wherein the frame rate of the night vision camera is higher than the frame rate of the display system. [C8] The method of any one of C1 to C7, wherein the field of view of the night vision camera is larger than the field of view of a display that displays the corrected image to a user. [C9] The method of any one of C1 to C8, wherein correcting the at least one image further comprises estimating a point spread function associated with the at least one image. [C10] The method of any one of C1 to C9, wherein modifying the at least one image further comprises estimating a measure of noise in the image. [C11] The method of any one of C1 to C10, wherein the movement data comprises an orientation of the night vision camera. [C12] 12. The method of any one of C1 to C11, further comprising applying a Fourier transform to the at least one image before modifying the at least one image based on motion data, the Fourier transform optionally being a discrete Fourier transform. [C13] 13. The method of any one of C1 to C12, further comprising applying an inverse Fourier transform to the modified image, the inverse Fourier transform optionally being an inverse discrete Fourier transform. [C14] A processing means, a first circuit configured to receive at least one image from a night vision camera; a second circuit configured to modify the at least one image based on motion data from a tracking system and a shutter time of the night vision camera to remove smear from the image; and a third circuit configured to output the modified at least one image; and A processing means comprising: [C15] The processing means of C14, wherein the second circuit is configured to align at least two sequential images based on motion data from the tracking system, and after aligning the at least two sequential images, the second circuit is configured to average pixel intensities across the aligned at least two sequential images. [C16] 16. The processing means of claim 14 or 15, wherein the second circuit is configured to apply a filter to the at least one image, filter coefficients of the filter being based on motion data from the tracking system. [C17] Processing means configured to carry out the method of any one of C1 to C13. [C18] A computer readable medium comprising instructions that, when executed, cause a processing means to perform the method of any one of C1 to C13.

Claims

1. 1. A method comprising: receiving at least one image from a night vision camera; correcting the at least one image based on motion data from a tracking system and a shutter time of the night vision camera to remove smear from the image; outputting the modified at least one image; Equipped with wherein the motion data comprises a change in orientation of the night vision camera, thereby determining a change in orientation of the at least one image; The at least one image comprises at least two sequential images, and correcting the at least one image comprises aligning the at least two sequential images based on motion data from the tracking system; averaging pixel intensities across the aligned at least two sequential images after aligning the at least two sequential images; and applying a filter to the averaged image, the filter having coefficients determined based on the motion data from the tracking system and a shutter time of the night vision camera. method.

2. The method of claim 1 , wherein aligning the at least two sequential images based on motion data from the tracking system comprises calculating a change in rotation of the night vision camera between the at least two sequential images.

3. 3. The method of claim 1, wherein the filter comprises a Wiener filter.

4. A method described in any one of claims 1 to 3, wherein the frame rate of the night vision camera is higher than the frame rate of the display system.

5. 5. The method of claim 1, wherein the field of view of the night vision camera is larger than the field of view of a display that displays the modified image to a user.

6. The method of claim 1 , wherein modifying the at least one image further comprises estimating a point spread function associated with the at least one image.

7. The method of claim 1 , wherein modifying the at least one image further comprises estimating a measure of noise in the image.

8. 8. The method of claim 1, further comprising applying a Fourier transform to the at least one image before modifying the at least one image based on motion data, the Fourier transform optionally being a discrete Fourier transform.

9. 9. The method of claim 1, further comprising applying an inverse Fourier transform to the modified image, the inverse Fourier transform optionally being an inverse discrete Fourier transform.

10. A processing means, a first circuit configured to receive at least one image from a night vision camera; a second circuit configured to modify the at least one image based on motion data from a tracking system and a shutter time of the night vision camera to remove smear from the image; and a third circuit configured to output the modified at least one image; Equipped with wherein the motion data comprises a change in orientation of the night vision camera, thereby determining a change in orientation of the at least one image; The second circuit is configured to align at least two sequential images based on motion data from the tracking system, and after aligning the at least two sequential images, the second circuit is configured to average pixel intensities across the aligned at least two sequential images and apply a filter to the averaged image, the filter having coefficients determined based on the motion data from the tracking system and a shutter time of the night vision camera. Processing means.

11. Processing means arranged to carry out a method according to any one of claims 1 to 9.

12. A computer readable storage medium comprising instructions which, when executed, cause a processing means to perform the method of any one of claims 1 to 9.

Images