Reconstruction of an image from a plurality of images

Abstract

An image capture method and apparatus are disclosed. An exemplary method includes focusing light by each of a plurality of beam shaping elements onto a respective different one of a plurality of photosensitive regions, each of the beam shaping elements being configured to capture an image for image reconstruction. The method further includes obtaining, by each respective photosensitive region, a respective image based on the light focused thereon, and deconvolving the images to generate a reconstructed image, the reconstructed image having an improved broadband image quality compared to the received images.

Description

Technical Field

[0001] Field of Disclosure The present disclosure relates to reconstructing an image from a plurality of images.

Background Art

[0002] Background Some optoelectronic imaging devices, such as cameras, have an optical channel that includes a beam shaping element (e.g., a lens) for focusing incident light onto a photosensitive area (e.g., a pixel) of an image sensor. The image sensor can be operable to convert the optical signal into a corresponding electrical signal, and the electrical signal can then be processed to reconstruct an image.

Summary of the Invention

Problems to be Solved by the Invention

[0003] Summary The present disclosure describes the reconstruction of an image from a plurality of images. An image capture method and apparatus are disclosed.

Means for Solving the Problems

[0004] For example, in one aspect, the present disclosure describes a method that includes focusing light by each of a plurality of beam shaping elements onto different respective photosensitive areas of a plurality of photosensitive areas, where each of the beam shaping elements is individually configured to capture an image for image reconstruction. The method further includes obtaining, by each respective photosensitive area, a respective image based on the light focused thereon, and deconvolving (inverse convolution) the images to generate a reconstructed image, where the reconstructed image has improved broadband image quality compared to the received images.

[0005] The present disclosure also describes an image capture device including at least one sensor having a plurality of photosensitive regions each operable to capture a respective image. The device also includes beam shaping elements, each of the beam shaping elements being operable to focus light onto different respective photosensitive regions, and each of the beam shaping elements being individually configured to capture an image for image reconstruction. An image processor is operable to receive signals representative of the respective images captured by the photosensitive regions. The image processor is further operable to deconvolve the received images and generate a reconstructed image based on the deconvolved images, the reconstructed image having an improved broadband image quality compared to the received images.

[0006] Some embodiments include one or more of the following features. For example, in some embodiments, the beam shaping element includes at least one of a diffractive lens or a metasurface lens. Optionally, each of the beam shaping elements may be configured for a specific wavelength or a narrow range of wavelengths.

[0007] In some embodiments, the image processor includes a neural network operable to deconvolve the received images. Optionally, each of the captured images has different respective sharp and blurred components, and the neural network is operable to combine the captured images to create a sharper color image.

[0008] In some embodiments, the image processor is operable to perform initial deblurring of the received images, detect sharp edges in each of the deblurred images, associate portions of each image within the detected sharp edges with features of one or more objects that reflect light of respective wavelengths, and combine portions of the deblurred images within the detected sharp edges to obtain a reconstructed image.

[0009] In some embodiments, the image capture device includes a plurality of optical sensors, each of which includes a different respective photosensitive region. The image capture device can, in some embodiments, include an optical projector operable to project an optical reference feature at a specific wavelength. A particular beam shaping element can be operable to focus the reflected light onto a specific photosensitive region, and at least one sensor can be operable to create an image that includes features corresponding to the optical reference feature. An image processor can be operable to determine whether the features in the image corresponding to the optical reference feature are sufficiently sharp. In some cases, the image processor can further be operable to generate a signal indicating that the image capture device should be recalibrated or adjusted in response to a determination that the features in the image corresponding to the optical reference feature are not sufficiently sharp.

[0010] Some embodiments include one or more of the following advantages. For example, in some embodiments, the beam shaping elements collectively form a sharp image, but in some cases, a single beam shaping element alone may not be operable to create an image of the same quality as the reconstructed image. In some cases, the methods and systems of the present disclosure can achieve an improvement in broadband image quality.

[0011] Details of one or more embodiments of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the specification, the accompanying drawings, and the claims.

Brief Description of the Drawings

[0012]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

[0013] Detailed Description A beam shaping element in a photoelectronic imaging device that focuses incident light onto a photosensitive area of an image sensor can be implemented, for example, as a diffractive lens or a metasurface. Such lenses may be designed to operate optimally at a specific wavelength (λ). On the other hand, light (e.g., visible, infrared) reflected from one or more objects in a scene from which an image will be acquired by the imaging device can include a plurality of different wavelengths or a spectrum of wavelengths. That is, some objects (or parts of objects) in the scene may predominantly reflect a first wavelength (λ1), while other objects (or parts of objects) in the scene may predominantly reflect a second different wavelength (λ2). Thus, for example, if a lens is optimized for operation with light of the first wavelength (λ1), portions of the scene that predominantly reflect light of other wavelengths (e.g., λ2) tend to be somewhat blurred as a result of diffraction. The present disclosure, in some embodiments, describes an imaging system and method that can help obtain a reconstructed image that more closely resembles the original scene from which the image was acquired. Further, the imaging system and method can be used regardless of whether the beam shaping element is operable for any particular wavelength or range of wavelengths to form a sharp image. That is, in some embodiments, the system and method can be used for reconstruction even if the acquired image is blurred at all wavelengths.

[0014] As shown in the example of FIG. 1, the image capture system 10 is operable to acquire images of one or more objects 14, 16 within the scene 12 and process the acquired images to obtain a reconstructed image 24 of the objects within the scene. The image capture system 10 can include a plurality of sensors 20A, 20B, each of which is operable to detect an optical signal and convert the detected optical signal into a corresponding electrical signal. Each sensor 20A, 20B has respective photosensitive regions (e.g., pixels) and can be implemented as a CCD sensor operable to output, for example, an electrical signal representative of a captured image. For example, as shown in FIG. 1, the first sensor 20A is operable to output a first image A, and the second sensor is operable to output a second image B. In some embodiments, instead of separate sensors 20A, 20B, a single large optical sensor can be provided, with a first photosensitive area of the sensor being used to capture and generate the first image A and a second photosensitive area of the sensor being used to capture and generate the second image B.

[0015] As further shown in FIG. 1, the image capture system 10 also includes respective beam shaping elements (e.g., lenses) 18A, 18B for focusing the light reflected by the objects in the scene 12 onto the respective sensors 20A, 20B. Each lens 18A, 18B can be implemented, for example, as a diffractive lens or a metalens. Further, each respective lens 18A, 18B can, in some cases, be designed for a specific respective wavelength or a narrow range of wavelengths. That is, each lens may be optimized to focus light, for example, in the visible region (e.g., red, blue, or green) or the infrared region of the electromagnetic spectrum. For example, the first lens 18A may be optimized for a first wavelength λ1 (e.g., a narrow range of wavelengths centered on λ1), while the second lens 18B may be optimized for a second different wavelength λ2 (e.g., a narrow range of wavelengths centered on λ2). In this context, a narrow range of wavelengths can be, for example, wavelengths within 10 nanometers (nm) of the central wavelength of the range. In some embodiments, the range may be different (e.g., larger or smaller). In some embodiments, the optical beam shaping element may create an image that is not sharp at any particular wavelength.

[0016] Since each beam shaping element 18A, 18B can, in some cases, be optimized for a specific respective wavelength or a narrow range of wavelengths, in some embodiments, it may not be necessary to use an optical filter with the lens. In such embodiments, since there is no filter in front of the sensor, light is not discarded, and thus it may be possible to capture more energy than a filter-based image sensor. This can, as a result, lead to a system with higher sensitivity. Nevertheless, in some cases, respective optical filters may be used in combination with one or more of the beam shaping elements 18A, 18B.

[0017] The image capture system 10 further includes an image processor 22 operable to receive images from sensors 20A, 20B and process the images together to obtain a reconstructed image 24. The reconstructed image 24 can be stored in a memory and / or displayed on a display 26 (e.g., a display monitor or a display screen).

[0018] Depending on the situation, different objects within the scene 12 may reflect different wavelengths of light. To illustrate an example of the operation of the image capture system, assume that the scene includes one or more objects 14 that reflect light of a first wavelength λ1. Further, for the sake of illustration, assume that the scene also includes one or more other objects 16 that reflect light of a second wavelength λ2. Nevertheless, in some cases, a particular object may reflect light that exceeds a certain wavelength (e.g., both wavelengths λ1 and λ2).

[0019] During operation, the controller 30 can generate one or more signals for triggering sensors 20A, 20B to acquire respective images of the scene 12 at a particular instant. The signals from the controller 30 may be generated, for example, in an automated manner or in response to user input. In response, a portion of the light reflected by the objects 14, 16 within the scene 12 passes through lenses 18A, 18B and is detected by sensors 20A, 20B, respectively. The signal output by the first sensor 20A represents a first image A, and the signal output by the second sensor 20B represents a second image B.

[0020] Both the first image A and the second image B may include features based on light detected at a first wavelength λ1 and a second wavelength λ2. That is, as shown in FIG. 2, the first image A (i.e., the output of the first sensor 20A) can include a first feature 14A based on light detected at the first wavelength λ1 and a second feature 16A based on light detected at the second wavelength λ2. Since the first lens 18A that focuses light onto the first sensor 20A is optimized for light of the first wavelength λ1, the first feature 14A tends to be sharper than the second feature 16A, which may be slightly blurred.

[0021] Similarly, as shown in FIG. 2, the second image B (i.e., the output of the second sensor 20B) can include a first feature 14B based on light detected at the first wavelength λ1 and a second feature 16B based on light detected at the second wavelength λ2. Since the second lens 18B that focuses light onto the second sensor 20B is optimized for light of the second wavelength λ2, the second feature 16B tends to be sharper than the first feature 14B, which may be slightly blurred.

[0022] Both image A and image B are provided as inputs to an image processor 22, which is operable to deconvolve image A and image B to obtain a reconstructed image 24. In this case, image deconvolution facilitates the reconstruction of latent images from two or more degraded images. That is, a relatively sharp color image can be created by combining a plurality of images (each having different sharp and blurred components).

[0023] As shown in FIG. 3, in some embodiments, the image processor 22 is operable to process an image to perform initial blur removal on individual images A and B (FIG. 3, 102). Thus, the blur removal operation can be applied, for example, as part of a preprocessing stage. If the point spread function (PSF) of the optical system (e.g., lenses 18A, 18B) is known or can be calculated, for example, non-blind deconvolution techniques can be used for blur removal. If the PSF of the optical system is unknown, blind deconvolution techniques can be used to remove the blur in the image (i.e., restore a relatively sharp version of the image from the blurred version). In some embodiments, other blur removal techniques can be used.

[0024] As further shown by FIG. 3, the method includes detecting sharp (e.g., high contrast) edges in each of the blur-removed images (FIG. 3, 104). Next, portions of each image within the detected sharp edges can be associated with features of the objects 14, 16 in the scene 12 that reflect light at corresponding wavelengths (e.g., λ1 or λ2) (FIG. 3, 106). That is, a portion of the first blur-removed image A within the detected sharp edge can be associated with features of the objects 14, 16 in the scene 12 that reflect light at the first wavelength λ1. Similarly, a portion of the second blur-removed image B within the detected sharp edge can be associated with features of the objects 14, 16 in the scene 12 that reflect light at the second wavelength λ2. Other portions of each blur-removed image may optionally be ignored (e.g., discarded). On the other hand, in some embodiments (e.g., those using a neural network), other portions of each blur-removed image may similarly be beneficial and thus may not be discarded. Next, portions of the blur-removed images within the detected sharp edges can be combined to obtain a reconstructed image 24 (FIG. 3, 108).

[0025] As shown in FIG. 4, in some embodiments, the image processor 22 includes a neural network 32 for deconvolving images A and B to obtain the reconstructed image 24. Thus, in some cases, the image processor 22 can incorporate artificial intelligence and may include iterative and / or deep learning techniques.

[0026] In the foregoing example, the latent image is reconstructed from two degraded images, but more generally, the latent image can be reconstructed from two or more degraded images (e.g., an array of images). For example, an image capture system can include an array of sensors (e.g., an array of 4 to 32 sensors), each having a respective associated beam shaping element (e.g., a diffractive lens or a metasurface) adjusted for operation at a different respective wavelength. Each sensor is operable, for example, to create an image of a scene. Further, depending on the wavelength of light reflected by various objects within the scene, different images can include some relatively sharp features and other somewhat blurred features. The image processor is operable to deconvolve the various images obtained from the sensors to obtain a reconstructed version of the latent image.

[0027] Some embodiments also include additional features to facilitate calibration or recalibration of the image capture system. For example, recalibration may be desirable to correct for thermal effects on the optical sensor. As shown in FIG. 5, the image capture system 10A can include an optical projector 36 operable to project an optical reference feature 40 onto the scene at a particular wavelength λ3. In addition to the first optical sensor 20A and the second optical sensor 20B and their associated lenses 18A, 18B, the image capture system 10A further includes a third optical sensor 20C and an associated third lens (e.g., a diffractive lens or a metasurface) 18C. The third lens 18C can be optimized for the wavelength λ3 (e.g., a narrow range of wavelengths centered on λ3) and is operable to focus the incident light onto the third sensor 20C.

[0028] As described above with respect to sensors 20A and 20B, the third sensor 20C can also be implemented as a CCD sensor having a photosensitive area (e.g., pixels) and operable to output, for example, an electrical signal representing a captured image. As shown in FIG. 6, the third sensor 20C is operable to output a third image C. In some embodiments, a single large optical sensor can be provided, where a first photosensitive area of the sensor is used to capture and generate a first image A based on light passing through the first lens 18A, a second photosensitive area of the sensor is used to capture and generate a second image B based on light passing through the second lens 18B, and a third photosensitive area of the sensor is used to capture and generate a third image C based on light passing through the third lens 18C.

[0029] Each of images A, B, and C can include features based on light detected at a first wavelength λ1, a second wavelength λ2, and a third wavelength λ3. That is, as shown in FIG. 6, the first image A (i.e., the output of the first sensor 20A) can include a first feature 14A based on light detected at the first wavelength λ1, a second feature 16A based on light detected at the second wavelength λ2, and a third feature 40A (e.g., corresponding to the reference feature 40) based on light detected at the third wavelength λ3. Since the first lens 18A that focuses light onto the first sensor 20A is optimized for light of the first wavelength λ1, the first feature 14A tends to be sharper than the second feature 16A and the third feature 40A, which may be slightly blurred.

[0030] Similarly, as shown in FIG. 6, the second image B (i.e., the output of the second sensor 20B) can include a first feature 14B based on the light detected at the first wavelength λ1, a second feature 16B based on the light detected at the second wavelength λ2, and a third feature 40B (e.g., corresponding to the reference feature 40) based on the light detected at the third wavelength λ3. Since the second lens 18B that focuses light onto the second sensor 20B is optimized for light of the second wavelength λ2, the second feature 16B tends to be sharper than the first feature 14B and the third feature 40B, which may be slightly blurred.

[0031] As further shown in the example of FIG. 6, the third image C (i.e., the output of the third sensor 20C) can include a first feature 14C based on the light detected at the first wavelength λ1, a second feature 16C based on the light detected at the second wavelength λ2, and a third feature 40C (e.g., corresponding to the reference feature 40) based on the light detected at the third wavelength λ3. Since the third lens 18C that focuses light onto the third sensor 20C is optimized for light of the third wavelength λ3, the third feature 40C corresponding to the reference feature 40 should be relatively sharp, while the other features 14C, 16C may be somewhat blurred. If the feature 40C in the image C is not sufficiently sharp, it may be necessary to adjust one or more optical components of the image capture system 10A. For example, the image processor may be operable to generate a signal indicating that the image capture device should be recalibrated or adjusted in response to a determination that the feature in the image corresponding to the optical reference feature is not sufficiently sharp.

[0032] The various aspects of the subject matter and functional operations described in this specification (e.g., the operations described in connection with the image processor 22, neural network 32, and / or controller 30) can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware. Accordingly, aspects of the subject matter described in this specification can be implemented, for example, as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by, or to control the operation of, a data processing apparatus. The computer-readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter that provides a machine-readable propagated signal, or one or more combinations of them. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware.

[0033] A computer program (also known as a program, software, software application, script, or code) can be written in any form of programming language, including compiled languages or interpreted languages, and can be deployed in any form, either as a stand-alone program or included as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program does not necessarily correspond to a file in a file system. The program can be stored as part of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), a single file dedicated to the program in question, or multiple cooperating files (e.g., files that store one or more modules, subprograms, or portions of code). A computer program can be deployed to be executed on one computer, or on one location, or distributed over multiple locations and executed on multiple computers interconnected by a communication network.

[0034] The processes and logical flows described herein can be implemented by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logical flows can also be implemented by, for example, a dedicated logic processing circuit such as a field programmable gate array (FPGA) or an application specific integrated circuit (ASIC), and the apparatus can also be implemented as a dedicated logic processing circuit.

[0035] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, as well as any one or more processors of any kind of digital computer. In general, a processor receives instructions and data from a read only memory or a random access memory or both. Essential elements of a computer are a processor for executing instructions, as well as one or more memory devices for storing instructions and data. Computer-readable media suitable for storing computer program instructions and data include, by way of example, all forms of non-volatile memory, media, and memory devices, such as semiconductor memory devices (e.g., EPROM, EEPROM, and flash memory devices), magnetic disks (e.g., internal hard disks or removable disks), magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and the memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0036] The image capture system described in this disclosure can be used for various applications, including, for example, the reconstruction of RGB images or hyperspectral images (e.g., medical, quality control, satellite images).

[0037] Although this specification contains many specific implementation details, these should not be construed as limitations on the scope of any invention or of what may be claimed, but rather as descriptions of features specific to particular embodiments. The particular features described in this specification in the context of separate embodiments can also be implemented in combination within a single embodiment. Conversely, the various features described in the context of a single embodiment can also be implemented separately in multiple embodiments or in any suitable sub-combination. Various modifications may be made to the foregoing examples. Accordingly, other embodiments are within the scope of the claims.

Claims

**Claim 1** An image capture device, comprising: At least one sensor having a plurality of photosensitive regions, each of the plurality of photosensitive regions being operable to capture a respective image; A beam shaping element, each of the beam shaping elements being operable to focus light on a respective one of the different photosensitive regions, and each of the beam shaping elements being individually configured to capture an image for image reconstruction; An image processor, the image processor being operable to receive signals representing the respective images captured by the photosensitive regions, the image processor being further operable to deconvolve the received images and generate a reconstructed image based on the deconvolved images, the reconstructed image having improved broadband image quality compared to the received images; An image capture device comprising the above components. **Claim 2** The image capture device according to claim 1, wherein the beam shaping element includes a diffractive lens. **Claim 3** The image capture device according to claim 1, wherein the beam shaping element includes a metasurface lens. **Claim 4** The image capture device according to any one of claims 1 to 3, wherein the image processor includes a neural network operable to deconvolve the received images. **Claim 5** The image capture device according to claim 4, wherein each of the captured images has distinct and blurred components, and the neural network is operable to combine the captured images to create a sharper color image. **Claim 6** The image processor: Performs initial blur removal on the received images; Detects sharp edges in each of the blur-removed images; Associates portions of each image within the detected sharp edges with features of one or more objects that reflect light of respective wavelengths; Is operable to combine the portions of the blur-removed images within the detected sharp edges to obtain the reconstructed image. The image capture device according to any one of claims 1 to 3. **Claim 7** The image capture device according to any one of claims 1 to 6, comprising a plurality of optical sensors, each of the plurality of optical sensors including a respective different photosensitive region.

8. Further comprising an optical projector operable to project an optical reference feature at a specific wavelength, The specific beam shaping element is operable to focus the reflected light on the specific photosensitive region, The at least one sensor is operable to create an image including a feature corresponding to the optical reference feature, The image capture device according to any one of claims 1 to 7, wherein the image processor is operable to determine whether the feature in the image corresponding to the optical reference feature is sufficiently sharp.

9. The image capture device according to claim 8, wherein the image processor is further operable to generate a signal indicating that the image capture device should be recalibrated or adjusted in response to a determination that the feature in the image corresponding to the optical reference feature is not sufficiently sharp.

10. An image capture method, comprising: Focusing light on different respective photosensitive regions of a plurality of photosensitive regions by each of a plurality of beam shaping elements, each of the beam shaping elements being individually configured to capture an image for image reconstruction; Acquiring respective images based on the light focused thereon by each respective photosensitive region; Performing deconvolution on the images to generate a reconstructed image, the reconstructed image having improved broadband image quality compared to the received images; An image capture method including the above steps.

11. The method according to claim 10, wherein the beam shaping element includes a diffractive lens.

12. The method according to claim 10, wherein the beam shaping element includes a metasurface lens.

13. The method according to any one of claims 10 to 12, including using a neural network to perform deconvolution on the images.

14. Each of the acquired images has different respective sharp and blurred components, and the method includes synthesizing the acquired images to create a sharper color image.

15. Performing initial blurring removal on the acquired images; detecting sharp edges in each of the defocus-removed images; associating each portion of the images within the detected sharp edges with features of one or more objects that reflect light of respective wavelengths; combining the portions of the defocus-removed images within the detected sharp edges to obtain the reconstructed image; The method according to any one of claims 10 to 14, further comprising the above steps.

16. projecting an optical reference feature at a specific wavelength; focusing the reflected light on the specific photosensitive area by the specific beam shaping element; creating an image based on a signal from the specific photosensitive area, the image including features corresponding to the optical reference feature; determining, by an image processor, whether the features in the image corresponding to the optical reference feature are sufficiently sharp; The method according to any one of claims 10 to 15, further comprising the above steps.

17. The method according to claim 16, further comprising generating a signal indicating the need for recalibration in response to a determination that the features in the image corresponding to the optical reference feature are not sufficiently sharp.

Images