Reconstructing image from plurality of images

Abstract

The invention discloses an image capturing method and device. An example method includes focusing, by each of a plurality of beam shaping elements, light onto a different respective one of a plurality of photosensitive regions, where each of the beam shaping elements is configured to capture an image for image reconstruction; the method further includes acquiring, by each respective one of the photosensitive regions, a respective image based on the light focused thereon, and deconvolving the images to generate a reconstructed image, where the reconstructed image has enhanced broadband image quality compared to the received image.

Description

Technical Field

[0001] The present disclosure relates to reconstructing an image from a plurality of images. Background Art

[0002] Some optoelectronic imaging devices, such as cameras, have an optical channel that includes a beam shaping element (e.g., a lens) to focus incident light onto a photosensitive area (e.g., a pixel) of an image sensor. The image sensor may be operable to convert the optical signal into a corresponding electrical signal, which may then be processed to reconstruct an image. Summary of the invention

[0003] The present disclosure describes reconstruction of an image from multiple images. An image capture method and apparatus are disclosed.

[0004] For example, in one aspect, the present disclosure describes a method that includes focusing light, by each of a plurality of beam shaping elements, onto a different corresponding photosensitive region of a plurality of photosensitive regions, wherein each of the beam shaping elements is individually configured to capture an image for image reconstruction. The method further includes acquiring, by each corresponding photosensitive region of the photosensitive regions, a corresponding image based on the light focused thereon, and deconvolving the image to generate a reconstructed image, wherein the reconstructed image has an enhanced broadband image quality compared to the received image.

[0005] The present disclosure also describes an image capture device, the image capture device comprising at least one sensor having a plurality of photosensitive regions, each of the plurality of photosensitive regions being operable to capture a respective image. The device further comprises a beam shaping element, each of the beam shaping elements being operable to focus light onto a different respective photosensitive region, wherein each of the beam shaping elements is individually configured to capture an image for image reconstruction. An image processor is operable to receive a signal representing the respective image captured by the photosensitive region. The image processor is further operable to deconvolve the received image and generate a reconstructed image based on the deconvolved image, wherein the reconstructed image has an enhanced broadband image quality compared to the received image.

[0006] Some embodiments include one or more of the following features. For example, in some embodiments, the beam shaping element includes at least one of a diffractive lens or a metalens. In some cases, each of the beam shaping elements can be configured for a specific wavelength or a narrow wavelength range.

[0007] In some embodiments, the image processor includes a neural network operable to deconvolve the received images, each of the captured images having a different corresponding sharp component and blurry component, wherein the neural network is operable to combine the captured images to produce a sharper color image.

[0008] In some embodiments, the image processor is operable to perform initial deblurring of received images; detect sharp edges within each of the deblurred images; associate a portion of each image within the detected sharp edges with features of one or more objects that reflect light of corresponding wavelengths; and combine the portions of the deblurred images within the detected sharp edges to obtain a reconstructed image.

[0009] In some embodiments, the image capture device includes a plurality of light sensors, each light sensor in the plurality of light sensors including a different corresponding light sensitive area in the light sensitive area. In some embodiments, the image capture device may include an optical projector operable to project the optical reference feature at a specific wavelength. A specific beam shaping element in the beam shaping elements is operable to focus the reflected light onto a specific light sensitive area in the light sensitive area, and the at least one sensor is operable to generate an image including a feature corresponding to the optical reference feature. The image processor may be operable to determine whether the feature in the image corresponding to the optical reference feature is sufficiently sharp. In some cases, the image processor is further operable to generate a signal indicating that the image capture device should be recalibrated or adjusted in response to determining that the feature in the image corresponding to the optical reference feature is not sufficiently sharp.

[0010] Some embodiments include one or more of the following advantages. For example, in some embodiments, the beam shaping elements collectively form a sharp image even though, in some cases, no individual one of the beam shaping elements is operable to produce an image of the same quality as the reconstructed image. In some cases, the disclosed methods and systems can achieve enhanced broadband image quality.

[0011] The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 An example imaging system according to the present disclosure is shown.

[0013] Figure 2 Shown are Figure 1 An example of an image generated by each of the multiple optical sensors in the system.

[0014] Figure 3 is a flow chart illustrating an example method for obtaining a reconstructed image according to some embodiments.

[0015] Figure 4 Another example imaging system according to the present disclosure is shown.

[0016] Figure 5 A further example imaging system according to the present disclosure is shown.

[0017] Figure 6 Shown are Figure 5 An example of an image generated by each of the multiple optical sensors in the system. DETAILED DESCRIPTION

[0018] The beam shaping element in an optoelectronic imaging device that focuses incident light onto a photosensitive area of ​​an image sensor may be implemented, for example, as a diffractive lens or metalens. Such a lens is sometimes designed for optimal operation at a specific wavelength (λ). On the other hand, light (e.g., visible light, infrared light) reflected from one or more objects in a scene whose image is to be acquired by the imaging device may include multiple different wavelengths or wavelength spectra. That is, some objects in the scene (or portions of an object) may primarily reflect a first wavelength (λ1), while other objects in the scene (or portions of an object) may primarily reflect a second different wavelength (λ2). Therefore, if the lens is optimized, for example, for operation at a first wavelength (λ1), then portions of the scene that primarily reflect light at other wavelengths (e.g., λ2) will tend to be somewhat blurred due to diffraction. The present disclosure describes an imaging system and method that, in some embodiments, can help obtain a reconstructed image that more closely resembles the original scene whose image was acquired. In addition, the imaging system and method can be used regardless of whether the beam shaping element is operable for any specific wavelength or wavelength range to form a sharp image. That is, in some embodiments, the systems and methods can be used to reconstruct even when the acquired image is blurry at all wavelengths.

[0019] like Figure 1 As shown in the example of , the image capture system 10 is operable to acquire images of one or more objects 14, 16 in a scene 12, and to process the acquired images to obtain a reconstructed image 24 of the objects in the scene. The image capture system 10 may include a plurality of sensors 20A, 20B, each of which is operable to detect an optical signal and convert the detected optical signal into a corresponding electrical signal. Each sensor 20A, 20B has a respective photosensitive area (e.g., a pixel) and may be implemented as, for example, a CCD sensor that is operable to output an electrical signal representing a captured image. For example, as Figure 1As shown, the first sensor 20A is operable to output a first image A, and the second sensor is operable to output a second image B. In some embodiments, instead of separate sensors 20A, 20B, a single large optical sensor may be provided, wherein a first photosensitive area of ​​the sensor is used to capture and generate the first image A, and a second photosensitive area of ​​the sensor is used to capture and generate the second image B.

[0020] like Figure 1 As further shown in , the image capture system 10 also includes corresponding beam shaping elements (e.g., lenses) 18A, 18B to focus light reflected by objects in the scene 12 onto corresponding sensors in the sensors 20A, 20B. Each lens 18A, 18B can be implemented as, for example, a diffractive lens or a metalens. Further, in some cases, each corresponding lens 18A, 18B can be designed for a specific corresponding wavelength or narrow wavelength range. That is, each lens can be optimized to focus light in, for example, the visible region (e.g., red, blue, or green) or the infrared region of the electromagnetic spectrum. For example, the first lens 18A can be optimized for a first wavelength λ1 (e.g., a narrow wavelength range centered on λ1), while the second lens 18B can be optimized for a different second wavelength λ2 (e.g., a narrow wavelength range centered on λ2). In this case, the narrow range of wavelengths can be, for example, wavelengths within ten nanometers (nm) of the center wavelength of the range. For some embodiments, the range can vary (e.g., more or less). In some embodiments, the beam shaping element may produce an image that is not clear at any particular wavelength.

[0021] Since, in some cases, each beam shaping element 18A, 18B may be optimized for a specific corresponding wavelength or narrow wavelength range, in some embodiments, it may not be necessary to use an optical filter in conjunction with a lens. In such an embodiment, since there is no filter in front of the sensor, no light is discarded, which may allow more energy to be captured than a filter-based image sensor. This in turn may result in a system with higher sensitivity. However, in some cases, a corresponding optical filter may be used in conjunction with one or more of the beam shaping elements 18A, 18B.

[0022] The image capture system 10 further includes an image processor 22 operable to receive the images from the sensors 20A, 20B and process the images together to obtain a reconstructed image 24. The reconstructed image 24 may be stored in a memory and / or may be displayed on a display 26 (eg, a display monitor or display screen).

[0023] In some cases, different objects 12 in the scene reflect different wavelengths, respectively. To illustrate an example of the operation of the image capture system, assume that the scene includes one or more objects 14 that reflect light of a first wavelength λ1. For the sake of illustration, further assume that the scene also includes one or more other objects 16 that reflect light of a second wavelength λ2. However, in some cases, a particular object may reflect light of more than one wavelength (e.g., both wavelengths λ1 and λ2).

[0024] In operation, the controller 30 may generate one or more signals to trigger the sensors 20A, 20B to acquire a corresponding image of the scene 12 at a particular moment. The signal 30 from the controller is generated, for example, in an automatic manner or in response to a user input. In response, some light reflected by the objects 14, 16 in the scene 12 passes through the lenses 18A, 18B and is detected by the sensors 20A, 20B, respectively. The signal output by the first sensor 20A represents the first image A, and the signal output by the second sensor 20B represents the second image B.

[0025] The first image A and the second image B may each include features based on light detected at the first wavelength λ1 and at the second wavelength λ2. Figure 2 As shown, the first image A (i.e., the output of the first sensor 20A) may include a first feature 14A based on light detected at a first wavelength λ1, and a second feature 16A based on light detected at a second wavelength λ2. Since the first lens 18A that focuses the light onto the first sensor 20A is optimized for light at the first wavelength λ1, the first feature 14A will tend to be sharper than the second feature 16A, which may be slightly blurred.

[0026] Likewise, Figure 2 As shown, the second image B (i.e., the output of the second sensor 20B) may include a first feature 14B based on light detected at the first wavelength λ1, and a second feature 16B based on light detected at the second wavelength λ2. Since the second lens 18B that focuses the light onto the second sensor 20B is optimized for light at the second wavelength λ2, the second feature 16B will tend to be sharper than the first feature 14B, which may be slightly blurred.

[0027] Both image A and image B are provided as inputs to an image processor 22, which is operable to deconvolve image A and image B to obtain a reconstructed image 24. In this case, image deconvolution facilitates reconstruction of a latent image from two or more degraded images. That is, multiple images (each image having a different corresponding sharp component and blurred component) can be combined to produce a relatively sharp color image.

[0028] like Figure 3As shown, in some embodiments, the image processor 22 is operable to process the images to perform initial deblurring of the individual images A and B ( Figure 3 , at 102). Thus, a deblurring operation may be applied, for example, as part of a preprocessing stage. If the point spread function (PSF) of the optical system (e.g., lenses 18A, 18B) is known or can be calculated, then, for example, a non-blind deconvolution technique may be used for deblurring. If the PSF of the optical system is unknown, then a blind deconvolution technique may be used to deblur the image (i.e., to recover a relatively sharp version of the image from a blurred version). For some embodiments, other deblurring techniques may be used.

[0029] like Figure 3 As further shown, the method includes detecting a sharp (eg, high contrast) edge within each of the deblurred images ( Figure 3 , at 104). The portion of each image that is within the detected sharp edge may then be associated with a feature of an object 14, 16 in the scene 12 that reflects light of a corresponding wavelength (e.g., λ1 or λ2) ( Figure 3 , at 106). That is, the portion of the first deblurred image A that is within the detected clear edge can be associated with features of objects 14, 16 in the scene 12 that reflect light of the first wavelength λ1. Similarly, the portion of the second deblurred image B that is within the detected clear edge can be associated with features of objects 14, 16 in the scene 12 that reflect light of the second wavelength λ2. In some cases, other portions of the corresponding deblurred image can be ignored (e.g., discarded). On the other hand, in some embodiments (e.g., those using a neural network), other portions of the corresponding deblurred image can also be beneficial and, therefore, may not be discarded. The portions of the deblurred images that are within the detected clear edge can then be combined to obtain a reconstructed image 24 ( Figure 3 , at 108).

[0030] like Figure 4 As shown, in some embodiments, the image processor 22 includes a neural network 32 to deconvolve the image A and the image B to obtain the reconstructed image 24. Therefore, in some cases, the image processor 22 can incorporate artificial intelligence and can include iterative and / or deep learning techniques.

[0031] Although in the foregoing examples, the latent image is reconstructed from two degraded images, more generally, the latent image can be reconstructed from two or more degraded images (e.g., an array of images). For example, an image capture system can include an array of sensors (e.g., an array of 4 to 32 sensors), each sensor in the array of sensors having a corresponding associated beam shaping element (e.g., a diffractive lens or metalens) that is customized for operation at a different corresponding wavelength. Each sensor is operable to produce, for example, an image of a scene. However, depending on the wavelength of light reflected by various objects in the scene, different images in the image may contain some features that are relatively sharp and other features that are slightly blurred. An image processor is operable to deconvolute the various images obtained from the sensors in order to obtain a reconstructed version of the latent image.

[0032] Some embodiments also include additional features to facilitate calibration or recalibration of the image capture system. Recalibration may be desirable, for example, to correct for thermal effects on the optical sensor. Figure 5 As shown, the image capture system 10A may include an optical projector 36 operable to project an optical reference feature 40 onto a scene at a particular wavelength λ3. In addition to the first and second optical sensors 20A, 20B and their associated lenses 18A, 18B, the image capture system 10A further includes a third optical sensor 20C and an associated third lens (e.g., a diffractive lens or metalens) 18C. The third lens 18C may be optimized for wavelength λ3 (e.g., a narrow wavelength range centered around λ3) and operable to focus incident light onto the third sensor 20C.

[0033] As described above with respect to sensors 20A, 20B, the third sensor 20C also has light-sensitive areas (eg, pixels) and may be implemented, for example, as a CCD sensor operable to output electrical signals representative of a captured image. Figure 6 As shown, the third sensor 20C is operable to output a third image C. In some embodiments, a single large optical sensor may be provided, wherein a first photosensitive area of ​​the sensor is used to capture and generate a first image A based on light passing through the first lens 18A, a second photosensitive area of ​​the sensor is used to capture and generate a second image B based on light passing through the second lens 18B, and a third photosensitive area of ​​the sensor is used to capture and generate a third image C based on light passing through the third lens 18C.

[0034] Each of the images A, B, and C may include features based on light detected at the first wavelength λ1, the second wavelength λ2, and the third wavelength λ3. Figure 6As shown, first image A (i.e., the output of first sensor 20A) may include first feature 14A based on light detected at first wavelength λ1, second feature 16A based on light detected at second wavelength λ2, and third feature 40A based on light detected at third wavelength λ3 (e.g., corresponding to reference feature 40). Because first lens 18A, which focuses light onto first sensor 20A, is optimized for light at first wavelength λ1, first feature 14A will tend to be sharper than second feature 16A and third feature 40A, which may be slightly blurred.

[0035] Likewise, Figure 6 As shown, the second image B (i.e., the output of the second sensor 20B) may include a first feature 14B based on light detected at the first wavelength λ1, a second feature 16B based on light detected at the second wavelength λ2, and a third feature 40B based on light detected at the third wavelength λ3 (e.g., corresponding to the reference feature 40). Since the second lens 18B that focuses the light onto the second sensor 20B is optimized for light at the second wavelength λ2, the second feature 16B will tend to be sharper than the first feature 14B and the third feature 40B, which may be slightly blurred.

[0036] like Figure 6 As further shown in the example of , the third image C (i.e., the output of the third sensor 20C) may include a first feature 14C based on light detected at the first wavelength λ1, a second feature 16C based on light detected at the second wavelength λ2, and a third feature 40C (e.g., corresponding to the reference feature 40) based on light detected at the third wavelength λ3. Since the third lens 18C that focuses light onto the third sensor 20C is optimized for light of the third wavelength λ3, the third feature 40C corresponding to the reference feature 40 should be relatively sharp, while the other features 14C, 16C may be slightly blurred. If the feature 40C in the image C is not sharp enough, one or more optical components of the image capture system 10A may need to be adjusted. For example, the image processor may be operable to generate a signal indicating that the image capture device should be recalibrated or adjusted in response to determining that a feature in the image corresponding to the optical reference feature is not sharp enough.

[0037] Various aspects of the subject matter and functional operations described in this specification (e.g., operations described in connection with image processor 22, neural network 32, and / or controller 30) may be implemented in digital electronic circuitry, or in computer software, firmware, or hardware. Thus, various aspects of the subject matter described in this specification may be implemented as one or more computer program products, i.e., one or more modules of computer program instructions encoded on a computer-readable medium for execution by a data processing apparatus or for controlling the operation of a data processing apparatus. The computer-readable medium may be a machine-readable storage device, a machine-readable storage substrate, a memory device, a combination of substances that affect a machine-readable propagated signal, or a combination of one or more thereof. In addition to hardware, the apparatus may include code that creates an execution environment for the computer program in question, e.g., code constituting processor firmware.

[0038] A computer program (also referred to as a program, software, software application, script, or code) may be written in any form of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for a computing environment. A computer program does not necessarily correspond to a file in a file system. A program may be stored in a portion of a file that holds other programs or data (e.g., one or more scripts stored in a markup language document), in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or portions of code). A computer program may be deployed to execute on one computer or on multiple computers located at one site or distributed over multiple sites and interconnected by a communications network.

[0039] The processes and logic flows described in this specification can be performed by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can also be performed by special purpose logic circuits (e.g., FPGAs (field programmable gate arrays) or ASICs (application specific integrated circuits)), and the apparatus can also be implemented as special purpose logic circuits.

[0040] For example, processors suitable for executing computer programs include general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Typically, the processor will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. Computer-readable media suitable for storing computer program instructions and data include various forms of non-volatile memory, media, and memory devices, including, for example, semiconductor memory devices, such as EPROM, EEPROM, and flash memory devices; magnetic disks, such as internal hard disks or removable disks; magneto-optical disks; and CD ROM and DVD-ROM disks. The processor and memory may be supplemented by or integrated into special purpose logic circuits.

[0041] The image capture systems described in the present disclosure may be used for a variety of applications, including, for example, for reconstruction of RGB images or hyperspectral images (eg, medical, quality control, satellite imagery).

[0042] Although this document contains many specific implementation details, these should not be interpreted as limitations on any invention or the scope of what may be claimed, but rather as descriptions of features specific to particular embodiments. Certain features described in this document in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented in multiple embodiments individually or in any suitable sub-combination. Various modifications may be made to the foregoing examples. Therefore, other implementations are also within the scope of the claims.

Claims

1. An image capturing device, comprising: at least one sensor having a plurality of photosensitive regions, each photosensitive region of the plurality of photosensitive regions being operable to capture a respective image; a beam shaping element, each of the beam shaping elements being operable to focus light onto a different respective one of the photosensitive areas, wherein each of the beam shaping elements is individually configured to capture an image for image reconstruction; An image processor operable to receive a signal representing a corresponding image captured by the photosensitive region, wherein the image processor is further operable to deconvolve the received image and generate a reconstructed image based on the deconvolved image, wherein the reconstructed image has enhanced broadband image quality compared to the received image.

2. The image capture device according to claim 1, wherein: The beam shaping element includes a diffractive lens.

3. The image capture device according to claim 1, wherein: The beam shaping element comprises a metalens.

4. The image capture device according to any one of claims 1 to 3, wherein: The image processor includes a neural network operable to deconvolve a received image.

5. The image capture device according to claim 4, wherein: Each of the captured images has a different corresponding sharp component and blurry component, wherein the neural network is operable to combine the captured images to produce a sharper color image.

6. The image capture device according to any one of claims 1 to 3, wherein: The image processor is operable to: performing initial deblurring on the received image; detecting a sharp edge within each of the deblurred images; associating a portion of each image within the detected sharp edges with features of one or more objects reflecting light of a corresponding wavelength; as well as Portions of the deblurred images that are within the detected sharp edges are combined to obtain the reconstructed image.

7. The image capture device of any one of claims 1 to 6, comprising a plurality of light sensors, each light sensor of the plurality of light sensors comprising a different respective one of the light sensitive areas.

8. The image capture device according to any one of claims 1 to 7, further comprising: an optical projector operable to project an optical reference feature at a specific wavelength, in: A specific one of the beam shaping elements is operable to focus the reflected light onto a specific photosensitive area within the photosensitive area, and The at least one sensor is operable to generate an image including features corresponding to the optical reference features, and The image processor is operable to determine whether a feature in the image corresponding to the optical reference feature is sufficiently sharp.

9. The image capture device according to claim 8, wherein: The image processor is further operable to generate a signal indicating that the image capture device should be recalibrated or adjusted in response to determining that a feature in the image corresponding to the optical reference feature is not sufficiently sharp.

10. An image capturing method, comprising: focusing the light onto a different corresponding photosensitive area of ​​the plurality of photosensitive areas by each of a plurality of beam shaping elements, wherein each of the beam shaping elements is individually configured to capture an image for image reconstruction; Acquiring, by each corresponding one of the photosensitive regions, a corresponding image based on the light focused thereon; and The image is deconvolved to generate a reconstructed image, wherein the reconstructed image has enhanced broadband image quality compared to the received image.

11. The method according to claim 10, wherein: The beam shaping element includes a diffractive lens.

12. The method according to claim 10, wherein: The beam shaping element comprises a metalens.

13. A method according to any one of claims 10 to 12, comprising using a neural network to deconvolve the image.

14. The method according to claim 13, wherein: Each of the acquired images has a different corresponding sharp component and blurry component, and the method includes combining the acquired images to produce a sharper color image.

15. The method according to any one of claims 10 to 14, further comprising: Performing initial deblurring on the acquired image; detecting a sharp edge within each of the deblurred images; associating a portion of each image within the detected sharp edges with features of one or more objects reflecting light of a corresponding wavelength; as well as Portions of the deblurred images that are within the detected sharp edges are combined to obtain the reconstructed image.

16. The method according to any one of claims 10 to 15, further comprising: Projecting an optical reference signature at a specific wavelength; focusing the reflected light onto a specific photosensitive area in the photosensitive area by a specific beam shaping element among the beam shaping elements; generating an image based on signals from specific ones of the photosensitive regions, wherein the image includes features corresponding to the optical reference features; as well as A determination is made by an image processor as to whether a feature in the image corresponding to the optical reference feature is sufficiently sharp.

17. The method according to claim 16, further comprising: In response to determining that a feature in the image corresponding to the optical reference feature is not sufficiently sharp, a signal is generated indicating that recalibration is required.

Images