System and method for digital optical aberration correction and spectral imaging

Abstract

Systems and methods are provided for digital optical aberration correction and spectral imaging. The optical system may include an optical imaging portion for forming an optical image near an image plane of the optical system, a wavefront imaging sensor portion disposed near the image plane, the wavefront imaging sensor portion providing raw digital data related to a light field near the image plane and an image output, and a controller for processing the raw digital data and the image output to provide a deblurred image output, the controller including a memory portion for storing instructions and a processor portion for executing the instructions to receive the image input and the raw digital data of the light field impinging on the wavefront imaging sensor and generate a deblurred image based on an analysis of an optical coherence function at the image plane.

Description

[Technical Field]

[0001] Cross-reference of related applications This application claims priority to U.S. Provisional Patent Application No. 63 / 023,287, filed on 12 May 2020, and Israeli Patent Application No. 276922, filed on 25 August 2020, both of which are incorporated herein by reference.

[0002] This invention relates to the fields of optical imaging, digital blur correction, digital aberration correction, digital adaptive optics, and spectral imaging. [Background technology]

[0003] Optical aberrations are a measure of an imperfect image, leading to a decrease in image quality and sharpness. In geometrical optics, perfect imaging is usually described as a state in which rays from each light point in object space travel through the imaging system and converge to a single corresponding point on the image plane. In practical implementation, the wave nature of light must be taken into account, which prevents focusing light to an infinitely small point. Rather, in physical wave optics, if rays of light are perfectly focused, diffraction effects produce a spot of finite size. A typical representative diameter of such a spot is given by the Airy disk 1.22·λ / NA, where λ is the typical wavelength and NA is the numerical aperture of the cone of the cone from which the rays converge. Perfect imaging is usually called the "diffraction limit," and in terms of optical optics, a sufficient criterion is that the rays converge within the Airy disk.

[0004] Optical aberrations can arise from a wide range of light sources. Some are inherent to optical imaging systems due to design compromises resulting from conflicting demands between imaging requirements and system complexity. On the one hand, operational requirement parameters such as field of view, working distance, magnification, aperture size, and spectral range impose further constraints on optical design. These typically manifest as increased system complexity, including the number and types of optical elements, the materials used, surface profiles (spherical / aspherical), and increased alignment and manufacturing tolerances. On the other hand, forces toward simplification, such as cost, size, weight, readily available materials and processes, relaxed manufacturing tolerances, and robustness against environmental factors, can lead to a decrease in image quality. The use of multiple complex aspherical surfaces to reduce aberrations is known in the art.

[0005] Additional aberrations are also a result of manufacturing and alignment tolerances during the construction of the optical imaging system. Manufacturing and alignment tolerances can lead to the actual imaging system parameters deviating from the nominal design values. Other aberrations may arise from drift / changes in the optical system due to environmental factors such as thermal changes, pressure changes, shocks, and / or vibrations.

[0006] Optical aberrations can also arise from external influences of the imaging system. Imaging through turbulent media such as atmospheric disturbances can introduce time-varying optical aberrations that reduce imaging quality, for example, the "seeing" effect in terrestrial astronomy and surveillance. In medical ophthalmic imaging, aberrations present in the target eye reduce imaging resolution and often impair diffraction-limited resolution in imaging fine cellular structures on the retina. Imaging of bulk biological tissues is also limited by aberrations induced by translucent tissues.

[0007] At the current level of technology, adaptive optics systems are used to compensate for aberrations. Adaptive optics systems are used, in particular, to compensate for aberrations that depend on the imaging medium outside the imaging system. Adaptive optics systems are further used to compensate for time-dependent variations in optical system parameters. These adaptive optics systems consist of optical elements that can dynamically modulate optical wavefronts, such as deformable mirrors or spatial light modulators, feedback wavefront sensors such as Shack-Hartmann sensors, and control units that use data from the wavefront sensors to control the wavefront modulation elements using closed-loop feedback.

[0008] Adaptive optics systems are highly complex to integrate into existing imaging systems. Integrating an adaptive optics system into an existing imaging system typically requires altering the optical path to incorporate wavefront sensors and wavefront modulators, usually in the form of additional optical relays and beam splitters. Furthermore, many adaptive optics systems are designed so that both the wavefront sensor and wavefront modulator are located at or near the pupil of the imaging system, thereby uniformly sampling and correcting optical wavefront aberrations across the entire imaging field. Spatially dependent adaptive optics corrections are typically performed in scanning optics imaging systems, further complicating the overall system design. Often, an additional "star-shaped guide" illumination system is required to provide sufficient feedback signals to the wavefront sensor, further complicating the system and limiting its usefulness and operating range.

[0009] On the one hand, the conflicting demands of simple optical design and high-quality imaging necessitate an effective aberration correction scheme. On the other hand, physically correcting wavefronts using adaptive optics is extremely complex, has limited use cases, and is prohibitively expensive. Therefore, the idea of ​​digital aberration correction (or digital adaptive optics system) is very appealing, namely, performing digital algorithmic de-blurring on captured image data.

[0010] In image processing, aberrations are evident as blur kernels applied to nominally ideal images, resulting in blurred images. Often, these blur kernels are spatially dependent, meaning different blur kernels are applied to different parts of the image. In many cases, the actual blur kernel is unknown, especially when optical aberrations are caused by external factors in the optical imaging system, such as a cloudy imaging medium, or when optical imaging changes due to environmental factors such as ambient temperature and pressure.

[0011] Simple inverse superposition de-blurring of blurred images yields limited results for several reasons. Firstly, actual optical aberrations often degrade the modulation transfer function (MTF) to such an extent that digitally reconstructing it introduces unacceptable levels of noise. Furthermore, the blur kernel is often unknown, leading to a host of blind inverse superposition / blurring algorithms with limited success.

[0012] Another known technique for digital blur and aberration correction is field imaging, which is based on the use of plenooptic cameras, angle-sensing pixels, and variations thereof to capture field data. Field data is a geometric description of the ray trajectories propagating from the scene to the imaging device. While conventional imaging records the number of rays that hit each sensor pixel regardless of the angle of incidence of each ray, field imaging aims to record the position of each ray (i.e., the pixel it hits) and its angle of incidence using a plenooptic camera configuration or a microlens array within angle-sensitive pixels. Conceptually, field imaging provides data on the position and angle of rays as they propagate from the imaged object through the main imaging aperture onto the image sensor. Theoretically, this recording of ray angular position data can be used to virtually ray-track, propagate, and manipulate rays to estimate the distance between the object and the camera, and to digitally correct for anomalous rays that did not converge to perfect focus due to aberrations or simply being far from the focal plane.

[0013] It is known that field imaging cannot tolerate digital deblurring and image quality restoration until diffraction-limited complete imaging is achieved. Due to its inherent angular-spatial resolution trade-off, field imaging can yield image resolutions that are multiples (typically 3–10) of the diffraction-limited Nyquist sampling limit, given by Δx = λ / 4NA. This is because the entrance pupil of a plenoptic system, imaged through the main imaging lens, is only a fraction of the main lens's entrance pupil, resulting in much lower optical resolution. The logic behind this configuration is that each such partial pupillary segment represents a different ray angular bundle. Without segmentation of the main imaging pupil, the plenoptic system would be unable to identify the angular information of the rays imaged through the system, rendering the entire virtual ray tracing and propagation scheme ineffective.

[0014] Therefore, the promise that field imaging can be used to correct optical aberrations remains unfulfilled, as classical field imaging operates far from the optical diffraction limit of the main imaging aperture. The intrinsic image resolution and quality of field imaging are far lower than the effective degradation resolution induced by typical aberrations to diffraction-limited optics, leaving nothing to gain.

[0015] Further known techniques are based on sparse sampling and undetermined reconstruction, such as encoded aperture and compressed field-of-view techniques. However, these methods rely on heuristic assumptions about the imaged scene and therefore produce a large number of imaging results for scenes that include elements not covered by the algorithm's training set. Furthermore, the required aperture coding and compressed field-of-view masks block a significant portion of the light, reducing the signal reaching the detector sensor array and lowering the system's overall signal-to-noise ratio (SNR). These types of imaging systems, as with field-of-view imaging, are designed to operate away from the diffraction limit of the primary imaging aperture.

[0016] Correcting aberrations using hardware-based adaptive optics systems is highly complex and expensive, potentially limiting their effectiveness and imposing restrictive constraints on the operating range of the imaging system. Digital aberration correction based on deblurring and inverse superposition techniques of blurred raw images is typically limited by the inability to know the noise and aberration blur kernels in advance. Furthermore, field-of-view-based methods cannot perform diffraction-limited digital aberration correction due to inherent spatial and angular resolution limitations. Sparse sampling techniques are known to introduce unacceptable imaging results, reduce system SNR, and typically function far beyond the diffraction limit.

[0017] Conventional color imaging typically uses color filter arrays, such as RGB Bayer filter arrays and their variations. This results in significant loss of light transmission to the sensor. Typically, 50–70% of the light is filtered out. While the use of dichroic prisms can overcome this light loss, it comes at the cost of additional image sensor and system complexity. Other multispectral or hyperspectral imaging devices that rely on diffraction gratings can typically only function as one-dimensional line-scan cameras with narrow entrance slits.

[0018] A system and method are needed to perform digital aberration correction selectively and without light loss, up to the diffraction limit, to produce deblurred output images with maximum SNR, regardless of the captured input scene. In some cases, it is necessary to perform RGB color, multispectral, or hyperspectral imaging with a 2D imaging array and minimal light loss in conjunction with digital aberration correction. [Overview of the Initiative]

[0019] According to an embodiment of the present invention, an optical system is provided, comprising: an optical imaging unit for forming an optical image near the image plane of the optical system; a wavefront imaging sensor unit disposed near the image plane, which provides raw digital data relating to the light field near the image plane and the image output; and a control unit for processing the raw digital data and the image output to provide a deblurred image output, wherein the control unit includes a storage unit for storing commands and a processing unit that receives an image input and raw digital data of the light field colliding with the wavefront imaging sensor, and executes commands to generate a deblurred image based on an analysis of the optical mutual interference function on the imaging plane.

[0020] The control unit may include the steps of: calculating the field of view characteristics of the light illumination field; identifying point light sources in the image output based on coherence and superimposed information within the field of view characteristics; estimating the degree of blur for each identified point light source; and reconstructing a deblurred image as a composite of the deblurred point light sources.

[0021] The field of view characteristics may be a Wigner distribution or an equivalent entity related to a Wigner distribution through mathematical transformation.

[0022] The wavefront imaging sensor unit may further include a color filter array having at least two color bands, and the control unit calculates the field of view characteristics corresponding to each color band, generates a plurality of chromatic field of view characteristics, and reconstructs a deblurred image based on the combination of chromatic field of view characteristics.

[0023] The wavefront imaging sensor unit may include at least two image sensors, each associated with a spectral filter. The control unit calculates the field of view characteristics corresponding to each of the at least two image sensors, generates a plurality of chromatic field of view characteristics, and reconstructs a deblurred image based on the combination of chromatic field of view characteristics.

[0024] According to an embodiment of the present invention, the wavefront imaging sensor unit includes a light modulator unit disposed near an image plane, and an image sensor unit disposed downstream of the light modulator unit with respect to a general propagation direction of an input light irradiation field passing through the system in order to acquire a raw digital image output. The light modulator unit modulates the light irradiation field using at least one of phase modulation and amplitude modulation.

[0025] The light modulator unit may include a plurality of unit cells, the image sensor unit may include an array of sensor cells, the array of sensor cells defines a plurality of sensor sub-array unit cells, each sensor sub-array corresponds to a unit cell of the plurality of unit cells of the light modulator unit, the light modulator unit applies a predetermined modulation to the input light collected by the image sensor unit, and each unit cell of the light modulator unit directs a part of the collected input light incident thereon to the corresponding sensor sub-array unit cell and one or more adjacent sensor sub-array unit cells within a predetermined proximity region.

[0026] According to an embodiment of the present invention, the number of raw pixels N R of the plurality of sensor sub-array unit cells of the image sensor unit N is greater than or equal to the number of Nyquist sampling points N

[0027] According to another embodiment, the number of raw pixels N R of the plurality of sensor sub-array unit cells of the image sensor unit and the number of Nyquist sampling points N N of the light modulator unit satisfy the relationship N R ≧N N +Σ i N i A where 1≦N i A ≦N N and N i A represents the spatial variability of dynamic aberration.

[0028] According to an embodiment of the present invention, the control unit calculates the field of view characteristics corresponding to each unit cell, generates a plurality of chromatic field of view characteristics, and reconstructs at least one of the group consisting of a full-color RGB image and a hyperspectral image based on a combination of chromatic field of view characteristics, without using a spectral filter or color filter.

[0029] According to another embodiment, the control unit calculates the chromatic field of view characteristics corresponding to each unit cell, generates a plurality of chromatic field of view characteristics, and outputs an image with a number of pixels N. O and the number of chromatic color viewing characteristics N C Reconstruct the output image having N O The Nyquist sampling limit N N Lower than N C ×N O ≤N R And N R This is the raw pixel count.

[0030] According to one aspect of the present invention, if the light field includes a plurality of separate or continuous wavelengths, the control unit performs one or both of the following steps: (1) estimate the spectral distribution of each identified point light source; and (2) reconstruct a spectral distribution map of the image.

[0031] The control unit further performs one or more of the following steps: (1) estimating the aberration intensity of each identified point light source; (2) estimating the depth based on the dynamic aberration intensity estimated for each identified point light source, generating a spatial map of power aberration intensity, and reconstructing a depth map based on the spatial map of power aberration intensity; and (3) restoring the diffraction-limited imaging resolution of the out-of-focus portion of the image based on the depth map.

[0032] At least the unit cells of the optical modulator section and the array of sensor cells of the image sensor section may be manufactured as one of a group consisting of monolithic integrated units and separate units, such that the unit cells are part of a processing stack used to manufacture the array of sensor cells.

[0033] The optical modulation section may be one of the following: a binary design having one disk section per unit cell; a multilayer design consisting of concentric rings with equally spaced radii; a multilevel design consisting of at least two concentric rings with unequally spaced radii; a binary design having two disks per unit cell located at the edge of the unit cell; and a multiring design having two sets of rings per unit cell located at the edge of the unit cell.

[0034] The unit cell of the optical modulation section applies phase modulation, amplitude modulation, or both phase and amplitude modulation.

[0035] The optical imaging unit may be one of the group consisting of a refractive optical imaging unit, a reflective optical imaging unit, a reflective-refractive optical imaging unit, a diffractive optical imaging unit, and combinations thereof.

[0036] The optical imaging unit may include an objective lens and at least one of the group consisting of a tube lens, a relay optical system, and a telescope optical system for forming an image on the image plane. The optical imaging unit may be used as a camera lens and may be one of the group consisting of a wide-angle lens, a normal lens, a telephoto lens, and a zoom lens. The optical imaging unit may be one of the group consisting of a refracting telescope system, a reflecting telescope system, or a reflecting-refracting telescope system.

[0037] The wavefront imaging sensor unit may be one of the group consisting of a 2D area sensor, a line scan sensor, a multi-line scan sensor, and a TDI sensor.

[0038] According to one aspect of the present invention, a method for digitally correcting optical aberrations of an image captured by an imaging unit near the imaging plane of an optical system is provided, comprising the steps of: providing raw digital data to a light field that collides with a wavefront imaging sensor unit located near the image plane and to an image output formed near the image plane; and having a control unit process the raw digital data and the image output to provide a deblurred image output based on an analysis of the optical mutual coherence function at the image plane.

[0039] In one embodiment, the wavefront imaging sensor unit comprises an optical modulator unit positioned near the image plane to acquire a raw digital image output, and an image sensor unit positioned downstream of the optical modulator unit with respect to the general propagation direction of the input light field through the system, the method further comprising the step of modulating the light field using at least one of phase modulation and amplitude modulation.

[0040] In another embodiment, the optical modulator section includes a plurality of unit cells, the image sensor section includes an array of sensor cells, the array of sensor cells defines a plurality of sensor subarray unit cells, each sensor subarray corresponds to a unit cell of the plurality of unit cells of the optical modulator section, and the method is to apply a predetermined modulation to the incident light collected by the image sensor section using the optical modulator section, and to direct a portion of the collected incident light incident on each unit cell of the optical modulator section to its corresponding sensor subarray unit cell and one or more adjacent sensor subarray unit cells within a predetermined proximity region. Includes the number of raw pixels N of the multiple sensor subarray unit cells in the image sensor section. R This is the Nyquist sampling point N of the optical modulator section. N The number of raw pixels N of multiple sensor subarray unit cells in the image sensor unit. R and the Nyquist sampling point N of the optical modulator section N The number of relation N R ≥N N +Σ i N i A This may also apply, where 1 ≤ N i A≤N N And N i A This indicates the spatial variability of dynamic aberrations.

[0041] In one embodiment of the present invention, the method further includes the steps of: calculating the field of view characteristics of a light field; identifying point light sources in the image output based on coherence and superimposed information within the field of view characteristics; estimating the blur of each identified point light source; and reconstructing a deblurred image as a synthesis of deblurred point light sources. The field of view characteristics may be a Wigner distribution or an equivalent entity related to a Wigner distribution by mathematical transformation.

[0042] In one embodiment of the present invention, the wavefront imaging sensor further comprises a color filter array having at least two color bands, and the method further includes the steps of calculating chromatic field characteristics corresponding to each color band, generating a plurality of chromatic field characteristics, and reconstructing a deblurred image based on a combination of chromatic field characteristics.

[0043] In another embodiment, the wavefront imaging sensor unit comprises at least two image sensors, each associated with a spectral filter, and the method further includes the steps of: calculating chromatic field characteristics corresponding to each of the at least two image sensors; generating a plurality of chromatic field characteristics; and reconstructing a deblurred image based on a combination of chromatic field characteristics.

[0044] In yet another embodiment, the raw pixel count N of the multiple sensor subarray unit cells of the image sensor unit. R This is the Nyquist sampling point N of the optical modulator section. N The method comprises the steps of: calculating a chromatic field of view characteristic corresponding to each unit cell; generating a plurality of chromatic field of view characteristics; and reconstructing at least one of the group consisting of a full-color RGB image, a spectral filter, or a hyperspectral image without a color filter, based on a combination of chromatic field of view characteristics.

[0045] If the light field includes multiple wavelengths that are different or continuous, the method may further include a step of estimating the spectral distribution for each identified point light source. The method may further include a step of reconstructing a spectral distribution map of the image.

[0046] A further aspect of the present invention provides a method for processing raw digital data representing elements of the optical mutual coherence function of the light illumination field near the image plane of an optical system, the method comprising the steps of: calculating field characteristics based on the optical mutual coherence function; identifying point light sources in an output image near the image plane based on coherence and superposition information in the field characteristics; and for each identified point light source, estimating its blur, generating a point light source without blur, and reconstructing a deblurred image as a synthesis of the deblurred point light sources.

[0047] The method may further include any or a combination thereof of the following steps: estimating the degree of aberration intensity of each point light source; estimating the spatial spectral distribution of each point light source to obtain a set of point-specific spectral distribution estimates; reconstructing a spatial map of the spectral distribution of the light field based on the set of point-specific spectral distribution estimates; and transforming the optical mutual coherence function into a Wigner distribution using the Wigner-Weyl transform, and performing identification, estimation, and calculation based on the Wigner distribution. The step of estimating the degree of aberration intensity of each point light source may further include classifying the aberrations as static or dynamic aberrations.

[0048] According to one aspect of the present invention, a method for image processing is provided, the method comprising the steps of generating raw digital data representing elements of the optical interference function of an optical field near the image plane of an optical system, the generating step of modulating an optical field with an optical modulator having a plurality of unit cells, acquiring a raw digital image output by an image sensor located downstream of the optical modulator with respect to the general propagation direction of an input optical field through the system, the image sensor including a sensor cell array, the sensor cell array defining a plurality of sensor subarray unit cells, each sensor subarray corresponding to direct a portion of the collected input light incident thereon by each unit cell of the optical modulator to its corresponding sensor subarray unit cell and one or more adjacent sensor subarray unit cells in a predetermined proximity region, and processing the raw data according to an embodiment of the present invention.

[0049] According to another aspect of the present invention, a method is provided for processing raw digital data representing elements of the optical mutual coherence function of the optical field near the image plane of an optical system, the method comprising the steps of: calculating field characteristics based on the optical mutual coherence function; identifying point light sources in an output image near the image plane based on coherence and superposition information in the field characteristics; and estimating the spatial spectral distribution of each identified point light source and generating a set of point-by-point spectral distribution estimates.

[0050] The method may further include any or a combination thereof of the following steps: reconstructing a spatial map of the spectral distribution of a light field based on a set of point-by-point spectral distribution estimates; estimating the blur of each identified point light source, generating deblurred point light sources, and reconstructing a deblurred image as a composite of the deblurred point light sources; estimating the degree of aberration intensity of each point light source; and transforming the optical mutual coherence function into a Wigner distribution using the Wigner-Weyl transform, and performing the identification, estimation, and calculations based on the Wigner distribution. The step of estimating the degree of aberration intensity of each point light source may further include classifying the aberrations as static or dynamic aberrations.

[0051] According to one aspect of the present invention, a wavefront imaging sensor system is provided for acquiring a raw digital image output, comprising an optical modulator unit and an image sensor unit positioned downstream of the optical modulator unit with respect to the general propagation direction of the input light field passing through the system, wherein the optical modulator unit comprises a plurality of unit cells, the image sensor unit comprises an array of sensor cells, the array of sensor cells defines a plurality of sensor sub-array unit cells, each sensor sub-array corresponds to a unit cell of the plurality of unit cells of the optical modulator unit, and the number of raw pixels of the plurality of sensor sub-array unit cells of the image sensor unit is N R This is the Nyquist sampling point N of the optical modulator section. N The number of pixels is greater than or equal to the number of pixels in the multiple sensor subarray unit cells of the image sensor unit. The optical modulator unit applies a predetermined modulation to the input light collected by the image sensor unit. Each unit cell of the optical modulator unit directs a portion of the collected input light incident thereto to its corresponding sensor subarray unit cell and one or more adjacent sensor subarray unit cells within a predetermined proximity area. The design of the unit cell is such that the number of raw pixels in the multiple sensor subarray unit cells of the image sensor unit is greater than or equal to the number of pixels in the multiple sensor subarray unit cells of the image sensor unit. R The Nyquist sampling point N of the optical modulator section NThe number of units must be greater than or equal to at least one of the following conditions: a binary design having one disk portion per unit cell; a multilayer design consisting of concentric rings with equally spaced radii; a multilevel design consisting of at least two concentric rings with unequally spaced radii; a binary design having two disks per unit cell located at the edge of a unit cell; a multiring design having two sets of rings per unit cell located at the edge of a unit cell; and a quasi-periodic design.

[0052] According to another aspect of the present invention, a method is provided for designing a wavefront imaging sensor system to acquire a raw digital image output, the wavefront imaging sensor system having an optical modulator and an image sensor unit positioned downstream of the optical modulator with respect to the general propagation direction of the input light field through the system, the method comprising the steps of providing a plurality of unit cells in the optical modulator and providing an array of sensor cells in the image sensor unit, wherein the array of sensor cells defines a plurality of sensor subarray unit cells, and each sensor subarray has a raw pixel count N of the plurality of sensor subarray unit cells in the image sensor unit. R The Nyquist sampling point N of the optical modulator section N The number of raw pixels N of the plurality of sensor subarray unit cells of the image sensor unit is greater than or equal to the number of such units, and the optical modulator unit applies a predetermined modulation to the input light collected by the image sensor unit, and each unit cell of the optical modulator unit directs a portion of the collected incident light incident therein to the corresponding sensor subarray unit cell and one or more adjacent sensor subarray unit cells within a predetermined proximity region, a step corresponding to the plurality of unit cells of the optical modulator unit, and the number of raw pixels N of the plurality of sensor subarray unit cells of the image sensor unit. R The Nyquist sampling point N of the optical modulator section NThe step of designing such a design that is greater than or equal to the number of units, and conforms to at least one of the following conditions: a binary design having one disk portion per unit cell, a multilayer design consisting of concentric rings having equally spaced radii, a multilevel design consisting of at least two concentric rings having unequally spaced radii, a binary design having two disks per unit cell located at the edge of a unit cell, a multiring design having two sets of rings per unit cell located at the edge of a unit cell, and a quasi-periodic design.

[0053] According to one aspect of the present invention, an optical system is provided, comprising: an optical imaging unit for forming an optical image near the image plane of the optical system; an adaptive optics system unit located between the optical imaging unit and the image plane, comprising: a wavefront imaging sensor unit positioned near the image plane; a wavefront imaging sensor unit providing raw digital data to the light field near the image plane and the image output; and a control unit for processing the raw digital data and the image output to provide a deblurred image output, wherein the control unit comprises: a storage unit for storing commands; a processing unit for receiving the image input and raw digital data of the light field colliding with the wavefront imaging sensor, and executing commands to generate a deblurred image based on an analysis of the optical mutual interference function at the imaging plane; and the adaptive optics system unit comprising: an adaptive optics wavefront sensor and an adaptive optics wavefront modulator positioned near the pupil plane of the imaging system; and an adaptive optics control unit, wherein the adaptive optics system wavefront sensor provides feedback for driving the wavefront modulator under the control of the adaptive optics system control unit. The adaptive optics system unit may further comprise a pupil relay and a tube lens. The control unit receives information from the adaptive optics wavefront sensor and provides coarse measurements of wavefront aberration. The optical imaging unit, wavefront sensor, and control unit, as well as the adaptive optical wavefront sensor, adaptive optical wavefront modulator, and adaptive optical control unit, can be implemented according to any aspect and embodiment of the present invention. [Brief explanation of the drawing]

[0054] For a better understanding of the present invention and embodiments thereof, refer to the accompanying drawings. In the drawings, similar reference numerals indicate corresponding entities throughout.

[0055] [Figure 1a] This is a schematic block diagram showing a digital optical aberration correction system according to one embodiment of the present invention. [Figure 1b] This is a schematic block diagram illustrating a digital optical aberration correction system according to another embodiment of the present invention. [Figure 2a] This is a schematic diagram illustrating an embodiment of the present invention. [Figure 2b] This is a schematic diagram illustrating an embodiment of the present invention. [Figure 2c] This is a schematic diagram illustrating an embodiment of the present invention. [Figure 3a] The present invention presents a method according to an embodiment of the present invention. [Figure 3b] The present invention presents a method according to an embodiment of the present invention. [Figure 4] This is a diagram showing a light source reference object – a Siemens star-shaped target. [Figure 5a] This figure shows exemplary characteristics of a system according to one embodiment of the present invention. [Figure 5b] This figure shows exemplary characteristics of a system according to one embodiment of the present invention. [Figure 6a] This figure shows the optical properties of a system according to one embodiment of the present invention. [Figure 6b] This figure shows the optical properties of a system according to one embodiment of the present invention. [Figure 7] This figure shows another optical property of the system according to one embodiment of the present invention. [Figure 8a] This figure shows a blurred image of the light source reference object in Figure 4, according to one embodiment of the present invention. [Figure 8b] This figure shows a de-blurred image of the light source reference object in Figure 4, according to one embodiment of the present invention. [Figure 9a]This figure schematically shows the arrangement of raw pixels according to an embodiment of the present invention. [Figure 9b] This figure schematically shows the arrangement of raw pixels according to an embodiment of the present invention. [Figure 10a] This figure schematically shows an embodiment of the optical modulator section according to the present invention. [Figure 10b] This figure schematically shows an embodiment of the optical modulator section according to the present invention. [Figure 10c] This figure schematically shows an embodiment of the optical modulator section according to the present invention. [Figure 10d] This figure schematically shows an embodiment of the optical modulator section according to the present invention. [Figure 11a] This figure schematically shows another embodiment of the optical modulator section according to the present invention. [Figure 11b] This figure schematically shows another embodiment of the optical modulator section according to the present invention. [Figure 12a] This figure schematically shows an embodiment of a wavefront imaging sensor according to the present invention. [Figure 12b] This is a block diagram of a system according to an embodiment of the present invention. [Figure 13a] This figure schematically shows a spectral band filter according to one embodiment of the present invention. [Figure 13b] This figure schematically shows the spectral distribution according to one embodiment of the present invention. [Figure 14] This is a schematic block diagram showing a digital optical aberration correction system according to one embodiment of the present invention. [Modes for carrying out the invention]

[0056] According to embodiments of the present invention, a system and method for digital optical aberration correction are described. The proposed system can correct optical aberrations to the diffraction limit without introducing imaging results. In some implementations, the proposed system can correct optical aberrations without reducing the light transmittance through the optical system. The ability to correct optical aberrations in an imaging system can result in a simplified optical system, improved operating parameters, and a wider range of applications.

[0057] According to one embodiment of the present invention shown in Figure 1a, the digital optical aberration correction system 10 comprises an optical imaging unit 102, also called an "imaging optical system." The optical imaging unit 102 is used to form an optical image on or near the image plane 104. It also includes a wavefront imaging sensor unit 106 and a control unit 108. The control unit 108 is for processing the raw digital image output of the object 100 to provide a deblurred image output. The control unit 108 may also measure the optical aberration intensity on the image plane 104.

[0058] Embodiments of the present invention can be interpreted to address several applications, such as microscopes, cameras, telescopes, and long-range imaging systems. For example, in a microscope application, the optical imaging unit 102 may consist of an objective lens, optionally a tube lens, and other relays or telescope optics (not shown in Figure 1) for forming an image on the image plane 104 of the system. In a camera application, the optical imaging unit 102 may be used as a camera lens, which may be a wide-angle lens, a standard lens, a telephoto lens, or a zoom lens (not shown in Figure 1a). In the case of telescopes and long-range imaging systems, the optical imaging unit 102 may consist of a refractive, reflecting, or reflector-refracting telescope system (not shown in Figure 1a).

[0059] An optical system 12 according to one embodiment of the present invention is shown in Figure 1b, where the wavefront imaging sensor unit 106 may consist of an optical modulator unit 110 near the image plane 104. The optical modulator unit 110 modulates light using phase modulation, amplitude modulation, or both. The wavefront imaging sensor unit 106 may further consist of an image sensor detector array 112, also called a downstream image sensor unit 112. The image sensor detector array 112 is for acquiring the raw digital image output.

[0060] The wavefront imaging sensor unit 106 can be configured as one of the following conventional imaging sensors, depending on the application: a 2D area sensor, a line scan sensor, a multi-line scan sensor, or a TDI sensor.

[0061] The control unit 108 (shown in Figures 1a and 1b) may consist of a processing unit and a storage unit (not shown). The processing unit may be used to perform digital algorithm processing, as will be described later. The storage unit may store pre-calculated, pre-measured, or otherwise predetermined data indicating static aberrations of the optical image unit, as will be described below.

[0062] The memory unit can also be used to store pre-calculated, pre-measured, or otherwise predetermined data indicating the chromatic characteristics of the optical image unit. The memory unit can store pre-calculated, pre-measured, or otherwise predetermined data indicating the chromatic characteristics of the wavefront imaging sensor unit 106 and optionally the optical modulator unit 110.

[0063] The optical imaging unit 102 can be of any type, including refraction, reflection, reflex-refractive, diffraction, or any combination thereof. The optical imaging unit 102 can be designed and configured with relaxed optical aberration specifications to simplify its design and manufacturing process while restoring image quality and sharpness by relying on the digital aberration correction function of the system described herein. The types of aberrations induced by the optical imaging unit 102 that can be corrected by system 10 include, but are not limited to, Seidel aberrations—spherical aberration, coma aberration, astigmatism, field curvature, and distortion, transverse and longitudinal chromatic aberrations, and optionally their higher-order counterparts.

[0064] The wavefront imaging sensor unit 106 plays a role in providing raw data to the control unit 108 in order to enable diffraction-limited deblurring reconstruction. The wavefront imaging sensor unit 106 can also provide measured values ​​of optical aberration intensity at the image plane 104. The term "raw data" refers to the digitization of light intensity measured by the image sensor 106. The raw data is further processed by the control unit 108, for example, as described below, to provide a measure of elements of the optical mutual interference function or equivalent entities such as the Wigner distribution. The wavefront imaging sensor 106 is designed to provide data on elements of the optical mutual interference function of the light field near the image plane 104 of the imaging system at diffraction-limited sampling resolution. Standard Shack-Hartmann sensors, plenooptic camera sensors, or angle-sensing pixel sensors cannot provide this data at diffraction-limited sampling resolution, as described in the background art section.

[0065] As used herein, the term “wavefront imaging sensor” refers to the ability to provide diffraction-limited sampling resolution. As used herein, the term “wavefront sensor” refers to conventional technologies such as Shack-Hartmann sensors, plenooptic cameras, or angle-sensing pixels that cannot achieve diffraction-limited sampling resolution.

[0066] According to one embodiment of the present invention, the wavefront imaging sensor unit 106 and the control unit 108 may operate in accordance with the principles described in International Publication No. 2018 / 185740, which is incorporated herein by reference. According to embodiments of the present invention, the wavefront imaging sensor 106 comprises an optical modulator 110 (also called an encoder) located near the image plane 104 of the imaging optical system and a downstream image sensor detector array 112. The optical modulator 110 (encoder) may be a static passive optical element that performs amplitude modulation, phase modulation, or both, on light reaching the image plane 104. If only phase modulation is performed, the optical modulator 110 does not cause any additional loss in the optical system 10. From there, the modulated light propagates to the downstream detector array 112, is sampled, and then digitized. The raw digital data—the output of the subpixel sensor cell array 112—is then processed by the control unit 106. The relationship of the encoder unit 110 to the Nyquist sampling of the light field at the image plane 104 is described below. The control unit 106 can be used to perform the functions of the control unit described in International Publication No. 2018 / 185740, as well as additional control functions, for example, as described below. These control functions may be performed by separate control units.

[0067] It should be noted that in some embodiments of the present invention, the optical modulator 110 may be a separate component from the detector array 112. In other embodiments, the optical modulator 110 may be monolithically integrated with the detector array 112. According to yet another embodiment, the optical modulator 110 may be part of a processing stack used to manufacture the sensor array 112, using, for example, a metallization process or a processing step similar to those used to manufacture a microlens array used to improve the pixel filling rate of the sensor array.

[0068] The wavefront imaging sensor unit 106 provides measured values ​​of the optical mutual interference function at the imaging surface 104 of the imaging system. The optical mutual coherence function p of the light irradiation field E(x) defines the time-averaged spatial correlation of the field.

[0069] (Math 1)p(x,x´)=〈E(x)E*(x´)〉

[0070] Here, in equation 1, x = (x, y) relates to the lateral spatial coordinates of the field. In general, the use of detector arrays and digitized data requires a specific discretization of mutual coherence, which provides a mutual coherence matrix of the following form.

[0071] (Number 2)p ij = <E(x i )E*(x j )〉

[0072] Here, in equation 2, E(xi) is a point In relation to the complex amplitude of the field in JPEG0007887173000001.jpg1340, E*(xj) is a point This relates to the complex conjugate of the field in JPEG0007887173000002.jpg1340. Note that the physically possible realizations of the mutual coherence matrix are Hermitian and non-negative.

[0073] As is known in the art, conventional imaging and detection techniques, or typical detection devices or systems, provide data showing diagonal elements corresponding to conventional intensity measurements, i.e., I i =p ii = <E(x i )E*(x i )〉. Measured intensity I i =p ii This indicates the number of photons / rays colliding with each detector pixel regardless of the angle of incidence, and provides a discretized measure of the lateral ray position x. Measured intensity I i =p ii This does not provide information about off-diagonal elements. The off-diagonal elements of the mutual coherence function include the phase and coherence relations of the light field.

[0074] The phase and coherence relationships in off-diagonal elements indicate the angular information of rays colliding with the wavefront imaging sensor 106 at the image plane 104 of the optical system. In physical optics, the correlation between wave position and angle is fully described by the Wigner distribution W(x, θ), where x=(x, y) relates to the lateral spatial coordinates of the field and θ=(θχ, θy) describes the ray trajectory angle. At the limits of ray optics, the Wigner distribution degenerates into a classical field description of geometric optics rays where the position x and propagation angle θ of each ray are recorded. However, the Wigner distribution is, in practice, a "quasi-probability" that can achieve negative values, demonstrating the wave-like phenomena of physical optics.

[0075] Mathematically, the Wigner distribution and the mutual coherence function are perfectly equivalent, and they are related by the Fourier transform (Wigner-Weyl transform).

[0076]

number

[0077] As is known in the art, classical field-of-light plenooptic cameras, angle-sensitive pixel sensors, or Shack-Hartmann sensors cannot provide diffraction-limited imaging resolution or aberration deblurring. According to embodiments of the present invention, the wavefront imaging sensor 106 can provide diffraction-limited sampling of the mutual coherence function. A Wigner distribution is obtained from the Wigner-Weyl transform using diffraction-limited spatial sampling. The angular sampling resolution of the resulting Wigner distribution is related to the maximum out-of-diagonal distance measured in the mutual coherence function.

[0078] According to embodiments of the present invention, the combined position angle information included in the Wigner distribution is then used to perform digital aberration correction. For illustrative purposes, the following simple example illustrates how digital aberration correction is performed. The simple example shown can be extended to cover all aberration types across any type of imaging scene.

[0079] The optical configuration 20 is shown in Figure 2a. In this simple example, consider imaging of two separate but close point light sources a and b, imaged using the complete optical system 202. The overlapping blurred spot a and blurred spot b images are shown in more detail in Figure 2b. The two point light sources a and b are positioned outside the focal plane of the imaging system, i.e., out of focus, and possibly even with different degrees of blur. In this example, blur and subsequent blurring are treated as examples representing imaging aberrations in general and can be extended to any type of aberration. The degree of blur and spatial separation is chosen so that the blurred images of the two point light sources a and b overlap.

[0080] If the point light sources are more widely spaced so that the blurred images do not overlap, known techniques may even be appropriate to digitally deconvolve each of those images and estimate the degree of their aberrations using phase retrieval or other techniques. In general, a blurred image contains many overlapping, arbitrarily different blur kernels, each corresponding to a point light source in the object space in which it was captured.

[0081] Therefore, it appears that the current common problem is how to perform a digital estimation of a blur kernel for two or more overlapping blur kernels that may have different degrees of aberration, as in the simple example in the description shown in Figure 2a. Here, we refer to the description of ray-tracing geometric optics for the problem at hand, shown in Figure 2c. We find that within the overlapping blur region 214 of two blurred blur spot images a and b, there is a point containing rays 216 converging toward both the image of the first point light source (point light source a) and the image of the second point light source (point light source b). In the particular example in Figure 2c, the convergence point is located behind the image plane 204 of the imaging system and is therefore the focused blurred point light source image. Without losing generality, the same considerations are valid if one or both point light sources are imaged as points in front of the image plane 204 (not shown). Within the overlapping region 214, each spatial point contains two rays 216 having different angles, one ray 216a from a binary point light source (point light source a) and the other ray 216b from a second point light source (point light source b).

[0082] Assuming the use of a conventional plenoptic light irradiation field sensor or an angle-sensing pixel imaging sensor disposed on the image plane 204, these types of sensors are configured to provide a one-to-one correspondence between the output data and the input light rays of the light impinging on the sensor. Each light irradiation field data point corresponds to a specific position within the space x having a unique light ray with propagation θ. The angle-sensing pixel detector also provides such one-to-one information. When a particular pixel lights up, it indicates the light ray position x and a single angle of incidence θ. On the other hand, as shown in the simple example illustrated in FIGS. 2a-2c earlier, within the overlapping region 214, there are positions that contain two distinct light rays having different angles passing through them. These types of double or multiple light ray points cannot be accurately recorded by conventional angle-sensitive or plenoptic light irradiation field sensors because the one-to-one correspondence is broken. Also, note that this one-to-one correspondence constraint is also part of the reason why conventional angle-sensing pixel sensors and plenoptic light irradiation field sensors cannot perform imaging up to the diffraction limit. When operating at a resolution coarser than the diffraction-limited spatial resolution, the ray tracing light irradiation field description of the light irradiation field does not include ambiguous regions where the one-to-one correspondence between the ray position and the angle is broken.

[0083] On the other hand, the Wigner distribution W(x, θ) can safely describe and quantify the ambiguity where the one-to-one angle position correspondence breaks down. The Wigner distribution is extracted from the mutual coherence function p(x, x') = <E(x)E*(x')〉 using the Wigner-Ville transform, and recall that the Wigner-Ville transform was estimated by the wavefront sensor section. The Wigner distribution is not only a wave optics analog of the geometric ray optics description of classical light irradiation fields. The Wigner distribution also includes coherence and interference information. The Wigner distribution accurately records and describes combinations of overlapping light rays, such as the light rays 216 described in the overlapping region 214 of the example shown in FIGS. 2a-2c above.

[0084] Using the Wigner distribution description, it is possible to both measure and identify overlapping but distinct rays within the overlapping blur region 214. By doing so, the problem can be reduced to a simpler one of non-overlapping blurred point sources. Blur correction can be performed, and the degree of aberration and focus blur that caused the blur can also be estimated.

[0085] The Wigner distribution contains data that shows the spectral components of the light field. This is in addition to data that shows the intensity, coherence, and phase of the light field. The spectral content can be accessed by the wavelength dependence described in Equation 3.

[0086] Wavefront imaging sensors may also be used to extract spectral information about the captured input scene. The spectral information contained in the Wigner distribution can be used to estimate the spectral distribution ("spectrum") of each point light source. The spectral information may optionally be used to reconstruct a spectral distribution map of the captured scene as either a color (RGB), multispectral, or hyperspectral image.

[0087] Figure 3a is a flowchart showing a method 30 according to one embodiment of the present invention for digital blur reduction, aberration estimation, and optionally spectral distribution estimation, using, for example, the configurations shown in Figures 1a-1b. For ease of explanation, the steps will be presented with reference to the imaging scenarios shown in Figures 2a-2c.

[0088] Method 30 begins with an operation 300 in which an image is captured and raw data is generated using an imaging wavefront sensor unit 106 located near the image plane 104 of the imaging system, as shown in Figures 1a and 1b. For example, the raw data generated by the imaging wavefront sensor unit 108 is passed to a control unit 108, as shown in Figures 1a and 1b, via a detector pixel array 112, as shown in Figure 1b.

[0089] Following operation 300, operation 302 is performed, for example by the control unit 108, to calculate field-of-view characteristics such as the Wigner distribution of the light field impacting the wavefront imaging sensor 106. According to one embodiment of the present invention, operation 302 is performed by performing operation 302a, which calculates the optical mutual interference function based on raw data, and operation 302b, which converts the optical mutual coherence function to an equivalent optical Wigner distribution using the Wigner-Weyl transform (Equation 3).

[0090] Operation 302 is followed by operation 304, which separates the individual point light sources. The combined coherence and angle information in the Wigner distribution is used to identify different point light sources, including the entire captured scene.

[0091] Operation 304 is followed by operation 306, which calculates the blur of each point light source.

[0092] Operation 306 may optionally be followed by operation 308, which estimates the degree of aberration intensity of each point light source.

[0093] Operation 306 (or, optionally, operation 308) is followed by operation 309, which estimates the spatial spectral distribution of each point light source.

[0094] Operation 306 (or, optionally, operation 308 or 309) is followed by operation 310, which reconstructs the de-blurred image. For example, the de-blurred image is reconstructed as a composite of de-blurred point light sources.

[0095] After operation 310, operation 312 can be optionally performed to reconstruct spatial maps of various aberration intensities from point-by-point aberration estimates.

[0096] Operation 310 (or, optionally, operation 312) is followed by operation 314, which optionally calculates the spectral distribution of each point light source.

[0097] After operation 314, operation 316 can be optionally performed to reconstruct a spatial map of the spectral distribution of the input light from the point-by-point spectral distribution estimates.

[0098] Figure 3b is a flowchart showing a method 34 according to one embodiment of the present invention for spectral distribution estimation using, for example, the configuration shown in Figures 1a-1b. For ease of explanation, the steps will be presented with reference to the imaging scenarios shown in Figures 2a-2c.

[0099] Method 34 begins with an operation 340 in which an image is captured and raw data is generated using an imaging wavefront sensor unit 106 located near the image plane 104 of the imaging system, as shown in Figures 1a and 1b. For example, the raw data generated by the imaging wavefront sensor unit 108 is passed to a control unit 108, as shown in Figures 1a and 1b, via a detector pixel array 112, as shown in Figure 1b.

[0100] Following operation 340, operation 342 is performed, for example by the control unit 108, to calculate field-of-view characteristics such as the Wigner distribution of the light field impacting the wavefront imaging sensor 106. According to one embodiment of the present invention, operation 342 is performed by performing operation 342a, which calculates the optical mutual interference function based on the raw data, and operation 342b, which converts the optical mutual coherence function to an equivalent optical Wigner distribution using the Wigner-Weyl transform (Equation 3).

[0101] Operation 342 is followed by operation 344, which separates the individual point light sources. The combined coherence and angle information in the Wigner distribution is used to identify different point light sources, including the entire captured scene.

[0102] Operation 344 is followed by operation 346, which estimates the spatial spectral distribution of each point light source.

[0103] Operation 346 is followed by operation 348, which reconstructs a spatial map of the spectral distribution of the input light from the point-by-point spectral distribution estimates.

[0104] Optionally, some or all of the following additional operations for aberration correction are performed: operation 350 to calculate the blur of each point light source, operation 352 to estimate the aberration intensity of each point light source, operation 354 to reconstruct the deblurred image, and operation 356 to reconstruct a spatial map of various aberration intensities from the point-by-point aberration estimates.

[0105] Without loss of generality, the Wigner distribution and the mutual coherence matrix are mathematically equivalent and related by a change in the representative basis, so that it is possible to perform methods 30 or 34 for calculating the field properties in any basis representation that includes the required coherence, angle, phase, and spectral information, respectively. Thus, operations 302, 304, and 306 of method 30 above, or operations 342, 344, and 346 of method 34 above, can be performed in a Wigner basis, a mutual coherence basis, or any other basis obtained as a combination of mutual coherence or Wigner distribution matrix elements.

[0106] For illustrative purposes, an example of a numerical simulation illustrating the method and system performance according to one embodiment of the present invention is shown. In this example, a Siemens star target shown in Figure 4 is used as the light source reference object. The target (i.e., Figure 4) has periodicity and extreme contrast values ​​as well as a geometric structure. The Siemens star target performance of an optical imaging system (e.g., as shown in Figure 1a or Figure 1b) for the Siemens star target is shown below.

[0107] An optical imaging system (e.g., system 10 or 12 shown in Figures 1a and 1b) may have an aberration represented by the point spreading function (PSF) 50 shown in Figure 5a. As is clear, this is very far from the diffraction-limited spot. This corresponds to the wavefront error 52 shown in Figure 5b, which is approximately four-wave peak-to-peak. Imaging of the Siemens star target (Figure 4) using such an anomalous imaging system yields the corresponding optical interference function as shown in Figure 6a (real part 60) and Figure 6b (imaginary part 62). The equivalent Wigner distribution 70 is obtained from the Wigner-Weyl transform and is shown in Figure 7. Figure 8a shows a blurred image 80 obtained from the anomalous optical imaging system. Figure 8b shows a deblurred image 82 reconstructed from the information in the Wigner distribution 70 using the method described above.

[0108] The methods according to embodiments of the present invention can be used to correct optical aberrations in a wide range of applications. According to embodiments of the present invention, several types of aberrations can be addressed.

[0109] Static aberrations induced by the design and manufacture of optical imaging systems. These static aberrations can be determined in advance by appropriate calculations, measurements, and characterization of the optical imaging system.

[0110] Dynamic aberrations can be induced by either variations in the optical imaging system due to environmental factors, or by changes in the medium in which imaging is performed, such as atmospheric turbulence or optic aberrations. These dynamic aberrations are not known in advance and may differ from image to image based on changes in external factors affecting individual imaging systems.

[0111] Chromatic aberration can be spectral aberration in either the transverse or longitudinal direction, a combination thereof, or a higher-order spectrally dependent variant. These types of chromatic aberration can also be further classified as static or dynamic according to the definitions above.

[0112] Referring again to method 30 in Figure 3a, according to embodiments of the present invention, the blur estimation operation 306 can take into account the classification of aberrations as static or dynamic aberrations. In the static case, the required blur is known in advance, and in the dynamic case, it is calculated quickly for each image instance. Static aberration parameters can be calculated or measured, or determined in advance and stored in the memory of the control unit.

[0113] The distinction between static and dynamic aberrations can also affect the number of pixels used in the wavefront imaging sensor (element 106 in Figures 1a and 1b). To explain this, consider the raw pixel count N of the image sensor. R The number of sampling points N required is determined according to the diffraction-limited Nyquist sampling limit given by Δχ = λ / 4NA. N It is necessary to distinguish between the two. In conventional imaging, a well-sampled image can be obtained if the number of sensor pixels is less than or equal to the Nyquist limit Δx = λ / 4NA, that is, if the number of raw pixels is at least equal to the number of Nyquist sampling points. This is due to the number of independent degrees of freedom of the intensity of the light field on the image plane. The number of independent degrees of freedom is equal to the Nyquist sampling points N N It can be expressed by the number of degrees of freedom. Therefore, to fully explain these degrees of freedom, they must be at least N R ≥N N The sampling must be performed at the sampling point.

[0114] This theoretical basis is carried over to the design of the wavefront imaging sensor (e.g., element 106 in Figures 1a and 1b). If all aberrations are static and known in advance, the number of unknown degrees of freedom is N. N Therefore, the required number of raw pixels is at least the number of Nyquist sampling points N. R ≥N N It must be so. However, if dynamic aberrations should be considered, there are further unknown degrees of freedom. Dynamic aberrations such as dynamism, coma aberration, and astigmatism A i Each type has a new set of degrees of freedom N i ARelated to this. The assumed spatial variation of each aberration is, for each N i A For spatially invariant aberrations, the value of -N is relevant. i A =1, in the case of dynamic aberrations where any intensity value can be taken at any point on the image plane -N i A =N N For aberrations with slow spatial dependence -N i A <N N , that is.

[0115] Therefore, in order to fully consider the degrees of freedom and dynamic aberrations of the image, the wavefront imaging sensor has at least N values ​​depending on the spatial variability of each dynamic aberration. R ≥N N +Σ i N i A It requires 1≦N i A ≤N N Therefore, naturally, capturing an excess of raw pixels increases redundancy and can improve the robustness and SNR of the wavefront imaging sensor.

[0116] In order to obtain a higher pixel count according to the above relationship, the actual raw pixels must be smaller than the Nyquist limit Δx = λ / 4NA. This is schematically shown in Figures 9a and 9b. Figure 9a shows the Nyquist sampling grid 90. Figure 9b shows the raw pixel sampling grid 92. In the case of a normal imaging sensor, this simply means oversampling of the optical image and does not bring new information. However, in the case of a wavefront imaging sensor according to an embodiment of the present invention, for example as shown in Figure 1b, the optical modulator creates new optical information that can be sampled by the image sensor detector array (e.g., element 112 in Figure 1b), generating raw data, which is used to perform deblurring and dynamic aberration intensity estimation, as described above.

[0117] The wavefront imaging sensor (for example, element 106 in Figures 1a and 1b) is N R <N NIt can also be used in a state where it is. In this case, in addition to functioning as an anti-aliasing filter and performing static aberration correction, a defocused image having a resolution lower than the Nyquist limit is output at a resolution N O (number of pixels in the output image) and generated at N R <N N +Σ i N i A When used together with the dynamic aberration in the regime of, the anti-aliasing effect still exists, and still generates a defocused image having a resolution lower than the Nyquist limit N O <N N is generated.

[0118] Also, at an output resolution N N lower than the number of Nyquist sampling points N O it is also possible to calculate the chromatic visual field characteristics (such as the spectral band in the output image, color channels, etc.). In these cases, N O <N N at the output resolution of the points, N C To extract the chromatic visual field characteristics (for example, the spectral band), N which is the number of raw pixels of the image sensor C ×N O ≦N R is required.

[0119] For example, in an embodiment of the present invention based on a wavefront imaging sensor described in International Publication No. 2018 / 185740 pamphlet, the encoder unit cell is related to the Nyquist sampling lattice, and optionally has a non-integer ratio between the size of the unit cell and the Nyquist lattice. The ratio between the size of the unit cell and the Nyquist lattice depends on the degree of sampling required, and can be sufficiently sampled at a ratio of 1:1. When the unit cell is smaller than the Nyquist lattice, oversampling can be performed, and when the unit cell is larger than the sampling lattice, undersampling can be performed. The number of sub-pixels in each unit cell may be a fixed integer, and the total number of raw data pixels may be equal to the number of unit cells × the number of sub-pixels per unit cell.

[0120] Even if the unit cell is larger than the Nyquist grid, if the total number of raw data pixels follows the sampling relationship described above, it is possible to recover sufficiently sampled mutual coherence and Wigner-distributed data.

[0121] It should also be noted that encoder unit cells, such as those described in International Publication No. 2018 / 185740, are described as periodic. However, in some embodiments, it is possible to use quasi-periodic or aperiodic structures to describe spatially dependent optical properties within the required field of view (FOV). For example, the encoder pitch can be adjusted to account for variations in the principal ray angle (CRA) across the FOV. Furthermore, the degree and type of aberrations may vary across the FOV, requiring different optimal optical encoder / modulator designs for each position within the FOV. Modulator designs vary according to these requirements, resulting in quasi-periodic or aperiodic designs.

[0122] Accordingly, according to embodiments of the present invention, the optical modulator section (for example, element 110 in Figure 1b) may include a plurality of unit cells (not shown in Figure 1b), the image sensor detector array section 112 may include an array of sensor cells (not shown in Figure 1b), the array of sensor cells of element 112 may define a plurality of sensor subarray unit cells (not shown in Figure 1b), each sensor subarray corresponds to a unit cell of the plurality of unit cells of the optical modulator section 110, and the optical modulator section 110 applies a predetermined modulation to the input light collected by the imaging sensor section 112. Each unit cell of the optical modulator section 110 can direct a portion of the collected incident light incident thereon to its corresponding sensor subarray unit cell and one or more adjacent sensor subarray unit cells within a predetermined proximity region.

[0123] Figures 10a to 10d schematically show various embodiments of the optical modulator 110 having a periodic cell configuration. The present invention is not limited by a specific cell arrangement of the optical modulator 110.

[0124] Figure 10a shows a binary design with one disk section per unit cell. The disk can use either phase modulation or amplitude modulation, or a combination of both. Note that in some cases, phase modulation is preferred because it does not induce light loss, i.e., 100% of photons reach the sensor pixel below. Typically, the disk can have a packing density of 25% to 75%, and the phase modulation can be 90 to 270 degrees at the design wavelength.

[0125] Figure 10b shows a multilayer design consisting of concentric rings. The rings can have equally spaced radii or other increments. Here again, phase modulation, amplitude modulation, or a combination of both are possible.

[0126] Figures 10c and 10d show binary disk or multi-ring designs with two disk / ring sets per unit cell, positioned at the edge of the unit cell. In practice, these are equivalent to the designs in Figures 10a and 10b rotated by 45 degrees and contracted by a factor of √2.

[0127] Figures 11a and 11b show the relationship between a unit cell of the optical modulator 110 and the corresponding sensor subarray of the detector array 112 according to one embodiment of the present invention. In Figure 11a, there are 2 × 2 sensor subarray pixels per unit cell, and in Figure 11b, there are 3 × 3 sensor subarray pixels per unit cell. Of course, it is possible to use a higher MxN pixel count in the sensor subarray, and the present invention is not limited by the number of pixels in the optical modulator 110 and the detector array 112.

[0128] Even when the optical modulator unit cell is larger than the Nyquist limit Δx = λ / 4NA, the number of raw pixels N is determined by the number of subarray pixels × the number of unit cells. R However, N for image recovery in the presence of dynamic aberrations R ≥N N , or N R ≥N N +Σ i Ni A It should be noted that diffraction-limited imaging resolution can be obtained as long as it is sufficient to guarantee [the following].

[0129] Chromatic aberration can be processed using wavefront imaging sensors designed with the required spectral sensitivity band, using techniques known in the art. One technique is to use a conventional color filter array, such as a Bayer filter. A non-limiting example is shown in Figure 12a, where each number represents a different spectral band filter. Figure 12b schematically shows an optical system 12 according to one embodiment of the present invention. System 12 includes elements 100, 102, 104, and 108 described with reference to Figures 1a and 1b. System 12 further includes a multisensor configuration 206 including wavefront imaging sensors 206λ1 to 206λ3 that provide spectral filtering and have a suitable dichroic or other beam splitter 114. The spectral sensitivity band can be implemented by using the multisensor configuration 206. Other possible configurations may include combinations of dichroic or other beam splitters and spectral filters. Each spectral element may be processed with a separate Wigner distribution, and then spectrally sensitive combined deblurring is applied.

[0130] It should be noted that the total number of raw pixels must include the necessary degrees of freedom as described above, and the spectral bandwidth must also be taken into consideration. According to one embodiment, the number of raw pixels required for a single spectral bandwidth is replicated for all spectral bandwidths, which can be seen with multiple raw pixels used in conjunction with a color filter array as shown in Figure 12a. According to another embodiment, when a multi-sensor configuration is used according to the example in Figure 12b, each sensor 206 should contain a sufficient number of raw pixels to cover the necessary degrees of freedom for the relevant spectral bandwidth.

[0131] According to embodiments of the present invention, hyperspectral imaging (HSI) is performed using, for example, the systems shown in Figures 1a, 1b, and 12b. Hyperspectral imaging (HSI) is a technique that analyzes light across a broad spectrum, not only by assigning a primary color (red, green, blue) to each pixel. The light hitting each pixel is decomposed into many different spectral bands to provide more information about what is being imaged.

[0132] According to embodiments of the present invention, hyperspectral imaging is achieved by providing some redundancy in the total number of raw sensor pixels for the measured spectral bands, as well as in the number and types of aberrations to be considered. The total number of raw sensor pixels is selected to be greater than the number of measured spectral bands, taking into account the estimated number and types of aberrations.

[0133] For example, using 3x3 (9) raw pixels per Nyquist pixel provides sufficient redundancy to extract the intensities of eight spectral bands and one additional depth-sensing or aberration-estimating channel. Under such conditions, it is possible to extract a Wigner distribution for additional spectral bands to be interpolated or extrapolated from the actual spectral filter used. This makes hyperspectral imaging possible with relatively simple color filter arrays or dichroic prism configurations. For example, in the case of an RGB Bayer filter, it is possible to extract hyperspectral imaging data for additional spectral bands in the red-green and green-blue spectral regions, as well as extrapolate to precise spectral sampling toward the violet and red regions of the spectrum.

[0134] Figure 13a schematically shows four spectral band filters 1-2-3-4 corresponding to the color filter array configuration shown in Figure 12a or the dichroic prism configuration shown in Figure 12b. The spectral information can be refined within each spectral band as described above, yielding spectral information for more than four wavelengths.

[0135] Furthermore, the sufficient redundancy of all raw pixels allows spectral information to be extracted into a wide spectral band, such as the entire visible range, 400-700 nm, or beyond, for example, the ultraviolet or infrared spectral regions. This enables full-color RGB imaging, multispectral imaging, or hyperspectral imaging without the use of any type of spectral filter or color filter array, and significantly improves acquisition efficiency because no photons are lost. Such sensors are useful in low-light conditions.

[0136] According to embodiments of the present invention, the sensor is configured to capture standard RGB imaging in the visible range (e.g., 400-700 nm) and near-infrared (IR) light. The near-infrared (IR) light may be in the range of 700-1000 nm, or even beyond that, within the SWIR range, depending on the spectral sensitivity of the underlying sensor pixels. Combined RGB+IR sensing is useful for a variety of applications, including discreet IR illumination. For example, combined RGB+IR sensing allows for the capture of full-color images while simultaneously illuminating the scene with invisible flood IR to improve low-light performance. In another example, the scene is illuminated with an invisible IR dot pattern to improve depth imaging.

[0137] Using a wide spectral bandwidth can lead to ambiguity in the extraction of spectral data. This ambiguity can be overcome by using a “comb” spectral filter that divides the spectrum reaching the sensor into a series of narrower spectral bandwidths, as shown in spectral distribution 1 in Figure 13b. Referring to the system shown in Figure 12b, this spectral filter is applied to the entire image and can be placed anywhere within the optical system 12 or as part of the image sensor 206. De-ambiguation of the spectrum may also be affected by other spectral patterns other than the comb pattern shown here, such as other spectral filters with spectral distribution 2 in Figure 13b.

[0138] Without loss of generality, as described above and shown in Figures 12b, 13a, and 13b, the control of the spectral content reaching the sensor 206 can be influenced by using a suitable illumination source in conjunction with a spectral filter, or in place of a spectral filter, to achieve the same effect. For example, a narrowband illumination source such as an LED can be used to illuminate RGB or other spectral distributions. The spectral content of the illumination source can be changed sequentially and simultaneously, as shown in Figure 13b, to yield settings essentially equivalent to those shown in Figures 12b and 13a.

[0139] The ability to digitally correct optical aberrations can alleviate constraints on the design and manufacture of imaging optical systems, potentially leading to cost reductions, size reductions, improved operating parameters, or increased robustness against environmental factors. Simplification of the optical system can be manifested, according to embodiments of the present invention, as one or more various combinations from the group consisting of the following operating parameters.

[0140] Reduction in the number of refractive and reflective optical elements: The greater degree of freedom in aberration reduces the need for complex combinations of optical elements that compensate for each other.

[0141] A simpler optical surface, namely a sphere, can be used, and optical designs can be simplified by utilizing a sphere instead of an aspherical or freeform surface.

[0142] Low-cost materials used in optical elements: reducing the need for special materials that are more expensive or require more expensive manufacturing processes.

[0143] Shorter optical paths: Reducing the number of elements, in some cases combined with a faster optical system, can result in shorter optical paths (see below).

[0144] Faster optical systems (lower f / #, higher NA): By relaxing aberration tolerances, it may be possible to use larger apertures or shorter focal lengths. In microscopy, this can result in higher numerical apertures (NA).

[0145] Relaxed manufacturing tolerances: These may appear as wider tolerances for individual optical elements, surfaces, or materials, resulting in higher yield or lower costs.

[0146] Relaxed alignment tolerances: A wider tolerance for misalignment of optical elements can simplify mounting and alignment procedures.

[0147] Relaxed system stability and repeatability requirements: The system can adapt to a wider range of environmental factors, such as heat, pressure, and shock, that may affect the alignment and performance parameters of the optical elements.

[0148] Extended depth of field: Compensation for focus blur to enable an extended depth of field range.

[0149] Larger field of view: Often, aberrations in optical systems increase towards the edges of the field of view. By relaxing the aberration tolerance, a larger field of view can be achieved.

[0150] Adaptation to stronger field curvature: Extended focal range can allow for stronger field curvature and, optionally, a larger field of view.

[0151] Larger working distance: In microscope objective lenses, a large working distance typically requires a large and complex design to overcome aberrations. The ability to digitally correct aberrations can enable simpler designs for a given working distance, or for increasing the working distance, or both.

[0152] Microscope slide cover slips, sample dishes, and volumetric measurement chambers: The microscope objective lens design allows for the use of relaxed tolerances for cover slips or thicker slips for microscope slides. This can also allow the inverted microscope to operate with higher NA and better penetration into the sample dish. It also allows for working with thicker slides or sample chambers. Slides or sample chambers with volumetric measurement or microfluidic channels and manipulable capabilities can be used.

[0153] Immersion targets: Increased tolerance for index matching, elimination of the requirement for index adjustment rings for immersion targets.

[0154] RGB color, multispectral, or hyperspectral imaging: The ability to perform real-time single-snapshot RGB color, multispectral, or hyperspectral imaging without using a color filter array. This can increase light sensitivity due to several factors, as photons are not filtered and all of them reach the pixel array of the detector. This function can be used in conjunction with the de-blurring function described above.

[0155] Many optical imaging systems have functions for adjusting various imaging parameters, such as controlling the aperture diaphragm, changing the focus, and changing the focal length by a zoom mechanism. According to embodiments of the present invention, if these mechanisms are repeatable to the required extent, the imaging system can re-evaluate the changes they affect and store them in the memory of the control unit as various sets of static aberrations, each such set corresponding to a specific imaging parameter configuration.

[0156] According to another embodiment of the present invention, for simplification and cost reduction, a particular optical imaging parameter mechanism may have a looser tolerance for reproducibility, which can be compensated for by using dynamic aberration blur correction.

[0157] The ability to calculate the dynamic aberration intensity at each point in the image plane can be used to estimate the depth at each point in the image plane, as well as to perform digital blurring to compensate for focus blur and thus obtain an extension of the depth of field range. In this scenario, the aberration in question is power, which depends on the distance from the imaging system to each point in object space. The spatial map of power aberration intensity is directly related to the depth map, and the algorithm's deblurring ability can restore diffraction-limited imaging resolution even for parts of the image that were originally out of focus. Furthermore, for a particular optical imaging system and its characteristic aberrations, the distance to each point in object space may induce other aberrations beyond a magnification, such as spherical aberration or other aberrations. Information on these additional aberrations can also be used to enhance the depth estimation of each point light source in object space.

[0158] Without loss of generality, this scheme can be extended to operate over a wide spectral range according to embodiments of the present invention, where chromatic aberration is used to estimate depth, and the wavefront imaging sensor can use one of the configurations shown in Figures 10a to 10b or other configurations. According to embodiments of the present invention, the system can also address static aberrations present in the optical imaging system.

[0159] According to embodiments of the present invention, depth estimation can be used in conjunction with a simplified imaging optical system.

[0160] According to embodiments of the present invention, in order to perform digital adaptive optics, the dynamic aberration at each point in the image plane is calculated to correspond to aberrations of varying degrees exceeding power. The digital adaptive optics system is used to perform digital blurring in situations where aberrations cannot be known in advance and to restore diffraction-limited imaging resolution.

[0161] Aberrations are unpredictable and cannot be known in advance, especially when they arise from various environmental factors that affect the imaging system in unpredictable ways, such as changes in heat or pressure, shocks, airflow within the optical system, and sagging due to the weight of the optical elements. External environmental factors, such as imaging through atmospheric turbulence, can also be at play. In many biological applications, microscopic imaging of bulk tissue in vivo or in the laboratory is limited by aberrations induced by light passing through the biological tissue. In ophthalmic microscopy through the subject's eye, corneal and lens aberrations cause a significant decrease in image clarity and quality.

[0162] At the current level of technology, these types of aberrations are partially corrected using complex, cumbersome, and expensive adaptive optics systems. Such systems are extremely complex to integrate into existing imaging systems and typically require changes to the optical path to incorporate wavefront sensors and wavefront modulators, usually in the form of additional optical relays and beam splitters.

[0163] Many adaptive optics systems are designed so that both the wavefront sensor and wavefront modulator are located at or near the pupil of the imaging system, thereby uniformly sampling and correcting optical wavefront aberrations across the entire imaging field. Spatially dependent adaptive optics corrections are typically performed in scanning optics imaging systems, further complicating the overall system design.

[0164] In many cases, an additional "star-shaped guide" illumination system is required to provide sufficient feedback signals to the wavefront sensor, further complicating the system and limiting its usefulness and operating range.

[0165] The dynamic aberration correction according to the embodiments of the present invention allows the original imaging system to be used with a wavefront imaging sensor, resulting in a much simpler optical system overall.

[0166] According to embodiments of the present invention, it is possible to provide spatially dependent blur reduction, which is a very complex feature, using a conventional adaptive optics system.

[0167] According to further embodiments of the present invention, the implementation of the embodiments of the present invention eliminates the need for the use of "star-shaped guides," resulting in further simplification.

[0168] According to embodiments of the present invention, for example, in situations with low light levels, the present invention can be enhanced by a "star-shaped guide" to improve the SNR of the digital blur algorithm.

[0169] According to embodiments of the present invention, the digital adaptive optics system can be implemented to compensate for changes in the optical imaging system due to environmental factors, as well as to compensate for blurring caused by external factors affecting the imaging medium.

[0170] According to embodiments of the present invention, the digital optical aberration correction system can be incorporated into an existing adaptive optics imaging system, and the wavefront imaging sensor can replace or be provided in addition to a conventional imaging sensor within the adaptive optics imaging system. This can provide the "final mile" for aberration correction, compensating for residual aberration errors left by the adaptive optics system and also enabling simpler compensation for spatially dependent aberrations. Such a configuration can be used with or without a "star guide".

[0171] Figure 14 is a schematic block diagram of an aberration correction system 14 having an adaptive optics system unit 140 according to an embodiment of the present invention. The adaptive optics system unit 140, enclosed in a dashed box, is shown in a general manner. According to an embodiment of the present invention, the adaptive optics system unit 140 includes a wavefront sensor 1410 and a wavefront modulator 1406 located near the pupil plane 1412 of the imaging system. This may involve an extended optical path including a pupil relay 1402 and a tube lens 1408. The adaptive optics wavefront sensor 1410 provides feedback for driving the wavefront modulator 1406 under the control of the adaptive optics control unit 1414. Without loss of generality, Figure 14 shows an adaptive optics system in which the adaptive optics wavefront sensor 1410 is positioned to measure the raw wavefront before wavefront correction by the wavefront modulator 1406. Other adaptive optics system configurations known in the art exist in which the adaptive optics system wavefront sensor 1410 measures the residual wavefront error after correction by the wavefront modulator 1406. This design variation of the adaptive optics system, and other similar forms, are within the scope of the present invention, and the adaptive optics system shown in Figure 14 serves as an illustrative example only.

[0172] According to embodiments of the present invention, the adaptive optics wavefront imaging sensor 1410 is positioned on the image plane 1414 of the system instead of the imaging sensor 206. The wavefront imaging sensor 306 and the control unit 108 can correct residual aberrations left by the adaptive optics system unit 140 and compensate for spatially dependent blur, as described with reference to, for example, Figures 1a, 1b, 3 and 12b, according to embodiments of the present invention.

[0173] According to embodiments of the present invention, the control unit 108 can function independently of the adaptive optics system unit 140.

[0174] According to another embodiment, the control unit 108 can also use information from the adaptive optics wavefront sensor 1410 to provide a rough measurement of wavefront aberration, and then perform fine, and possibly spatially dependent, digital adaptive optics system correction, as described above with reference to Figure 3.

[0175] The digital adaptive optics system 14 can be used not only to provide depth estimation, but also in conjunction with the simplified optical scheme described above.

[0176] Embodiments of the present invention can be used for a variety of applications.

[0177] According to embodiments of the present invention, the digital aberration correction system can be used in cameras having shorter lens tracks and fewer optical elements, such as mobile phone cameras that do not protrude from the phone case, miniature cameras for wearable devices, drones, and robots.

[0178] According to embodiments of the present invention, the digital aberration correction system can be used in cameras having RGB, multispectral, or hyperspectral capabilities, which do not have a color filter array for improving light sensitivity and may also have digital blur processing and extended depth of field functions.

[0179] According to embodiments of the present invention, the digital aberration correction system may be used in a camera hidden behind a display panel such as a mobile phone or a vehicle display screen. Digital aberration correction can compensate for aberrations caused by imaging through the display device layer. In addition, color RGB, multispectral, or hyperspectral imaging can be obtained without a color filter array for improved light sensitivity and compensate for light loss and potential spectral imbalances caused by imaging through a display device.

[0180] According to embodiments of the present invention, the digital aberration correction system can be used in a simplified ophthalmoscope having higher imaging resolution. The simplified ophthalmoscope may enable simpler area imaging instead of a scanning configuration. The simplified ophthalmoscope can improve the resolution of retinal scans for biometric purposes. The simplified ophthalmoscope can be used as a "final mile" precision correction in place of or added to conventional adaptive optics systems.

[0181] According to embodiments of the present invention, the digital aberration correction system can be used in various combinations for use in microscopic inspection. Microscopes using the digital aberration correction system according to embodiments of the present invention may offer one or more of the following advantages: a simplified objective design, a wider field of view; a greater working distance; improved operating parameters related to sample handling such as slides, coverslips, volumetric chambers, and microfluidic channels; increased manufacturing tolerances for immersion objective lenses; and extended focus and digital aberration correction in turbid media.

[0182] The digital aberration correction system according to an embodiment of the present invention can be implemented as part of a conventional imaging microscope, as part of a confocal microscope, or for use in a light-sheet microscope.

[0183] The digital aberration correction system according to embodiments of the present invention can be used in microscopes, particularly for applications requiring RGB color, multispectral, or hyperspectral sensitivity for nonlinear imaging.

[0184] The digital aberration correction system according to the embodiment of the present invention can be used in microscopes for brightfield, darkfield, DIC, phase contrast, quantitative phase imaging and tomography, as well as other imaging techniques.

[0185] The digital aberration correction system according to the embodiment of the present invention can be used in microscopes for conventional fluorescence, two-photon, multi-photon, and nonlinear techniques.

[0186] The digital aberration correction system according to the embodiment of the present invention can be used as a "final mile" precision correction in place of a conventional adaptive optics system, or added to an existing adaptive optics system.

[0187] The digital aberration correction system according to embodiments of the present invention can be used in long-range monitoring and imaging systems. Examples include aerial, space, ocean, land, and space telescopes and imaging systems.

[0188] By using various embodiments of the present invention, such imaging systems can utilize a simplified and lightweight imaging optical system.

[0189] By using various embodiments of the present invention, such imaging systems can provide broader resistance to environmental factors affecting the imaging system, such as temperature, pressure, shock, and vibration.

[0190] Using various embodiments of the present invention, such imaging systems can utilize a digital adaptive optics system to compensate for atmospheric turbulence and fluctuations in the imaging system under environmental factors.

[0191] By using various embodiments of the present invention, such imaging systems can utilize digital aberration correction as a "final mile" precision correction in place of conventional adaptive optics systems or as an addition to existing systems.

[0192] The digital aberration correction system according to an embodiment of the present invention can be used in imaging systems for applications requiring RGB color, multispectral, and hyperspectral sensitivity without a color filter array to enhance sensitivity.

Claims

1. An optical system, An optical imaging unit configured to form an optical image near the image plane of the optical system, A wavefront imaging sensor unit is positioned near the image plane, wherein the wavefront imaging sensor unit comprises: An optical modulator unit comprising a plurality of unit cells and positioned near the image plane, wherein the plurality of unit cells are arranged in a periodic pattern, and the optical modulator unit is configured to collectively apply a predetermined periodic modulation to the input light, An image sensor unit comprising an array of sensor cells positioned downstream of the optical modulator unit with respect to the general propagation direction of the input light field passing through the optical system, An optical system comprising, The wavefront imaging sensor unit is configured to provide raw digital data relating to the light illumination field near the image plane and the image output, and the raw digital data has a raw pixel count N Ｒ A wavefront imaging sensor unit comprising, A control unit configured to process the raw digital data and provide an output image which is at least one of the group consisting of a full-color RGB image and a hyperspectral image, without the use of a spectral filter or a color filter, wherein the control unit comprises a storage unit and a processing unit, the storage unit is configured to store instructions, the processing unit receives raw digital data of the light field of illumination colliding with the wavefront imaging sensor unit, processes the raw digital data to determine the chromatic field of view characteristics based on the analysis of the optical mutual interference function in the image plane, and the number of output image pixels N Ｏ and the number of chromatic color field characteristics N Ｃ It is configured to reconstruct an output image having the following: Ｏ ×N Ｃ ≤N Ｒ The control unit and Equipped with, The array of sensor cells is divided into a plurality of sensor subarray unit cells, and each sensor subarray unit cell includes a plurality of sensor pixels that collectively correspond to one unit cell of the optical modulator section. An optical system wherein the optical modulator unit is configured to apply a predetermined periodic modulation to the input light collected by the image sensor unit, each unit cell of the optical modulator unit is configured to direct a portion of the collected input light incident thereon to its corresponding sensor subarray unit cell and one or more adjacent sensor subarray unit cells within a predetermined proximity region, and the predetermined periodic modulation is at least one of phase modulation and amplitude modulation.

2. The control unit, A step of calculating the field of view characteristics of the light irradiation field, The steps include identifying the points between light sources in the image output based on interference and superposition information within the field of view characteristics, For each identified point light source, the step is to estimate its degree of blurring, The steps include: reconstructing the blur-free image as a composite of point light sources with the blur removed; The optical system according to claim 1, further configured to perform the following.

3. The optical system according to claim 2, wherein the field of view characteristics are a Wigner distribution or an equivalent entity related to the Wigner distribution by mathematical transformation.

4. The number of raw pixels N of the multiple sensor subarray unit cells in the image sensor unit. Ｒ The number of Nyquist sampling points in the optical modulator section is N. Ｎ The optical system according to claim 1, which is equal to or greater than .

5. The optical system according to claim 2, wherein the light field comprises a plurality of wavelengths, which are separate or continuous, and the control unit is configured to perform one or both of the following steps: (1) estimating the spectral distribution of each identified point light source, and (2) reconstructing a spectral distribution diagram of the optical image.

6. The optical system according to claim 1, wherein the control unit is further configured to estimate the depth at each point in the image plane.

7. A method for correcting digital optical aberrations in an image formed by an imaging unit near the image plane of an optical system, A light irradiation field that collides with a wavefront imaging sensor unit disposed near the image plane, and a step of providing raw digital data for image output formed near the image plane, wherein the raw digital data has a raw pixel count N Ｒ A step comprising: The control unit processes the raw digital data and determines the chromatic field of view characteristics based on the analysis of the optical interference function in the image plane without using a spectral filter or color filter. Number of pixels in the output image: N Ｏ and the number of chromatic color field characteristics N Ｃ A step of reconstructing an output image having the following: Ｏ ×N Ｃ ≤N Ｒ The output image is at least one of the group consisting of a full-color RGB image and a hyperspectral image, Equipped with, The wavefront imaging sensor unit is An optical modulator unit comprising a plurality of unit cells and positioned near the image plane, wherein the plurality of unit cells are arranged in a periodic pattern, and the optical modulator unit is configured to collectively apply a predetermined periodic modulation to the input light, An image sensor unit comprising an array of sensor cells positioned downstream of the optical modulator unit with respect to the general propagation direction of the input light field passing through the optical system, and configured to acquire a raw digital image output, Equipped with, The array of sensor cells is divided into a plurality of sensor subarray unit cells, and each sensor subarray unit cell includes a plurality of sensor pixels that collectively correspond to one unit cell of the optical modulator section. The aforementioned method, The steps include: applying a predetermined periodic modulation to the input light collected by the image sensor unit using at least one of phase modulation and amplitude modulation with the optical modulator unit; The steps include: directing a portion of the collected input light incident on each unit cell of the optical modulator to the corresponding sensor subarray unit cell and one or more adjacent sensor subarray unit cells within a predetermined proximity area; A way to prepare further.

8. The method according to claim 7, wherein the wavefront imaging sensor unit comprises an optical modulator unit positioned near the image plane for acquiring a raw digital image output, and an image sensor unit positioned downstream of the optical modulator unit with respect to the general propagation direction of the input light field through the optical system, the method further comprising the step of modulating the light field using at least one of phase modulation and amplitude modulation.

9. A step of calculating the field of view characteristics of the light irradiation field, The steps include identifying the points between light sources in the image output based on interference and superposition information within the field of view characteristics, For each identified point light source, the step is to estimate its blur, The steps include: reconstructing the blur-free image as a composite of point light sources with the blur removed; The method according to claim 8, further comprising:

Images