Imaging system for capturing wide field images

Abstract

An imaging system for capturing wide field images, comprising: an objective lens assembly configured to receive light from a scene; a diffractive optical element (DOE) positioned along an optical path between the objective lens assembly and an image sensor, wherein the DOE is located off-aperture, offset from an aperture stop plane of the objective lens assembly; an image sensor configured to capture encoded light modulated by the objective lens assembly and the DOE; and a processor comprising a reconstruction processing engine trained with optimization parameters of the DOE to decode the captured encoded light into a reconstructed image of the scene; wherein the off-aperture positioning of the DOE enables spatially varying modulation of incident light based on angle of incidence, providing local wavefront control on the image sensor to enhance image fidelity over a wide field of view.

Description

Technical Field

[0001] This invention generally relates to imaging systems and methods. More specifically, it relates to wide-field imaging systems employing computational imaging techniques involving optical coding elements and associated reconstruction processing, particularly those utilizing deep learning. Background Technology

[0002] Imaging systems are ubiquitous in modern technology, with applications ranging from consumer photography to scientific instruments and autonomous systems. A key challenge in optical design is achieving high image fidelity characterized by sharpness, good contrast, and minimal aberrations over a wide field of view (FoV) while maintaining a compact and cost-effective form factor.

[0003] Traditional methods typically rely on complex objective lens assemblies, which include multiple precisely shaped refractive elements to correct aberrations, especially those that become severe over wide angles (off-axis aberrations). While effective, such compound lens systems can be bulky, heavy, and expensive. Summary of the Invention

[0004] This invention relates to a high-fidelity imaging system that achieves wide field-of-view (FoV) performance by integrating off-aperture diffractive optical elements (DOEs) with deep learning techniques. Positioning the DOE far from the system aperture enables local wavefront control, effectively mitigating off-axis aberrations prevalent in wide-angle imaging. The system employs end-to-end optimization, jointly designing the optical encoder (DOE and potential hybrid refractive lens) and the deep neural network decoder. This approach allows for the capture of high-resolution images (and potential depth maps) over a wide FoV (e.g., verified up to 45°), with significantly improved sharpness and detail compared to conventional or aperture-on-aperture encoding systems.

[0005] According to a first aspect of the invention, an imaging system for capturing wide field-of-view images is provided, comprising: an objective lens assembly configured to receive light from a scene; a diffractive optical element (DOE) positioned along an optical path to receive light from the objective lens assembly; an image sensor configured to capture an coded image modulated by the objective lens assembly and the DOE; and a reconstruction processing engine arranged to decode the captured coded light into a reconstructed image of the scene; wherein parameters of the DOE and parameters of the reconstruction processing engine are jointly optimized in an end-to-end process.

[0006] According to the first aspect, the DOE is located away from the aperture and offset from the aperture stop plane of the objective lens assembly.

[0007] According to the first aspect, the DOE's aperture positioning can spatially modulate the incident light based on the incident angle, thereby providing local wavefront control on the image sensor.

[0008] According to the first aspect, the DOE is positioned at a distance between 0.1 and 0.4 of the effective focal length of the objective lens assembly, measured from the aperture stop plane of the objective lens assembly toward the image sensor.

[0009] According to the first aspect, the objective lens assembly includes a single refractive lens element.

[0010] According to the first aspect, the objective lens assembly includes a plurality of refractive lens elements forming a compound lens assembly.

[0011] According to the first aspect, at least one refractive lens element in the composite lens assembly has an aspherical surface.

[0012] According to the first aspect, the composite lens assembly includes a Cook three-piece configuration.

[0013] According to the first aspect, the reconstruction algorithm includes a deep neural network.

[0014] According to the first aspect, the deep neural network includes a multi-head architecture with a shared feature extractor and at least two separate decoding heads.

[0015] According to the first aspect, the first decoding head is configured to reconstruct a color image, and the second decoding head is configured to estimate a depth map of the scene.

[0016] According to the first aspect, the end-to-end process utilizes the differentiability simulation of the image formation process of the imaging system.

[0017] According to the first aspect, the differentiable simulation combines differentiable ray tracing for the objective lens assembly and microwave-compatible propagation for the DOE.

[0018] According to the first aspect, the microwave propagation utilizes the angular spectrum method (ASM) suitable for off-axis propagation.

[0019] According to the first aspect, the system is configured to capture images with a wide field of view of at least 45 degrees.

[0020] According to the first aspect, the DOE is a pure phase modulation element.

[0021] According to the first aspect, the joint optimization is performed using a loss function that includes at least one of mean squared error (MSE) loss, perceptual loss, L1 loss for depth estimation, or point spread function (PSF) focusing loss.

[0022] According to a second aspect of the invention, a method for capturing a wide field-of-view image is provided, comprising the steps of: receiving light from a scene via an objective lens assembly; modulating the light with a diffractive optical element (DOE) to form modulated light; capturing the modulated light as an encoded image using an image sensor; and processing the encoded image using a reconstruction algorithm to generate a reconstructed image of the scene; wherein the parameters of the DOE and the parameters of the reconstruction algorithm are jointly optimized in an end-to-end process.

[0023] According to the second aspect, the step of modulating the light includes positioning the DOE at a distance from the aperture, offset from the aperture stop plane of the objective lens assembly, to achieve spatial variation modulation for local wavefront control.

[0024] According to the second aspect, the step of processing the encoded image to reconstruct a depth map is performed in real time to provide data for autonomous navigation or obstacle avoidance.

[0025] According to the second aspect, the steps of processing the encoded image include reconstructing both a color image and a depth map from the encoded image using a multi-head neural network.

[0026] According to the second aspect, the joint optimization is performed using a differentiable simulation of the image formation process.

[0027] According to the second aspect, the differentiable simulation combines ray tracing for the objective assembly and wave-optical modeling for the DOE.

[0028] According to the second aspect, the joint optimization includes training a multi-task reconstruction algorithm using dynamic weighted averaging (DWA) to automatically adjust the weights of multiple loss functions.

[0029] According to a third aspect of the invention, an imaging system for capturing wide field-of-view images is provided, comprising: an objective lens assembly configured to receive light from a scene; a diffractive optical element (DOE) positioned along an optical path between the objective lens assembly and an image sensor, wherein the DOE is located off-aperture and offset from the aperture stop plane of the objective lens assembly; an image sensor configured to capture coded light modulated by the objective lens assembly and the DOE; and a processor including a reconstruction processing engine trained together with optimized parameters of the DOE to decode the captured coded light into a reconstructed image of the scene; wherein the off-aperture positioning of the DOE enables spatial variation modulation of the incident light based on the incident angle, thereby providing local wavefront control on the image sensor to enhance image fidelity over a wide field of view.

[0030] According to the third aspect, the objective lens assembly includes a single refractive lens element.

[0031] According to a third aspect, the objective lens assembly includes a plurality of refractive lens elements forming a compound lens assembly.

[0032] According to the third aspect, the composite lens assembly includes a Cook three-piece configuration.

[0033] According to the third aspect, at least one lens element in the composite lens assembly has an aspherical surface.

[0034] According to the third aspect, the DOE is positioned at a distance between 0.1 and 0.4 of the effective focal length of the objective lens assembly, measured from the aperture stop plane toward the sensor.

[0035] According to the third aspect, the DOE is positioned at approximately 0.24 of the effective focal length of the objective lens assembly, measured from the aperture stop plane toward the sensor.

[0036] According to the third aspect, the DOE is a pure phase modulation element.

[0037] According to the third aspect, the DOE includes a height map defining its phase modulation profile.

[0038] According to the third aspect, the height map of the DOE is optimized using an end-to-end design process involving differentiable simulation of the imaging system and the reconstruction processing engine.

[0039] According to the third aspect, the reconstruction processing engine includes a deep neural network.

[0040] According to the third aspect, the deep neural network includes a multi-head architecture with a shared feature extractor and at least two separate decoding heads.

[0041] According to the third aspect, at least two separate decoding heads include a first decoding head configured to reconstruct a color image and a second decoding head configured to estimate a depth map of the scene.

[0042] According to the third aspect, the processor is configured to reconstruct both a color image and a depth map from the captured coded light.

[0043] According to the third aspect, the wide field of view is at least 28 degrees.

[0044] According to the third aspect, the wide field of view is at least 45 degrees.

[0045] According to the third aspect, the differentiable simulation used in the end-to-end design process combines differentiable ray tracing for the objective assembly with microwave propagation for the DOE and subsequent propagation to the sensor.

[0046] According to the third aspect, the microwave propagation utilizes the angular spectrum method (ASM) suitable for off-axis propagation.

[0047] According to the third aspect, the DOE is manufactured using photolithography and etching techniques.

[0048] According to the third aspect, the reconstruction processing engine is trained using a loss function including at least one of mean squared error (MSE) loss, perceptual loss, L1 loss for depth, or point spread function (PSF) focusing loss.

[0049] According to the third aspect, during training, the weights used for different loss functions are dynamically adjusted using Dynamic Weighted Average (DWA).

[0050] According to a fourth aspect of the invention, a method for capturing a wide field-of-view image is provided, comprising: receiving light from a scene via an objective lens assembly; modulating the light with a diffractive optical element (DOE) positioned in an optical path between the objective lens assembly and an image sensor, wherein the DOE is located at an aperture offset from the aperture stop plane of the objective lens assembly, thereby enabling spatially variable modulation based on the incident angle to achieve local wavefront control; capturing the modulated light as coded light using the image sensor; and processing the captured coded light using a reconstruction processing engine, the reconstruction processing engine being trained together with optimized parameters of the DOE to decode the captured coded light into a reconstructed image of the scene having enhanced image fidelity over a wide field of view.

[0051] According to the fourth aspect, processing the captured coded light includes reconstructing both a color image and a depth map of the scene.

[0052] According to the fourth aspect, the method further includes using end-to-end differentiable simulation of the image formation process to jointly optimize the parameters of the DOE and the reconstruction processing engine.

[0053] According to the fourth aspect, the differentiable simulation combines ray tracing and wave-optical modeling.

[0054] According to a fifth aspect, a non-transitory computer-readable medium is provided that stores instructions, when executed by a processor of an imaging system, to cause the processor to perform a method comprising: receiving an coded image captured by an image sensor, the coded image being modulated by an objective lens assembly and a diffractive optical element (DOE); and decoding the coded image into a reconstructed image using a reconstruction algorithm, wherein the reconstruction algorithm includes a neural network having parameters determined by an end-to-end optimization process in conjunction with parameters of the DOE.

[0055] According to a sixth aspect, a method for designing a computational imaging system is provided, the method comprising: defining a differentiable simulation of an optical path, the optical path including models of objective lens assemblies and diffractive optical elements (DOEs); defining a model of a reconstruction algorithm configured to process the output of the differentiable simulation; and jointly optimizing a first set of parameters corresponding to the DOE and a second set of parameters corresponding to the reconstruction algorithm using an end-to-end process to minimize a predefined loss function. Attached Figure Description

[0056] Figure 1A This is a schematic diagram depicting potential locations for integrating diffractive optical elements (DOEs) for encoding purposes, including locations away from the aperture, according to embodiments of the present invention.

[0057] Figure 1B Cross-sectional views of two custom-made optical imaging systems are shown, illustrating the DOE and aperture located on the separation plane.

[0058] Figure 1C The analysis shows a comparison between the resolved wide field of view (FoV) imaging results and the original coded measurements.

[0059] Figure 1D The results of color and depth reconstruction using a compound lens system are shown.

[0060] Figure 2A This is a block diagram illustrating an imaging system for capturing wide field-of-view images according to an embodiment of the present invention.

[0061] Figure 2B This is a schematic diagram illustrating a hybrid differentiable imaging pipeline that combines ray tracing, wave propagation, and multi-head decoding networks according to an embodiment of the present invention.

[0062] Figure 3A This is a schematic diagram showing the system setup including a thin lens and a distance-from-aperture DOE.

[0063] Figure 3B This is a graph showing the peak signal-to-noise ratio (PSNR) of the images recovered at various DOE locations to determine the optimal placement trade-off between aperture and sensor.

[0064] Figure 3C The comparison of the point spread function (PSF) amplitudes at several FoVs is shown when the DOE is placed at different positions along the optical path.

[0065] Figure 4 Simulated wide FoV imaging results with and without aperture DOE configurations are shown in the embodiment of the present invention.

[0066] Figure 5AOptimized PSF is shown using a large aperture FoV extended depth of field (EDoF) imaging system at various depths (from 0.8 m to 10 m) and FoV (up to 28 degrees).

[0067] Figure 5B This is a 3D model of an optimized compound lens system with ray tracing, wave propagation (LS-ASM), and DOE placement near and far from the aperture, according to an embodiment of the present invention, as well as optimized DOE height diagrams for near and far aperture settings.

[0068] Figure 6 The results of simulated wide FoV full-focus (AiF) images and depth maps comparing near-aperture and far-aperture DOE systems according to embodiments of the present invention are shown.

[0069] Figure 7A The photograph shows the optical components used in the first experimental prototype, including a biconvex thin lens, a DOE, and an aperture.

[0070] Figure 7B The photograph shows the optical components used in the second experimental prototype, including the Cook three-piece refractive lens assembly, DOE, and retaining ring.

[0071] Figure 7C It is a photograph showing the experimental laboratory setup used for the imaging system.

[0072] Figure 8 Experimental results from real-world scenarios are shown for wide FoV simple lens imaging applications according to embodiments of the present invention.

[0073] Figure 9 Experimental results from real-world scenes are shown for wide FoV AiF and depth imaging applications according to embodiments of the present invention.

[0074] Figure 10 Experimental results comparing a simple wide FoV lens prototype with a pinhole camera setup are shown. Detailed Implementation

[0075] Through their experimentation and research, the inventors devised alternative methods for computational imaging, attempting to simplify optics and compensate for defects using post-processing techniques. Some methods utilize coded elements, such as mask or diffractive optics (DOEs), placed at or near the aperture stop (pupil plane) of the imaging system. These elements can modulate the incident light to encode scene information (e.g., depth or spectral data) into the captured image, which is then decoded by a reconstruction engine. However, placing the coded elements at the aperture typically only provides global control over the wavefront, limiting the ability to efficiently resolve spatially varying aberrations over a wide FoV.

[0076] Achieving high fidelity over a wide angular range using simple, compact optics remains a significant challenge. Improved imaging systems are needed that can capture high-quality, wide-field-of-view images using compact hardware and advanced processing techniques.

[0077] According to embodiments of the present invention, a high-fidelity wide-field-of-view imaging system and method are provided to overcome the aforementioned drawbacks. This is achieved by positioning the diffractive optical element (DOE) at an "off-aperture" position within the optical path and employing an end-to-end (E2E) design method that jointly optimizes optical hardware and a deep learning-based reconstruction processing engine. The present invention enables local wavefront control, thereby achieving superior aberration correction and image quality over a wide angle, even using relatively simple optics.

[0078] The inventors have designed optics with an end-to-end (E2E) design, known in many studies as Deep Optics, which shows great potential in a wide range of applications, including low-level visual tasks in photography, 3D vision and computational imaging, and holographic displays. Unlike traditional optical systems that rely on heuristic methods, Deep Optics enhances the coding capabilities of camera systems, allowing their measurements to be decoded via post-processing algorithms. This coding-decoding capability is co-designed to achieve optimal imaging goals. Specifically, optical elements such as refractive lenses and superlenses are integrated into the camera to modulate complex incident wavefronts. Parameters controlling these optics, such as curvature, heightmaps, or atomic diameters, are updated along with differentiable algorithms (typically neural networks) employing backpropagation.

[0079] The inventors are particularly interested in hybrid optics that combine a classical refractive lens (which provides high diffraction efficiency for focused light) with a diffractive optical element (DOE) for optical coding. Research on this topic has focused on optimizing the parameters of the coding element, neglecting the question of where to place the DOE relative to the aperture (or pupil plane). For simplicity, conventional coded aperture systems place the optical code on or near the aperture plane, creating a global modulation across the entire wavefield. For an idealized aberration-free system, this results in a displacement-invariant PSF across the entire image plane. However, in the presence of off-axis aberrations, the global effect of the DOE parameters on the entire image becomes an obstacle to local aberration correction. To address these issues, the inventors explored separating the coding plane from the aperture plane so that different regions of the DOE produce different coding characteristics based on the incident light direction.

[0080] Figure 1AThree potential locations for integrating diffractive optical elements (DOEs) for coding purposes in imaging systems are shown. At the aperture, a DOE has many degrees of freedom to influence each beam, but these are all applied globally across the entire image plane. On the other hand, a DOE near the sensor provides localized control over the PSF, but with far fewer degrees of freedom per beam. Off-aperture DOEs strike the optimal balance between these two extremes. Figure 1B Cross-sectional views of two custom-made optical imaging systems are shown. The DOE and aperture are located on separate planes. Figure 1C The diagram shows a comparison between the wide field of view (FoV) results of App.1's resolution and the coded measurements; Figure 1D The color and depth results for App.2 are shown.

[0081] It is worth noting that in most imaging scenarios, the sensor size significantly exceeds the pupil size, and placing the DOE closer to the sensor increases the total degrees of freedom as well as the locality of wavefront control. However, placing the DOE directly on the sensor is not optimal, as the ability to spatially redistribute light is diminished at this location (Figure 1). In this work, the inventors investigated this inherent trade-off between the locality of control and the ability to redirect light.

[0082] Local control of wavefront shape is particularly important for wide FoV imaging, where off-axis aberrations require local phase modulation for PSF shaping. In this work, the inventors first demonstrate that pairing a simple thin lens with an off-aperture DOE enables aberration-free image recovery at 45°, achieving a peak signal-to-noise ratio (PSNR) more than 5 dB higher than an on-aperture DOE system. Building on this achievement, the inventors introduce a more sophisticated composite optics setup to address the challenging task of simultaneously performing color imaging and depth recovery over a wide FoV of approximately 28°. This extension leverages the synergistic optimization of an off-aperture DOE with a Cooke tri-lens configuration—comprising three refractive lenses located near the aperture plane that partially correct for large-angle aberrations. This setup utilizes the high diffraction efficiency of the refractive lenses to focus light while allowing the DOE to encode high-frequency information crucial for depth estimation. This hybrid refractive-diffractive system is modeled in two steps: a differentiable ray-tracing engine (dO) for light propagation through the refractive lenses, followed by two steps of off-axis wave propagation using a minimum-sampled ASM (LS-ASM). This hybrid design preserves the precise diffraction properties of the optics, a feature not found in purely refractive systems.

[0083] When demonstrating applications of multiple optimization objectives, the inventors designed a lightweight multi-head neural network architecture that efficiently extracts different information channels (e.g., color and depth) through separate decoding heads. The inventors designed an integrated, shared pre-trained feature extractor with a single head dedicated to a specific task. This framework not only reduces the training burden compared to traditional single-head networks but also facilitates efficient information encoding. The inventors' contributions are further elaborated below.

[0084] The inventors introduce an E2E-optimized distance-of-aperture coding framework tailored for wide FoV computational imaging applications. Specifically, the inventors explore the key effects of distance-of-aperture (DOE) placement.

[0085] The inventors’ method involves a differentiable refractive-diffractive hybrid imaging pipeline that integrates accurate off-axis wave propagation modeling, enabling the co-design of off-aperture DOEs and multi-head image processing networks.

[0086] Through simulations, the inventors demonstrated high-fidelity lightweight lens imaging capabilities at 45° with a PSNR gain of over 5 dB compared to the system at 28°, as well as RGBD imaging capabilities at 28°.

[0087] The inventors developed two camera prototypes: one equipped with an off-the-shelf convex lens, and the other employing a custom-designed composite lens incorporating a custom-machined aspherical lens geometry using CNC turning. Both prototypes utilized custom-made nanofabricated DOEs as encoding elements. These systems were rigorously validated using a variety of indoor and outdoor scenarios.

[0088] Refer to Figure 1 to Figure 10 An embodiment of an imaging system for capturing wide field-of-view images is provided, comprising: an objective lens assembly configured to receive light from a scene; a diffractive optical element (DOE) positioned along an optical path between the objective lens assembly and an image sensor, wherein the DOE is located off-aperture and offset from the aperture stop plane of the objective lens assembly; an image sensor configured to capture coded light modulated by the objective lens assembly and the DOE; and a processor including a reconstruction processing engine trained together with optimized parameters of the DOE to decode the captured coded light into a reconstructed image of the scene; wherein the off-aperture positioning of the DOE enables spatial variation modulation of the incident light based on the incident angle, thereby providing local wavefront control on the image sensor to enhance image fidelity over a wide field of view.

[0089] Preferably, the interface and processor can be implemented by a computer with a suitable user interface. The computer can be implemented by any computing architecture, including portable computers, tablet computers, standalone personal computers (PCs), smart devices, Internet of Things (IoT) devices, edge computing devices, client / server architectures, "dumb" terminal / mainframe architectures, cloud computing-based architectures, or any other suitable architecture. The computing device can be appropriately programmed to implement the present invention.

[0090] Furthermore, computers or computing devices can provide the computing power required to operate or interface with machine learning networks (such as neural networks) to provide various functions and outputs. Neural networks can be implemented locally, or accessed or partially accessed via servers or cloud-based services. Machine learning networks can also be untrained, partially trained, or fully trained, and / or can be retrained, adjusted, or updated over time.

[0091] Reference Figure 2A An exemplary computational imaging system 100 according to an embodiment of the present invention is illustrated. The system is designed to capture light from a scene and generate a high-fidelity reconstructed image through a co-design of optical hardware and software-based reconstruction algorithms. In operation, the input to the system is light from a scene 102, and the output is a reconstructed image 116, which may be a high-fidelity color image, a depth map, or both.

[0092] System 100 is defined by an optical path and a data path. Light from scene 102 (or object) first passes through objective lens assembly 104, which performs initial focusing. The light then passes through diffractive optical element (DOE) 106, which modulates the light to encode information about the scene. This modulated light is captured by image sensor 108, which converts the optical information into an encoded image data signal.

[0093] Encoded image data is transmitted along a data path from image sensor 108 to a reconstruction processing engine, which may be a processor 110 configured to execute reconstruction algorithm 112 to decode the captured data and generate a final reconstructed image 116. Furthermore, once trained, reconstruction algorithm 112 can be stored as executable instructions on a non-transitory computer-readable medium. When these instructions are executed by processor 110, they cause the processor to perform the decoding method described herein.

[0094] The co-design of optical hardware and reconstruction software / algorithms is preferably achieved through an end-to-end optimization process 114. As shown in Figure 1, the parameters of DOE 106 and reconstruction algorithm 112 are jointly optimized in the end-to-end process 114. This co-design ensures that the optical encoding performed by DOE 106 and the decoding capabilities of reconstruction algorithm 112 are optimally matched, thereby achieving performance that cannot be achieved by traditional sequential design methods and maximizing the overall system performance. This process forms the basis of the imaging system and the image acquisition method.

[0095] Further reference Figure 2B An embodiment of the imaging system includes an objective lens assembly, a diffractive optical element (DOE), an image sensor, and a processor including a reconstruction processing engine.

[0096] Preferably, the objective lens assembly can be configured to receive light from the scene and perform initial focusing. In some embodiments, the objective lens assembly can be simple, comprising a single refractive lens element, such as... Figure 7A The biconvex lens shown and used in the first application described herein. Advantageously, this configuration provides maximum compactness. Alternatively, the objective assembly may include multiple refractive lens elements forming a compound lens assembly, which can provide better correction of inherent aberrations before light reaches the DOE.

[0097] Objective lens assembly 104 can be implemented in various configurations. In a simple embodiment, it may include a single refractive lens element. In a more complex embodiment, it may include multiple refractive lens elements forming a compound lens assembly, such as a Cooke three-element configuration. To further enhance aberration correction over a wide field of view, at least one refractive lens element in the compound lens assembly may be aspherical. Different embodiments of this objective lens assembly 104 will be described further later in this disclosure.

[0098] Preferably, refer to Figure 5B and Figure 7B The compound lens assembly may include a Cooke three-element configuration to balance performance and complexity. Furthermore, at least one lens element in the compound lens assembly may have an aspherical surface, as employed in the second application described herein, for better control of off-axis rays. The objective lens assembly defines the aperture stop plane, which limits the light cone passing through the system.

[0099] Reference Figure 2A and 2B The DOE is preferably positioned "away from the aperture," unlike coded aperture systems where the coding element is typically placed at or very close to the aperture stop. This means that the DOE is positioned along the optical path between the objective assembly and the image sensor, but significantly offset from the aperture stop plane.

[0100] Figure 1AThe diagram schematically illustrates the differences between placement on the aperture, near the sensor, and the preferred placement away from the aperture. In this example, placing the encoding element in the aperture or pupil plane only results in global control of the wavefront. In the presence of off-axis aberrations, this global effect becomes an obstacle to local aberration correction. On the other hand, by positioning the DOE away from the aperture, the system can achieve spatial demixing of degrees of freedom. Different regions of the DOE intercept and modulate light rays corresponding to different field-of-view angles in different ways. Advantageously, this provides local control of the wavefront across the entire image plane, which is particularly advantageous for correcting spatially varying off-axis aberrations in wide FoV systems and for achieving complex point spread function (PSF) shaping.

[0101] In a preferred embodiment, DOE 106 is located away from the aperture, meaning it is physically offset from the aperture stop plane of objective lens assembly 104.

[0102] This off-aperture positioning is a key feature, enabling spatial modulation of the incident light angle. The functional result is local wavefront control on the image sensor 108, which is particularly effective for correcting off-axis aberrations in wide field-of-view systems. For optimal performance, the DOE 106 can be positioned at a distance between 0.1 and 0.4 of the effective focal length of the image sensor 108, measured from the aperture stop plane of the objective assembly 104.

[0103] The optimal position represents a trade-off between the locality of control (preferred to be closer to the sensor) and the ability to spatially redirect light or demultiplex angles (preferred to be closer to the aperture). Preferably, optimal performance is achieved when the DOE is positioned at a distance between 0.1 and 0.4 of the effective focal length (EFL) from the aperture stop plane of the objective assembly toward the sensor, and more preferably, the DOE can be positioned at approximately 0.24 of the EFL, falling within... Figure 3B Within the high-performance region identified in the text.

[0104] In a preferred embodiment, the DOE can be a pure phase modulation element, manipulating the phase of light passing through it without significantly attenuating its amplitude, for example, by creating a height map on the substrate material. The phase shift introduced by the DOE at a given point (p,q) is related to the height h(p,q) and the refractive index n of the material, as follows: DOE(p,q)=k(nλ0 1) h(p,q), where k is the wavenumber and λ0 is the design wavelength. An exemplary optimized DOE height diagram is shown in... Figure 5B In the absence of theoretical constraints, DOEs can be fabricated using standard microfabrication techniques, such as iterative photolithography and dry etching processes (e.g., reactive ion etching), to create multiple discrete phase levels on substrates such as fused silica.

[0105] The DOE 106 is preferably a pure phase modulation element that manipulates the phase of light without significantly attenuating its amplitude. The parameters of the jointly optimized DOE may include a height map that defines the phase modulation profile. After the ideal height map is determined in the optimization process, the physical DOE is produced by manufacturing a substrate with a surface relief corresponding to these optimized parameters.

[0106] An image sensor, such as a standard CMOS or CCD image sensor, can be positioned after the DOE to capture light that has been modulated by both the objective lens assembly and the DOE. This captured light is formed by coded measurements, which may appear blurry or have artifacts due to optical encoding.

[0107] Preferably, the processor may include a reconstruction processing engine to decode the captured coded light. Preferably, the reconstruction processing engine may include a deep neural network (DNN) processing engine, leveraging the capabilities of deep learning to solve complex inverse problems. The parameters of the DOE (specifically its heightmap) and the parameters of the reconstruction processing engine (the weights of the DNN) are jointly optimized in an end-to-end (E2E) manner. This E2E optimization ensures an optimal match between the optical encoding performed by the DOE and the decoding capabilities of the DNN, thereby maximizing the performance of the target imaging task. The processor itself can be any suitable computing device, such as a CPU, GPU, FPGA, or dedicated AI accelerator, integrated internally or externally connected to the camera.

[0108] The reconstruction engine, or reconstruction algorithm 112, is preferably implemented as a deep neural network. To handle multiple imaging tasks simultaneously, the deep neural network can employ a multi-head architecture with a shared feature extractor and at least two separate decoding heads. For example, in RGBD imaging applications, a first decoding head is configured to reconstruct a color image, and a second decoding head is configured to estimate a depth map of the scene from the same coded image.

[0109] For multi-task imaging, such as simultaneous color image reconstruction and depth estimation (RGBD imaging), DNNs preferably employ a multi-head architecture, such as... Figure 2B This is illustrated schematically. This can involve a shared feature extractor network (e.g., based on ResNet18) that processes the encoded measurements of the input, followed by multiple separate decoder heads, each dedicated to a specific task. Preferably, one decoder head reconstructs a high-fidelity color image, while another estimates a depth map. Advantageously, compared to a single-output network, this architecture can efficiently handle the different characteristics of the color image (dense features) and the depth map (sparse features), and alleviates the training burden. Preferably, the processor reconstructs both color and depth information.

[0110] System 100 is particularly advantageous for wide-angle imaging and is configured to capture images with a wide field of view of at least 45 degrees. The reconstruction algorithm 112 can be trained using a loss function including at least one of mean squared error (MSE) loss, perceptual loss, L1 loss for depth estimation, or point spread function (PSF) focusing loss. When multiple loss functions are used for multi-task training, dynamic weighted averaging (DWA) can be employed to automatically adjust the weights of the different loss functions during the optimization process.

[0111] Preferably, the overall system architecture, combining an objective lens (which can be simple or compound / hybrid refractive-diffractive), a uniquely positioned off-aperture DOE for local wavefront control, a sensor, and a co-designed deep learning processor, enables high-fidelity imaging over a wide field of view of at least 28 degrees, more preferably at least 45 degrees, overcoming the limitations of existing systems. Experimental results show that the off-aperture DOE significantly enhances image quality (e.g., at 45° FoV, the PSNR of the simple lens system is improved by more than 5 dB) and improves depth recovery accuracy (e.g., at 28° FoV, the MAE of the compound lens system is reduced by up to 24% compared to the near-aperture system).

[0112] Figure 2B An imaging pipeline of an imaging system according to an embodiment of the present invention is illustrated. The inventors use a combination of ray tracing and wave propagation to model the light propagation of the camera. The compound lens module is pre-optimized and remains fixed during E2E training, while the entire system is fully differentiable. A DOE placed between the lens module and the sensor facilitates information encoding. A multi-head decoding network based on the ResNet architecture is integrated to support optimization for multiple vision tasks.

[0113] The optical component of an imaging system consists of a refractive lens and a DOE, such as Figure 2B As shown on the left. The inventors demonstrate this design paradigm through two configurations tailored to specific imaging tasks. Initially, as a basic application, the inventors used a simple primary lens and a DOE placed at a distance from the aperture to achieve wide FoV imaging. Building upon this, a second application not only aims to achieve wide FoV imaging capability but also utilizes depth estimation facilitated by a pre-optimized Cooke three-element lens and an end-to-end designed DOE.

[0114] To collaboratively design these optical elements and the image processing neural network, the inventors developed an image formation model that simulates light propagation from an object to an image sensor. In this model, the inventors first resort to geometrical optics to obtain the response of the refractive lens, i.e., the modified complex amplitude. Specifically, the inventors assume that propagation through a simple lens is dispersive and model it using a wavelength-dependent quadratic phase, while the propagation of the composite optics is derived via ray tracing. After obtaining the response, the inventors employ wave optics theory to accurately calculate the DOE-modulated PSF. Finally, a wide FoV image is synthesized through convolution of the spatial displacement variation between the object and the PSF.

[0115] Complex lens elements are widely used in imaging systems to enhance performance, particularly in mitigating off-axis aberrations. To accommodate this design scenario, the inventors used geometric optics to simulate complex lenses.

[0116] The lens can be optimized or predefined and then simulated using the differentiable ray tracing engine dO to obtain the complex field before applying wave optics. Rays are traced from the front plane of the lens module's bounding box to the rear plane, which is tangent to the outside of the lens module and is assumed to be perfectly refractive within its boundaries. The rear plane serves as a transition plane, connecting refractive and diffractive optics. The complex field on this plane represents the transfer function of the compound lens at various angles.

[0117] Ray tracing involves tracing rays between continuous surfaces—the lens, the aperture, and the front and back planes. At each point on surface i... At point i, where i = 0, ..., S – 2, and S is the total number of surfaces, and the number of rays is... exist It intersects with the subsequent surface i + 1. Here, the direction... It is from the previous direction of light. The unit vector, determined by the surface material, follows Snell's law. It is worth noting that for... Corresponding to the direction of the incident light, at each surface intersection, the distance traveled by the light ray is calculated using an iterative root-finding technique such as Newton's method. .

[0118] The total phase shift of the light ray is related to the optical path length (OPL) through the lens and is given by the sum of the OPL niti in each propagation medium. The light ray reaches scattering points on the transition plane, but subsequent processes (such as wave propagation or sensing) require measurements on a uniform grid. Therefore, these points are interpolated onto a regular grid (ξ,η) using Clough-Tocher interpolation (denoted as g). The phase at wavelength λ is then... Represented as: (1) The inventors neglected energy attenuation during geometric light propagation and assumed that the energy of each ray was uniform. Intensity on the transition plane. It is obtained by bilinear interpolation to adjacent grid points of the accumulated ray energy. The transfer function is defined as follows: ,in .

[0119] The inventors' two-step wave propagation process begins at the transition plane, passes through the off-aperture DOE, and finally reaches its apex on the sensor plane. To simulate highly off-axis wave propagation, the inventors employed the widely accepted angular spectrum method (ASM). Input field The finite region propagating from a distance z to the plane (x, y) is represented as: (2) in It is the frequency domain transfer function, k = 2π / λ is the wave number, and F represents the Fourier transform (FT) operator.

[0120] Connecting two wave propagation paths at a large angle off-aperture DOE can be computationally demanding. To address this, the inventors employed LS-ASM, which introduces a compensation term into the phase of the input field, thereby generating a quasi-coaxial field. This reduces the sampling requirements for simulating off-axis propagation. For the input field at coordinates (x, y), the compensation term is defined as... ,in( The ) represents the effective spectral center. Then, the wavefield reaching the DOE plane is calculated as follows: (3) The DOE I learned Modulated incident field, in which is the refractive index of the substrate material at the nominal wavelength, and h is the height diagram of the designed DOE. From the origin of the principal ray on the DOE... and direction exist Cut out the window This includes most of the diffraction energy. Therefore, the complete wave propagation is formulated as: (4) For a point source located at (x0, y0, z0), the PSF is calculated as the squared amplitude of the complex field reaching the sensor plane within a specified window (x, y): (5) in .

[0121] To simulate camera measurements, the inventors convolved the PSF with local image patches, assuming local displacement invariance due to the relatively smooth variation of the PSF across the entire FoV. The inventors considered multiple depths. And it utilizes an established occlusion perception model to provide a continuous and seamless representation of depth variations: (6) in * indicates element-wise multiplication. The image is quantized into D depth layers, represented as... And accompanied by a binary mask Each depth layer consists of... Normalization was performed, and Gaussian noise N was incorporated into the simulation. The sensor's spectral response was also applied to image I to address the sensor's sensitivity to different wavelengths.

[0122] The reconstruction network is responsible for extracting compressed information from sensor measurements and decoding it into both image and depth data. Figure 2B However, color images and depth maps have fundamentally different spatial properties: images are characterized by dense color features, while depth maps exhibit sparse features. Furthermore, sharp image reconstruction leads DOE to generate similar PSFs across various depths, while depth estimation prefers significant PSF variations along the depth axis. This inherent difference necessitates extracting different types of information from the encoded features. Therefore, employing a separate decoding head for each task is crucial.

[0123] To simplify training and reduce computational overhead, the inventors employed a shared feature extractor for both decoding heads. Specifically, they utilized a lightweight pre-trained ResNet18, similar to Monodepth2, and fine-tuned it to better align with the encoded features in sensor measurements. Each decoding head consists of four upsampled blocks, each integrating bilinear interpolation and residual convolutional layers. Furthermore, skip connections were applied between the feature maps of the ResNet encoder and the corresponding layers of the decoder, facilitating efficient gradient flow and enhanced feature decoding across all levels.

[0124] During training, the decoder in the network generates multi-scale images to compute losses tailored to each task. For sharp image reconstruction, the inventors employ pixel-level mean squared error (MSE) and VGG16 perceptual loss Lpercep to ensure that the reconstruction closely matches ground reality while maintaining visual fidelity. Given an image patch of size M × N, the MSE loss is calculated as follows: For depth estimation, the inventors utilize the L1 loss, which promotes sparsity, defined as... To enhance focusing capability, the inventors defined a region O where most of the PSF energy should be concentrated. This region is a circle with a diameter equal to half the size of the PSF. PSF loss is defined as... .

[0125] It is worth noting that multi-task reconstruction requires precise adjustment of the loss function weights for each task. The inventors employ Dynamic Weight Averaging (DWA) instead of relying on heuristic-based adjustment methods, dynamically adjusting the weights based on the average gradient from previous iterations. This strategy avoids complex hyperparameter tuning when all losses have similar importance. Therefore, the total loss function is a weighted sum of the individual losses, as shown below: (7) Wide FoV imaging is becoming increasingly challenging when aiming to achieve compact lens shape factors (e.g., a single refractive lens element). Most state-of-the-art coded aperture and / or depth optics imaging solutions assume spatially invariant PSF behavior and have demonstrated FoVs up to 30° in experiments.

[0126] The wide FoV imaging setup comprises a simple focusing lens with a FoV of approximately 45° and a rotationally symmetric DOE, effectively compensating for most aberrations. Due to computational limitations, the camera was configured with an effective focal length (EFL) of 35 mm and an f-number of 12 for proof-of-concept purposes. The object was focused at a depth of 1.4 meters.

[0127] The system was trained on a total of 26,281 images from three datasets, including 21,818 images from the simulated Sceneflow FlyingThings3D (SF), 2,506 images from the real-world Dualpixels (DP), and 1,957 images from Instereo2K (IS).

[0128] During testing, the inventors utilized 1,000 images from the SF dataset, 684 images from the DP dataset, and 50 images from the IS dataset. All images were upsampled and cropped to match the sensor resolution. For this scenario, the network used a single image head with λ² = 0. The learning rates for both the network and the DOE were initialized to 1 × 10⁻⁶. -4 and 6 × 10 - ³. Training is performed for 50 epochs, with a cosine annealing scheme introduced starting from the 35th epoch to reduce the learning rate to 1 / 10 of its initial value. Afterward, the system undergoes an additional 10 epochs of fine-tuning at the adjusted learning rate, and triggers early stopping upon convergence.

[0129] The inventors randomly sampled two angles for each forward iteration in the aperture simulation. This sampling strategy facilitates gradient calculation within the overlapping region between each beam. The training process typically takes one day on an Nvidia RTX 4090 GPU.

[0130] Figure 3A The system setup of a ready-made thin lens and a distance-from-aperture DOE is shown. Figure 3B The fitted PSNR plots of the recovered images at various DOE locations are shown, indicating that the optimal location is a trade-off between aperture and sensor performance. The dashed line indicates the optimal DOE location for each dataset, which is approximately 0.24. Figure 3C The diagram shows a comparison of PSF amplitudes at several FoVs when the DOE is placed in different locations. For visualization purposes, the PSF is cropped by the central 99 pixels.

[0131] To evaluate the trade-off between angle demultiplexing and local degrees of freedom, the inventors analyzed the system's performance at various DOE positions, as shown in Figure 3. The DOEs were optimized at uniform intervals along the optical path from 0 to 1, where 0 corresponds to the principal plane of light refraction and 1 corresponds to the sensor plane. Considering assembly difficulties and the potential performance degradation caused by placing the DOE too close to the sensor (where local degrees of freedom are compromised), the analysis focused on DOE positions between 0 and 0.4, corresponding to distances of 35 to 14 mm from the sensor plane. A smoothing factor of 1.7 was used to fit spline curves to the sampled data points.

[0132] The optimized PSF shown on the right side of Figure 3 implies that the angular tradeoff decreases as the DOE moves further away from the aperture (top row). For a larger FoV of 40°, the PSF is most concentrated near positions 0.2 and 0.3. The quantitative results depicted in the figure reinforce this observation, indicating the image PSNR values ​​of the DOE optimized at different locations when evaluated on all three datasets. The aperture-based DOE system achieves PSNR values ​​of 35.49 dB, 35.44 dB, and 31.74 dB on the SF, IS, and DP datasets, respectively. This is 5.35 dB (SF), 4.41 dB (IS), and 4.13 dB (DP) higher than the system on the aperture, respectively.

[0133] The observed trend highlights the trade-off between local and global degrees of freedom, implying that the optimal DOE location should be between the aperture and the sensor plane, but not at either extreme.

[0134] It is worth noting that although the DOE position can be used as a parameter for co-optimization, the above analysis shows that the performance impact is minimal when the DOE is between 0.1 and 0.3 of the EFL. Therefore, the DOE position is unstable in multiple optimization runs, but all found solutions exhibit a comparable overall performance level. Therefore, for the physical prototype, the inventors chose to forgo automatic optimization of the DOE position and instead select a location within the high-performance region that also facilitates mechanical installation.

[0135] The present invention also includes a method for capturing wide-field-of-view images using the system. This method involves several key steps, implemented by the system architecture and end-to-end (E2E) design.

[0136] First, the objective lens assembly receives light from the scene. Second, this light is modulated by a diffractive optics (DOE) positioned off-aperture. As previously mentioned, this off-aperture placement enables spatially variable modulation based on the incident angle, providing the local wavefront control needed to correct aberrations and effectively encode information over a wide FoV. Third, the modulated light, now carrying encoded information about the scene (potentially including depth), is captured by the image sensor as an encoded measurement.

[0137] Fourth, the captured coded light is processed by a reconstruction processing engine. This reconstruction processing engine, preferably a DNN, decodes the measurements to generate a reconstructed image of the scene. Due to optimized optical coding and a powerful decoding network, the reconstructed image exhibits enhanced image fidelity (e.g., sharpness, reduced aberrations) over a wide field of view. Preferably, if a multi-head network architecture is employed, the processing steps may involve reconstructing both the color image and the depth map from a single coded measurement.

[0138] Preferably, the E2E optimization process is used to design system components. This involves jointly optimizing the parameters of the DOE (e.g., its height map) and the parameters of the reconstruction processing engine (e.g., DNN weights). This joint optimization is performed using a differentiable simulation of the entire image formation process from the object to the final reconstructed image / depth map. The differentiable simulation preferably combines differentiable ray tracing and waveoptic modeling.

[0139] refer to Figure 2A and 2B Ray tracing (e.g., using an engine like dO) can simulate propagation through the refracting objective assembly, while microwaveable propagation simulates the effects of the DOE and propagation to the sensor. Accurate modeling of wave propagation, especially off-axis rays in wide FoV systems, is also preferred. Preferably, microwaveable propagation can utilize angle spectral methods (ASM), suitable for efficient and accurate off-axis simulations, such as minimum sampling ASM (LS-ASM). Differentiability of the entire pipeline allows gradients to backpropagate from the defined loss function to simultaneously update DOE parameters and network weights, enabling co-design.

[0140] The end-to-end optimization process 114 is facilitated by a differentiable simulation of the entire image formation process of the imaging system 100. This simulation provides a computational model that allows gradients to backpropagate from the final loss function to update hardware and software parameters. The differentiable simulation preferably combines differentiable ray tracing to simulate light propagation through the objective assembly 104, and microwave propagation to simulate the effects of the DOE 106 and subsequent propagation to the image sensor 108. Microwave propagation can utilize angular spectrum methods (ASM) suitable for off-axis propagation, such as least-sampled ASM (LS-ASM). Therefore, the present invention also provides a method for designing such a computational imaging system by defining and jointly optimizing these simulation models.

[0141] The training process for the reconstruction processing engine can utilize a combination of loss functions tailored to specific imaging tasks. For image reconstruction, this could include pixel-level losses like mean squared error (MSE) and perceptual losses like VGG loss to ensure fidelity and visual quality. For depth estimation, losses like L1 loss are typically used to handle the sparsity of depth maps. PSF focusing losses might also be included to encourage the DOE to generate sharp PSFs. Balancing these losses is important when multiple tasks are involved. Furthermore, Dynamic Weighted Averaging (DWA) can be employed to automatically adjust the weights of different loss components during training, avoiding manual adjustments.

[0142] Advantageously, E2E optimization and aperture coding strategies enable this method to achieve results that are unattainable by traditional systems or simpler computational imaging methods. For example, Figure 4 Simulation results are shown, demonstrating that for wide FoV (~45°) imaging, the DOE at the aperture is significantly improved in image sharpness compared to the DOE at the aperture. Figure 6 Simulation results of RGBD imaging (~28° FoV) are shown, where the near-aperture system provides a clearer AiF image and a more accurate depth map compared to the near-aperture system. Figure 8 and Figure 9 Experimental results from the physical prototype (components shown in Figure 7) are presented, validating the high performance of simple lens RGB imaging and composite lens RGBD in real-world indoor and outdoor scenarios.

[0143] Figure 4 This demonstrates a 4K resolution real-world scene reconstructed using wide FoV imaging in a simulation. The DOE (DoE) at the aperture results in sharper results in both the center and peripheral FoV regions compared to the DOE at the aperture itself.

[0144] The inventors divided the entire FoV into smaller local blocks, where displacement invariance could be reasonably assumed. Each block consisted of 256 x 256 pixels, and the inventors stitched 16 x 10 blocks together to generate the final 4K resolution image. To mitigate stitching artifacts, the inventors incorporated a 32-pixel overlap at each boundary. Figure 4 We compared reconstructed images based on on-aperture and off-aperture DOEs using the Instereo2K dataset. These results show that off-aperture DOEs significantly enhance visual quality in both the central and peripheral regions of the FoV, consistent with the corresponding metrics and optimized PSF properties.

[0145] While basic applications are demonstrated using simple thin lenses, most imaging systems employ compound optics. These systems, while advantageous, also present several challenges. First, the aperture plane typically does not coincide with the principal plane where light refraction occurs. Second, simulating the DOE before ray tracing the lens complicates the optimization process. Third, the joint optimization of refractive and diffractive lenses has received limited attention in previous research.

[0146] To address these challenges, the inventors introduced a composite optics prototype designed for wide FoV depth and color imaging. The system incorporates an optimized Cooke three-element lens as the focusing lens module, achieving a practical balance between compactness and aberration correction. The lens profile was initially optimized in Zemax OpticStudio to minimize aberrations at focal length across the entire FoV. For ease of manufacturing, the latter two lenses were replaced with the closest commercially available options, while the first lens was fine-tuned. Furthermore, the first surface of the first lens was designed with an aspherical shape for better control over off-axis rays. A distance-to-aperture (DOE) is applied after the lens module.

[0147] The camera is configured with a 35mm EFL and f / 7. To effectively integrate the lens module, the inventors optimized the position of the DOE within the back focal length, placing it 22mm from the distance sensor, but before the pre-assembled sensor assembly, to facilitate simulation and assembly. The system is designed to capture depth and color information of objects in the range of 0.8m to 10m, corresponding to approximately 1.2 diopters, with a maximum FoV of approximately 30 degrees.

[0148] For this multi-task application, the network architecture employs two decoding heads: one for AiF imaging and the other for depth estimation. Based on an optimized compound lens module for AiF imaging and insights into balancing multiple tasks, the inventors chose to assign higher loss weights to the more challenging depth estimation task. Therefore, the inventors allocated depth estimation weights on top of the DWA (Depth Estimation and Depth Assessment) algorithm. An additional weight of 2 = 1, while setting the weight for other parts. 0 = 3 = 0.05.

[0149] To manage GPU memory limitations associated with larger apertures, the inventors set... 1 = 0 was used to disable perceptual loss. Experimental results show that a relatively balanced performance was achieved between the two tasks. The training process retained the settings of the previous simulation, but the cosine annealing scheduler was started from the 15th epoch to gradually reduce the learning rate to 1 / 100 of its initial value, ensuring training stability. Training on the same GPU as before typically takes two days due to the increased sampling.

[0150] While off-axis wave propagators (LS-ASMs) have proven effective in generating PSFs for simple lenses, they are insufficient to simulate the PSFs of composite optical systems. Similarly, dOs can simulate the geometric propagation of composite systems but cannot simulate the fine details of light diffraction. The fusion of these two components has not been previously demonstrated, and the lack of ground-based data on such systems hinders insightful comparisons. The inventors compare dO PSFs with PSFs generated from fused systems, demonstrating a consistent shape and structure across the entire FoV.

[0151] To save computational memory and reduce runtime, the inventors chose not to train the refractive lens together with the DOE E2E, as double-wave propagation at large angles requires a very high sampling rate. While E2E optimization of the fusion system is theoretically feasible, as demonstrated in concurrent work, this is orthogonal to the scope of this work. Instead, the inventors propose a proof-of-concept system demonstrating the effectiveness of the image system according to embodiments of the invention while keeping complexity to a minimum.

[0152] Figure 5A The optimized PSF using a large-aperture FoV EDOF imaging system is shown at depths from 0.8 m to 10 m and FoVs up to 28°. The depth layers are sampled uniformly. Figure 5B The optimized DOE height map and the 3D model of the optimized system are shown, with the dashed line indicating the location of the DOE placed near the aperture (approximately 0.11) and far from the aperture (approximately 0.24).

[0153] The inventors first designed a Cook three-element lens using Zemax OpticStudio, which minimizes aberrations at a depth of focus of 1.7m across discrete angles (0°, 5°, 10°, 14°). This optimization resulted in a lens module with relatively low aberrations across the entire FoV, although not at all depths. Using this optimized lens as a foundation, the inventors integrated a DOE primarily for encoding depth information.

[0154] The inventors positioned the DOE at approximately 0.24 of the EFL, or 8.4 mm from the aperture plane. Due to the challenges of simulating DOE placement before refractive optics, the inventors compared this off-aperture configuration with a near-aperture setting, where the DOE is located on the tangential surface of the last lens, approximately 4 mm from the aperture plane (0.11 of the EFL). Figure 5 presents the optimized PSF of the off-aperture system, uniformly sampled across six depth directions and three angular directions. Notably, the system was trained at seven discrete angles (0°, 2.5°, 5°, 7.5°, 10°, 12°, and 14°) and included intermediate angles of 1.25°, 3.75°, 6.25°, 8.75°, 11°, and 13° during testing. The optimized DOE in both the near-aperture and off-aperture systems exhibits distinct characteristics in its outer ring, highlighting the ability of the off-aperture DOE to effectively encode high-frequency information of large FoV rays without internal FoV interference. This represents a key advantage of aperture design.

[0155] Table 1 presents a quantitative comparison of near-aperture and far-aperture systems. The far-aperture configuration achieved approximately 1 dB higher PSNR for color images and reduced the mean absolute error (MAE) of the depth map by up to 24% on three representative datasets. The greater improvement in depth maps can be attributed to two factors: first, the near-aperture system incurs fewer drawbacks at the DOE placement at 0.11 of the EFL compared to 0.24; second, the Cooke triple lens is primarily optimized for image performance, leaving more room for improvement in depth recovery. The inventors further compared the multi-head network with a simple U-Net-based architecture, which outputs both image and depth from the same layer as the multiple channels. The mode of Image System I significantly outperforms the U-Net architecture in both image and depth recovery.

[0156]

[0157] Table 1. Evaluation of depth maps in wide FoV depth and color imaging applications when optimizing DOE at two locations closer to and further from the aperture, and when using a simple network, for color images with increased PSNR (dB) and depth maps with decreased MAE. These evaluations were performed on three datasets. The percentage reduction in depth map MAE is shown.

[0158] Figure 6 This paper presents a restored 4K resolution real-world image and depth map from a large FoV EDOF imaging system. Magnified blocks of AiF images from near-aperture and far-aperture DOE systems are presented, and the differences between the obtained depth maps are highlighted.

[0159] Reference Figure 6A visual comparison of near-aperture and far-aperture systems was performed at 4K resolution. The far-aperture system exhibited significantly superior sharpness compared to its near-aperture counterpart in the recovered color images. This enhanced image quality was evident across various depths and throughout the entire FoV (including both internal and external regions). Furthermore, the depth maps generated by the far-aperture system showed significantly enhanced fidelity, particularly over a wider FoV. These results are also consistent with previous observations that a distance between 0.2 and 0.3 represents the optimal choice for far-aperture designs.

[0160] In Application 1, the inventors utilized a readily available anti-reflective (AR) coated N-BK7 biconvex lens (Thorlabs LB1811-A) with a focal length of 35mm. Considering the wide FoV, the inventors employed a Sony A6400 mirrorless camera equipped with an APS-C sensor. The effective diameter of the DOE is approximately 12mm, and the total diameter is 16mm. It is mounted with the biconvex lens along with a 1-inch diameter 3D-printed circular bracket. The distance between the lens center plane and the DOE plane is set to 10.5mm. The total length of the imaging system outside the camera body is only 21.5mm, resulting in a compact and lightweight wide FoV imaging system. A mounting adapter was designed and 3D-printed to connect the SM1 lens tube to the camera body.

[0161] In Application 2, the inventors utilized an AR-coated N-SF11 biconcave lens with a focal length of -15mm (LD2060-A) and a CaF2 positive meniscus lens with a focal length of 50mm (LE5243). Importantly, an aspherical lens was fabricated using a computer numerical control (CNC) machining system capable of 5-axis single-point diamond turning, similar to the technology used in state-of-the-art work. Considering the limitations of available turning tools, the inventors chose polymethyl methacrylate (PMMA) as the substrate material, which translates to a refractive index of 1.492 at the dominant wavelength of 550 nm. The inventors employed a FLIR GS3-U3-123S6C-C sensor with 4K resolution.

[0162] Figure 7A The thin lens, DOE, and aperture are shown in Application 1. Figure 7B The three refractive lenses, DOE, and other components in Application 2 are shown. Figure 7C The experimental setup in the laboratory is shown.

[0163] The DOE fabrication employs an iterative photolithography and dry etching process to create 24 discrete phase levels on a fused silica wafer, with a chromium layer serving as an optical baffle. The photolithography and reactive ion etching steps are repeated four times to preserve the high-frequency spatial characteristics crucial for the DOE design. The inventors then assembled all the optics using a custom-designed focus adjustment bracket. Setup diagrams and photographs of the components in both applications are shown in Figures 1 and 7.

[0164] The inventors used a white laser source (LS-WL1 from Edmund Optics) coupled to a collimating lens and a 25 µm pinhole as a point light source. In the first application, PSF captured on the sensor at a distance of 1.4 m was measured at 24 different locations within the quadrant of the image plane. Each measured PSF was spaced approximately 512 sensor pixels apart. In the second application, the inventors captured five FoV PSFs at each depth plane in nine depths ranging from 0.8 m to 5 m.

[0165] After capturing the PSF, the inventors simulated measurements and fine-tuned the network using three datasets. However, the inventors observed a strong haze effect in the measurements of Application 2, which was likely caused by defects in the manufacturing and assembly process. To address this issue and enable the network to learn optical encoding from damaged measurements, the inventors manually perturbed the captured PSF by introducing a zero-frequency term and applying random scaling to simulate real measurement conditions.

[0166] Furthermore, the inventors enhanced the adjustment process by incorporating a high-performance semantic-based depth estimation model, "Depth Anything." Nevertheless, the optical coding system successfully predicted a planar depth map from a fake scene displayed on the screen, a feat the semantic model failed to achieve, highlighting the continued importance of optical coding in solving RGBD imaging problems.

[0167] Experimental results using real-world indoor and outdoor scenes captured using a custom camera prototype are shown in Application 1. Figure 8 For application 2 shown Figure 9 And Figure 1. In particular, Figure 8 Indoor and outdoor experimental results for wide FoV simple lens imaging applications are shown. Shutter speed is indicated in the lower right corner of the reconstructed image. The equivalent f-number is 12. ISO was set to 100 for all captures. Figure 9 Indoor and outdoor experimental results are shown for Application 2: Wide FoV AiF and depth imaging. For each scene, the inventors present half of the captured and reconstructed color images (top), along with magnified patches of them (bottom right) and the estimated depth map (bottom left). The equivalent f-number is 7, and the gain of all captures is set to 0.

[0168] For Application 1, the reconstructed image shows significant improvement over the original measurement in both internal and external FoV. High-frequency details, such as text and edges, are noticeably sharper in the reconstructed image. For example, even small characters, such as those on a bottle label, can be accurately recovered, highlighting the method's ability to effectively reconstruct wide FoV information.

[0169] In Application 2, the AiF image exhibits sharpness across the entire depth range. Notably, distant objects, such as leaves that are so blurred they are unrecognizable to the human eye, are faithfully reconstructed. Furthermore, the recovered depth map aligns well with visual cues, showing clear transitions between depth planes. For example, the scene on the left shows a distinct edge between the box and its background, while the scene on the right highlights the effectiveness of the inventors' method in wide FoV AiF imaging and depth encoding.

[0170] Figure 10 Further experimental results comparing the noise reduction of a wide FoV RGB imaging prototype with those of a pinhole camera setting (0.5 mm diameter) are shown. The invention restores the correct tones (reference captured by iPhone 15) and sharper edges. Both scenes were captured indoors under artificial lighting at a shutter speed of 0.1 seconds and ISO 500.

[0171] The inventors further evaluated the system's wide FoV imaging performance in Application 1 by comparing it to a theoretically aberration-free pinhole setup. In this experiment, indoor scenes were captured under fixed exposure conditions, such as... Figure 10 As shown. The corresponding pinhole setup uses the same convex lens, but with a 0.5mm iris aperture. While this configuration minimizes severe off-axis aberrations over a wide angle, it also significantly reduces the signal-to-noise ratio. The inventors observed that the measurement results appeared extremely dim. Even after applying state-of-the-art low-light image restoration algorithms to the pinhole measurements, the results showed poorer detail recovery compared to the method according to embodiments of the invention, which maintained sharp edge fidelity across the entire FoV and ensured more realistic color reproduction. Importantly, the prototype measurements preserved more fidelity natural tones, as evidenced by the color checkerboard pattern captured by a mobile phone camera.

[0172] Furthermore, while pinholes can reduce off-axis aberrations, large aperture settings remain crucial for computational vision tasks that rely on optical coding, such as RGBD imaging demonstrated in Application 2. In this case, the pinhole configuration inherently extends the depth of field to near infinity, resulting in low efficiency for PSF-based depth recovery. This limitation restricts the system to a purely semantic-based depth estimation approach, which is orthogonal to the range of off-aperture coding exploration in the embodiments of the present invention.

[0173] In summary, the inventors present an off-aperture E2E imaging system designed for wide FoV imaging applications. By positioning the DOE on a plane different from the aperture or pupil plane, the inventors achieve enhanced angle-dependent encoding through an additional design parameter. Analysis of the impact of DOE placement provides valuable insights into how system performance is affected by DOE location. By decoupling light from different angles, the inventors can improve the PSNR of off-axis aberration correction by more than 5 dB. Importantly, the inventors have validated the imaging system with two custom-designed and assembled camera prototypes: one employing a simple lens and the other incorporating multiple refractive optics, both equipped with an E2E-optimized, off-aperture positioned DOE. These prototypes have demonstrated the system's effectiveness in RGB-only and RGBD imaging applications, achieving FoVs up to 45° and 28°, respectively.

[0174] The inventors have devised an aperture coding paradigm with great potential for a wider range of applications, such as compact and wide FoV cameras tailored for AR / VR, autonomous systems, and edge devices, where enhanced angular information coding capabilities are crucial.

[0175] In one exemplary embodiment, consider an autonomous delivery drone navigating a complex urban environment. Equipped with the aforementioned imaging system (using a compound lens and an off-aperture DOE optimized for RGBD), the drone can use its wide FoV capability to continuously monitor its surroundings. The system can capture coded snapshots. The onboard processor in the imaging system can then run a multi-head DNN: one head reconstructs a sharp, wide-angle color image for identifying landmarks, reading signs, and detecting potential hazards (e.g., other drones, birds); a second head simultaneously generates a depth map of the scene.

[0176] Advantageously, this depth map can aid in real-time obstacle avoidance, path planning, and precise altitude control, particularly during landing sequences. The compactness of the imaging system also minimizes payload weight and aerodynamic drag. Furthermore, E2E optimization ensures reliable operation of the system under various lighting conditions encountered during the day.

[0177] When configured for RGBD imaging, one example application of this system is in autonomous systems such as drones, vehicles, or robots. In such applications, the coded images can be processed in real time to reconstruct depth maps, providing data for autonomous navigation or obstacle avoidance. The system's compact and powerful features make it well-suited for such resource-constrained environments.

[0178] These embodiments may be advantageous because the imaging systems and methods described herein can offer significant advantages in applications requiring high-quality imaging across a wide field of view, particularly where compactness and computational power are relevant. The ability to achieve high resolution and potential depth information using simplified optics opens up possibilities across various fields.

[0179] Advantageously, embodiments of the invention can be implemented in a variety of applications, including: Integrating compact, high-performance wide-angle cameras into mobile devices such as smartphones and tablets improves photography and potential AR / VR capabilities. The E2E design allows for system tailoring to specific mobile imaging tasks.

[0180] It provides extensive situational awareness for drones, autonomous vehicles, and robots. The system can provide clear images for visual recognition tasks (object detection, tracking) and potentially provide depth information for navigation and obstacle avoidance, all from a single compact sensor in robotics and autonomous system applications.

[0181] In surveillance and security applications, wide-area monitoring can be achieved with fewer cameras while maintaining high image detail throughout the scene, even under challenging conditions that can be addressed through end-to-end optimization.

[0182] Compact, wide FoV cameras can be used in head-mounted displays for inside-out tracking, environment mapping, or capturing the user's view. Off-aperture coding paradigms support AR / VR system operation, where enhanced angular information encoding is crucial.

[0183] Imaging systems can also be implemented in medical devices, such as endoscopes or other medical imaging modalities that require a wide field of view and high detail, although they may need to be adapted to specific domain characteristics.

[0184] Furthermore, the combination of optimized optics and a potentially lightweight DNN makes the system suitable for deployment on resource-constrained edge devices.

[0185] While not strictly necessary, the embodiments described with reference to the accompanying drawings can be implemented as an application programming interface (API) or as a series of libraries for developers, or can be included in another software application, such as a terminal or personal computer operating system or a portable computing device operating system. Typically, since program modules include routines, programs, objects, components, and data files that assist in performing specific functions, those skilled in the art will understand that the functionality of a software application can be distributed across multiple routines, objects, or components to achieve the same functionality desired herein.

[0186] It should also be understood that any suitable computing system architecture can be used when the methods and systems of the present invention are implemented entirely or partially by a computing system. This will include stand-alone computers, network computers, and dedicated hardware devices. When the terms "computing system" and "computing device" are used, these terms are intended to cover any suitable computer hardware arrangement capable of implementing the described functions.

[0187] Those skilled in the art will understand that many variations and / or modifications can be made to the invention as illustrated in the specific embodiments without departing from the spirit or scope of the invention as broadly described. Therefore, these embodiments should be considered illustrative rather than restrictive in all respects.

[0188] Any references to prior art contained herein should not be construed as an admission that the information is common general knowledge, unless otherwise stated.

Claims

1. An imaging system for capturing wide field-of-view images, characterized in that, It includes: An objective lens component configured to receive light from the scene; A diffractive optical element (DOE) positioned along the optical path to receive light from the objective lens assembly; An image sensor configured to capture coded images modulated by the objective lens assembly and the DOE; as well as A reconstruction processing engine configured to convert the captured coded image into a reconstructed image of the scene; The parameters of the DOE and the parameters of the reconstruction processing engine are jointly optimized in the end-to-end process.

2. The imaging system according to claim 1, characterized in that, in, The DOE is located away from the aperture and is offset from the aperture stop plane of the objective lens assembly.

3. The imaging system according to claim 2, characterized in that, in, The DOE's aperture positioning can spatially modulate the incident light based on the incident angle, thereby providing local wavefront control on the image sensor.

4. The imaging system according to claim 2, characterized in that, in, The DOE is positioned at a distance between 0.1 and 0.4 of the effective focal length of the objective assembly, measured from the aperture stop plane of the objective assembly toward the image sensor.

5. The imaging system according to claim 1, characterized in that, in, The objective lens assembly includes a single refractive lens element.

6. The imaging system according to claim 1, characterized in that, in, The objective lens assembly includes multiple refractive lens elements that form a compound lens assembly.

7. The imaging system according to claim 6, characterized in that, in, The compound lens assembly includes a Cook three-element configuration.

8. The imaging system according to claim 1, characterized in that, in, The reconstruction processing engine includes a deep neural network.

9. The imaging system according to claim 8, characterized in that, in, The deep neural network includes a multi-head architecture with a shared feature extractor and at least two separate decoding heads.

10. The imaging system according to claim 9, characterized in that, in, The first decoding head is configured to reconstruct a color image, and the second decoding head is configured to estimate a depth map of the scene.

11. The imaging system according to claim 1, characterized in that, in, The end-to-end process utilizes the differentiability of the image formation process of the imaging system.

12. The imaging system according to claim 11, characterized in that, in, The differentiable simulation combines differentiable ray tracing for the objective assembly and microwave propagation for the DOE.

13. The imaging system according to claim 12, characterized in that, in, The microwave propagation method utilizes the angular spectrum method (ASM) suitable for off-axis propagation.

14. The imaging system according to claim 1, characterized in that, in, The system is configured to capture images with a wide field of view of at least 45 degrees.

15. A method for capturing wide field-of-view images, characterized in that, Includes the following steps: Light is received from the scene via the objective lens assembly; The light is modulated using a diffractive optical element (DOE) to form modulated light; The modulated light is captured as an encoded image using an image sensor; and The encoded image is processed using a reconstruction algorithm to generate a reconstructed image of the scene; The parameters of the DOE and the parameters of the reconstruction algorithm are jointly optimized in the end-to-end process.

16. The method according to claim 15, characterized in that, in, The step of modulating the light includes positioning the DOE at a distance from the aperture, offset from the aperture stop plane of the objective lens assembly, to achieve spatial variation modulation for local wavefront control.

17. The method according to claim 15, characterized in that, in, The steps of processing the encoded image include reconstructing both a color image and a depth map from the encoded image using a multi-head neural network.

18. The method according to claim 15, characterized in that, in, The joint optimization is performed using a differentiable simulation of the image formation process.

19. The method according to claim 18, characterized in that, in, The differentiable simulation combines ray tracing for the objective assembly and wave-optical modeling for the DOE.

20. The method according to claim 15, characterized in that, in, The joint optimization includes training a multi-task reconstruction algorithm using Dynamic Weighted Averaging (DWA) to automatically adjust the weights of multiple loss functions.

Images