Super-resolution image display and free-space communication using a diffractive decoder

Abstract

A deep learning-enabled system for displaying or projecting high-resolution images is disclosed. The system is based on a jointly trained pair of electronic encoder and all-optical decoder networks to synthesize / project super-resolution images using a low-resolution wavefront modulator. The electronic encoder network rapidly preprocesses a high-resolution image of interest, encodes its spatial information into a low-resolution (LR) modulation pattern, and projects it through a low-SBP wavefront modulator. The all-optical decoder network processes this LR-encoded information using a thin, transparent layer structured using deep learning, and projects the all-optically synthesized super-resolution image into an output FOV. Results show that this diffractive image display system achieves a super-resolution factor of approximately 4, with an SBP increase of approximately 16 times. This system can be extended to operate at visible wavelengths and can be used for compact, low-power, computationally efficient, large-viewing-angle, high-resolution displays.

Description

[Technical Field]

[0001] Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 352,045, filed June 14, 2022, and U.S. Provisional Patent Application No. 63 / 497,052, filed April 19, 2023, which are incorporated herein by reference. Priority is claimed pursuant to 35 U.S.C. § 119 and other applicable laws and regulations.

[0002] STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH AND DEVELOPMENT

[0002] This invention was made with government support under DE-SC0023088 awarded by the U.S. Department of Energy. The government has certain rights in this invention.

[0003]

[0003] The technical field relates to diffractive super-resolution displays. This display design uses a deep learning-enabled diffractive display design based on a jointly trained pair of electronic encoder and diffractive optical decoder to synthesize / project super-resolution images using a low-resolution wavefront modulator. The technical field also relates to a free-space optical communication system and method for transmitting information even through obstructions that block the optical path. [Background technology]

[0004] Over the past decade, augmented reality / virtual reality (AR / VR) systems have attracted significant interest, aiming to provide immersive and enhanced user experiences in a wide range of fields, including human-computer interaction, visual media, art, and entertainment consumption, as well as biomedical applications and instrumentation. However, the realization of AR / VR systems has primarily relied on fixed-focus stereoscopic display architectures, which have been partially limited in performance in terms of power efficiency, device form factor, and support for the natural depth cues of the human visual system. Holographic displays, which use spatial light modulators (SLMs) and coherent illumination such as lasers, are a promising alternative that enable precise control and manipulation of optical wavefronts and can simplify the optical setup between the SLM and the human eye. Furthermore, this approach can emulate the wavefronts emanating from a desired 3D scene to provide human visual depth cues, potentially eliminating sources of user discomfort (e.g., vergence-accommodation conflict) associated with fixed-focus stereoscopic displays.

[0005] Despite these advantages, holographic displays generally have a relatively modest spatial-bandwidth product (SBP) due to the limitations of current wavefront modulator technology, which is directly determined by the number of individually addressable pixels on the SLM. As a result, current holographic display systems are unable to meet the spatiotemporal requirements of AR / VR devices due to the limited range of synthesized image sizes and corresponding viewing angles. In fact, previous research on this subject has shown that a wavefront modulator for a wearable AR / VR device should ideally have approximately 50K × 50K pixels, with a pixel pitch smaller than the wavelength of visible light. Given that state-of-the-art SLMs can provide resolutions up to 4K (e.g., 3,840 pixels horizontally and 2,160 pixels vertically), with pixel pitches typically 5–20 times the wavelength of light in the visible light range, such an SLM is not feasible with current technology. Even if a new SLM architecture capable of supporting such a large SBP were developed, additional challenges would arise in terms of power consumption, memory usage, computational load, form factor, and system complexity.

[0006] To maximize the potential of holographic displays, significant efforts have been devoted over the years to increasing the SBP of SLM technology, including various designs using spatial multiplexing of wavefront modulators arranged in application-specific configurations. While such multiplexed systems offer significantly larger SBPs than single SLMs, utilizing multiple SLMs increases power consumption, memory usage, and computational load, resulting in bulky optical architectures with tedious adjustment and synchronization procedures. Besides spatial multiplexing, many temporal multiplexing methods have been developed to improve the SBP of holographic displays, but they often rely on rotating mirrors or other moving opto-mechanical components, complicating the optical setup. Another method for improving the SBP of holographic displays without spatial and / or temporal multiplexing was proposed by Yu et al. The authors exploited random speckle patterns generated by multiple optical scattering events by introducing complex modulation media (e.g., multiple random diffusers) into the optical signal path and exciting only a few optical modes based on wavefront shaping. See H. Yu, K. Lee, J. Park, Y. Park, "Ultrahigh-definition dynamic 3D holographic display by active control of volume speckle fields", Nature Photon, 11, 186-192 (2017). This method provides a relatively large viewing angle, but the random nature of the diffuser creates background noise and speckle, degrading the resulting image quality. A similar approach has been developed for AR displays by introducing a periodic grating instead of a random diffuser into the optical path between the SLM and the lens acting as the eyepiece.See X. Duan, J. Liu, X. Shi, Z. Zhang, J. Xiao, "Full-color see-through near-eye holographic display with 80°field of view and expanded eye box", Opt.Express, OE. 28, 31316-31329 (2020).

[0007] Recently, advances in machine learning have been extended to provide deep learning-enabled solutions to some of the aforementioned challenges related to holographic displays. Various deep neural network architectures have been used to learn the transformation from a given target image to a corresponding phase-only pattern on an SLM, with the goal of replacing traditional iterative hologram computation algorithms with faster and better alternatives. Deep neural networks have been utilized to parameterize wave propagation models between the SLM modulation pattern and the synthesized image, calibrating the forward model to partially account for physical error sources and aberrations present in the optical setup. Summary of the Invention

[0008]

[0008] In one embodiment, a deep-learning-enabled diffractive super-resolution (SR) image display system is disclosed, based on a jointly trained pair of electronic encoder and all-optical decoder, that projects a super-resolution image at the output while preserving the image field of view (FOV) size, thereby exceeding the SBP limitations imposed by wavefront modulators or SLMs. This diffractive SR display also significantly reduces computational load and data transfer / storage by encoding a high-resolution image (projected / displayed image) into a compact low-resolution representation with fewer pixels per image, where k>1 defines the target SR coefficient during training of the diffractive SR image display system. In this computational image display approach, the main function of the electronic encoder network (i.e., a front-end based on a convolutional neural network (CNN)) is to digitally pre-process a high-resolution image to compute a low-resolution (LR) SLM modulation pattern and encode the LR representation of the input information. The all-optical decoder "backend" of this SR display is implemented by a passive diffractive network jointly trained with an electronic encoder CNN. It processes the input waves generated by the SLM pattern and projects a super-resolution image by decoding the encoded LR representation of the input image. In other words, the all-optical diffractive decoder realizes super-resolution image projection at its output FOV by processing the coherent waves generated by the LR-encoded representation of the input image computed by the jointly trained encoder CNN. This diffractive decoder forms the all-optical backend of the SR image display system, consuming no power other than the illumination light for the low-resolution SLM, and instantly computing the super-resolution image via light propagation within a thin diffractive volume.

[0009] The SR capabilities of this unique diffractive display design have been demonstrated using a lensless image projection system, as shown in Figures 1A, 1B, 6A, and 6B. The diffractive SR display can achieve an SR factor of approximately 4, or an approximately 16-fold increase in SBP, using a five-layer diffractive decoder network. The success of this diffractive SR display framework was experimentally demonstrated based on a 3D-fabricated diffractive decoder operating in the THz portion of the spectrum. This diffractive SR image display system can be scaled to operate in any portion of the electromagnetic spectrum, including visible wavelengths, and can be used for SBP-enhanced image display solutions, forming a component of next-generation 3D display technologies, including head-mounted AR / VR devices.

[0010]

[0010] In one embodiment, a system or apparatus for displaying or projecting high-resolution images comprises at least one electronic encoder network including a trained deep neural network configured to receive one or more high-resolution images and generate a low-resolution modulation pattern or image representing the one or more high-resolution images using one of a display, projector, screen, spatial light modulator (SLM), or wavefront modulator; and an all-optical decoder network having one or more optically transmissive and / or reflective substrate layers arranged in an optical path, each of the optically transmissive and / or reflective substrate layers having a plurality of physical features formed on or within the one or more optically transmissive and / or reflective substrate layers and having different transmission and / or reflection properties as a function of local coordinate across each substrate layer, wherein the one or more optically transmissive and / or reflective substrate layers and the plurality of physical features receive light arising from the low-resolution modulation pattern or image representing the one or more high-resolution images and optically generate a corresponding high-resolution image projection at an output field of view.

[0011]

[0011] In another embodiment, a device for decoding high-resolution images from low-resolution modulation patterns or images representing one or more high-resolution images comprises an all-optical decoder network including one or more optically transmissive and / or reflective substrate layers positioned in an optical path, each of the optically transmissive and / or reflective substrate layers having a plurality of physical features formed on or within the one or more optically transmissive and / or reflective substrate layers and having different transmission and / or reflection properties as a function of local coordinate across each substrate layer, wherein the one or more optically transmissive and / or reflective substrate layers and a plurality of physical features receive the low-resolution modulation patterns or images representing the one or more high-resolution images and optically generate a corresponding high-resolution image projection at an output field of view.

[0012] In some embodiments, the all-optical decoder network is incorporated into a wearable device, goggles, or glasses. Thus, the electronic encoder network front end of the system may be separated from the all-optical decoder network portion or back end of the system or device. For example, a pattern or image is created using at least one electronic encoder network. A separate all-optical decoder network is then used to reconstruct a high-resolution image that was encoded using the at least one electronic encoder network.

[0013]

[0013] In another embodiment, a method for projecting high-resolution images onto a field of view provides a system or device comprising at least one electronic encoder network including a trained deep neural network configured to receive one or more high-resolution images and generate a low-resolution modulation pattern or image representing the one or more high-resolution images using one or more of a display, projector, screen, spatial light modulator (SLM), or wavefront modulator, and an all-optical decoder network including one or more optically transmissive and / or reflective substrate layers arranged in an optical path, each of the optically transmissive and / or reflective substrate layers having a plurality of physical features formed on or within the one or more optically transmissive and / or reflective substrate layers and having different transmission and / or reflection properties as a function of local coordinate across each substrate layer, the one or more optically transmissive and / or reflective substrate layers and the plurality of physical features receiving light arising from the low-resolution modulation pattern or image representing the one or more high-resolution images and optically generating a corresponding high-resolution image projection at an output field of view. The method includes inputting one or more high-resolution images into an electronic encoder network to generate a low-resolution modulation pattern or image representing the one or more high-resolution images, and optically generating a corresponding high-resolution image projection at an output field of view.

[0014]

[0014] In another embodiment, a method for communicating information to one or more humans includes the steps of transmitting a low-resolution modulation pattern or image representing one or more high-resolution images containing information using one or more of a display, projector, screen, spatial light modulator (SLM), or wavefront modulator, and all-optically decoding the low-resolution modulation pattern or image with one or more optically transmissive and / or reflective substrate layers positioned in an optical path, each of the optically transmissive and / or reflective substrate layers having a plurality of physical features formed on or within the one or more optically transmissive and / or reflective substrate layers and having different transmission and / or reflection properties as a function of local coordinate across each substrate layer, and the one or more optically transmissive and / or reflective substrate layers and the plurality of physical features receive light arising from the low-resolution modulation pattern or image representing the one or more high-resolution images and generate a corresponding high-resolution image projection containing the information at an output field of view.

[0015] In another embodiment, a communication system for transmitting messages or signals in space includes at least one electronic encoder network and an all-optical decoder network. The at least one electronic encoder network includes a trained deep neural network configured to receive a message or signal and generate a phase-encoded and / or amplitude-encoded optical representation of the message or signal transmitted along an optical path. The all-optical decoder network is disposed in the optical path with the encoder network and includes one or more optically transmissive and / or reflective substrate layers at least partially obscured and / or blocked by an opaque obscurant and / or diffusive medium, each of the optically transmissive and / or reflective substrate layers including a plurality of physical features formed on or within the one or more optically transmissive and / or reflective substrate layers and having different transmission and / or reflection characteristics as a function of local coordinate across each substrate layer, the one or more optically transmissive and / or reflective substrate layers and the plurality of physical features receiving secondary light waves scattered by the opaque obscurant and / or diffusive medium and optically generating the message or signal in an output field of view.

[0016]

[0016] In another embodiment, a device for decoding an encoded optical message or signal comprises an all-optical decoder network positioned in the optical path of the encoded optical message or signal and including one or more optically transmissive and / or reflective substrate layers at least partially shielded and / or blocked by an opaque obstructing and / or diffusive medium, each of the optically transmissive and / or reflective substrate layers including a plurality of physical features formed on or within the one or more optically transmissive and / or reflective substrate layers and having different transmission and / or reflection properties as a function of local coordinate across each substrate layer, the one or more optically transmissive and / or reflective substrate layers and a plurality of physical features receiving secondary light waves scattered by the opaque obstructing and / or diffusive medium and optically generating the message or signal in an output field of view.

[0017] In another embodiment, a method for transmitting a message or signal through space in the presence of an opaque obstruction and / or a diffusive medium is disclosed. The method includes providing a system including at least one electronic encoder network and an all-optical decoder network. The at least one electronic encoder network includes a trained deep neural network configured to receive a message or signal and generate a phase-encoded and / or amplitude-encoded optical representation of the message or signal transmitted along an optical path. The all-optical decoder network includes one or more optically transmissive and / or reflective substrate layers disposed in the optical path, each of the optically transmissive and / or reflective substrate layers including a plurality of physical features formed on or within the one or more optically transmissive and / or reflective substrate layers and having different transmission and / or reflection characteristics as a function of local coordinate across each substrate layer, the one or more optically transmissive and / or reflective substrate layers and the plurality of physical features receiving secondary light waves scattered by the opaque obstruction and / or diffusive medium and optically generating the message or signal in an output field of view. One or more messages or signals are input to an electronic encoder network, which generates a phase-encoded and / or amplitude-encoded optical representation of the messages or signals, and optically generates the messages or signals at an output field of view. [Brief explanation of the drawings]

[0018] [Figure 1]FIG. 1A is a schematic diagram of a system for displaying or projecting high-resolution images. The system can display high-resolution / super-resolution images using a front-end digital encoder and a back-end all-optical diffractive decoder. FIG. 1A shows an all-optical decoder network integrated into a wearable device, such as a wearable headset for virtual reality or augmented reality applications. FIG. 1B is a schematic diagram of how the apparatus of FIG. 1A can be used to project high-resolution / super-resolution images onto a surface, such as a mammalian eye. [Figure 2] 2 illustrates a single substrate layer of an all-optical decoder network. The substrate layer may be made of an optically transparent material (for transmission mode) or an optically reflective material (for reflection mode). In some embodiments, the substrate layer is formed as a substrate or plate with surface features formed throughout the substrate layer. The surface features form a patterned surface (e.g., an array) with different values of transmission (or reflection) properties as a function of the abscissa across each substrate layer. These surface features function as artificial "neurons" that connect to other "neurons" in other substrate layers of the optical neural network via optical diffraction (or reflection) and change the phase and / or amplitude of the light waves. [Figure 3] 3 shows a schematic cross-sectional view of a single substrate layer of an all-optical decoder network according to one embodiment. In this embodiment, the surface topography is created by adjusting the thickness of the substrate layers that form the all-optical decoder network. These different thicknesses may define peaks and valleys in the substrate layers that act as artificial "neurons." [Figure 4] Figure 4 shows a schematic cross-sectional view of a single substrate layer of an all-optical decoder network according to another embodiment. In this embodiment, the material composition or material properties of a single substrate layer are varied at different lateral positions across the substrate layer to form different surface topographies. This can be achieved by doping the substrate layer with dopants or by incorporating other optical materials into the substrate layer. Metamaterials or plasmonic structures may also be incorporated into the substrate layer. [Figure 5]FIG. 5 shows a schematic cross-sectional view of a single substrate layer of an all-optical decoder network according to another embodiment. In this embodiment, the substrate layer can be reconfigured by, for example, changing the optical properties of various artificial neurons through the application of stimuli (such as current or electric fields). One example is a spatial light modulator (SLM) whose optical properties can be changed. In this embodiment, the neuron structure is not fixed but can be dynamically changed or adjusted as needed. This embodiment can provide a learning or tunable optical neural network that can be changed on the fly (e.g., over time) to, for example, improve performance, correct aberrations, or change to another task. [Figure 6] Figures 6A-6B are schematic diagrams illustrating a super-resolution (SR) image display system composed of an all-electronic encoder and an all-optical decoder network. Figure 6A illustrates the components of a super-resolution image display system composed of an all-electronic encoder network and an all-optical decoder network containing five diffractive modulation layers. The all-electronic encoder network is used to create low-resolution representations of the input image, which are then super-resolved using a diffractive optical decoder to achieve the desired SR coefficient (k>1). Figure 6B shows the optical layout of the five-layer diffractive decoder network, with d1 = 2.667λ, d2 = 66.667λ, and d3 = 80λ. [Figure 7] Figure 7 shows the image projection results of a diffractive SR display using a phase-only SLM. The top row shows the image projection results of an SR display using five diffractive layers (L=5). The middle row shows the image projection results of an SR display using three diffractive layers (L=3). The bottom row shows the image projection results of an SR display using one diffractive layer (L=1). For comparison, a lower-resolution version of the same image using the same number of pixels as the corresponding wavefront modulator is shown on the right side of the figure. [Figure 8]Figures 8A-8B quantify the image projection performance of the diffractive SR display as a function of k and L. The test image dataset contains 6,000 images, each containing multiple EMNIST handwritten characters. Figure 8A shows the average PSNR values for phase-only (left) and complex-valued encoding (right). Figure 8B shows the average SSIM values for phase-only (left) and complex-valued encoding (right). [Figure 9] Figure 9 shows the image resolution analysis of a diffractive SR display using a phase-only SLM. The projection of vertical and horizontal line pairs with a linewidth of 2.132λ is shown for different SR coefficients (k = 4, 6, 8). A diffractive all-optical decoder network with different numbers of diffractive layers (L = 1, 3, 5) projects a super-resolution image at the output. For comparison, a low-resolution (LR) version of the same object using the same number of pixels and the corresponding wavefront modulator is shown on the right side of Figure 9. The diffractive SR system was trained using handwritten characters; the training dataset did not include any resolution test targets or line pairs. [Figure 10] Figures 10A-10D show the experimental setup for a three-layer diffractive SR decoder. The three-layer diffractive SR decoder is vaccinated for lateral (Δxy = ~0.334λ) and axial (Δz = ~0.533λ) misalignments and trained for an SR coefficient k = 3. An all-electronic encoder creates a phase-only LR representation of the projected image. Figure 10A shows the phase profile of the trained diffractive decoder layer used in the experiments. Figure 10B shows the optical layout of the three-layer diffractive SR decoder. Figure 10C shows a photograph of the 3D-printed diffractive SR decoder network. Figure 10D shows a schematic diagram of the experimental setup using continuous-wave THz illumination. [Figure 11] Figure 11 shows experimental results for an L=3 layer diffractive SR image display system. An all-electronic encoder is used to obtain a phase-only representation of the encoded object. An all-optical diffractive decoder projects the super-resolution image. For comparison, a lower-resolution version of the same image using the same number of pixels as the corresponding wavefront modulator is shown at the bottom of Figure 11. [Figure 12]Figures 12A-12C show the experimental setup for a one-layer diffractive SR decoder. The one-layer diffractive SR decoder is vaccinated against lateral (Δxy = ~0.334λ) and axial (Δz = ~0.533λ) misalignments and trained with an SR coefficient k = 3. An all-electronic encoder creates a phase-only LR representation of the projected image. Figure 12A shows the phase profile of the trained diffractive decoder layer used in the experiments. Figure 12B shows the optical layout of the one-layer diffractive SR decoder. Figure 12C shows a photograph of the 3D-printed diffractive SR decoder network. [Figure 13] Figure 13 shows experimental results for an L=1 layer diffractive SR image display system. An all-electronic encoder is used to obtain a phase-only representation of the encoded object. An all-optical diffractive decoder projects the super-resolution image. For comparison, a lower-resolution version of the same image using the same number of pixels as the corresponding wavefront modulator is shown at the bottom of Figure 13. [Figure 14] Figure 14 shows the quantization analysis of the phase-only wavefront modulation of the composite image. Image projection results for an SR display with five diffractive layers (L = 5) are shown at different phase quantization levels (16, 8, 6, 4, and 2 bits). The encoding-decoding framework is trained on 16-bit phase quantization of the SLM pattern and blind-tested on lower quantization levels. [Figure 15] Figure 15 shows the image projection results of a diffractive SR display using complex-valued image encoding. The top row shows the image projection results of an SR display using five diffractive layers (L=5). The middle row shows the image projection results of an SR display using three diffractive layers (L=3). The bottom row shows the image projection results of an SR display using one diffractive layer (L=1). For comparison, a lower-resolution version of the same image is shown on the right side of Figure 15. [Figure 16] Figure 16 shows the image projection results of a diffractive SR display using an amplitude-only SLM. This is the image projection result of an SR display with five diffractive layers (L=5). For comparison, a lower-resolution version of the same image using the same number of pixels as the corresponding wavefront modulator is shown on the right side of Figure 16. [Figure 17]Figure 17 shows the image resolution analysis of a diffractive SR display using complex-valued image encoding. The projection of vertical and horizontal line pairs with a line width of 2.132λ was tested with different SR coefficients (k = 4, 6, 8). A diffractive all-optical decoder network with different numbers of diffractive layers (L = 1, 3, 5) projects the super-resolved image onto the output FOV. For comparison, a low-resolution version of the same image is shown on the right side of Figure 15. The diffractive SR system was trained using handwritten characters, and the training dataset did not include any resolution test targets or line pairs. [Figure 18] Figure 18 shows the image generation for the EMNIST display dataset. Different numbers of EMNIST handwritten characters were randomly selected and augmented with a set of predefined operations, including scaling (K~U(0.84,1)), rotation (θ~U(-5,°5°)), and translation (Dx, Dy~U(-1.06λ,1.06λ)). These randomly selected augmented handwritten characters were placed at randomly selected locations in a 3x3 grid for each image in the EMNIST display dataset. [Figure 19] Figure 19 shows the phase profile of the trained diffractive decoder layers, using a phase-only SLM at the input of each decoder. The size of each diffractive layer is 106.66λ × 106.66λ, and the size of the diffractive neurons is 0.533λ × 0.533λ. [Figure 20]Figure 20A shows a schematic diagram of an optical communication framework around a fully opaque obstruction using electronic encoding and diffractive all-optical decoding. The electronic neural network encoder and all-optical diffractive decoder are jointly trained to communicate around the opaque obstruction. For a message / object to transmit, the electronic encoder outputs an encoded 2D phase pattern, which is transmitted to a plane wave at the transmitter aperture. After being obstructed and scattered by the fully opaque obstruction, the phase-encoded wave proceeds to the receiver, where the diffractive decoder processes the encoded information all-optically to reproduce the message on the output FOV. Figure 20B shows the architecture used for the convolutional neural network (CNN) electronic encoder network. Figure 20C shows a visualization of various processes, such as the obstruction of the transmitted phase-encoded wave by a wo-wide obstruction and the subsequent all-optical decoding performed by the diffractive decoder. The diffractive decoder contains L surfaces (S1, ..., S1) with phase-only diffractive properties. Figure 20C shows L=3 as an example. Figure 20D shows a comparison of the encoding-decoding scheme (diffractive decoder output) to conventional lens-based imaging (lens image). [Figure 21] Figure 21 shows the generalization of the trained encoder-decoder pair to previously unseen handwritten digit objects. For comparison, the performance of the trained encoder-decoder pair with various numbers of decoder layers (L) for various values of occlusion width wo is shown. [Figure 22] Figures 22A and 22B quantify the performance of encoder-decoder pairs with different numbers of decoder layers (L) trained to increase the occlusion width (wo) in terms of PSNR (Figure 22A) and SSIM (Figure 22B) between the diffraction decoder output and the ground truth message. PSNR and SSIM values are calculated by averaging over 10,000 MNIST test images. wt denotes the transmitter aperture width. [Figure 23]Figure 23 is similar to Figure 21, except that these results reflect external generalization for object types different from those used during training. [Figure 24] Figure 24 shows the output resolution of the diffraction decoder corresponding to the L=1, L=3, and L=5 designs trained with different occluder widths (wo). For the object, the vertical / horizontal spacing between the inner edges of the dots is 2.12λ for the top test pattern and 4.24λ for the bottom test pattern. The diffraction decoder output is accompanied by cross sections along the vertical / horizontal lines. [Figure 25] Figure 25A shows the effect of phase bit depth and diffraction layer features of the encoded object on the performance of the trained encoder-decoder pair. Qualitative performance of designs trained assuming a particular phase quantization bit depth bq,tr is reported as a function of the bit depth bq,te used during testing (b). Figure 25B shows PSNR and SSIM values plotted as a function of bq,te for different bq,tr. PSNR and SSIM values are evaluated by averaging the results for 10,000 test images from the MNIST dataset. [Figure 26] Figures 26A-26C show the output efficiency of an electronic encoding and diffractive decoding scheme for optical communication around a fully opaque occlusion. Figure 26A is a graph of the diffraction efficiency (DE) of the same design as Figures 22A and 22B. Figure 26B shows the tradeoff between DE and SSIM achieved by varying the training hyperparameter η, i.e., the weight of the additional loss term used to penalize less efficient designs. For these designs, wo = 32λ and L = 3 were used. DE and SSIM values are calculated by averaging over 10,000 MNIST test images. Figure 26C shows the performance of a portion of the design shown in Figure 26B trained with different η values. [Figure 27]Figures 27A-27E show the performance of encoder-decoder pairs trained on different opaque occluder shapes. We show the performance of four designs trained on various occluder shapes (square, circle, rectangle, and arbitrary shape). These fully opaque occluders have approximately equal areas. [Figure 28] Figure 28A shows the terahertz setup consisting of the source and detector, along with the 3D-printed components used as the encoded phase object, obscurant, and diffractive layer. Experimental results for an L = 1 design for an obscurant width wo = 32λ, operating at a wavelength λ = 0.75 mm. Figure 28B shows the assembly of the encoded phase object, obscurant, diffractive layer, and output aperture using a 3D-printed holder. Figure 28C shows the encoded phase object (example), obscurant, and diffractive layer separately, housed within a support frame. Figure 28D shows the experimental diffractive decoder output (bottom row) for 10 handwritten digit objects (top row) and the corresponding simulated lens image (second row) and diffractive decoder output (third row). [Figure 29] FIG. 29 shows an example of a custom-created training image. [Figure 30] FIG. 30 shows histograms of the average SSIM values of the diffraction decoder outputs of the four designs in FIGS. 27A-27E, computed for 10,000 test images from the MNIST dataset (internal generalization) and the Fashion-MNIST dataset (external generalization). [Figure 31] FIG. 31 illustrates the transmission training of the CNN encoder at the transmitter, without modifying the diffraction decoder at the receiver, for successful communication when the size of the opaque occluder blocking the transmitter's field of view increases / changes. [Figure 32] Figure 32 shows the importance of diffractive decoding for optical communication around a fully opaque obscurant. A design without a diffractive decoder and a design with an L=1 layer diffractive decoder are compared for two different obscurant width (wo) sizes. DETAILED DESCRIPTION OF THE INVENTION

[0019]

[0058] FIG. 1A illustrates one embodiment of a system 10 for displaying or projecting a high-resolution image 100. System 10, in some embodiments, includes aspects that can be incorporated into a portable or wearable device 11. For example, FIG. 1A illustrates a portion of system 10 embodied in a headset (or glasses) as portable or wearable device 11, which can be used, for example, for virtual reality or augmented reality applications. Of course, system 10 is not limited thereto. Other applications of system 10 include displays used in transportation and delivery (e.g., heads-up displays, console displays, etc.). System 10 can also be used for advertising (digital billboards, digital signage, security settings, surgical procedures, etc.).

[0020]

[0059] The system 10 uses a digital version or model of a jointly trained pair of electronic encoder networks 12 and all-optical decoder networks 14. In one aspect, the electronic encoder network 12 includes a trained deep neural network, which in a preferred embodiment is a trained convolutional neural network (CNN). The trained electronic encoder network 12 receives one or more high-resolution images 100 and generates, via an associated image generator 16, corresponding low-resolution modulation patterns or images 104 that represent the one or more high-resolution images 100. The low-resolution modulation patterns or images 104 are generated by the image generator 16. Examples of image generators 16 include, by way of example and not limitation, a display, a projector, a screen, a spatial light modulator (SLM), or a wavefront modulator. The low-resolution modulation patterns or images 104 may include phase-only modulation, amplitude-only modulation, or a complex modulation. The low-resolution modulation pattern or image 104 is then input to a physical all-optical decoder network 14 that includes one or more optically transmissive and / or reflective substrate layers 18 (also referred to herein as diffractive layers) disposed in the optical path. The optical path may be straight or curved. Each of the optically transmissive and / or reflective substrate layers 18 includes a plurality of physical features 20 (e.g., FIGS. 2-5 ) formed on or within the one or more optically transmissive and / or reflective substrate layers 18 and having different transmission and / or reflection properties as a function of local coordinates (e.g., length and width) across each substrate layer 18. In the experimental system 10 described herein, the all-optical decoder network 14 operates in a transmission mode, where light is transmitted / diffracted through the substrate layer 18. In other embodiments, the all-optical decoder network 14 operates in a reflection mode, where light is reflected / diffracted by the substrate layer 18. Furthermore, in some embodiments, the system 10 may include a substrate layer 18 that operates in both a transmission mode and a reflection mode.

[0021]

[0060] 2-5, physical features 20 on or within substrate layer 18 form neurons of all-optical decoder network 14. In some embodiments, each individual physical feature 20 may define a distinct physical location on substrate layer 18, while in other embodiments, multiple physical features 20 may jointly or collectively define a physical region having specific transmission (or reflection) properties. One or more substrate layers 18 arranged along an optical path collectively generate a reconstructed high-resolution / super-resolution image 106. During system operation, one or more optically transmissive and / or reflective substrate layers 18 with multiple physical features 20 receive light resulting from a low-resolution modulation pattern or image 104 representing one or more high-resolution images 100 and optically generate a corresponding high-resolution image reconstruction or projection 106 at an output field of view. All-optical decoder network 14 projects the high-resolution / super-resolution reconstruction or projection 106 to the output while maintaining the size of the image field of view (FOV), thereby overcoming the SBP limitations imposed by the wavefront modulator or SLM. System 10 may operate at any number of wavelengths within the electromagnetic spectrum, including, for example, ultraviolet wavelengths, visible wavelengths, infrared wavelengths, THz wavelengths (used in the experiments described herein), millimeter wavelengths, etc.

[0022]

[0061] FIG. 3 illustrates one embodiment of a method for forming various physical features 20 in the substrate layer 18. In this embodiment, the substrate layer 18 has different thicknesses (t) of material at different lateral locations along the substrate layer 18. In one embodiment, the different thicknesses (t) modulate the phase of light passing through the substrate layer 18. The different thicknesses of material in the substrate layer 18 form multiple discrete "peaks" and "valleys" that control the transmission characteristics of neurons formed in the substrate layer 18. The different thicknesses of the substrate layer 18 can be formed using additive manufacturing techniques (e.g., 3D printing) or lithography methods used in semiconductor processing. For example, the design of the substrate layer 18 is saved in a stereolithography file format (e.g., stl file format) and used to 3D print the substrate layer 18 that forms the all-optical decoder network 14. Other manufacturing techniques include well-known wet and dry etching processes that can form very small lithographic features on the substrate layer 18. Lithography methods can be used to form very small, high-density physical features 20 on the substrate layer 18 that are usable with short wavelengths of light. As shown in FIG. 3, in this embodiment, the physical features 20 are permanently fixed (ie, the surface shape is established and remains the same once completed).

[0023]

[0062] FIG. 4 illustrates another embodiment in which physical features 20 are created or formed within the substrate layer 18. In this embodiment, the substrate layer 18 has a substantially uniform thickness, but different regions of the substrate layer 18 may have different optical properties. For example, the refractive index (or reflectivity) of the substrate layer 18 can be varied by doping the substrate layer 18 with dopants (e.g., ions, etc.) to form neuron regions in the substrate layer 18 with controlled transmission properties (and / or absorption and / or spectral features). In yet other embodiments, optical nonlinearities can be incorporated into deep optical network designs using various optical nonlinear materials (e.g., crystals, polymers, semiconductor materials, doped glasses, polymers, organic materials, semiconductors, graphene, quantum dots, carbon nanotubes, etc.) incorporated into the substrate layer 18. Neurons may also be formed on the substrate layer 18 using masking layers or coatings that partially transmit or partially block light at different lateral locations on the substrate layer 18.

[0024]

[0063] Alternatively, the physical features 20 or transfer functions of neurons can be engineered using metamaterials, metasurfaces (e.g., surfaces with subwavelength, nanoscale structures that result in special optical properties), and / or plasmonic structures. These techniques may also be used in combination. In other embodiments, non-passive components such as spatial light modulators (SLMs) can be incorporated into the substrate layer 18. SLMs are devices that provide modulation that spatially varies the phase, amplitude, or polarization of light. SLMs can be optically addressed or electrically addressed. Electrical SLMs include liquid crystal-based technologies that use thin-film transistors (transmission applications) or silicon backplanes (reflection applications) for switching. Another example of an electrical SLM is a magneto-optical device that uses pixelated aluminum garnet crystals switched by an array of magnetic coils utilizing the magneto-optic effect. Other electronic SLMs use nano-fabricated deformable or movable mirrors that are electrostatically controlled to selectively deflect light.

[0025]

[0064] FIG. 5 shows a schematic cross-sectional view of a single substrate layer 18 of an all-optical decoder network 14 according to another embodiment. In this embodiment, the substrate layer 18 is reconfigurable as a function of time, in that the optical properties of the various physical features 20 that form the artificial neurons can be altered, for example, by the application of a stimulus (such as an electric current or an electric field). One example is the spatial light modulator (SLM) described above, whose optical properties can be changed. The substrate layer 18 can incorporate at least one nonlinear optical material. In another embodiment, the layer uses the DC electro-optic effect to introduce optical nonlinearity into the substrate layers 18 of the all-optical decoder network 14, requiring a DC electric field across each substrate layer 18. This electric field (or current) can be applied externally to each substrate layer 18. Alternatively, polarized materials (such as polarized crystals or polarized glass) can be used, which have very strong electric fields built into them. In this embodiment, the neuron structure is not fixed but can be dynamically altered or adjusted (i.e., changed on demand) as needed. This embodiment can provide a learning or configurable all-optical decoder network 14 that can be modified on the fly, for example, to improve performance, correct aberrations, or modify for another task.

[0026]

[0065] In some embodiments, the high-resolution reconstruction or projection image 106 can be projected onto a viewing surface or surface, including, for example, the surface of a mammalian eye. For example, the all-optical decoder network 14 of system 10 can be integrated into a headset, goggles, eyeglasses, or other portable electronic device 11 (FIG. 1A) and projected onto a user's eye 108, as shown in FIG. 1B. FIG. 1B illustrates system 10 used to display directional guidance to a user. In this embodiment, a high-resolution image 100 is encoded into a low-resolution modulation pattern 104 by electronic encoder network 12 and then decoded by all-optical decoder network 14 to generate a high-resolution image reconstruction or projection 106 for display to the user. For example, system 10 can be integrated into head-mounted AR / VR devices of next-generation display technologies. In some embodiments, the high-resolution reconstruction or projection image 106 can be projected onto a field of view (FOV) captured by one or more optical detectors.

[0027]

[0066] Exemplary materials that may be used for the substrate layer 18 include polymers and plastics (e.g., those used in additive manufacturing techniques such as 3D printing), as well as semiconductor-based materials (e.g., silicon and its oxides, gallium arsenide and its oxides), crystalline materials, or amorphous materials such as glass, and combinations thereof. Metallized materials may also be used for the reflective substrate layer 18.

[0028]

[0067] The pattern of physical locations formed by the physical features 20 may, in some embodiments, define an array located across the surface of the substrate layer 18. In one embodiment, the substrate layer 18 is a two-dimensional, generally planar substrate having a length (L), a width (W), and a thickness (t), all of which may vary depending on the particular application. In other embodiments, the substrate layer 18 may be non-planar, e.g., curved. Additionally, while a rectangular or square substrate layer is depicted in FIG. 2, it should be understood that different shapes are contemplated. The physical features 20 and the physical regions formed thereby function as artificial “neurons” that connect to other “neurons” in other substrate layers 18 of the all-optical decoder network 14 via optical diffraction (or reflection) and alter the phase and / or amplitude of light waves. The specific number and density of physical features 20 or artificial neurons formed on each substrate layer 18 may vary depending on the type of application. In some embodiments, a total number of hundreds or thousands of artificial neurons may be sufficient, while other embodiments may use hundreds of thousands or millions of neurons or more. Similarly, the number of substrate layers 18 used in a particular all-optical decoder network 14 will typically range from at least one substrate layer 18 to fewer than ten substrate layers 18, but may vary.

[0029]

[0068] The system 10 may be used to transmit information, messages, or data to individuals. For example, an image generator 16, such as a display, projector, screen, spatial light modulator (SLM), or wavefront modulator, may generate a low-resolution modulation pattern or image 104 (or multiple patterns or images 104) from a high-resolution image 100. Anyone can view the low-resolution modulation pattern or image 104, but no useful information can be discerned from it. However, anyone with access to the all-optical decoder network 14 can reconstruct a high-resolution image 106 encoded in the low-resolution modulation pattern or image 104. This may be an image of a scene, a text message, an advertisement, a guide, or the like. It may also be a series of images forming a video or image clip. In some embodiments, groups of people or individuals may have their own all-optical decoder network 14 so that secure communications can be tailored to specific groups or individuals. Additionally, in some embodiments, the low-resolution modulation pattern or image 104 may be generated as a watermark or an overlapping image on another image.

[0030]

[0069] experiment

[0070] result

[0071] The operating principles and components of the diffractive SR imaging system 10 are illustrated in Figures 1 and 6A-6B. According to the forward model described in Figures 6A-6B, an electronic encoder network 12 (e.g., a CNN) is trained to extract spatial features of a (projected) high-resolution image 100 and encode this spatial information into a reduced-size, low-dimensional representation 104 equal to the number of pixels physically available on the wavefront modulator. An input beam, assumed to be a uniform plane wave (see Figure 6A), is modulated by the output pattern of the encoder network 12 on the SLM. The resulting wave is then all-optically processed by an all-optical decoder network 14, aiming to recover a high-resolution reconstruction or projection 106 of the original image at its output FOV, effectively creating a high-resolution display through all-optical super-resolution.

[0031]

[0072] Figure 7 shows the super-resolution image projection performance (blind test results) of diffractive SR display system designs trained with SR coefficients k = 4, k = 6, and k = 8 in both the x and y directions. Details of training the diffractive SR displays with different configurations are described in the "Methods" section. In each case (k = 4, 6, 8), the input and output fields of view, i.e., the size of the wavefront modulator and output image, are kept identical. Therefore, the pixel size of the wavefront modulator at each SR coefficient is given as k × 0.533λ, which corresponds to 2.132λ, 3.198λ, and 4.264λ for k = 4, 6, and 8, respectively. Another important design parameter besides the SR coefficient (k) is the number L of substrate layers 18 used in the design of the all-optical decoder network 14. Figure 7 also shows a comparison between various decoders using diffractive layers 18 with L = 1, 3, and 5 trained with SR coefficients k = 4, 6, and 8. In the results shown in Figure 7, the wavefront modulator 16 is assumed to provide only the phase modulation of the incident field. The results of a similar analysis using a complex-valued SLM at the input of each all-optical decoder network 14 are also shown in Figure 15. Additionally, Figure 16 shows the results for an amplitude-only wavefront modulator 16 used in the electronic encoder network 12.

[0032]

[0073] Figures 7, 15, and 16 show that when k ≥ 4, the SLM 16 exhibits a very low resolution with a large pixel size and a small pixel count. The original resolution is insufficient to directly represent most of the fine details of the test object (EMNIST handwritten characters) within the FOV. On the other hand, these spatial features can be all-optically recovered through the all-optical decoder network 14, which projects the SR image 106 into its output FOV, as shown in Figures 7 and 15-16. We also observed that for a fixed SR coefficient k, the mismatch between the desired high-resolution image and the optically synthesized intensity distribution in the output FOV of the all-optical decoder network 14 becomes smaller as the number L of diffractive layers 18 increases, demonstrating the advantage of a deeper all-optical decoder network 14 providing better image projection.

[0033]

[0074] Beyond the visual inspection and comparison provided in Figure 7 and Figures 15 and 16, the effectiveness of the diffractive SR display framework is also confirmed by quantifying image quality using structural similarity index measure (SSIM) and peak signal-to-noise ratio (PSNR) metrics. As part of this quantitative analysis, Figures 8A and 8B compare the overall image synthesis performance of phase-only and complex-valued wavefront modulation at the input plane of the all-optical decoder network 14. On average, complex-valued wavefront modulation provides slightly better PSNR and SSIM values at the output of the diffractive decoder compared to phase-only modulation / encoding due to its additional degrees of freedom. Figures 8A and 8B also support the conclusion in Figure 7 that the deeper all-optical decoder network 14, with its greater number of diffractive layers 18, produces a higher-fidelity output image projection 106.

[0034]

[0075] To gain a deeper understanding of the success of the all-optical decoder network 14 in synthesizing super-resolution images 106, we conducted additional blind tests using images of various lines with subpixel linewidths compared to the original phase-only SLM resolution, as shown in Figure 9. It is important to emphasize that the training of the diffractive SR system 10 relied entirely on the EMNIST handwritten character dataset. Therefore, these new images 100 of resolution test lines represent a blind test dataset that is statistically different from the training data. The phase-encoding-only resolution test results summarized in Figure 9 reveal that even with very small pixel linewidths, individual lines of both horizontal and vertical structures can be resolved at the output of the five-layer all-optical decoder network 14. On the other hand, an all-optical decoder network 14 with a single diffractive layer 18 (L = 1) is unable to resolve individual lines with a linewidth of 2.132λ for k = 8 (Figure 9) due to the limited generalization capability offered by the one-layer diffractive decoder architecture. Figure 17 shows the same resolution test analysis, except for the diffractive SR display system 10, which uses complex-valued encoding in the SLM, and reaches similar conclusions. It should be understood that in other embodiments, the low resolution modulation pattern or image 104 may be encoded in amplitude only.

[0035]

[0076] These results, summarized in Figure 9, show that a test image 100 with a linewidth of 2.132λ, consisting of vertical and horizontal line pairs, can be resolved through an L=5 diffractive decoder trained with an SR coefficient of k=8 using a phase-only wavefront modulator 16 with an original pixel size of 4.264λ, i.e., k × 0.533λ. This indicates that the effective pixel size at the output face of this all-optical decoder network 14 is ~1.066λ (half the minimum resolvable linewidth), which corresponds to a pixel super-resolution coefficient of ~4x and an SBP increase of ~16x. For comparison, as shown in the right column of Figure 9, a test target image 100 with the same resolution but a linewidth of 2.132λ cannot be resolved, as expected, on a lower-resolution display with a pixel size of 2.132λ or larger. However, a diffractive SR display system 10 with L=5 and k=8 successfully resolved these lines using a pixel size of 4.264λ with a phase-only wavefront encoder. This corresponds to a ∼16-fold increase in SBP for the image display system 10. This SBP increase is k 2 It is noteworthy that the sigma is smaller than . This indicates that the training image set (handwritten EMINST characters) did not have a high enough resolution feature representation to guide the joint training of the encoder-decoder pair to achieve a higher resolution image representation. Furthermore, such resolution test targets consisting of lines and grids were not included in the training data.

[0036]

[0077] Next, to experimentally demonstrate the success of the presented SR image display system 10, we designed two different all-optical diffractive decoder networks 14 to operate in the THz portion of the spectrum (see the Methods section for details). To achieve image SR, the first all-optical diffractive decoder network 14 uses a three-layer diffractive decoder design (Figures 10A-10D and 11), while the second all-optical diffractive decoder network 14 (Figures 12A-12C) relies on only a single diffractive plane L = 1. These all-optical diffractive decoder networks 14 were 3D printed and physically assembled / tuned to operate under continuous-wave THz illumination with λ = ~0.75 mm (see the Methods section). The experimental setup, the 3D-printed substrate layer 18 of the all-optical diffractive decoder network 14, and the phase profile of the fabricated optimized substrate layer 18 are shown in Figures 10A-10D and Figures 12A-12C for the three-layer and one-layer all-optical diffractive decoder networks 14, respectively. As detailed in the Methods section, the training loss function of these fabricated all-optical diffraction decoder networks14 includes an additional penalty term to regularize the output diffraction efficiency, resulting in an average output diffraction efficiency of 2.39% and 3.29% for the 3-layer and 1-layer all-optical diffraction decoder networks14, respectively, for blind test images. Furthermore, these all-optical diffraction decoder networks14 were trained to be robust to misalignments between layers in the x, y, and z directions using a vaccine strategy (outlined in the Methods section) that randomly introduces 3D misalignments during the training process, resulting in the creation of misalignment-robust diffractive designs.

[0037]

[0078] Experimental results for a diffractive SR image display system 10 with L = 3 layers are shown in Figure 11, clearly demonstrating the super-resolution capability at the output FOV of the all-optical diffractive decoder network 14 and providing very good agreement between the numerical forward model results and experimental measurements. Similarly, Figure 13 reports successful experimental results obtained using an SR image display system 10 with a single substrate layer 28 (L = 1), also achieving super-resolution at the output of the all-optical diffractive decoder network 14. Despite using a single diffractive / substrate layer 18, the all-optical diffractive decoder network 14's jointly trained encoding and decoding framework optically synthesized target test characters at the output FOV. In these experiments, the diffractive decoder achieved average PSNR values of 13.134 ± 1.368 dB at L = 3 and 12.151 ± 2.138 dB at L = 1. These results are consistent with the previous analysis reported in Figures 7–9 and confirm the benefits of a deeper all-optical diffractive decoder network 14 for improved image synthesis at the output FOV.

[0038]

[0079] Finally, Figure 14 shows the robustness of the SR image display system 10 to different quantization levels of wavefront modulation. For this analysis, a diffractive SR image display system 10 with L = 5 substrate layers 18 trained for 16-bit quantization of the phase-only wavefront modulator was blind-tested for lower quantization levels of 8, 6, 4, and 2 bits. Figure 14 shows that the proposed diffractive SR image display system 10 can successfully synthesize a super-resolution reconstructed or projected image 106 at its output, even with 6-bit quantization of the encoded phase profile. The overall image synthesis performance for 8-bit quantization (18.58 dB PSNR and 0.58 SSIM) and 6-bit quantization (18.20 dB PSNR and 0.55 SSIM) of the phase modulator / encoder demonstrates the robustness of the diffractive system 10, considering that the 16-bit phase quantization case yields 18.61 dB PSNR and 0.58 SSIM. The diffractive SR image display system 10 is unable to synthesize a clear image in the output FOV with 2-bit phase quantization, but is partially successful with 4-bit phase quantization (Figure 14). For such low-bit-depth phase quantization, the presented encoding-decoding framework can be trained from scratch to further improve image projection performance even with limited phase encoding accuracy.

[0039]

[0080] Consider

[0081] A diffractive SR image display system 10 is disclosed, which is based on a jointly trained pair of an electronic encoder network 12 and an all-optical diffractive decoder network 14, which collectively improve the SBP of an image projection system. The deep learning-engineered diffractive display system 10 synthesizes and projects / reconstructs a super-resolution image 106 at an output FOV by encoding each high-resolution image 100 of interest into a low-resolution representation 104 with fewer pixels per image. As a result, the all-optical decoding capability of the all-optical diffractive decoder network 14 not only improves the effective SBP of the image projection system 10, but also reduces data transmission and storage requirements due to the use of a low-resolution image generator 16, such as a wavefront modulator. Because the all-optical diffractive decoder network 14 is an all-optical diffractive system constructed with a passive structural substrate layer 18, it consumes no computing power other than that of the illumination light. Similarly, because the all-optical synthesized image 106 is calculated at the speed of light propagation between the encoder SLM plane and the output FOV of the all-optical diffractive decoder network 14, the computational bottleneck in terms of speed and power consumption is only the inference portion of the front-end CNN encoder 12.

[0040]

[0082] As shown in the experimental results (Figures 11 and 13), there are some relatively small discrepancies between the numerical output images of the forward model and the corresponding experimentally measured output images 106. There are potential error sources that cause these discrepancies. First, the numerical forward model used in training assumes a uniform plane wave incident on the surface of the wavefront modulator 16. This assumption can potentially be violated in the experimental setup due to wavefront distortions of the THz source used. Due to the limited resolution of the 3D printer used to fabricate the diffractive / substrate layers 18, additional errors may be introduced during the fabrication of each diffractive / substrate layer 18. Furthermore, inaccuracies in the refractive index properties of the 3D-printed material at the illumination wavelength may also contribute to the slight discrepancies between the numerical and experimental results.

[0041]

[0083] While these conceptual proof-of-concept experiments use the THz portion of the electromagnetic spectrum, the main design principles and conclusions provided herein also apply to display systems 10 operating at visible wavelengths. Extending the design of the SR display system 10 to visible wavelengths is feasible using various nanofabrication techniques, such as two-photon polymerization and lithography, that provide subwavelength capabilities. Furthermore, we investigated the ability of jointly trained encoder and decoder networks 12, 14 in synthesizing SR images 106 at small axial distances (~150–350 λ) from the wavefront modulation surface of the encoder 12. The training procedure and design principles can also be extended to synthesize 3D super-resolution object fields with extended working distances at the output of the all-optical diffraction decoder network 14.

[0042]

[0084] While the results of the SR image display system 10 described herein were obtained with a single illumination wavelength, the design principles of the all-optical diffractive decoder network 14 can be extended to operate with multiple wavelengths and incorporate spectral information into the projected image 106. The high-resolution image projection 106 in the output field of view may also display the color information of the corresponding input image 100. To optically synthesize full-color (RGB) images, some conventional holographic display systems use sequential operation (i.e., illuminating one illumination wavelength at a time, followed by another), which spatially utilizes all pixels of the SLM for each wavelength but reduces the frame rate. Another approach involves spatially multiplexing SLM pixels between different illumination wavelength channels, but this further sacrifices the SBP of the display between the different color channels, limiting the output image size and resolution. By incorporating the dispersion and refractive index information of the wavefront modulation medium (e.g., liquid crystal) and the all-optical diffractive decoder network 14 materials as part of the optical forward model of the design, the diffractive display system 10 can be extended to synthesize super-resolution images at groups of illumination wavelengths. In this case, the jointly trained encoder network 12 can be optimized to drive the SLM 16 simultaneously or sequentially at multiple wavelengths based on assumptions made during the training process of the encoder-decoder pair. In either mode of operation, a multi-wavelength SR image display using the all-optical diffractive decoder network 14 requires more diffractive features / neurons for a given output FOV and SR coefficients compared to a monochrome version to be able to process independent spatial features in different illumination wavelengths or color channels of the input image 100.

[0043]

[0085] The SR image display system 10 can be thought of as a hybrid autoencoder framework, including a digital encoder network 12 used to create a low-dimensional representation 104 of a target high-resolution image 100, and an all-optical diffraction decoder network 14 (co-trained with the encoder network 12) that synthesizes a super-resolution image 106 at its output FOV from the diffraction patterns of the low-resolution encoded pattern 104 generated by the encoder network 12. Joint optimization and communication between the electronic front-end and the diffractive optical back-end of the SR image display system 10 is crucial to improving the SBP of the image formation model, influencing the design of new compact, low-power, and computationally efficient high-resolution camera and display systems.

[0044]

[0086] method

[0087] All-optical decoder design for SR image display

[0088] In the optical forward model, the diffractive modulation layer (e.g., substrate layer 28) is oriented along the x and y axes as w x and w y Each point on the grid is called a "diffraction neuron" and has a transmittance coefficient t of the smallest feature of each modulation layer. l The electric field transmittance I≧1 of the diffractive layer is defined as follows:

[0089] (Formula 1) TIFF2025527099000002.tif16104

[0090] where τ(λ) = n(λ) + jκ(λ) is the complex refractive index of the optical material used to manufacture the diffractive layer, λ denotes the wavelength of the coherent illumination, and n a = 1 denotes the refractive index of the medium surrounding the modulation layer (air in this case), and h l [m,n] represents the material thickness of the corresponding neuron and is defined as follows:

[0091] (Formula 2) TIFF2025527099000003.tif17104

[0092] where o l [m,n] is [hb ,h m These auxiliary variables are used to calculate the material thickness between [ ] and [ ]. l [m,n] and material thickness values h for all m, n, and I l [m,n] is optimized using backpropagation based on stochastic gradient descent and deep learning.

[0045]

[0093] 2D modulation function T for continuous coordinates (x,y) l (x,y) is the transmittance coefficient t l [m,n] and a 2D rectangular sampling kernel p l It can be written in terms of (x,y) as:

[0094] (Formula 3) TIFF2025527099000004.tif18123

[0095] where p l (x,y) is defined as follows:

[0096] (Formula 4) TIFF2025527099000005.tif22103

[0046]

[0097] Light propagation between successive diffractive layers is modeled by a Fast Fourier Transform (FFT)-based implementation of the Rayleigh-Sommerfeld diffraction integral using the angular spectrum method. This diffraction integral is defined as the propagation kernel w(x,y,z) and the input wavefront It can be represented as a 2D convolution of TIFF2025527099000006.tif1328.

[0098] (Formula 5) TIFF2025527099000007.tif14107

[0099] TIFF2025527099000008.tif1474

[0100] TIFF2025527099000009.tif1895

[0101] TIFF2025527099000010.tif1656

[0047]

[0102] Diffraction Decoder Vaccine

[0103] To mitigate the effects of potential misalignments during the experiments, possible physical error sources were incorporated as part of the optical forward model of the all-optical diffractive decoder network 14. During training of the experimentally tested diffractive designs, these errors were represented by random 3D displacement vectors D that indicate the deviation of the position of the diffractive layer I from its ideal position. l =(D x ,D y ,D z ), where D x , D y , D z is defined as an independent, uniformly distributed random variable.

[0104] (Formula 6) TIFF2025527099000011.tif4247

[0048]

[0107] In Equation 6, the variable Δ x , Δ y , Δ z , and , respectively, denote the maximum displacement along the corresponding axis. Therefore, the position of the diffractive layer I at the i-th iteration is TIFF2025527099000012.tif1215 is defined as follows:

[0108] (Formula 7) TIFF2025527099000013.tif16109

[0049]

[0109] Encoder CNN network design for SR image display

[0110] A CNN-based electronic encoder network 12 was used to compress high-resolution input images of interest into a low-dimensional latent representation 104 that can be provided using a low-SBP wavefront modulator or SLM 16. The CNN network architecture is shown in Figure 6A. It contains four convolution blocks, followed by a flattening operation, a fully connected layer, a rectified linear unit (ReLU)-based activation function, and an unflattening operation. Each convolution block contains three sets of 4x4 convolution filters (with the same padding configuration) with Leaky ReLU (gradient α = 0.1). The i-th convolution block contains 2 1+i There is a channel. Finally, a fully connected layer is utilized to reduce the channel dimensionality and obtain a low-dimensional representation at the output of the electronic encoder CNN.

[0050]

[0111] Preparing the training and test datasets

[0112] To train and test the diffractive SR display system 10, a training image dataset, i.e., the EMNIST display dataset, was created. As shown in Figure 18, each image in this display dataset was generated using a different number of images selected from the EMNIST handwritten characters. The selected characters were scaled (K~U(0.84,1)), rotated (θ~U(-5°,5°)), and translated (D x ,D yThe images were augmented by predefined geometric operations, including the λ(-1.06λ, 1.06λ) approximation. These selected and augmented images were then randomly arranged in a 3x3 grid. This procedure was used for each image in the display dataset. The original EMNIST handwritten character dataset contains 88,000 and 14,800 character images for training and testing, respectively. The original size of these characters is 28x28 pixels. Before applying the above tiling procedure, each image was interpolated to 32x32 using bicubic interpolation. For the training dataset, 60,000 images (96x96 pixels) containing the handwritten characters 1, 2, 3, and 4 were created using the EMNIST character training dataset (15,000 images in each case). For the validation dataset, 6,000 images (96x96 pixels) containing the handwritten characters 1, 2, 3, and 4 were created using the EMNIST character training dataset (1,500 images in each case). For the test dataset, 6,000 images (96x96 pixels) containing handwritten characters 6, 7, 8, and 9 were created using the EMNIST character test dataset (1,500 in each case).

[0051]

[0113] In the experimentally tested design, two complementary image sets containing 80,000 and 8,800 different handwritten characters from the EMNIST character training dataset were used as the training and validation datasets, respectively. The EMNIST character test dataset, containing 14,800 handwritten characters, was used as the test dataset. The handwritten characters were resized to 15 × 15 pixels using bicubic downsampling (with an anti-aliasing filter) based on the effective pixel size used in the measurement (output) plane.

[0052]

[0114] Implementation details of a numerically tested diffractive SR display system

[0115] The width of the diffractive neuron in the transparent layer (w x ,w y) and the sampling period of the light propagation model were chosen as 0.533λ. The size of each diffractive layer was 106.66λ × 106.66λ (200 × 200 pixels). The input and output FOV of the diffractive decoder was 51.168λ × 51.168λ (96 × 96 pixels). To avoid aliasing in the light forward model, these matrices were padded with zeros to 400 × 400 pixels. In the light forward propagation model, the material absorption was assumed to be zero (κ(λ) = 0). Therefore, the transmittance of each feature in the diffractive layer can be written as follows:

[0116] (Formula 8) TIFF2025527099000014.tif17150

[0053]

[0117] Phase coefficient θ of each diffractive layer of the decoder l [m,n] was optimized using deep learning and backpropagation. l [m,n] was initialized as 0.

[0054]

[0118] In the results, different all-optical diffractive decoder networks 14 containing one, three, and five transmission layers (substrate layers 18) were analyzed. The axial distances from the input surface to the first diffractive layer d1, from one diffractive layer to another diffractive layer d2, and from the last layer to the output surface d3 (Figure 6B) were empirically optimized (see Table 1). The trained phase profiles of the transmission layers of the diffractive decoder using a phase-only SLM are shown in Figure 19.

[0055] TIFF2025527099000015.tif51156

[0056]

[0119] Implementation details of an experimentally tested diffractive SR display system

[0120] In the experiments, a monochromatic THz illumination source (λ = ~0.75 mm) was used. The diffractive neuron size in the substrate layer 18 and the sampling period of the light propagation model were set to ~0.667λ, resulting in a size of 66.7λ × 66.7λ (5 cm × 5 cm) for each layer 18. The effective pixel size in the measurement plane was set to ~2.67λ in the experiments. The size of the phase-only wavefront modulator 16 was selected to be 40λ × 40λ (3 cm × 3 cm), which is also equal to the size of the output FOV. Based on the pixel size of ~2.67λ in the output FOV, the number of pixels in the output image was set to 15 × 15, and the LR wavefront modulator 16 was selected as a 5 × 5 pixel phase-only layer with a pixel pitch of 8λ × 8λ. This corresponds to a desired SR coefficient of k = 15 / 5 = 3 targeted during the training of these models, i.e., a 9-fold SBP enhancement by the all-optical diffractive decoder network 14. To avoid spatial aliasing in the optical forward model, the matrix was padded with zeros to 300 × 300 pixels.

[0121] The complex refractive index T(λ) of the 3D printed material used to fabricate the substrate layer 18 and the phase-encoding input was measured to be ~1.6518 + j0.0612. In this model, the material thickness h of each diffractive neuron l The [m,n] was optimized in the range of [0.5mm, ~1.64mm], which corresponds to the [-π,π] phase modulation. l [m,n] was initialized as 0.

[0057]

[0122] In our experiments, we fabricated and tested two all-optical decoder networks 14, one with L = 3 and one with L = 1. The axial distances of these models are shown in Table 1. As part of the training process of the all-optical diffractive decoder network, independent random displacements along the axial and lateral coordinates were added to the substrate layer 18 position and input FOV during the training phase, as detailed in Equation (7). During the training of these experimentally tested models, Δ x , Δ y , Δ zwere set to ~0.334λ, ~0.334λ, and ~0.533λ, respectively. The resulting optimized thickness map and phase-only encoded representation of substrate layer 18 were converted into STL files using MATLAB and fabricated using a 3D printer (Objet30 Pro, Stratasys Ltd.).

[0058]

[0123] Experimental setup

[0124] A schematic diagram of the experimental setup is shown in Figures 10C-10D. The THz plane wave incident on the object was generated via a WR2.2 modular amplifier / multiplier chain (AMC) 50 with a compatible diagonal horn antenna 58 (Virginia Diode Inc.). The AMC 50 was powered by an RF synthesizer 52 at 11.111 GHz (f RF1 The AMC receives a 10 dBm RF input signal at 11.083 GHz (f) and multiplies it by 36 to generate continuous wave (CW) radiation at 0.4 THz. The AMC output is modulated at a 1 kHz rate via signal generator 56, and low-noise output data is resolved by lock-in detection with lock-in amplifier 54. The exit aperture of the horn antenna 58 is positioned approximately 60 cm from the object plane of the 3D-printed all-optical decoder network 14. Diffracted THz radiation at the output plane is detected by a single pixel mixer / AMC (Virginia Diode Inc.) 60. The AMC is then coupled to a 11.083 GHz (f RF2A 10 dBm RF signal from 64 dBm was fed to the detector as a local oscillator for mixing, downconverting the detected signal to 1 GHz. The detector was mounted on an XY positioning stage containing two linear motor stages (Thorlabs NRT100). The output FOV was scanned with a 1 mm step size using a 0.5 × 0.25 mm detector. 2 × 2 pixel binning was used to improve the SNR and closely match the design output pixel size (~2.67 λ). The downconverted signal was sent to a cascaded low-noise amplifier 64 (Mini-Circuits ZRL-1150-LN+) to provide 40 dB of amplification. A 1 GHz (+ / - 10 MHz) bandpass filter 66 (KL Electronics 3C40-1000 / T10-O / O) was then used to remove noise from unwanted frequency bands. The amplified and filtered signal passed through a tunable attenuator 68 (HP 8495B) for linear calibration and a low-noise power detector 70 (Mini-Circuits ZX47-60). The output voltage signal was read by a lock-in amplifier 54 (Stanford Research SR830). A modulated signal from a signal generator 56 was used as the reference signal for the lock-in amplifier 54. Based on the calibration results, the lock-in amplifier measurements were converted to a linear scale. The bottom 5% and top 5% of all pixel values for each measurement were saturated, and the remaining pixel values were mapped to a dynamic range of 0–1.

[0059]

[0125] Training loss functions and performance comparison metrics

[0126] In the joint training of an electronic CNN-based encoder network12 and an all-optical diffractive decoder network14, an efficiency penalty term The mean absolute error function along with TIFF2025527099000016.tif1118 was used as the training loss function (L), defined as follows:

[0127] (Formula 9) TIFF2025527099000017.tif1778

[0128] TIFF2025527099000018.tif2135

[0129] TIFF2025527099000019.tif1741

[0130] where TIFF2025527099000020.tif109 and TIFF2025527099000021.tif1310 represents the high-resolution image of the target (ground truth) and the output intensity of the all-optical decoder network, respectively. N is the number of pixels in each image, and σ is a normalization term. P i and TIFF2025527099000022.tif1434 represents the light intensity incident on the input and output FOVs, respectively. The power efficiency of the all-optical diffraction decoder network 14 can be adjusted by adjusting γ. In the experimentally demonstrated training of the all-optical diffraction decoder network 14, γ was set to 0.005 for L=1, 0.015 for L=3, and γ=0 for other designs.

[0060]

[0131] During the joint training of the CNN encoder12 and the all-optical diffraction decoder network14, several image data augmentations were used, including random image rotations (0°, 90°, 180°, and 270°), random image flips, and random contrast adjustments. The joint training was implemented in Python (v3.6.12) and TensorFlow (v1.15.4, Google LLC). During training, the Adam optimizer was used with a learning rate of 0.001 for the all-optical diffraction decoder14 and 0.0005 for the CNN-based encoder12. All networks12,14 were trained on a GeForce RTX 3090 GPU (Nvidia Corp.) and an AMD Ryzen Threadripper 3960X CPU (Advanced Micro Devices Inc.) with 264 GB of RAM. Each network12,14 was trained for up to 17 hours (500 epochs) with a batch size of 40.

[0132] For quantitative comparison of the results, we calculated the PSNR and SSIM values for each target image in the test set. PSNR is calculated as follows:

[0133] (Formula 10) TIFF2025527099000023.tif2189

[0061]

[0134] SSIM is computed using TensorFlow's standard implementation with a maximum value of 1. For visual comparison, a low-resolution version of the target image was obtained using k-fold downsampling with a bicubic kernel (and an anti-aliasing filter).

[0062]

[0135] FIG. 20A illustrates another embodiment of a communication system 30, including an electronic encoder network 12 and an all-optical decoder network 14, capable of spatially transmitting a message or signal 70 from the electronic encoder network 12 to the all-optical decoder network 14, even though the optical path is at least partially obstructed by an opaque obstruction and / or diffusing medium 32. Here, the obstruction and / or diffusing medium 32 is located between the electronic encoder network 12 and the all-optical decoder network 14, yet the message or signal 70 (which may include an image in some embodiments) is still analyzed by the all-optical decoder network 14. The electronic encoder network 12 is used to create an encoded message or wavefront 72, which is transmitted through free space to the all-optical decoder network 14, which generates a decoded output 74 containing the transmitted message or signal 70. In this method, the electronic encoder network 12 and the all-optical decoder network 14 are jointly trained using deep learning to transmit an optical message or signal 70 of interest around an opaque obstruction / diffusing medium 32 of any shape. The all-optical decoder network 14 includes a spatially designed, continuous passive surface (i.e., the substrate layer 18) that processes optical information through light-matter interactions. After training, the encoder-decoder pair can communicate any optical information or signal around an opaque occluder or diffusive medium 32, where information decoding occurs at the speed of light propagation. For occluders or diffusive media 32 that change size and / or shape and / or properties as a function of time, the electronic encoder network 12 can be retrained to successfully communicate with an existing all-optical decoder network 14 without modifying the already deployed physical substrate layer 18. This system 30 was experimentally verified in the terahertz spectrum using a 3D-printed all-optical decoder network 14 to communicate around a fully opaque occluder. This method, which operates scalably across any wavelength region, is particularly useful for emerging high-data-rate free-space communication systems.

[0063]

[0136] In this embodiment, the electronic encoder network 12, trained together with the all-optical decoder network 14, effectively bypasses the opaque obstruction and / or diffusing medium 32 and uses passive diffraction through the thin structured substrate layer 18 to encode a message / signal of interest 70 (i.e., the encoded message) in an encoding wavefront 72 for decoding at a receiver by the all-optical decoder network 14. This all-optical decoding is performed on the encoding wavefront 72 carrying the optical message / signal of interest 70 after being obstructed by an opaque obstruction 32 of arbitrary shape. The all-optical decoder network 14 uses a passive smart material consisting of continuous spatially engineered surfaces to process secondary waves scattered through the edges of the opaque obstruction 32 and perform reconstruction of the hidden information to produce a decoded output 74 at the propagation speed of light through a thin diffractive volume less than 100 × λ (λ is the wavelength of the illumination light) in the axial direction.

[0064]

[0137] The combination of electronic encoding and all-optical decoding enables direct optical communication between the electronic encoder network 12 (i.e., transmitter) and the all-optical decoder network 14 (i.e., receiver) even when an opaque obstruction 32 completely blocks the transmitter's field of view (FOV). This system 30 is highly power-efficient, with a diffraction efficiency of over 50% at the output. In the case of opaque obstructions in the diffusing medium 32 that change in size / shape and / or properties over time, the electronic encoder network 12 can be retrained to successfully communicate with an existing all-optical decoder network 14 without modifying the physical structure already deployed. This makes the system highly dynamic and easily adaptable to external, uncontrollable changes that may occur between the transmitter and receiver apertures. The system 30 can be extended to operate in various portions of the electromagnetic spectrum, enabling applications in emerging high-data-rate free-space communication technologies in scenarios where various undesirable structures block the direct communication channel between the transmitter and receiver.

[0065]

[0138] FIG. 20A shows a schematic diagram of an optical communication system 30 capable of transmitting around an opaque obstruction 32 with zero optical transmittance. A message or signal 70 to be transmitted, e.g., an image of an object, is fed into an electronic encoder neural network 12, which outputs a phase-encoded optical representation 72 of the message. It should be understood that in other embodiments, the message representation may be amplitude-encoded only or complex-valued encoded. This code is imposed on the phase of a plane-wave illumination and transmitted toward an all-optical decoder network 14. In the experimental setup, the plane-wave illumination passes through an aperture 40 (FIGS. 28B, 28C) partially or fully obstructed by an opaque obstruction 32. Scattered waves from the edge of the opaque obstruction 32 travel as secondary waves toward a receiver aperture 42 (FIG. 28A), where the all-optical decoder network 14 all-optically decodes the received light to directly reproduce the message / object 70 in its output FOV. This decoding operation is completed while the light propagates through a thin decoder substrate layer 18. Additionally, in embodiments including one or more reflective substrate layers 18, light may reflect off of the substrate layers 18. In this cooperative encoding / decoding scheme, the electronic encoder network 12 and the all-optical decoder network 14 are digitally trained jointly in a data-driven manner to provide effective optical communication, bypassing any completely opaque obstructions placed between the transmitter aperture and receiver.

[0066]

[0139] Figures 20B and 20C show in more detail the respective architectures of the constructed electronic encoder network 12 and all-optical decoder network 14. As shown in Figure 20B, the convolutional neural network (CNN)-based electronic encoder network 12 consists of several convolutional layers followed by a dense layer representing the encoded output. The output of this dense layer is rearranged into a 2D array corresponding to the spatial grid that maps the phase-encoded transmitter aperture. Unless otherwise noted, we assume that both the desired message and the phase code to be transmitted contain 28 x 28 pixels. The architecture of the electronic encoder network 12 is the same for all designs reported in this paper. Figure 20C shows the architecture of the all-optical decoder network 14, which decodes the transmitted and shielded phase-encoded wave. Figure 20C shows the all-optical decoder network 14 including L = 3 spatially designed substrate layers 18 (i.e., S1, S2, S3). However, for comparison, results are also reported for designs including all-optical decoder networks 14 with L = 1 and L = 5 substrate layers 18. The spatial features of the diffractive plane (i.e., substrate layer 18) of the all-optical decoder network 14, along with the encoder CNN parameters, are optimized to decode the encoded, blocked / obscured wavefront and generate a decoded output 74. In the tested embodiment, phase-only diffractive features 20 were considered; that is, only the phase values of the features 20 at each diffractive plane were trainable (see the "Materials and Methods" section for details). Figure 20D compares the performance of the presented electronic encoding and diffractive decoding system 30 with that of a lens-based camera. As shown in Figure 20D, in contrast to the decoded output 74, the lens image reveals a significant loss of information caused by opaque occlusions 32 in a standard camera system, demonstrating the scale of the problem addressed in this embodiment.

[0067]

[0140] For all models reported herein, data-driven joint digital training of the CNN-based electronic encoder network 12 and the all-optical decoder network 14 was achieved by minimizing a structural loss function defined between the objects (ground truth messages) and the all-optical decoder network output using 55,000 images of handwritten digits from the MNIST training dataset and an additional 55,000 custom-generated images (examples shown in Figure 29). All results are from blind tests using objects / messages never used during training. Digital training involves training digital models of the encoder network 12 and the all-optical decoder network 14 to create design parameters for the substrate layer 18, which is used to create a physical implementation of the optimized all-optical decoder network 14.

[0068]

[0141] Numerical analysis of diffractive optical communication around opaque obstacles

[0142] First, we compare the performance of trained encoder-decoder pairs with different diffractive decoder architectures in terms of the number of diffractive planes (i.e., substrate layers 18) employed, for various levels of opaque occluders 32. Specifically, we compare the width value of each occluder, i.e., w o = 32.0λ, w o = 53.3λ, w o For L = 74.7λ, we designed three encoder-decoder pairs with L = 1, L = 3, and L = 5 diffractive substrate layers 18 in the all-optical decoder network 14 and compared the performance of these designs on a new handwritten digit in Figure 21. This blind test represents "internal generalization" because the specific test object was not used in training but was derived from the same dataset. As shown in Figure 21, the L = 1 design can also faithfully decode the optical message 70, avoiding these various levels of occlusion. Furthermore, as the number of layers 18 in the all-optical decoder network 14 increases from L = 3 to L = 5, the quality of the output also improves. The performance of the L = 1 design is wo Although there is a slight degradation in performance as the width of the transmitter aperture increases, the L=3 and L=5 designs do not show any qualitative degradation in performance for such large obstructions. t Note that = 59.73λ. Therefore, the size of the occlusion is w o For λ = 74.7λ, the opaque obscurant completely blocks the opening of the encoding transmitter aperture, so none of the ballistic photons can reach the receiver aperture. However, scattering from the obscurant edge is sufficient for the encoder-decoder pair to communicate faithfully. It should be understood that the transmit and receive apertures in the experimental setup are not required to implement communication system 30 and may be omitted.

[0069]

[0143] To complement the qualitative results shown in Fig. 21, we consider the width of the occlusion (w o We compared the performance of different encoder-decoder pairs designed to increase w in terms of peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM) averaged over 10,000 (previously unusable) handwritten digits from the MNIST test set. See Figures 22A-22B, respectively. o As w increases, the performance degradation is larger for the L=1 design compared to the L=3 and L=5 designs. o The decoder performance for L=1 and L=3 improves slightly when wt=59.73λ (the transmitter aperture width). This improved performance level is achieved by o >w t The reason for this is explained in this paper.

[0070]

[0144] Next, for the same designs reported in Figure 21, the external generalization of these encoder-decoder pairs was investigated by testing their performance on object types not represented in the training set (see Figure 23). For this analysis, two images of fashion products were randomly selected from the Fashion-MNIST test set (top) and two additional images were randomly selected from the CIFAR-10 test set (bottom). As shown in Figure 23, the encoder-decoder design demonstrates excellent generalization across these completely different object types. o = 53.3λ and w o Although the all-optical decoder network output for the L = 1 decoder design for = 74.7λ is slightly degraded, it is still able to recognize objects at the output plane even when the transmitter aperture is completely blocked by an obstruction.

[0071]

[0145] We also investigated the ability of these designs to resolve closely spaced individual features in their output. To this end, we transmitted a test pattern consisting of four closely spaced dots, and the corresponding all-optical decoder network output is shown in Figure 24. In the top (bottom) pattern, the vertical / horizontal spacing between the inner edges of the dots is 2.12λ (4.24λ). While none of the designs were able to resolve the dots spaced 2.12λ apart, dots spaced 4.24λ apart were resolved with good contrast for all encoder-decoder designs, as can be seen in the cross-sectional views accompanying the output images in Figure 24. Note that this 4.24λ resolution limit is due to the output pixel size, which was set to 2.12λ in the simulations. The effective resolution of the encoder-decoder communication system 30 can be further improved within the diffraction limit of light by using high-resolution objects and reducing the pixel size during training.

[0072]

[0146] Performance impact of phase bit depth

[0147] where the finite bit depth b of the encoder plane qThe effect of phase quantization was investigated. In the results presented so far, the bit depth of phase quantization was assumed to be infinite. o = 32.0λ, L = 3 design (infinite bit depth b q tr = ∞), the first line of Figure 25A shows the q te This shows the effect of quantizing the encoded phase pattern rather than just the phase values of the diffractive layer. This shows that the electronic encoder network 12 and the all-optical decoder network 14 can be trained without such a phase bit depth restriction, in other words, b q tr =∞, and b q te It represents an "attack" on the design because it was tested at a finite level of b. q tr For the design of =∞, the output quality is b q te = 8 was not affected, but b q te At = 4, significant deterioration is observed, and b q te =3 and b q te = 2 caused a complete failure. However, q te This sharp performance drop with decreasing w can be corrected by taking finite bit depth into account during training. o = 32.0λ and L = 3, b q tr =4 and b q tr Two additional designs were trained assuming a finite bit depth of b = 3. Both of these designs q te = 3 (8-level phase quantization at the encoder and decoder layers). q tr =∞, b q tr =4, b q tr = 3) performance (PSNR and SSIM) for different b q te These quantitative comparisons lead to the same conclusion: low b q tr By training at a bit depth of b, the output performance is sacrificed relatively little. q te This results in a robust encoder-decoder design that maintains optical communication quality even when the power consumption decreases.

[0073]

[0148] Output Efficiency

[0149] Next, we investigated the output efficiency of the optical communication system 30 around an opaque obstruction 32 using a jointly trained electronic encoder and diffraction decoder pair. In this analysis, diffraction efficiency (DE) was defined as the ratio of the light intensity at the output FOV to the light intensity exiting the transmitter aperture. Figure 26A plots the diffraction efficiency of the same design as Figures 22A-22B as a function of obstruction size. These values were calculated by averaging 10,000 MNIST test images. These results reveal that, as expected, the diffraction efficiency decreases monotonically with increasing obstruction width. Furthermore, the diffraction efficiency is relatively low, below or around 1% even for small obstructions. However, this issue of low diffraction efficiency can be addressed during the design phase by adding a loss term to the training loss function that penalizes low diffraction efficiency (see Equation 1 in the Materials and Methods section). Figure 26B shows how increasing the weight (η) of this additional loss term during the training phase results in improved diffraction efficiency. For example, the designs with η = 0.02 and η = 0.1 yield average diffraction efficiencies of 27.43% and 52.52%, respectively, while still being able to resolve various features in the target image, as shown in Figure 26C. This additional loss term weight η therefore provides a powerful mechanism for significantly improving output diffraction efficiency at a relatively small sacrifice in image quality, as illustrated in Figures 26B-26C.

[0074]

[0150] Shape of shield

[0151] Above, we considered square-shaped opaque occluders 32 symmetrically arranged around the optical axis. However, the communication system 30 is not limited to square-shaped occluders and can be used to communicate around virtually any occluder shape. Figures 27A-27E show a comparison of the performance of four different trained encoder-decoder pairs for four different occluder shapes, where the area of the opaque occluders 32 is kept approximately the same. We can see that the shape of the occluder 32 does not noticeably affect the output image quality. The average SSIM values calculated for these four models are plotted in Figure 30 across 10,000 MNIST test images (internal generalization) and 10,000 Fashion-MNIST test images (external generalization), further supporting the success of our embodiment for various opaque occluder 32 configurations, including the randomly shaped occluder shown in Figure 27E.

[0075]

[0152] Experimental verification

[0153] An electronic encoding and diffractive decoding system 30 for communication around an opaque obstruction 32 in the terahertz (THz) portion of the spectrum (λ = 0.75 mm) was experimentally verified using a 3D-printed single-layer 18 (L = 1) all-optical decoder network 14 (see the "Materials and Methods" section for details). The setup used for this experimental verification is shown in Figure 28A. Figures 28B and 28C show the 3D-printed parts used to implement the encoding (phase) pattern, the opaque obstruction 32, and the diffractive decoder substrate layer 18. As shown in Figure 28C, the width of the transmitter aperture (dashed square in the "encoding object" image) into which the encoding phase pattern enters is w t ≒59.73λ, and the width of the opaque occluder (the dashed rectangle in the "occluder" image) is w o ≒32.0λ, and the width of the diffractive decoder layer (the dashed square in the "diffractive layer" image) is w tThe axial distances between the encoding object and the occluder 32, between the occluder and the diffractive layer 18, and between the diffractive layer 18 and the output FOV were chosen to be ~13.33λ, ~106.67λ, and ~40λ, respectively. Figure 28D shows the input object / message, simulated lens image, and simulated and experimental diffractive decoder output images for 10 randomly selected handwritten digits from the test dataset. The experimental results show that CNN-based phase encoding followed by diffractive decoding successfully conveyed the intended object / message around the opaque occluder 32 (see the bottom row of Figure 28D).

[0076]

[0154] The optical communication system 30 using CNN-based encoding and diffractive all-optical decoding is useful for optical communication of information around opaque obstructions 32 caused by existing or changing structures. Even if such obstructions 32 change slowly over time (e.g., grow in size as a function of time), the same all-optical decoder network 14 deployed as part of the communication link can be used simply by updating the electronic encoder network 12. To illustrate this, in Figure 31, we show an example of an optical communication system 30 that initially uses the CNN-based encoding and diffractive all-optical decoding. o The encoder-decoder design trained with an L=3 occlusion size of 2.0λ is shown, and the L=3 occlusion size remains the same (i.e., w o = 32.0λ), the input message is successfully communicated between the CNN-based phase transmitter aperture and the output FOV of the all-optical decoder network 14 (top). Figure 31 also shows that when the size of the opaque occluder is w o = 40.0λ, this encoder-decoder pair fails (center). This failure due to the (unexpected) increased size of the occlusion can be repaired by simply retraining the CNN encoder part, without changing the deployed diffractive decoder layer (see Figure 31 (bottom)). Here, the all-optical decoder network 14 remains unchanged, but the electronic encoder network 12 is retrained by w o =40.0λ.

[0077]

[0155] The speed of optical communication through the encoder-decoder pair is limited by the speed at which the encoding phase pattern (CNN output) can be refreshed or the speed of the output detector array, whichever is smaller. The transmission and decoding of the desired optical information / message occurs at the propagation speed of light through the thin substrate layer 18 (i.e., the diffractive layer) and consumes no external power (except for the illumination light). Therefore, the primary power consumers in this architecture are the CNN inference, the transmission of the encoding phase pattern, and the operation of the detector array.

[0078]

[0156] Communication around a shield 32 using system 30 works even when the shield width is larger than the transmitter aperture width because it utilizes CNN-based phase encoding of information to effectively exploit scattering from the edges of the shield. Surprisingly, the shield width is much smaller than the transmitter aperture width (w t), the performance of the L=1 and L=3 designs improves slightly, as shown in Figures 22A-22B. This relative improvement can be explained by the switching between the operating modes of the encoder-decoder pairs. When the opaque occluder is smaller than the transmitter aperture, pixels at the transmitter edge transmit directly to the receiver aperture and therefore dominate the power balance. In this operating regime, as the occluder size increases, the effective number of pixels in the transmitter aperture that communicate directly with the receiver / decoder decreases, resulting in poor performance of the diffractive decoder. However, when the occluder becomes larger than the transmitter aperture, none of the input pixels can communicate directly at the receiver to dominate the power balance. Instead, all pixels in the encoder plane are forced to contribute indirectly to the receiver aperture through the occluder's edge scattering. This improves performance for occluders larger than the transmitter aperture. This is because substantially more pixels in the encoder plane can contribute to the receiver aperture without significant power imbalance between secondary wave-based contributions (due to edge scattering). This performance upset (i.e., switching between these two modes of operation) is not observed when the diffractive decoder has a deeper architecture (e.g., L = 5) because the deeper decoder can effectively balance the ballistic photons transmitted from the edge pixels. As a result, because the multiple substrate layers 18 of the deeper all-optical decoder network 14 function as a universal mode processor, the edge pixels of the transmitter aperture do not dominate the output signal, even when they can "see" the receiver aperture directly.

[0079]

[0157] Finally, the success of the simpler all-optical decoder network 14 design with L = 1 layers, as shown in Figures 21-24 and 28A-28D, raises the question of whether such optical communication around opaque obstructions is feasible with electronic encoding only, i.e., without diffractive decoding. To address this question, we developed a two-encoder-only design. o = 32.0λ and w o= 53.3λ and compare its performance with the L = 1 design in Figure 32. The encoder-only architecture performs well with w o = 32.0λ, and w o = 53.3λ, while the L = 1 design provides significantly better performance. This demonstrates the importance of complementing electronic encoding with diffractive decoding for effective communication around opaque obstructions.

[0080]

[0158] The communication system 30 can operate at multiple wavelengths, for example, the message or signal 70 to be encoded / decoded can be transmitted at any of the following wavelengths: ultraviolet, visible, infrared, terahertz, or millimeter wavelengths.

[0081]

[0159] Materials and Methods - Optical Communication Around Opaque Obstructions

[0160] Model

[0161] In the model used, a message / object m to be transmitted is fed into a CNN-based electronic encoder network 12, which generates a phase-encoded representation ψ of the message. in ×N in = 28 × 28 pixels. The encoded phase ψ is N out ×N out Assume that the dimensions are 28 × 28. N out ×N out The phase element of w t ×w t is distributed over the transmitter aperture of area w t ≈59.73λ, where λ is the illumination wavelength. Therefore, the lateral width of each phase element / pixel is TIFF2025527099000024.tif1246. The phase-encoded input wave exp(jψ) is TIFF2025527099000025.tif1338 is propagated to the surface of an opaque occluder, and its amplitude is modulated by the occluder function o(x,y) as follows:

[0162] TIFF2025527099000026.tif2385

[0082]

[0163] The encoded wave, after being blocked and scattered by the occlusion 32, travels through free space to the receiver, where an all-optical decoder network 14 processes and decodes the input wave all-optically, resulting in an all-optical reconstruction of the original message m at the output FOV. Generate TIFF2025527099000027.tif1112. The receiver aperture coincides with the first layer of the diffraction decoder and is ol ≈106.67λ. The effective size of the independent diffractive features in each transmissive layer (substrate layer 18) is assumed to be 0.53λ × 0.53λ, and each of the L layers contains 200 × 200 diffractive features 20, so the lateral width of the diffractive layer 20 is w l ≒106.67λ. The distance between layers is d ll = 40λ. The output FOV of the diffractive decoder 14 is 40λ away from the last diffractive layer 18 and has an area w d ×w d (Here w d is assumed to be 59.73λ.

[0083]

[0164] Diffraction decoding at the receiver involves sequentially modulating the received wave with L diffractive layers 18, followed by propagation through free space. We assume that the modulation of the incident light wave on the diffractive layers 18 is achieved passively by changing their height. The complex transmittance of the passive diffractive layers is TIFF2025527099000028.tif1220 is related to the height h(x,y) by the following formula:

[0165] TIFF2025527099000029.tif12130

[0084]

[0166] where n and k are the refractive index and extinction coefficient, respectively, TIFF2025527099000030.tif18106 are the amplitude and phase of the complex electric field transmittance, respectively. In the numerical simulations, unless otherwise specified, the diffractive layer 18 is assumed to be lossless, i.e., k=0, a-1.

[0085]

[0167] The propagation of an optical field through free space is modeled using the angular spectrum method, according to which the transformation of an optical field u(x,y) after propagating an axial distance d can be calculated as follows:

[0168] TIFF2025527099000031.tif16152

[0169] where TIFF2025527099000032.tif1225 is a two-dimensional Fourier (inverse Fourier) transform operator, TIFF2025527099000033.tif1331 is the free space transfer function for propagation over an axial distance d, defined as follows:

[0170] TIFF2025527099000034.tif30169

[0086]

[0171] In the numerical analysis, the optical field was sampled at intervals of δ ≈ 0.53λ along both the x and y directions, and the Fourier (inverse Fourier) transform was implemented using the fast Fourier transform (FFT) algorithm. In the lens-based imaging simulations reported in this paper, the plane-wave illumination is assumed to be amplitude modulated by an object placed in the transmitter aperture, the (thin) lens is placed in the same plane as the plane of the first diffractive layer 18 in the encoding-decoding scheme, and the diameter of the lens aperture is equal to the width of the diffractive layer, i.e., w l ≒106.67λ.

[0087]

[0172] training

[0173] The all-optical decoder network 14 is characterized by the latent variable h latent and the feature height h is h = According to TIFF2025527099000035.tif1753 latent Related to, where h max is a hyperparameter that indicates the maximum height. max Then, the corresponding maximum phase modulation Φ max The result was as follows: TIFF2025527099000036.tif1433TIFF2025527099000037.tif1236

[0088]

[0174] The parameters of the phase features of the encoder CNN 12 and the diffraction decoder 14 are optimized by minimizing the following loss function:

[0175] (1) TIFF2025527099000038.tif1555

[0176] where: TIFF2025527099000039.tif1219 is the pixel of the desired message m and the (scaled) decoded light intensity. is the mean squared error (MSE) between the pixels of TIFF2025527099000040.tif1329, i.e.

[0177] TIFF2025527099000041.tif17102

[0089]

[0178] The scaling factor σ is defined as: TIFF2025527099000042.tif2355

[0090]

[0180] An additive loss term scaled by the weight η TIFF2025527099000043.tif1539 is used to penalize models with low diffraction efficiency. DE is the diffraction efficiency and is calculated as follows: TIFF2025527099000044.tif2975

[0091]

[0181] The training data contained 110,000 examples, including 55,000 images from the MNIST training set and 55,000 custom-created images (see example in Figure 29). Of the 60,000 MNIST training images, the remaining 5,000, plus an additional 5,000 custom images, for a total of 10,000 images, were used for validation. After each epoch, the average loss on the validation images was calculated, and the model state corresponding to the minimum validation loss was selected as the final design.

[0092]

[0182] The digital model of the electronic encoder and diffraction decoder was implemented in TensorFlow version 2.4 using the Python programming language and trained on a machine equipped with an Intel® Core™ i7-8700 CPU @ 3.20 GHz and an NVIDIA GeForce GTX 1080Ti GPU. The Adam optimizer was used to minimize the loss function for 50 epochs with a batch size of 4. The training rate was initially 1e-3 and decreased by 0.99 every 10,000 optimization steps. For other parameters of the Adam optimizer, default TensorFlow settings were used. Training time varied depending on the model size; for example, training a diffraction decoder model with L = 3 took approximately 8 hours.

[0093]

[0183] Message m and scaled diffractive decoder output Native TensorFlow implementations of PSNR and SSIM were used to calculate image comparison metrics between TIFF2025527099000045.tif6170.

[0094]

[0184] Experimental design

[0185] In the experiment, the operating wavelength was λ=0.75 mm. A single substrate layer 18 was used for the all-optical decoder network 14, i.e., L=1 and N=200. 2There are 20 independent features of λ, each with a width of ∼0.53λ ≈ 0.40 mm, resulting in a diffractive layer of ∼80 mm × 80 mm. The width of the transmitter aperture containing the encoded phase message is w t ≒59.73λ≒44.8mm, and the width of the output FOV is w d The width of the shield is w o ≒32λ≒24mm. The distance from the transmitter aperture to the shielding surface is d to ≒13.33λ≒10 mm, and the diffraction layer 18 is ol ≈106.67λ≈80 mm. The output FOV is 40λ≈30 mm away from the diffractive layer 18.

[0095]

[0186] The diffractive layer 18 and the phase-encoded message (CNN output) were fabricated using a 3D printer (Objet30Pro, Stratasys Ltd). Similar to the implementation of the phase of the diffractive layer, the phase-encoded message was implemented by height variation according to the following method. TIFF2025527099000046.tif1643The height variations were applied on a uniform base thickness of 0.2 mm, which was used for mechanical support. Shielding was achieved by attaching aluminum to the 3D printed substrate (see Figure 9). The complex refractive index n+jk of the 3D printed material at λ=0.75 mm was measured to be 1.6518+j0.0612.

[0096]

[0187] During training of the experimental model, the weight η of the loss term related to the diffraction efficiency was set to zero. To make the experimental design robust to misalignment, random lateral and axial misalignments of the encoded object, occluder 32, and diffractive layer 18 were incorporated into the optical forward model during training. The random misalignments were modeled using the following uniformly distributed random variables: TIFF2025527099000047.tif8125This represents the displacement of the encoded object, occluder, and diffractive layer from their nominal positions in the x, y, and z directions, respectively.

[0097]

[0188] Terahertz experimental setup

[0189] Using a WR2.2 modular amplifier / multiplier chain (AMC) 50 in combination with a compatible diagonal horn antenna from Virginia Diodes Inc., a 10 dBm RF input signal is fed through an RF synthesizer 52 to f RF1 = 11.1111 GHz 36 times to generate 0.4 THz continuous wave (CW) radiation. The AMC output was filtered via signal generator 56 to resolve the low noise output data by lock-in detection in lock-in amplifier 54. M0D The horn antenna 58 was modulated at a rate of f = 1 kHz. The exit aperture of the horn antenna 58 was positioned ~60 cm from the input (encoded object) face of the 3D-printed all-optical decoder network 14 so that the incident THz wavefront was nearly planar. To detect the diffracted THz radiation at the output face, a single pixel Mixer / AMC 60, also from Virginia Diodes Inc., was used. To downconvert the detected signal to 1 GHz, a single pixel Mixer / AMC 60 was used via an RF synthesizer 62. RF1 A 10 dBm local oscillator signal at 11.0833 GHz was fed to the detector. The detector was mounted on a Thorlabs NRT100 XY positioning stage consisting of two linear motor stages, and the 0.5 × 0.1 mm detector was used to scan the output FOV with a 2 mm scan interval. The downconverted signal was amplified 40 dB using a Mini-Circuits ZRL-1150-LN+ cascaded low-noise amplifier 64 and passed through a 1 GHz (+ / - 10 MHz) bandpass filter 66 (KL Electronics 3C40-1000 / T10-O / O) to remove noise from unwanted frequency bands. The filtered signal was attenuated with a tunable attenuator (HP8495B) 68 for linearity calibration and detected with a low-noise power detector 70 (Mini-Circuits ZX47-60). The output voltage signal was read out using a lock-in amplifier (Stanford Research SR830), where f MODA modulation signal with a frequency of 1 kHz was used as the reference signal. The lock-in amplifier measurements were converted to a linear scale according to the calibration results. To increase the signal-to-noise ratio (SNR), 2 × 2 binning was applied to the THz measurements. The contrast of the measurements was digitally enhanced by saturating the top and bottom 1% of pixel values using the built-in function imadjust in MATLAB and mapping the resulting image to a dynamic range of 0–1.

[0098]

[0190] While embodiments of the present invention have been shown and described, various modifications can be made without departing from the scope of the present invention. For example, while the all-optical decoder network 14 is shown in a transmissive mode (light passing through the substrate layer 18), it should be understood that, as described herein, the all-optical decoder network 14 may include one or more substrate layers 18 that reflect light. Embodiments are contemplated that utilize both transmission and reflection in the all-optical decoder network 14. Furthermore, in some embodiments, multiple electronic encoder networks 12 may be used to generate a low-resolution modulation pattern or image 104. Thus, the system 10 may include one or more electronic encoder networks 12. Accordingly, the present invention should not be limited except by the claims and their equivalents.

Claims

1. 1. A system for displaying or projecting high resolution images, comprising: at least one electronic encoder network having a trained deep neural network configured to receive one or more high resolution images and generate a low resolution modulation pattern or image representative of the one or more high resolution images using one of a display, a projector, a screen, a spatial light modulator (SLM), or a wavefront modulator; 1. A system comprising: an all-optical decoder network having one or more optically transmissive and / or reflective substrate layers disposed in an optical path, each of said optically transmissive and / or reflective substrate layers having a plurality of physical features formed on or within said one or more optically transmissive and / or reflective substrate layers and having different transmission and / or reflection properties as a function of local coordinate across each substrate layer; wherein said one or more optically transmissive and / or reflective substrate layers and said plurality of physical features receive light arising from a low-resolution modulation pattern or image representing said one or more high-resolution images and optically generate a corresponding high-resolution image projection at an output field of view.

2. The system of claim 1 , wherein the low-resolution modulation pattern or image comprises a phase-only modulation, an amplitude-only modulation, or a complex-valued modulation.

3. 10. The system of claim 1, wherein the trained deep neural network comprises a trained convolutional neural network (CNN).

4. 10. The system of claim 1, wherein the trained deep neural network and a plurality of physical features formed on or in the one or more optically transmissive and / or reflective substrate layers are jointly trained.

5. The system of claim 1 , wherein the all-optical decoder network comprises a single optically transmissive substrate layer or a single reflective substrate layer.

6. The system of claim 1 , wherein the low-resolution modulation pattern or image comprises one of ultraviolet wavelengths, visible wavelengths, infrared wavelengths, or terahertz wavelengths.

7. The system of claim 1 , wherein the generated high-resolution image projection at the output field of view exhibits color information of the corresponding image.

8. The system of claim 1 , wherein the high-resolution image projection generated at the output field of view comprises a moving image.

9. The system of claim 1 , wherein one or more detectors, observation planes, surfaces, or eyes are positioned in the output field of view.

10. The system of claim 1 , wherein the all-optical decoder network is integrated into a wearable device, goggles, or glasses.

11. 1. A device for decoding high resolution images from low resolution modulation patterns or images representing one or more high resolution images, comprising: a device comprising: an all-optical decoder network including one or more optically transmissive and / or reflective substrate layers disposed in an optical path, each of the optically transmissive and / or reflective substrate layers having a plurality of physical features formed on or within the one or more optically transmissive and / or reflective substrate layers and having different transmission and / or reflection properties as a function of local coordinate across each substrate layer, the one or more optically transmissive and / or reflective substrate layers and plurality of physical features receiving a low-resolution modulation pattern or image representative of the one or more high-resolution images and optically generating a corresponding high-resolution image projection at an output field of view.

12. The device of claim 11 , wherein the all-optical decoder network is integrated into a wearable device, goggles, or glasses.

13. 1. A method for projecting a high resolution image onto a field of view, comprising: A device, at least one electronic encoder network including a trained deep neural network configured to receive one or more high resolution images and generate a low resolution modulation pattern or image representative of the one or more high resolution images using one or more of a display, a projector, a screen, a spatial light modulator (SLM), or a wavefront modulator; providing an all-optical decoder network having one or more optically transmissive and / or reflective substrate layers disposed in an optical path, each of said optically transmissive and / or reflective substrate layers having a plurality of physical features formed on or within said one or more optically transmissive and / or reflective substrate layers and having different transmission and / or reflection properties as a function of local coordinate across each substrate layer, said one or more optically transmissive and / or reflective substrate layers and said plurality of physical features receiving light arising from a low-resolution modulation pattern or image representing said one or more high-resolution images and optically generating a corresponding high-resolution image projection at an output field of view; inputting one or more high resolution images into the electronic encoder network to generate a low resolution modulation pattern or image representing the one or more high resolution images, and optically generating a corresponding high resolution image projection at the output field of view.

14. The method of claim 13 , wherein the low-resolution modulation pattern or image comprises a phase-only modulation, an amplitude-only modulation, or a complex-valued modulation.

15. 14. The method of claim 13, wherein the trained deep neural network comprises a trained convolutional neural network (CNN).

16. 14. The system of claim 13, wherein the trained deep neural network and the plurality of physical features formed on or in the one or more optically transmissive and / or reflective substrate layers are jointly trained.

17. The method of claim 13 , wherein the corresponding high resolution image projection in the output field of view is projected onto a viewing plane or surface or eyeball.

18. The method of claim 13 , wherein the generated high-resolution image projection at the output field of view exhibits color information of the corresponding image.

19. The method of claim 13 , wherein the high-resolution image projection generated at the output field of view comprises a moving image.

20. 1. A method of communicating information with one or more humans, comprising: transmitting a low-resolution modulation pattern or image representing one or more high-resolution images containing information using one or more of a display, projector, screen, spatial light modulator (SLM), or wavefront modulator; and all-optically decoding the low-resolution modulation pattern or image with one or more optically transmissive and / or reflective substrate layers disposed in an optical path, each of the optically transmissive and / or reflective substrate layers having a plurality of physical features formed on or within the one or more optically transmissive and / or reflective substrate layers and having different transmission and / or reflection properties as a function of local coordinate across each substrate layer, the one or more optically transmissive and / or reflective substrate layers and the plurality of physical features receiving light arising from the low-resolution modulation pattern or image representing the one or more high-resolution images and generating a corresponding high-resolution image projection containing the information at an output field of view.

21. 21. The method of claim 20, wherein the corresponding high resolution image projection in the output field of view is projected onto a viewing plane, surface, or eyeball.

22. 21. The method of claim 20, wherein the corresponding high-resolution image projection in the output field of view exhibits color information.

23. 21. The method of claim 20, wherein the corresponding high resolution image projection in the output field of view comprises a moving image.

24. 21. The method of claim 20, wherein the one or more optically transmissive and / or reflective substrate layers are integrated into a wearable device, goggles, or eyeglasses.

25. 1. A communications system for transmitting messages or signals in space, comprising: at least one electronic encoder network including a trained deep neural network configured to receive a message or signal and generate a phase-encoded and / or amplitude-encoded optical representation of the message or signal transmitted along an optical path; and an all-optical decoder network disposed in the optical path with the encoder network and including one or more optically transmissive and / or reflective substrate layers at least partially obscured and / or blocked by an opaque obstruction and / or diffusive medium, each of the optically transmissive and / or reflective substrate layers including a plurality of physical features formed on or within the one or more optically transmissive and / or reflective substrate layers and having different transmission and / or reflection properties as a function of local coordinate across each substrate layer, the one or more optically transmissive and / or reflective substrate layers and the plurality of physical features receiving secondary light waves scattered by the opaque obstruction and / or diffusive medium and optically generating the message or signal in an output field of view.

26. 26. The communication system of claim 25, wherein the phase-encoded and / or amplitude-encoded optical representation of the message or signal is transmitted at any of ultraviolet, visible, infrared, terahertz, or millimeter wavelengths.

27. 1. A device for decoding an encoded optical message or signal, comprising: an all-optical decoder network disposed in an optical path of the encoded optical message or signal and including one or more optically transmissive and / or reflective substrate layers at least partially obscured and / or blocked by an opaque obscuring and / or diffusive medium, each of the optically transmissive and / or reflective substrate layers including a plurality of physical features formed on or within the one or more optically transmissive and / or reflective substrate layers and having different transmission and / or reflection properties as a function of local coordinate across each substrate layer, the one or more optically transmissive and / or reflective substrate layers and a plurality of physical features receiving secondary light waves scattered by the opaque obscuring and / or diffusive medium and optically generating the message or signal in an output field of view.

28. 28. The device of claim 27, wherein the all-optical decoder network is integrated into a wearable device, goggles, or glasses.

29. 1. A method for transmitting a message or signal through space in the presence of an opaque barrier and / or a diffusive medium, comprising:

1. A system comprising: at least one electronic encoder network including a trained deep neural network configured to receive a message or signal and generate a phase-encoded and / or amplitude-encoded optical representation of the message or signal transmitted along an optical path; providing a system comprising: an all-optical decoder network including one or more optically transmissive and / or reflective substrate layers disposed in the optical path, each of the optically transmissive and / or reflective substrate layers including a plurality of physical features formed on or within the one or more optically transmissive and / or reflective substrate layers and having different transmission and / or reflection properties as a function of local coordinate across each substrate layer, the one or more optically transmissive and / or reflective substrate layers and a plurality of physical features receiving secondary light waves scattered by the opaque obstruction and / or diffusive medium and optically generating a message or signal in an output field of view; inputting one or more messages or signals into the electronic encoder network such that a phase-encoded and / or amplitude-encoded optical representation of the messages or signals is generated, and the messages or signals are optically generated at an output field of view.

30. 30. The method of claim 29, wherein the at least one electronic encoder network and the all-optical decoder network are jointly trained and optimized.

Images