Novel image reconstruction methods

Abstract

The present invention relates to novel image reconstruction methods and technologies. Embodiments of the invention specifically relate to image reconstruction using a flat lensless camera, or any other imaging device and / or imaging system, and a pre-trained diffusion model with a control network and a learned separable transformation. Additional embodiments of the invention include the addition of textual descriptions and / or text captions of the captured scene to further enhance image reconstruction.

Description

[0001]File No. FRAMO-P057-WO NOVEL IMAGE RECONSTRUCTION METHODS CROSS-REFERENCE TO RELATED APPLICATION This application claims priority to U.S. Provisional Application No.63 / 673,983, filed July 5 22, 2024, which is hereby incorporated by reference in its entirety. FIELD The present invention relates to novel image reconstruction methods. More specifically, the invention relates to flat camera image reconstruction using a strong image prior that relies on a 10 pre-trained diffusion model. BACKGROUND Cameras have become very popular and common in recent years, especially in small handheld devices. Despite that, reducing the size of the camera remains a difficult problem since 15 a camera requires lenses and optical elements to get a high-quality image. Flat cameras is a computational photography method to reduce camera size by replacing the camera lens with a diffuser (e.g., an amplitude mask placed very close to the sensor). Thus, the image on the sensor consists of multiplexed projections of the scene reflections on all the sensors area such that the captured image is not visually understandable. Using a computational algorithm, the scene image 20 can be retrieved. However, reconstructing high-quality images from flat camera measurements is still a difficult task. Known approaches have attempted to reconstruct the scene image using different techniques, including direct optimization and deep learning. Despite these attempts, the resulting images are not of sufficient quality. High-quality image reconstruction from flat camera 25 measurements is still not achievable using known methods, systems, and / or devices, with better algorithms needed to reproduce better images. SUMMARY It is to be understood that both the following summary and the detailed description are 30 exemplary and explanatory and are intended to provide further explanation of the invention as File No. FRAMO-P057-WO claimed. Neither the summary nor the descriptions that follow is intended to define or limit the scope of the invention to the particular features mentioned in the summary or in the description. Rather, the scope of the invention is defined by the appended claims. In certain embodiments, the disclosed embodiments may include one or more of the features described herein. 5 In at least one embodiment, a method for image reconstruction is disclosed. The method comprises inputting flat lensless camera measurements into a separable transformation, followed by an adapter or network (e.g., a ControlNet adapter), which controls image generation via a diffusion model. The reconstructed image may be achieved by, for instance, T iterative diffusions. In at least one example, the separable transformation comprises a separable linear transformation. 10 In at least one example, the diffusion model is a pre-trained latent diffusion model (LDM). In at least one example, the method further comprises inputting text guidance and / or text captions. In at least one example, training of the diffusion model comprises optimizing weights in the separable transformation and adapter / network. In at least one example, weights in the diffusion 15 model are pre-trained and fixed. In at least one example, the dataset for training and testing is captured using a prototype flat camera by projecting images on to a screen. The flat camera may be used to capture a measurement image, which is not visually understandable. An image is then reconstructed, e.g., using any method described herein, from the measurements. The reconstructed image may be 20 compared to a reference image. In at least one example, the method comprises using a separation loss value and / or a diffusion training loss value. In at least one embodiment, a pre-trained image generation model is used, which is trained on a large amount of images. Additionally, a strong prior may be used for natural images 25 perceptually, facilitating the reconstruction of state-of-the-art quality images from flat camera measurements. At least one embodiment of the invention uses lensless imaging, which can be a more compact, lighter, and more cost-effective imaging system than other types of imaging systems and / or methods. In at least one example, an imaging arrangement is used that includes a static 30 amplitude mask placed near the sensor. The mask pattern may be designed in a separable manner File No. FRAMO-P057-WO such that the linear imaging model can be simplified to a separable operation. In at least one example, a separable pattern obtained from an outer product of M-sequence binary signals in the length of 255 was used for the amplitude mask. In at least one example, the mask is printed by lithography on a chrome plate on glass with a thickness of about 0.2 mm. Each feature in the 5 pattern is a single pinhole having a size of about 25 µm. At least one embodiment of the invention uses one or more data-driven algorithms that incorporate deep learning to reconstruct high-quality images from multiplexed measurements. In at least one example, learned separable transforms are used, followed by an architecture (e.g., Unet) that perceptually enhances the resulting image in a generative adversarial network (GAN) 10 approach. In at least one embodiment, the diffusion model is a parameterized Markov chain that produces samples of a certain data distribution after a predefined T number of steps. In each step of the forward diffusion process, a Gaussian noise is added to the data, such that after a large number of steps, the sample is mapped to an isotropic Gaussian. In the reverse step, a deep neural 15 network is trained to denoise the clean image from the noisy sample. In at least one example, to sample an image, T steps are applied in the reverse direction starting with pure Gaussian noise, which is subsequently denoised until a clean, natural image is obtained from the learned distribution of the data. At least one embodiment of the invention utilizes a latent diffusion model (LDM), which 20 is a method for applying the diffusion process in a latent space instead of the pixel space, thereby obtaining a more computationally efficient model for high-resolution images. In at least one example, a pre-trained auto-encoder is used, which comprises encoder E and decoder D, for training a diffusion model in a low-dimension (latent) space. In at least one embodiment, the adapter (e.g., ControlNet) adds an image encoder to control 25 the diffusion process. This control network is initiated as the pretrained encoder of the diffusion model (e.g., UNet) with additional zero convolutions. The output features of the control network are then added to the pre-trained diffusion model features and, due to the zero convolutions, the initial performance of the diffusion model is not degraded. After training, ControlNet weights are optimized such that the added control network features manipulate the diffusion process for the 30 target task. File No. FRAMO-P057-WO In at least one embodiment, the image acquired by the flat camera comprises multiplexed measurements of the light reflected from the scene across the sensor area. The light projections on the sensor create long-range correspondences in the captured images. To convert the image from the projections (that is, ”projection space”) to the pixel space of the target image, the at least one 5 embodiment applies a learned separable linear transformation to the measurements. This transformation is crucial since the input image serves as guidance for the diffusion model process. The model may, in at least one example, be ignorant about the flat camera mask projections on the image. In at least one example, a control network is used to guide the reconstruction process. In at least one embodiment, a pre-trained diffusion model is used as an image prior. The 10 model is a strong prior for natural images since the model is trained on a large number of samples for image generation tasks. In at least one embodiment, the generation process of the diffusion model is controlled so that the model can generate the captured scene from the measurements of the flat camera. For instance, a network (e.g., ControlNet network Cψ) can be trained. The network is initialized as a 15 copy of the encoder of the diffusion model (e.g., a UNet model) with zero convolutions, such that the pre-trained weights performance is not affected and that, during training, non-zero weights are learned for the reconstruction task. The input of the control network is the output of the separable transform (Co) and, and the control network loss can be applied for both ψ, the set of parameters of the control network Cψ, and the learned separable transform weights. The diffusion model 20 parameters θ can be pre-trained and fixed. In at least one embodiment, to guide the training for better results, a separable reconstruction loss term is added to the diffusion conventional loss. This loss is applied to the output of the separable transformation. In at least one embodiment, the diffusion model used is a text-guided model for image 25 generation. Thus, this text can be input into a reconstruction algorithm. In at least one embodiment, the separable transform and control network are trained in a supervised way. Accordingly, pairs of RGB images are used with their corresponding measurements of the flat camera. In at least one example, the flat camera imaging process is simulated to obtain realistic measurements. 30 In at least one embodiment, to obtain a large dataset using the flat camera, images were File No. FRAMO-P057-WO captured from one or more datasets (e.g., the LAION-aesthetics dataset, which includes a large amount of high-resolution images and their corresponding textual captions). The dataset images were projected on a screen (e.g., a personal computer (PC) screen having a size of 34 cm x 34 cm) at a distance (e.g., about 60 cm) from the camera. After exposure (e.g., about 12 ms exposure time), 5 the raw Bayer pattern images were saved (e.g., in 12-bit depth). In at least one example, to compensate for stray light, a black screen with no image projected on it was captured. In the post- processing stage, the measured black levels were then subtracted from the captured measurements. The novel methods described herein can be evaluated using several image quality and text similarity metrics. The results of such methods can be compared to ground-truth images using, for 10 instance, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and Learned Perceptual Image Patch Similarity (LPIPS) metrics. Since a text-guidance approach is also described herein, a CLIP score can also be measured and compared to a ground-truth text description of the captured scene. At least one embodiment of the invention performs better than known methods (e.g., 15 FlatNet, Tikhonov et al.), as measured by one or more of the metrics described above herein. In at least one embodiment, text guidance and / or text captions are added to the image reconstruction process. Without text input, the image reconstruction is driven by the flat camera measurements only. In the at least one embodiment, text captions result in the reconstructed image having details that are more similar to the true captured scene. The text captions also contribute to 20 the high frequency details in the reconstructed images. Further, any one or more of the methods described herein, and / or any one or more steps thereof, may be executed, in whole or in part, on any of the computing devices and / or computing systems described herein. For instance, computing systems are described for performing image reconstruction. 25 In at least one embodiment, any one or more of the methods described herein further includes transmitting (e.g., by at least one processor) an alert to at least one computing device, the alert containing one or more reconstructed images. Therefore, based on the foregoing and continuing description, the subject invention in its various embodiments may comprise one or more of the following features in any non-mutually- 30 exclusive combination: File No. FRAMO-P057-WO • A system comprising a lensless camera to capture data comprising raw sensor multiplexed measurements of light reflected from a scene across a sensor area, a memory storing computer-readable instructions, and at least one processor to execute the instructions to reconstruct an output image from the raw sensor 5 multiplexed measurements using a deep neural network to denoise and generate the output image from the raw sensor multiplexed measurements; • The deep neural network being trained to control a latent diffusion model; • The deep neural network being trained to control the latent diffusion model using a library of images comprising pairs of a source image and corresponding raw sensor 10 multiplexed measurements of light reflected from a display of the source image across a sensor area; • The at least one processor further to execute the instructions to train the deep neural network to control the latent diffusion model using the library of images and corresponding measurements; 15 • The at least one processor further to execute the instructions to train the deep neural network using a large language model (LLM) that provides automatic image captioning for each pair of the source image and corresponding measurements; • The at least one processor further to execute the instructions to receive input text for the scene and reconstruct the output image from the raw sensor multiplexed 20 measurements using the deep neural network and the input text; • The deep neural network being initialized with a copy of an encoder for the latent diffusion model; • The deep neural network having a separable reconstruction loss term, which is applied to an output of a separable transformation and added to a diffusion 25 conventional loss term; • The camera comprising a diffuser comprising an amplitude mask; • The source image comprising one of an RGB image, a depth map, and polarization data; • A method comprising capturing, by a lensless camera, data comprising raw sensor 30 multiplexed measurements of light reflected from a scene across a sensor area, and File No. FRAMO-P057-WO reconstructing, by at least one processor, an output image from the raw sensor multiplexed measurements by applying a deep neural network to denoise and generate the output image from the raw sensor multiplexed measurements; • The deep neural network being trained to control a latent diffusion model; 5 • The deep neural network being trained to control the latent diffusion model using a library of images comprising pairs of a source image and corresponding raw sensor multiplexed measurements of light reflected from a display of the source image across a sensor area; • The deep neural network being trained to control the latent diffusion model using 10 the library of images and corresponding measurements; • The deep neural network being trained using a LLM that provides automatic image captioning for each pair of the source image and corresponding measurements; • The method further comprising receiving, by the at least one processor, input text for the scene; 15 • The method further comprising reconstructing, by the at least one processor using the deep neural network and the input text, the output image from the raw sensor multiplexed measurements; • The method further comprising initializing, by the at least one processor, the deep neural network with a copy of an encoder for the latent diffusion model; 20 • The deep neural network having a separable construction loss term, which is applied to an output of a separable transformation and added to a diffusion conventional loss term; • The camera comprising a diffuser comprising an amplitude mask; • The source image comprising one of an RGB image, a depth map, and polarization 25 data; • A non-transitory computer-readable storage medium, having instructions stored thereon that, when executed by a computing device, cause the computing device to perform operations, the operations comprising obtaining data comprising raw sensor multiplexed measurements of light reflected from a scene across a sensor 30 area; and reconstructing an output image from the raw sensor multiplexed File No. FRAMO-P057-WO measurements by applying a deep neural network to denoise and generate the output image from the raw sensor multiplexed measurements; • The data being captured by a lensless camera; • The deep neural network being trained to control a latent diffusion model; 5 • The deep neural network being trained to control the latent diffusion model using a library of images comprising pairs of a source image and corresponding raw sensor multiplexed measurements of light reflected from a display of the source image across a sensor area; • The deep neural network being trained to control the latent diffusion model using 10 the library of images and corresponding measurements; • The deep neural network being trained using a LLM that provides automatic image captioning for each pair of the source image and corresponding measurements; • The operations further comprising receiving input text for the scene; • The operations further comprising reconstructing, by the deep neural network and 15 the input text, the output image from the raw sensor multiplexed measurements; • The operations further comprising initializing, by a copy of an encoder for the latent diffusion model, the deep neural network; • The deep neural network having a separable construction loss term, which is applied to an output of a separable transformation and added to a diffusion 20 conventional loss term; • The camera comprising a diffuser comprising an amplitude mask; and • The source image comprising one of an RGB image, a depth map, and polarization data. These and further and other objects and features of the invention are apparent in the 25 disclosure, which includes the above and ongoing written specification, as well as the drawings. BRIEF DESCRIPTION OF THE DRAWINGS The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate exemplary embodiments and, together with the description, further serve to 30 enable a person skilled in the pertinent art to make and use these embodiments and others that will File No. FRAMO-P057-WO be apparent to those skilled in the art. The invention will be more particularly described in conjunction with the following drawings wherein: Figure 1 is a diagram illustrating an overview of a novel image reconstruction method, according to at least one embodiment of the present invention. 5 Figures 2A-2D show a prototype flat camera (Figure 2A) for capturing measurement images, an example of such a measurement image (Figure 2B), a reconstructed image from the measurement image of Figure 2B (Figure 2C), and a reference image to which the reconstructed image can be compared (Figure 2D), according to at least one embodiment of the present invention. Figures 3A-3B show a non-limiting example of a specific mask pattern used with a flat 10 camera (Figure 3A), and the measured camera point spread function that is achieved with the mask of Figure 3A attached to the sensor hot mirror (Figure 3B), according to at least one embodiment of the present invention. Figure 4 shows non-limiting examples of images generated using one or more of the image reconstruction methods described herein, according to at least one embodiment of the present 15 invention. Figure 5 shows image comparisons between known image reconstruction methods and one or more of the image reconstruction methods described herein, according to at least one embodiment of the present invention. Figure 6 shows image comparisons between known image reconstruction methods and one 20 or more of the image reconstruction methods described herein, where the images are reconstructed from real scenes, according to at least one embodiment of the present invention. Figure 7 shows image reconstructions of real scenes using a prototype flat camera described herein, according to at least one embodiment of the present invention. Figure 8 shows visual ablation images, according to at least one embodiment of the present 25 invention. Figure 9 is a block diagram of a computing system for performing image reconstruction, according to at least one embodiment of the present invention. Figure 10 is a block diagram of a computing device, according to at least one embodiment of the present invention. 30 Figure 11 shows an example of a system for implementing certain aspects of the present File No. FRAMO-P057-WO technology. Figure 12 shows a further example of a system for implementing certain aspects of the present technology. 5 DETAILED DESCRIPTION This specification discloses one or more embodiments that incorporate features of the invention. The following description is exemplary in that several embodiments are described (e.g., by use of the terms “preferably,” “for example,” or “in one embodiment”); however, such should not be viewed as limiting or as setting forth the only embodiments of the present invention, as the 10 invention encompasses other embodiments not specifically recited in this description, including alternatives, modifications, and equivalents within the spirit and scope of the invention. Further, the use of the terms “invention,” “present invention,” “embodiment,” and similar terms throughout the description are used broadly and not intended to mean that the invention requires, or is limited to, any particular aspect being described or that such description is the only manner in which the 15 invention may be made or used. Additionally, the invention may be described in the context of specific applications; however, the invention may be used in a variety of applications not specifically described. The embodiment(s) described, and references in the specification to “one embodiment”, “an embodiment”, “an example embodiment”, etc., indicate that the embodiment(s) described may 20 include a particular feature, structure, or characteristic. Such phrases are not necessarily referring to the same embodiment. When a particular feature, structure, or characteristic is described in connection with an embodiment, persons skilled in the art may effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described. In the several figures, like reference numerals may be used for like elements having like 25 functions even in different drawings. The embodiments described, and their detailed construction and elements, are merely provided to assist in a comprehensive understanding of the invention. Thus, it is apparent that the present invention can be carried out in a variety of ways, and does not require any of the specific features described herein. Also, well-known functions or constructions are not described in detail since they would obscure the invention with unnecessary detail. Any 30 signal arrows in the drawings / figures should be considered only as exemplary, and not limiting, File No. FRAMO-P057-WO unless otherwise specifically noted. Further, the description is not to be taken in a limiting sense, but is made merely for the purpose of illustrating the general principles of the invention, since the scope of the invention is best defined by the appended claims. It will be understood that, although the terms first, second, etc. may be used herein to 5 describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. Purely as a non-limiting example, a first element could be termed a second element, and, similarly, a second element could be termed a first element, without departing from the scope of example embodiments. As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items. As used 10 herein, “at least one of A, B, and C” indicates A or B or C or any combination thereof. As used herein, the singular forms "a", "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It should also be noted that, in some alternative implementations, the functions and / or acts noted may occur out of the order as represented in at least one of the several figures. Purely as a non-limiting example, two figures shown in succession 15 may in fact be executed substantially concurrently or may sometimes be executed in the reverse order, depending upon the functionality and / or acts described or depicted. As used herein, ranges are used herein in shorthand, so as to avoid having to list and describe each and every value within the range. Any appropriate value within the range can be selected, where appropriate, as the upper value, lower value, or the terminus of the range. 20 “About” means a referenced numeric indication plus or minus 10% of that referenced numeric indication. For example, the term “about 4” would include a range of 3.6 to 4.4. All numbers expressing quantities of ingredients, reaction conditions, and so forth used in the specification are to be understood as being modified in all instances by the term “about.” Accordingly, unless indicated to the contrary, the numerical parameters set forth herein are 25 approximations that can vary depending upon the desired properties sought to be obtained. At the very least, and not as an attempt to limit the application of the doctrine of equivalents to the scope of any claims, each numerical parameter should be construed in light of the number of significant digits and ordinary rounding approaches. The words “comprise,” “comprises,” and “comprising” are to be interpreted inclusively 30 rather than exclusively. Likewise, the terms “include,” “including,” and “or” should all be File No. FRAMO-P057-WO construed to be inclusive, unless such a construction is clearly prohibited from the context. The terms “comprising” or “including” are intended to include embodiments encompassed by the terms “consisting essentially of” and “consisting of.” Similarly, the term “consisting essentially of” is intended to include embodiments encompassed by the term “consisting of.” Although having 5 distinct meanings, the terms “comprising,” “having,” “containing,” and “consisting of” may be replaced with one another throughout the description of the invention. Conditional language, such as, among others, “can,” “could,” “might,” or “may,” unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, 10 certain features, elements and / or steps. Thus, such conditional language is not generally intended to imply that features, elements and / or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether these features, elements and / or steps are included or are to be performed in any particular embodiment. 15 Wherever the phrase “for example,” “such as,” “including” and the like are used herein, the phrase “and without limitation” is understood to follow unless explicitly stated otherwise. “Typically” or “optionally” means that the subsequently described event or circumstance may or may not occur, and that the description includes instances where said event or circumstance occurs and instances where it does not. 20 In general, the word “instructions,” as used herein, refers to logic embodied in hardware or firmware, or to a collection of software units, possibly having entry and exit points, written in a programming language, such as, but not limited to, Python, R, Rust, Go, SWIFT, Objective-C, Java, JavaScript, Lua, C, C++, or C#. A software unit may be compiled and linked into an executable program, installed in a dynamic link library, or may be written in an interpreted 25 programming language such as, but not limited to, Python, R, Ruby, JavaScript, or Perl. It will be appreciated that software units may be callable from other units or from themselves, and / or may be invoked in response to detected events or interrupts. Software units configured for execution on computing devices by their hardware processor(s) may be provided on a computer readable medium, such as a compact disc, digital video disc, flash drive, magnetic disc, or any other tangible 30 medium, or as a digital download (and may be originally stored in a compressed or installable File No. FRAMO-P057-WO format that requires installation, decompression or decryption prior to execution). Such software code may be stored, partially or fully, on a memory device of the executing computing device, for execution by the computing device. Software instructions may be embedded in firmware, such as an EPROM. It will be further appreciated that hardware modules may be comprised of connected 5 logic units, such as gates and flip-flops, and / or may be comprised of programmable units, such as programmable gate arrays or processors. Generally, the instructions described herein refer to logical modules that may be combined with other modules or divided into sub-modules despite their physical organization or storage. As used herein, the term “computer” is used in accordance with the full breadth of the term as understood by persons of ordinary skill in the art and includes, 10 without limitation, desktop computers, laptop computers, tablets, servers, mainframe computers, smartphones, handheld computing devices, and the like. In this disclosure, references are made to users performing certain steps or carrying out certain actions with their client computing devices / platforms. In general, such users and their computing devices are conceptually interchangeable. Therefore, it is to be understood that where 15 an action is shown or described as being performed by a user, in various implementations and / or circumstances the action may be performed entirely by the user’s computing device or by the user, using their computing device to a greater or lesser extent (e.g. a user may type out a response or input an action, or may choose from preselected responses or actions generated by the computing device). Similarly, where an action is shown or described as being carried out by a computing 20 device, the action may be performed autonomously by that computing device or with more or less user input, in various circumstances and implementations. In this disclosure, various implementations of a computer system architecture are possible, including, for instance, thin client (computing device for display and data entry) with fat server (cloud for app software, processing, and database), fat client (app software, processing, and 25 display) with thin server (database), edge-fog-cloud computing, and other possible architectural implementations known in the art. Generally, the present disclosure is directed towards novel image reconstruction methods. In particular, a novel method for flat camera image construction is described in at least one embodiment. The method may, in at least one example, use a strong image prior that relies on a 30 pre-trained diffusion model. File No. FRAMO-P057-WO Figure 1 provides an overview of at least one example of the novel method described herein. Specifically, in method 100, flat lensless camera measurements 102 are input into a separable transformation 104 (e.g., a separable linear transformation), followed by an adapter 106 (e.g., a ControlNet adapter), which controls image generation via a diffusion model 108 (e.g., a 5 pre-trained latent diffusion model (LDM)). Optionally, text guidance 110 is inputted into the reconstruction as well. During training, weights in the separable transformation 104 and the adapter 106 are optimized, with pathways 112, 114, 128, 130, 132, and 134 being training losses. Weights in the diffusion model 108 are pre-trained and fixed. In at least one example, the dataset for training and testing was captured using a prototype flat camera by projecting images on to a 10 screen. An additional loss 116 (lsep) is shown, in addition to the diffusion training loss 118 (lC) for increased convergence. Also shown is target image 120, encoder 122, and decoder 124, which will be described in further detail below. The reconstructed image 126 is achieved by T iterative diffusions. An additional convolution layer 117 is also shown, which, in at least one example, has learnable parameters added to enable application of the aforementioned lseploss 116. The layer 15 117 can perform a light transformation to the data before comparison to the target image 120. In at least one embodiment, a pre-trained image generation model is used, which is trained on a large amount of images. Additionally, a strong prior may be used for natural images perceptually, facilitating the reconstruction of state-of-the-art quality images from flat camera measurements. Figure 2A shows a prototype flat camera 200, as described in further detail herein. 20 Specifically shown is a connector (e.g., a Universal Serial Bus (USB) connector) 202, a flat camera made up of three layers 204, 206, and 208, and a mask holder 210. The mask itself is positioned within the hole 212 at the center of the holder. The flat camera 200 may be used to capture measurement image 220, which is not visually understandable (Figure 2B). An image 240 is then reconstructed, e.g., using any method described herein, from the measurements, as shown in 25 Figure 2C. The reconstructed image 240 may be compared to a reference image 260 (Figure 2D), as described in further detail herein. In at least one embodiment, the diffusion model includes a text-guidance property for generating an even further improved image reconstruction (e.g., via a description of the captured scene, which may be provided by a user or photographer). 30 In at least one embodiment, stable diffusion is used and an adapter (e.g., ControlNet) is File No. FRAMO-P057-WO trained for the aforementioned measurement image from the flat camera. Comparisons between the flat camera prototype and existing datasets known in the art are described below herein. At least one embodiment of the invention uses lensless imaging, which can be a more compact, lighter, and more cost-effective imaging system than other types of imaging systems 5 and / or methods. In at least one example, an imaging arrangement is used that includes a static amplitude mask placed near the sensor. The mask pattern may be designed in a separable manner such that the linear imaging model can be simplified to a separable operation, specifically: Y = ΦLXΦR, (Equation 1) 10 where ΦL and ΦR are the separable operations on the image’s rows and columns, respectively, X is the scene intensity, and Y represents the sensor measurements. In at least one embodiment, the measurements of the camera comprise multiplexed projections of the scene reflectance such that reconstruction becomes an ill-posed problem. Known 15 methods for image reconstruction from flat camera measurements were designed as an optimization problem, and such methods are heavily dependent on the imaging model and rely on accurate calibration, leading to low-quality image reconstructions. At least one embodiment of the invention overcomes the aforementioned limitations by using one or more data-driven algorithms that incorporate deep learning to reconstruct high-quality 20 images from multiplexed measurements. In at least one example, learned separable transforms are used, followed by an architecture (e.g., Unet) that perceptually enhances the resulting image in a generative adversarial network (GAN) approach. As described herein, at least one embodiment utilizes one or more diffusion models, which are generative models that can learn a natural image distribution, which is then used for image 25 generation, segmentation, inpainting, image super-resolution, and / or general image reconstruction. Generally, a diffusion model is a parameterized Markov chain that produces samples of a certain data distribution after a predefined T number of steps. In each step of the forward diffusion process, a Gaussian noise is added to the data, such that after a large number of steps, the sample is mapped to an isotropic Gaussian. In the reverse step, a deep neural network is trained to denoise the clean 30 image from the noisy sample. In at least one example, to sample an image, T steps are applied in File No. FRAMO-P057-WO the reverse direction starting with pure Gaussian noise, which is subsequently denoised until a clean, natural image is obtained from the learned distribution of the data. At least one embodiment of the invention utilizes a latent diffusion model (LDM), which is a method for applying the diffusion process in a latent space instead of the pixel space, thereby 5 obtaining a more computationally efficient model for high-resolution images. In at least one example, a pre-trained auto-encoder is used, which comprises encoder E and decoder D, for training a diffusion model in a low-dimension (latent) space. In at least one example, data samples can be denoted as x ∼ q(x), where q is the data distribution learned, and the latent sample is obtained by the encoder as z = E(x). A denoiser 10 network ^θ is then trained to predict the added noise (at the backward diffusion step) from the noisy sample zt at the timestamp t ∈ [0,...,T]. With an additional text input y for text guidance in image generation, the training loss for the LDM is: lldm = EE(x),^∼N(0,I),t∼U(0,T)h||^ − ^θ(zt,t,y)||i. (Equation 2) 15 In Equation 2, x is a real image sample from the dataset, E(x) is the encoder that maps the image x to a latent representation z of the image (that is, z = E(x) as described above herein). Epsilon (^) is random Gaussian noise sampled from a standard normal distribution N(0,I). The variable t is a diffusion timestep randomly sampled from a uniform distribution over [0,T]. The 20 variable zt is the noisy latent variable at time t, which is produced by adding the noise ^ to z in the forward diffusion process. The variable y is a text input used to guide the image generation process. The variable ^θ is the denoising network parameterized by θ, while ^θ(zt,t,y) is the output of the denoising neural network that predicts the noise added at step t, conditioned on the noisy latent zt, timestep t, and text y. The variable E[…] is the expectation (e.g., average) over samples from the 25 data distribution, Gaussian noise, and diffusion steps. Accordingly, lldm is the loss function used to train the latent diffusion model, which minimizes the difference between the true noise ^ and the predicted noise ^θ. In at least one embodiment, the adapter (e.g., ControlNet) adds an image encoder to control the diffusion process. This adapter can enable controllable generation such as, for instance, 30 generating an image of a cat in a specific pose, or following the lines of a sketch of a specific cat. File No. FRAMO-P057-WO Accordingly, the adapter can also be referred to herein as a “control network,” which is initiated as the pretrained encoder of the diffusion model (e.g., UNet) with additional zero convolutions. The output features of the control network are then added to the pre-trained diffusion model features and, due to the zero convolutions, the initial performance of the diffusion model is not 5 degraded. After training, ControlNet weights are optimized such that the added control network features manipulate the diffusion process for the target task. As described herein, a flat camera is used in at least one embodiment. A separable pattern obtained from an outer product of M-sequence binary signals in the length of 255 was used for the amplitude mask. An M-sequence, also known as a Maximum Length Sequence, can be generated 10 and initialized as known in the art. In at least one example, the mask is printed by lithography on a chrome plate on glass with a thickness of about 0.2 mm. Each feature in the pattern is a single pinhole having a size of about 25 µm. Figure 3A shows a non-limiting example of a specific mask pattern 300, while Figure 3B shows the measured camera point spread function (PSF) 350 that is achieved with the mask of Figure 3A attached to the sensor hot mirror. 15 In at least one embodiment, the image acquired by the flat camera comprises multiplexed measurements of the light reflected from the scene across the sensor area (e.g., measurements 102). The light projections on the sensor create long-range correspondences in the captured images. To convert the image from the projections (that is, ”projection space”) to the pixel space of the target image, the at least one embodiment applies a learned separable linear transformation (e.g., 20 transformation 104) to the measurements. This transformation is crucial since the input image serves as guidance for the diffusion model process (e.g., model 108, which is trained and worked in the domain of natural images). The model may, in at least one example, be ignorant about the flat camera mask projections on the image. In at least one example, a control network is used to guide the reconstruction process, which may exhibit better results when operating in the image 25 pixel domain rather than the long-range projections. In at least one embodiment, the RAW image captured by the camera (e.g., in RGGB Bayer pattern) is split into four color channels: R, Gr, Gb and B. Each channel Ck is linearly transformed as: (Equation 3) File No. FRAMO-P057-WO where k ∈ [R,Gr,Gb,B], each color channel Ck of size hi × wi and ϕl ∈ Rho×hiand ϕr ∈ Rwi×woare two learnable matrices. In at least one example, output features are stacked onto a single four-channel tensor Co ∈ R4×ho×wo. 5 In at least one embodiment, a pre-trained diffusion model is used as an image prior. The model is a strong prior for natural images since the model is trained on a large number of samples for image generation tasks. In at least one example, the generation process of the diffusion model is controlled so that the model can generate the captured scene from the measurements of the flat camera. For instance, 10 a network (e.g., ControlNet network Cψ 106) can be trained. The network is initialized as a copy of the encoder 122 of the diffusion model 108 (e.g., a UNet model) with zero convolutions, such that the pre-trained weights performance is not affected and that, during training, non-zero weights are learned for the reconstruction task. The input of the control network is the output of the separable transform (Co), and the control network loss is lc 218, which is defined as: 15 lC = EE(x),^∼N(0,I),t∼U(0,T)h||^ − ^θ(zt,t,y,Cψ(Co))||2i, (Equation 4) where the variables in Equation 4 are the same as described above herein with respect to Equation 2. The network loss can be applied for both ψ, the set of parameters of the control network Cψ, and 20 the learned separable transform weights. The diffusion model parameters θ can be pre-trained and fixed. In at least one example, to guide the training for better results, a separable reconstruction loss term is added to the diffusion conventional loss (Equation 4). This loss is applied to the output of the separable transformation as lsep116, which is defined as: 25 lsep = ||I − fconv(Co)||2, (Equation 5) where fconv is a learned 3 × 3 convolution layer (e.g., convolution 117) which maps the 4-channels Coto a 3-channels image, and I is the target image 120 (or ground-truth image) in RGB channels 30 format. Further discussion of lsep will be provided below herein. File No. FRAMO-P057-WO In at least one example, the diffusion model used (e.g., model 108) is a text-guided model for image generation. Thus, text can be input into a reconstruction algorithm. In at least one embodiment, the separable transform (e.g., transform 104) and control network (e.g., network 106) are trained in a supervised way. Accordingly, pairs of RGB images 5 are used with their corresponding measurements of the flat camera. In at least one example, the flat camera imaging process is simulated to obtain realistic measurements. In at least one example, to obtain a large dataset using the flat camera, images were captured from one or more datasets (e.g., the LAION-aesthetics dataset, which includes a large amount of high-resolution images and their corresponding textual captions). The dataset images 10 were projected on a screen (e.g., a personal computer (PC) screen having a size of 34 cm x 34 cm) at a distance (e.g., about 60 cm) from the camera. After exposure (e.g., about 12 ms exposure time), the raw Bayer pattern images were saved (e.g., in 12-bit depth). In total, about 55,000 images were captured. About 500 images were saved for testing, while the remainder were used for training. In at least one example, to compensate for stray light, a black screen with no image projected on it 15 was captured. In the post-processing stage, the measured black levels were then subtracted from the captured measurements. In addition to the screen images, real objects were also captured to show that the novel methods described herein are not restricted only to objects on dataset images projected on a screen. In at least one example, the diffusion model is pre-trained using stable diffusion (e.g., 20 stable-diffusion 2.1) for about 500,000 steps on the captured dataset, using the losses in Equation 2 and Equation 4, and a learning rate of 5 · 10−5with, e.g., the AdamW optimizer. The novel methods described herein can be evaluated using several image quality and text similarity metrics. The results of such methods can be compared to ground-truth images using, for instance, Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index Measure (SSIM), and 25 Learned Perceptual Image Patch Similarity (LPIPS) metrics. Since a text-guidance approach is also described herein, a CLIP (Contrastive Language-Image Pretraining) score can also be measured and compared to a ground-truth text description of the captured scene. Such a score evaluates the similarity of an image to a text description. In at least one example, the latest and most accurate model (ViT-L / 14@336px) was used. 30 In at least one example, the novel methods described herein achieved a PSNR of about File No. FRAMO-P057-WO 21.89, an SSIM of about 0.541, a LPIPS of about 0.276, and a CLIP score of about 24.38. Figure 4 presents examples of these image reconstruction results. Specifically, sets of images 402, 404, 406, and 408 are shown for scenes 410, 412, 414, and 416. Images 402 are the prototype measurements, images 404 show the results of the novel image reconstruction without text 5 guidance, images 406 show the results of the novel image reconstruction with text guidance, and images 408 are the ground truth images. Scene 410 is described as “a large dining room table with chairs around it, a fireplace, and a hanging light.” Scene 412 is described as “the Bridal Veil waterfall, with a large rock and a tree in the background.” Scene 414 is described as “a landscape with a river, trees, and a cabin in the woods.” Scene 416 is described as “two men riding horses in 10 a fenced area.” In at least one embodiment, the novel image reconstruction described herein was compared to known methods (e.g., FlatNet, Tikhonov et al.) by using the same dataset as those methods. In at least one example, the results of the novel image reconstruction methods described herein were compared to FlatNet by, for instance, training the methods on the published FlatNet dataset. This 15 dataset includes about 10,000 images for training and about 100 for testing, and does not contain text captions for the images. Thus, to train the novel method with text guidance, a large language model (LLM) was used for the auto image captioning process. In at least one example, the llava1.5 LLM was used, and text captions for all images in the dataset were generated. To prevent the problem of incorrect captions, which is also known as “LLM hallucinations,” the generated 20 captions were verified for accuracy. In at least one example, the model was trained for about 700,000 steps with a similar optimizer setup as descried above herein. In at least one example, the prototype camera was an Allied Vision 1800 U-500 board-level camera with a pixel size of 2.2 µm and 5 megapixels. Method PSNR SSIM LPIPS CLIP Tikhonov et al. 10.98 0.318 0.736 16.76 FlatNet-R 18.58 0.487 0.332 21.53 FlatNet-L 18.64 0.518 0.322 21.68 Novel method 20.25 0.601 0.242 22.81 (without text) File No. FRAMO-P057-WO Novel method 20.20 0.604 0.238 23.50 (with text) Novel method 20.43 0.612 0.237 23.53 (fine-tuned with text) Table 1: Comparison between novel image reconstruction methods and known methods As can be seen in Table 1 above, the novel image reconstruction methods described herein generally outperformed both FlatNet and Tikhonov et al. For instance, the novel method without text guidance achieved a higher (or better) PSNR, a higher (or better) SSIM, a lower (or better 5 LPIPS), and a higher (or better) CLIP score than both FlatNet and Tikhonov et al. Additionally, the novel method with text guidance also produced comparable or better results as the novel method without text guidance. Though a minor and insignificant loss of PSNR was observed, the addition of text can improve perceptual and textual results (including, e.g., a higher CLIP score). Further, fine-tuning the model with text guidance produces the best PSNR, 10 SSIM, LPIPS, and CLIP score results. The image comparisons are shown in Figure 5. Specifically, sets of images 502, 504, 506, and 508 are shown for scenes 510, 512, 514, 516, and 516. Images 502 show the results from the FlatNet method, images 504 show the results of the novel image reconstruction without text guidance, images 506 show the results of the novel image reconstruction with text guidance, and 15 images 508 are the ground truth images. Scene 510 is described as “a green parrot with a black beak.” Scene 512 is described as “a ladybug on a white surface.” Scene 514 is described as “a bookstore with people walking by.” Scene 516 is described as “a stone castle with houses around.” Scene 518 is described as “a traffic light with a blue arrow pointing to the left.” Turning now to Figure 6, results are presented for image reconstructions of real scenes 20 (that is, scenes not captured from a screen). The novel reconstruction methods described herein are compared to FlatNet and Tikhonov et al. Specifically, sets of images 602, 604, 606, and 608 are shown for scenes 610, 612, 614, and 616. Images 602 show the results from the method of Tikhonov et al., images 604 show the results from the FlatNet method, images 606 show the results of the novel image reconstruction without text guidance, and images 608 show the results of the 25 novel image reconstruction with text guidance. Scene 610 is described as “a Pooh bear doll on a File No. FRAMO-P057-WO table.” Scene 612 is described as “a candy box with text.” Scene 614 is described as “a puppy on a table.” Scene 616 is a zoomed-in version of scene 614 (that is, “a puppy on a table”). Figure 7 presents real scene reconstructions using the prototype camera described herein. Specifically, sets of images 702, 704, and 706 are shown for scenes 708, 710, 712, and 714. Images 5 702 show the results of the novel image reconstruction without text guidance, images 704 show the results of the novel image reconstruction with text guidance, and images 706 show the respective reference images (not aligned with the other images) captured with a Canon80D camera. Scene 708 is described as “a plant with green and yellow leaves.” Scene 710 is described as “black palm trees on a red, orange, and yellow background.” Scene 712 is described as “sheep and grass 10 in a field with a background of gray panels.” Scene 714 is described as “a red shirt with the Barcelona football club, a yellow banana, and a red, green, and blue colored cap.” In some text caption datasets (e.g., the LAION dataset), the captions for the images are not always accurate or relevant. Accordingly, in at least one embodiment, these captions are checked to verify accuracy and correctness, and to identify possible inaccuracies. This verification can be 15 used to improve text guidance reconstruction results and CLIP scores. Ablation results according to at least one embodiment of the invention are presented below herein. Method PSNR SSIM LPIPS CLIP Without 9.84 0.175 0.666 23.22 separation loss Without text 21.44 0.512 0.304 22.50 training Novel method 21.58 0.541 0.276 24.38 Table 2: Ablation results on a dataset captured by a prototype camera 20 Table 2 above shows ablation results for three separate methods, specifically (1) an image reconstruction method without separation loss, (2) an image reconstruction method without text training, and (3) the novel reconstruction method that includes both separation loss and text training. As can be seen, without the separation loss described herein, PSNR values are lower File No. FRAMO-P057-WO (worse), SSIM values are lower (worse), and LPIPS values are higher (worse). Although reconstructed images contain text information, as shown by a relatively high CLIP score, there is no additional information extracted from the camera measurements for the reconstruction process (that is, the reconstructions become independent of the input camera measurements). By contrast, 5 when separation loss is included, as in the novel method, these camera measurements are taken into account. Adding text information improves the reconstruction even further, as can be seen by comparing the PSNR, SSIM, LPIPS, and CLIP values of the “without text training” method with the better values of the novel method. 10 Figure 8 presents visual ablation images that show these ablation results. Specifically, sets of images 802, 804, 806, and 808 are shown for scenes 810 and 812. Images 802 show the results of image reconstruction using an incorrect text caption, images 804 show the results of the novel image reconstruction without text guidance, images 806 show the results of the novel image reconstruction with (correct) text guidance, and images 808 are the ground truth images. The 15 correct description of scene 810 is “a train track surrounded by trees and a tunnel,” while the incorrect text caption of scene 810 is “a painting of a table with three pears and one apple.” The correct description of scene 812 is “a small village with a dirt road, a pond, and a bridge,” while the incorrect text caption of scene 812 is “a large elephant with tusks walking in front of two smaller elephants.” 20 The images show the contribution of text captions to image reconstruction, in at least one embodiment. Without text input, the image reconstruction is driven by the flat camera measurements only. In the at least one embodiment, text captions result in the reconstructed image having details that are more similar to the true captured scene. The text captions also contribute to the high frequency details in the reconstructed images. This is also noticeable when an incorrect 25 text caption is provided. For example, the image reconstruction may gain a “painting” style when the caption incorrectly mentions a painting. As another example, elephant shapes may be visible in the reconstruction when elephants are incorrectly mentioned in the text caption. One or more of the examples and / or embodiments described herein can be implemented on one or more computing systems, as described in further detail below. 30 Figure 9 is a block diagram of a computing system 900 for image reconstruction using any File No. FRAMO-P057-WO one or more of the methods described herein, according to an example embodiment. Thus, the computing system 900 may perform, for instance, any of the steps and / or calculations described above herein. The system 900 comprises one or more computing devices 902 that may execute one or 5 more applications to perform any one or more image reconstruction methods described herein (“Image Reconstruction Application”), and / or any one or more steps of such one or more methods. Additionally, the applications can be used to send information to, or receive information from, a specific user or users. The applications can further be capable of scheduled or triggered communications or commands when various events occur (e.g., when one or more calculations 10 results in an error, when more information is needed, and the like). The one or more computing devices 902 can be used to store acquired computational data, as well as other data in memory and / or a database. The memory may be communicatively coupled to one or more hardware processing devices. The one or more computing devices 902 may further be connected to a communications 15 network 904, which can be the Internet, an intranet, or another wired or wireless communication network. For example, the communication network 904 may include a Mobile Communications (GSM) network, a code division multiple access (CDMA) network, 3rdGeneration Partnership Project (GPP) network, an Internet Protocol (IP) network, a wireless application protocol (WAP) network, a Wi-Fi network, a satellite communications network, or an IEEE 802.11 standards 20 network, as well as various communications thereof. Other conventional and / or later developed wired and wireless networks may also be used. The one or more computing devices 902 include at least one processor to process data and memory to store data. The processor processes communications, builds communications, retrieves data from memory, and stores data to memory. The processor and the memory are hardware. The 25 memory may include volatile and / or non-volatile memory, e.g., a computer-readable storage medium such as a cache, random access memory (RAM), read only memory (ROM), flash memory, or other memory to store data and / or computer-readable executable instructions related to the image reconstruction application. In addition, the one or more computing devices 902 further include at least one communications interface to transmit and receive communications, messages, 30 and / or signals. File No. FRAMO-P057-WO Thus, information processed by the one or more computing devices 902, or the applications executed thereon, may be sent to another computing device, such as a remote computing device, via the communication network 904. Figure 10 illustrates a block diagram of a computing device 902 according to an example 5 embodiment. The computing device 902 includes computer readable media (CRM) 1004 in memory on which an image reconstruction application 1006 or other user interface or application is stored. The computer readable media may include volatile media, nonvolatile media, removable media, non-removable media, and / or another available medium that can be accessed by the processor 1002. By way of example and not limitation, the computer readable media comprises10 computer storage media and communication media. Computer storage media includes non- transitory storage memory, volatile media, nonvolatile media, removable media, and / or non- removable media implemented in a method or technology for storage of information, such as computer / machine-readable / executable instructions, data structures, program modules, or other data. Communication media may embody computer / machine-readable / executable instructions, 15 data structures, program modules, or other data and include an information delivery media or system, both of which are hardware. The image reconstruction application 1006 can include an image reconstruction module 1008 that is operable to perform various functions, such as, for instance, any one or more image reconstruction methods described herein, and / or any one or more steps of such one or more 20 methods. The module 1008 may also be operable to obtain data from other sources, such as a user, a database, and the like, and to process that data to correct for imperfect, inaccurate, or absent calculations and / or models regarding the aforementioned image reconstruction method(s). Using a local high-speed network, the computing device 902 may receive the aforementioned data in near real time, process the data, and generate calculations. These 25 calculations may be executed by one or more algorithms within the image reconstruction application 1006 or other stored applications. Measured or calculated data may be monitored to generate an event and an alert if something is out of range (e.g., one or more calculations and / or equations is incorrect or throws an error, one or more additional data points or definitions is needed, etc.). As mentioned herein, an 30 alert may also be sent after one or more image reconstructions has been made. Such alerts may be File No. FRAMO-P057-WO sent in real-time or near real-time using an existing uplink or dedicated link. The alerts may be sent using email, SMS, push notification, or using an online messaging platform to end users and computing devices, among others. The image reconstruction application 1006 may provide data visualization using a user 5 interface module 1010 for displaying a user interface on a display device. As an example, the user interface module 1010 generates a native and / or web-based graphical user interface (GUI) that accepts input and provides output viewed by users of the computing device 902. The computing device 902 may provide real-time automatically and dynamically refreshed information on the image reconstruction method(s) and / or any one or more steps therein. The user interface module 10 1010 may send data to other modules of the image reconstruction application 1006 of the computing device 902, and retrieve data from other modules of the image reconstruction application of the computing device 902 asynchronously without interfering with the display and behavior of the user interface displayed by the computing device 902. Further, one or more computing systems can implement one or more aspects of the 15 technology and / or methods described herein. Figure 11 shows an example of such a computing system 1102, which may include one or more computing devices (e.g., computing device 902) and / or processing units, which include one or more processors and software. The one or more computing devices (e.g., computing device 902) may execute one or more applications to operate one or more applications, such as, for example, the image reconstruction application 1006 20 described above herein, or one or more portions thereof. The computing system 902 may further control, monitor, and / or extract data from, for instance, one or more imaging devices 1104 (e.g., one or more cameras, including any camera described herein). The computing system can further comprise a graphical user interface (GUI) so that a user may control the system or portions thereof, such as, for instance, the imaging device 1104. 25 Figure 12 shows an example of computing system 1200, which can be for example any computing device such as the computing device 902, or any component thereof in which the components of the system are in communication with each other using connection 1205. Connection 1205 can be a physical connection via a bus, or a direct connection into processor 1210, such as in a chipset architecture. Connection 1205 can also be a virtual connection, 30 networked connection, or logical connection. File No. FRAMO-P057-WO In some embodiments, computing system 1200 is a distributed system in which the functions described in this disclosure can be distributed within a datacenter, multiple data centers, a peer network, etc. In some embodiments, one or more of the described system components represents many such components each performing some or all of the function for which the 5 component is described. In some embodiments, the components can be physical or virtual devices. Example system 1200 includes at least one processing unit (CPU or processor) 1210 and connection 1205 that couples various system components including system memory 1215, such as read-only memory (ROM) 1220 and random access memory (RAM) 1225 to processor 1210. Computing system 1200 can include a cache of high-speed memory 1212 connected directly with, 10 in close proximity to, or integrated as part of processor 1210. Processor 1210 can include any general purpose processor and a hardware service or software service, such as services 1232, 1234, and 1236 stored in storage device 1230, configured to control processor 1210 as well as a special-purpose processor where software instructions are incorporated into the actual processor design. Processor 1210 may essentially be a completely self- 15 contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric. To enable user interaction, computing system 1200 includes an input device 1245, which can represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech, etc. Computing 20 system 1200 can also include output device 1235, which can be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems can enable a user to provide multiple types of input / output to communicate with computing system 1200. Computing system 1200 can include communications interface 1240, which can generally govern and manage the user input and system output. There is no restriction on operating on any particular 25 hardware arrangement, and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed. Storage device 1230 can be a non-volatile memory device and can be a hard disk or other types of computer readable media which can store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, 30 cartridges, random access memories (RAMs), read-only memory (ROM), and / or some File No. FRAMO-P057-WO combination of these devices. The storage device 1230 can include software services, servers, services, etc., that when the code that defines such software is executed by the processor 1210, it causes the system to perform a function. In some embodiments, a hardware service that performs a particular function 5 can include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as processor 1210, connection 1205, output device 1235, etc., to carry out the function. For clarity of explanation, in some instances, the present technology may be presented as including individual functional blocks comprising devices, device components, steps or routines 10 in a method embodied in software, or combinations of hardware and software. Any of the steps, operations, functions, or processes described herein may be performed or implemented by a combination of hardware and software services or services, alone or in combination with other devices. In some embodiments, a service can be software that resides in memory of a client device and / or one or more servers of a content management system and perform 15 one or more functions when a processor executes the software associated with the service. In some embodiments, a service is a program or a collection of programs that carry out a specific function. In some embodiments, a service can be considered a server. The memory can be a non-transitory computer-readable medium. In some embodiments, the computer-readable storage devices, mediums, and memories 20 can include a cable or wireless signal containing a bit stream and the like. However, when mentioned, non-transitory computer-readable storage media expressly exclude media such as energy, carrier signals, electromagnetic waves, and signals per se. Methods according to the disclosures herein can be implemented using computer- executable instructions that are stored or otherwise available from computer-readable media. Such 25 instructions can comprise, for example, instructions and data which cause or otherwise configure a general purpose computer, special purpose computer, or special purpose processing device to perform a certain function or group of functions. Portions of computer resources used can be accessible over a network. The executable computer instructions may be, for example, binaries, intermediate format instructions such as assembly language, firmware, or source code. Examples 30 of computer-readable media that may be used to store instructions, information used, and / or File No. FRAMO-P057-WO information created during methods according to described examples include magnetic or optical disks, solid-state memory devices, flash memory, Universal Serial Bus (USB) devices provided with non-volatile memory, networked storage devices, and so on. Devices implementing methods according to these disclosures can comprise hardware, 5 firmware and / or software, and can take any of a variety of form factors. Typical examples of such form factors include servers, laptops, smartphones, small form factor personal computers, personal digital assistants, and so on. The functionality described herein also can be embodied in peripherals or add-in cards. Such functionality can also be implemented on a circuit board among different chips or different processes executing in a single device, by way of further example. 10 The instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are means for providing the functions described in these disclosures. Generally, the novel image reconstruction methods described herein significantly improve image reconstruction from flat camera measurements and, as a result, make such cameras more 15 practical. The methods described herein achieve high-quality reconstructions both with and without text guidance. In at least one embodiment, the method leverages the strong capabilities of a pre-trained diffusion model for an image prior. Such an approach can be integrated and / or applied into other imaging systems and / or arrangements to improve the reconstructions. Accordingly, embodiments of the invention provide the following contributions over 20 known methods: (1) a novel computational photography method based on a lensless flat camera and a reconstruction algorithm based on a diffusion model image prior, (2) state-of-the-art results for flat camera reconstruction quality in all evaluated metrics, (3) a novel approach to using text guidance to improve imaging results of an optical system, and (4) a deep control network with an intermediate separable loss for improving convergence and results. 25 While specific preferred embodiments and examples of novel image reconstruction methodologies according to the invention have been illustrated and described, it will be clear that the invention is not so limited. Numerous modifications or alterations, changes, variations, substitutions and equivalents will occur to those skilled in the art without deviating from the spirit and scope of the invention, and are deemed part and parcel of the invention disclosed herein. 30 Further, the invention should be considered as comprising all possible combinations of File No. FRAMO-P057-WO every feature described in the instant specification, appended claims, and / or drawing figures which may be considered new, inventive and industrially applicable. Multiple variations and modifications are possible in the embodiments of the invention described here. Although certain illustrative embodiments of the invention have been shown and 5 described here, a wide range of modifications, changes and substitutions is contemplated in the foregoing disclosure. While the above description contains many specifics, these should not be construed as limitations on the scope of the invention, but rather as exemplifications of one or another preferred embodiment thereof. In some instances, some features of the present invention may be employed without a corresponding use of the other features. 10 Accordingly, it is appropriate that the foregoing description be construed broadly and understood as being given by way of illustration and example only, the spirit and scope of the invention being limited only by the claims which ultimately issue.

Claims

File No. FRAMO-P057-WO What is claimed is:

1. A system, comprising: a lensless camera to capture data comprising raw sensor multiplexed measurements of 5 light reflected from a scene across a sensor area; a memory storing computer-readable instructions; and at least one processor to execute the instructions to: reconstruct an output image from the raw sensor multiplexed measurements using a deep neural network to denoise and generate the output image from the raw sensor 10 multiplexed measurements; wherein the deep neural network is trained to control a latent diffusion model.

2. The system of claim 1, wherein the deep neural network is trained to control the latent diffusion model using a library of images comprising pairs of a source image and 15 corresponding raw sensor multiplexed measurements of light reflected from a display of the source image across a sensor area.

3. The system of claim 2, the at least one processor further to execute the instructions to train the deep neural network to control the latent diffusion model using the 20 library of images and corresponding measurements. .

4. The system of claim 3, the at least one processor further to execute the instructions to train the deep neural network using a large language model (LLM) that provides automatic image captioning for each pair of the source image and corresponding measurements. 25 5. The system of any of the preceding claims, the at least one processor further to execute the instructions to receive input text for the scene and reconstruct the output image from the raw sensor multiplexed measurements using the deep neural network and the input text. 30 6. The system of any of the preceding claims, wherein the deep neural network isFile No. FRAMO-P057-WO initialized with a copy of an encoder for the latent diffusion model.

7. The system of any of the preceding claims, wherein the deep neural network has a separable reconstruction loss term, which is applied to an output of a separable transformation 5 and added to a diffusion conventional loss term.

8. The system of any of the preceding claims, wherein the camera comprises a diffuser comprising an amplitude mask. 10 9. The system of any of the preceding claims, wherein the source image comprises one of an RGB image, a depth map, and polarization data.

10. A method comprising: capturing, by a lensless camera, data comprising raw sensor multiplexed measurements 15 of light reflected from a scene across a sensor area; and reconstructing, by at least one processor, an output image from the raw sensor multiplexed measurements by applying a deep neural network to denoise and generate the output image from the raw sensor multiplexed measurements, wherein the deep neural network is trained to control a latent diffusion model. 20 11. The method of claim 10, wherein the deep neural network is trained to control the latent diffusion model using a library of images comprising pairs of a source image and corresponding raw sensor multiplexed measurements of light reflected from a display of the source image across a sensor area. 25 12. The method of claim 11, wherein the deep neural network is trained to control the latent diffusion model using the library of images and corresponding measurements.

13. The method of claim 12, wherein the deep neural network is trained using a large 30 language model (LLM) that provides automatic image captioning for each pair of the sourceFile No. FRAMO-P057-WO image and corresponding measurements.

14. The method of any of claims 10-13, further comprising: receiving, by the at least one processor, input text for the scene; and 5 reconstructing, by the at least one processor using the deep neural network and the input text, the output image from the raw sensor multiplexed measurements.

15. The method of any of claims 10-14, further comprising: initializing, by the at least one processor, the deep neural network with a copy of an 10 encoder for the latent diffusion model.

16. The method of any of claims 10-15, wherein the deep neural network has a separable construction loss term, which is applied to an output of a separable transformation and added to a diffusion conventional loss term. 15 17. The method of any of claims 10-16, wherein the camera comprises a diffuser comprising an amplitude mask.

18. The method of any of claims 10-17, wherein the source image comprises one of 20 an RGB image, a depth map, and polarization data.

19. A non-transitory computer-readable storage medium, having instructions stored thereon that, when executed by a computing device, cause the computing device to perform operations, the operations comprising: 25 obtaining data comprising raw sensor multiplexed measurements of light reflected from a scene across a sensor area; and reconstructing an output image from the raw sensor multiplexed measurements by applying a deep neural network to denoise and generate the output image from the raw sensor multiplexed measurements, 30 wherein the data is captured by a lensless camera, andFile No. FRAMO-P057-WO wherein the deep neural network is trained to control a latent diffusion model.

20. The non-transitory computer-readable storage medium of claim 19, wherein the deep neural network is trained to control the latent diffusion model using a library of images 5 comprising pairs of a source image and corresponding raw sensor multiplexed measurements of light reflected from a display of the source image across a sensor area.

21. The non-transitory computer-readable storage medium of claim 20, wherein the deep neural network is trained to control the latent diffusion model using the library of images 10 and corresponding measurements.

22. The non-transitory computer-readable storage medium of claim 21, wherein the deep neural network is trained using a large language model (LLM) that provides automatic image captioning for each pair of the source image and corresponding measurements. 15 23. The non-transitory computer-readable storage medium of any of claims 19-22, wherein the operations further comprise: receiving input text for the scene; and reconstructing, by the deep neural network and the input text, the output image from the 20 raw sensor multiplexed measurements.

24. The non-transitory computer-readable storage medium of any of claims 19-23, wherein the operations further comprise: initializing, by a copy of an encoder for the latent diffusion model, the deep neural 25 network.

25. The non-transitory computer-readable storage medium of any of claims 19-24, wherein the deep neural network has a separable construction loss term, which is applied to an output of a separable transformation and added to a diffusion conventional loss term. 30File No. FRAMO-P057-WO 26. The non-transitory computer-readable storage medium of any of claims 19-25, wherein the camera comprises a diffuser comprising an amplitude mask.

27. The non-transitory computer-readable storage medium of any of claims 19-26, 5 wherein the source image comprises one of an RGB image, a depth map, and polarization data.

Images