Fresnel aperture coding imaging method and device based on two-stage reconstruction and noise matching

Abstract

The invention discloses a Fresnel aperture coding imaging method and device based on two-stage reconstruction and noise matching, and the method comprises the steps: carrying out the convolution operation of a preprocessed real image and a Fresnel aperture, and obtaining a simulation image; brightness conversion and noise fitting operation are carried out on the simulation image, so that noise distribution of the simulation image is close to real data, and a coding graph is obtained; performing back propagation reconstruction on the coded image to obtain an initial reconstructed image; according to the fast Fourier transform module and the convolutional neural network, constructing a target FFTConv-UNet model; and inputting the initial reconstructed image into the target FFTConv-UNet model to obtain a target reconstructed image. The method can improve the quality of picture reconstruction, and can be widely applied to the technical field of image reconstruction.

Description

Technical Field

[0001] The present invention relates to the technical field of image reconstruction, and in particular to a Fresnel aperture coding imaging method and device based on two-stage reconstruction and noise matching. Background Art

[0002] Fresnel aperture coded imaging is a novel optical computational imaging method. Traditional optical imaging technology relies on a lens system to form an image by focusing light. For example, a camera or optical microscope usually contains multiple groups of lens elements. Although this imaging method can provide high-quality images, its equipment cost is high and the structure is complex. In Fresnel aperture coded imaging, the imaging system relies on the encoding and image reconstruction of optical modulation elements (i.e., Fresnel aperture mask plates), getting rid of the dependence on complex lenses. Therefore, it has the advantages of simple structure, low cost, and easy integration. At present, the image reconstruction methods of Fresnel aperture coded imaging technology include traditional iterative optimization methods and data-driven deep learning methods.

[0003] Traditional iterative optimization methods perform iterative reconstruction on captured, encoded images. During this process, the image estimate is continuously updated, gradually approximating the true image, resulting in highly accurate reconstruction results. However, this method is computationally intensive, especially for high-resolution, large-data-volume images. The reconstruction process consumes significant time and computing resources, limiting its use in applications requiring high real-time performance. Furthermore, the algorithm may become trapped in local optimal solutions, failing to find the globally optimal reconstruction result. Parameter selection also significantly impacts the reconstruction results.

[0004] Existing deep learning methods for binarized Fresnel aperture imaging are primarily based on an end-to-end neural network architecture, which takes as input an encoded image and directly outputs a reconstructed, clear image. Currently, due to the lack of real-world data for network training, network training primarily relies on simulated data. However, the noise distribution in real data is more complex than that in simulated data, resulting in poor image reconstruction quality from models trained on simulated data. Furthermore, existing methods fail to fully utilize the optical model, resulting in poor image reconstruction quality. Summary of the Invention

[0005] In view of this, the main purpose of the embodiments of the present invention is to provide a Fresnel aperture coded imaging method and device based on two-stage reconstruction and noise matching, in order to solve at least one of the problems of the prior art. The present invention can improve the image reconstruction quality.

[0006] To achieve the above objectives, an embodiment of the present invention provides a Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching, comprising:

[0007] Perform convolution operation on the pre-processed real image and the Fresnel aperture to obtain a simulated image;

[0008] Performing brightness conversion and noise fitting operations on the simulated image to obtain a coding image;

[0009] Performing back-propagation reconstruction on the coding image to obtain an initial reconstructed image;

[0010] Construct the target FFTConv-UNet model based on the fast Fourier transform module and convolutional neural network;

[0011] The initial reconstructed image is input into the target FFTConv-UNet model to obtain a target reconstructed image.

[0012] In some embodiments, before performing a convolution operation on the pre-processed real image and the Fresnel aperture to obtain a simulated image, the following steps are included:

[0013] A preprocessing operation of magnification and black filling is performed on the real image to obtain a preprocessed real image.

[0014] In some embodiments, performing brightness conversion and noise fitting operations on the simulated image to obtain a coding image includes the following steps:

[0015] Collecting an n-level grayscale original image of the real image through a sensor;

[0016] Constructing an exponential function according to the grayscale original image and the real image;

[0017] Obtaining a first mean and a first variance of the grayscale original image;

[0018] Performing linear fitting on the first mean and the first variance to obtain Gaussian noise;

[0019] According to the exponential function and the Gaussian noise, brightness conversion and noise fitting operations are performed on the simulation image to obtain the coding image.

[0020] In some embodiments, the back-propagation reconstruction of the coding image is performed to obtain an initial reconstructed image, and the formula used includes:

[0021]

[0022] Where, O R represents the initial reconstructed image; O represents the real image; I represents the coding image; H -1 represents the inverse of the transfer function of Fresnel aperture coding in the frequency domain; represents the Hadamard product; F{·} represents the Fourier transform operation; F-1 {·} denotes an inverse Fourier transform operation.

[0023] In some embodiments, constructing a target FFTConv-UNet model based on a fast Fourier transform module and a convolutional neural network comprises the following steps:

[0024] Using an encoder-decoder structure, the fast Fourier transform module is integrated with the convolutional neural network to obtain an initial FFTConv-UNet model;

[0025] Construct the target loss function based on pixel loss, Laplace loss and fast Fourier transform loss;

[0026] The initial FFTConv-UNet model is trained according to the coding graph and the target loss function to obtain the target FFTConv-UNet model.

[0027] In some embodiments, the target loss function is constructed based on pixel loss, Laplace loss, and fast Fourier transform loss, and the formula used includes:

[0028] L=L piexl +λ F L FFT +λ L L laplacian

[0029] Where,

[0030]

[0031] Among them, L represents the target loss function; L piexl Indicates pixel loss; L FFT represents the Laplace loss; L laplacian represents the fast Fourier transform loss; λ F represents the weight of Laplace loss; λ L represents the weight of the fast Fourier transform loss; O represents the real image; O R represents the initial reconstructed image; N represents the total number of pixels in the image; F(·) represents the Fourier transform operation; ‖·‖2 represents the L2 norm operation; ‖·‖1 represents the L1 norm operation; represents the Laplace operator.

[0032] To achieve the above objectives, another aspect of an embodiment of the present invention provides a Fresnel aperture coded imaging device based on two-stage reconstruction and noise matching, the device comprising:

[0033] The first module is used to perform a convolution operation on the pre-processed real image and the Fresnel aperture to obtain a simulated image;

[0034] The second module is used to perform brightness conversion and noise fitting operations on the simulated image to obtain a coding image;

[0035] The third module is used to perform back-propagation reconstruction on the coding image to obtain an initial reconstructed image;

[0036] The fourth module is used to build the target FFTConv-UNet model based on the fast Fourier transform module and convolutional neural network;

[0037] The fifth module is used to input the initial reconstructed image into the target FFTConv-UNet model to obtain a target reconstructed image.

[0038] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the aforementioned Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching.

[0039] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the aforementioned Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching.

[0040] To achieve the above objectives, another aspect of an embodiment of the present invention provides a computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching.

[0041] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a Fresnel aperture coded imaging method and apparatus based on two-stage reconstruction and noise matching, wherein the scheme performs a convolution operation on a preprocessed real image and a Fresnel aperture to obtain a simulated image; performs brightness conversion and noise fitting operations on the simulated image so that its noise distribution is close to real data, thereby obtaining a coding image; performs backpropagation reconstruction on the coding image to obtain an initial reconstructed image; constructs a target FFTConv-UNet model based on a fast Fourier transform module and a convolutional neural network; and inputs the initial reconstructed image into the target FFTConv-UNet model to obtain a target reconstructed image, thereby improving the quality of image reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0043] Figure 1 is a flow chart of a Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching provided by an embodiment of the present invention;

[0044] Figure 2 is a 16-level grayscale original image captured by the sensor provided in an embodiment of the present invention;

[0045] Figure 3 This is a flow chart of obtaining simulation data and adding noise to the simulation data provided by an embodiment of the present invention;

[0046] Figure 4 Schematic diagram of the FFTConv-UNet network architecture provided by an embodiment of the present invention;

[0047] Figure 5 Schematic diagram of the FFTConv Block structure provided by an embodiment of the present invention;

[0048] Figure 6 Schematic diagram of the Conv Block structure provided by an embodiment of the present invention;

[0049] Figure 7 is a flow chart of a two-stage algorithm provided by an embodiment of the present invention;

[0050] Figure 8 Schematic diagram of a Fresnel aperture-coded imaging system provided by an embodiment of the present invention;

[0051] Figure 9 Schematic diagram of an experimental device provided by an embodiment of the present invention;

[0052] Figure 10 is a schematic diagram of brightness calibration fitting results provided by an embodiment of the present invention;

[0053] Figure 11 Schematic diagram of noise distribution fitting results provided by an embodiment of the present invention;

[0054] Figure 12 is a schematic diagram comparing a target test image and a one-stage BP reconstructed image provided by an embodiment of the present invention;

[0055] Figure 13 1 is a schematic diagram comparing the first-stage reconstructed image and the second-stage reconstructed image provided by an embodiment of the present invention;

[0056] Figure 14 2. It is a schematic diagram comparing the image reconstruction effects of the method according to the embodiment of the present invention and the prior art;

[0057] Figure 15 It is a schematic diagram of the hardware structure of the electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely for the purpose of explaining the present invention and are not intended to limit the present invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are merely examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the appended claims.

[0059] It should be noted that although the functional modules are divided in the system schematic and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100" and "second / S200" in the specification and claims and the above-mentioned figures may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to a determination".

[0060] The terms "at least one", "plurality", "each", "any", etc. used in the present invention include at least one, two or more, multiple, two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present invention pertains. The terms used herein are for the purpose of describing embodiments of the present invention only and are not intended to limit the present invention.

[0062] Before describing the embodiments of the present invention in detail, some nouns and terms involved in the embodiments of the present invention are first described. The nouns and terms involved in the embodiments of the present invention are subject to the following explanations.

[0063] Fresnel Aperture Coded Imaging: Fresnel Aperture Coded Imaging is a lensless imaging technique that operates on the principle of Fresnel diffraction. When light waves pass through a specific coded aperture, Fresnel diffraction occurs, altering their propagation path and intensity. The encoded light signal contains more information, which can be decoded using algorithms to reconstruct the target image.

[0064] Binarization: Binarization means that the transmittance only contains two values: 1 and -1. In theory, Fresnel aperture coding uses a zone plate with a continuously sinusoidally varying transmittance. However, zone plates with continuously varying transmittance are difficult to manufacture, so in practice, they are processed using a binary method.

[0065] Noise matching: Noise matching refers to matching the noise distribution of simulated data with the actual captured data.

[0066] like Figure 1 As shown, an embodiment of the present invention provides a Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching, which may include but is not limited to steps S100 to S500:

[0067] Step S100, performing a convolution operation on the pre-processed real image and the Fresnel aperture to obtain a simulated image;

[0068] Step S200, performing brightness conversion and noise fitting operations on the simulated image to obtain a coding image;

[0069] Step S300, performing back propagation reconstruction on the coding image to obtain an initial reconstructed image;

[0070] Step S400, constructing a target FFTConv-UNet model based on the fast Fourier transform module and the convolutional neural network;

[0071] Step S500: input the initial reconstructed image into the target FFTConv-UNet model to obtain a target reconstructed image.

[0072] In some embodiments, before step S100, the method further includes performing a preprocessing operation of magnifying and black filling on the real image to obtain a preprocessed real image, and then performing a convolution operation on the preprocessed real image and the Fresnel aperture to obtain a simulated image.

[0073] In some embodiments, step S200 may include but is not limited to steps S210 to S250:

[0074] Step S210, collecting an n-level grayscale original image of the real image through a sensor;

[0075] Step S220, constructing an exponential function according to the grayscale original image and the real image;

[0076] Step S230, obtaining a first mean and a first variance of the grayscale original image;

[0077] Step S240, performing linear fitting on the first mean and the first variance to obtain Gaussian noise;

[0078] Step S250 , performing brightness conversion and noise fitting operations on the simulated image according to the exponential function and the Gaussian noise to obtain the coding image.

[0079] In step S210 of some embodiments, the real image is replaced with a uniform gray square with increasing pixel values ​​every several pixels, and an n-level grayscale original image is generated and displayed on the LCD screen, and the n-level grayscale original image is captured by a sensor. For example, Figure 2 As shown, the real image is replaced with uniform gray squares with increasing pixel values ​​every 16 times. The pixel values ​​are 0, 15, 31, ..., 239, 255, for a total of 16 grayscale images. That is, a 16-level grayscale original image is generated and displayed on the LCD screen. The 16-level grayscale original image is then captured by the sensor.

[0080] In step S220 of some embodiments, the brightness conversion is performed using exponential function fitting, which has the following formula:

[0081] O0=aO b +c

[0082] Where O0 represents the sensor data; O represents the real image; and a, b, and c are the parameters to be fitted. a is the exponential term coefficient of the fitting, which physically translates to the mapping of the normalized grayscale values ​​of the LCD display to the normalized grayscale values ​​of the sensor data. b is the exponential power coefficient, reflecting the nonlinear relationship of brightness calibration. c represents the sensor's response to weak ambient light.

[0083] In some embodiments, in steps S230 to S240, a linear fit is performed on the first mean and first variance of the n-th order grayscale original image data collected by the sensor to obtain a gain coefficient and Gaussian noise. The relationship between the noise and the mean of the grayscale original image can be obtained based on the noise distribution, as shown in the following formula:

[0084]

[0085] Where, σ(O0) 2 represents the first variance of the sensor data, and the variance represents the noise level; E(O0) represents the first mean of the sensor data; g represents the gain coefficient; represents Gaussian noise.

[0086] In step S250 of some embodiments, an exponential function is used to perform brightness conversion on the simulated image, and noise fitting is performed on the simulated image by adding corresponding Gaussian noise to obtain noisy simulated data, namely, a coding image.

[0087] In steps S100 to S200 of some embodiments, as Figure 3 As shown in the figure, a 256×256 real target image is magnified and black-filled to 1500×1500 pixels. The simulated image is then processed through Fresnel diffraction (i.e., convolved with the Fresnel aperture (FZA)) to obtain a simulated image. The simulated image then undergoes brightness conversion and noise fitting to obtain noisy simulated data that is close to the real data, namely the coding map.

[0088] In step S300 of some embodiments, in the first stage of image reconstruction, back-propagation reconstruction is performed on the obtained coding map to obtain an initial reconstructed image. The formula used includes:

[0089]

[0090] Where, O R represents the initial reconstructed image; O represents the real image; I represents the coding image; H -1 represents the inverse of the transfer function of Fresnel aperture coding in the frequency domain; represents the Hadamard product; F{·} represents the Fourier transform operation; F -1 {·} represents the inverse Fourier transform operation. Substituting the encoded image I into the formula yields the initial reconstructed image. In this step, noise in I is allowed to enter the solution. Although the BP algorithm cannot handle noise, it can quickly recover the basic structure and some details of the image. In other words, the introduction of optical prior information in the first stage completes the initial reconstruction of the image.

[0091] In some embodiments, step S400 may include but is not limited to steps S410 to S430:

[0092] Step S410, using an encoder-decoder structure to fuse the fast Fourier transform module with the convolutional neural network to obtain an initial FFTConv-UNet model;

[0093] Step S420, constructing a target loss function according to pixel loss, Laplace loss and fast Fourier transform loss;

[0094] Step S430: Training the initial FFTConv-UNet model according to the coding graph and the target loss function to obtain the target FFTConv-UNet model.

[0095] In some embodiments, in steps S410 to S430, the coding graph after noise matching is divided into a training set and a test set, and the initial FFTConv-UNet model is trained for inference. Figure 4 As shown in the figure, the initial FFTConv-UNet model deeply integrates the fast Fourier transform (FFT) with the convolutional neural network, adopts an encoder-decoder structure, and includes a total of 5 levels of downsampling (reducing the input image size M by half, implemented by Maxpool Block (maximum pooling layer)) and upsampling paths (doubling the input size M, implemented by UpConv Block (deconvolution layer)). The number of channels C is expanded from 32→64→128→256→512, and the encoder and decoder are connected by Skip Connection.

[0096] In some embodiments, as Figure 5 The FFTConv Block (Fast Fourier Transform Block) shown in the figure is the core component of the FFTConv-UNet model. It combines spatial domain convolution and FFT-ReLU flow (a module formed by combining FFT and ReLU) to achieve spatial-frequency domain collaborative learning through a dual-path architecture. The spatial domain flow branch adopts the original Conv Block structure in UNet, as shown in the figure. Figure 6 As shown in the figure, it contains two 3×3 convolutional layers, each followed by BatchNorm (batch normalization) and ReLU activation function. The main function of the spatial flow is to extract local spatial features of the image and capture the details and local structure in the image.

[0097] In some embodiments, the loss function of the model is a function used in deep learning to measure the difference between the model prediction results and the actual data. The target loss function in the FFTConv-UNet model consists of pixel loss, Laplace loss, and fast Fourier transform loss. The expression of the target loss function is as follows:

[0098] L=L piexl +λ F L FFT +λ L L laplacian

[0099] Where, pixel loss L piexl The most basic loss function is used to measure the pixel-level difference between the reconstructed image and the real image. The pixel loss is defined as:

[0100]

[0101] The fast Fourier transform loss function reflects the difference in the amplitude spectrum after calculating the Fourier transform of the reconstructed image and the real image, and is defined as:

[0102]

[0103] The Laplace loss function uses a 3x3 Laplace kernel to perform a convolution operation on the image, and then calculates the absolute difference between the reconstructed image and the real image after Laplace transform. The Laplace loss is defined as follows:

[0104]

[0105] Among them, L represents the target loss function; L piexl Indicates pixel loss; L FFT represents the Laplace loss; L laplacian represents the fast Fourier transform loss; λ F represents the weight of the Laplace loss, and optionally, λ F =0.1;λ L represents the weight of the fast Fourier transform loss, and optionally, λ L =0.08; O represents the real image; O R Represents the initial reconstructed image; N represents the total number of pixels in the image, for example, if the image pixels are 200×200, then N is 40000; F(·) represents the Fourier transform operation; ‖·‖2 represents the L2 norm operation; ‖·‖1 represents the L1 norm operation; represents the Laplace operator.

[0106] In step S500 of some embodiments, in the second stage of image reconstruction, the initial reconstructed image is input into the target FFTConv-UNet model to obtain a target reconstructed image.

[0107] In some embodiments, steps S300 to S500 are as follows: Figure 7 As shown in the figure, a coded image after noise matching has a pixel size of 1500×1500. After the first stage of back propagation reconstruction, the approximate outline and details of the image are restored, and the initial reconstructed image of the first stage is obtained, with a pixel size of 512×512. After that, a clear image is reconstructed through the second stage FFTConv-UNet network, and the target reconstructed image is output, with a pixel size of 256×256.

[0108] In some optional embodiments, a Fresnel aperture coded imaging system composed of a CMOS image sensor, an LCD screen, a computer, and other optical components is used to perform Fresnel aperture coded imaging based on two-stage reconstruction and noise matching, but the components used are not limited thereto. The CMOS image sensor is used to collect image data, the LCD screen is used as a light source and to display images, the computer is used to perform image reconstruction, and other optical components may include a Fresnel aperture mask (a mask made based on the Fresnel diffraction principle). For example, Figure 8 As shown in the figure, in a Fresnel aperture-coded imaging system, an FZA mask is secured 3 mm in front of the CMOS image sensor via a custom 3D-printed housing. The mask and camera form the Fresnel aperture-coded camera. The target image is displayed in the center of an LCD screen, with the area outside the target image covered in black (i.e., the LCD displays the pre-processed, real image). The LCD screen is placed 300 mm parallel to the plane of the Fresnel aperture-coded imaging camera. Image data captured by the Fresnel aperture-coded camera is then transferred to a computer, which uses a two-stage reconstruction algorithm to recover the object image.

[0109] In some embodiments, a black and white camera is selected as the shooting camera, the model of which is QHY163m, using a 16-megapixel Panasonic 4 / 3-inch COMS. The experimental device is shown in the figure Figure 9 As shown. The CMOS size is 3.8μm pixel size, the output resolution is 4656×3522, and the experiment uses single exposure shooting without pixel merging. The dataset uses the large-scale scene recognition dataset Places365-Standard dataset developed by the MITCSAIL team (a dataset with fine annotations covering 365 scene categories, such as bedrooms, streets, parks, airports, etc.). The target image on the LCD display is replaced with a uniform gray square with an increment of 16 pixel values, with values ​​of 0, 15, 31, ..., 239, 255. A total of 16 grayscale images are displayed on the LCD display for shooting, as shown Figure 2 As shown. Then, after brightness calibration and noise distribution fitting, the noise matching result is obtained. Among them, the brightness calibration result is as follows Figure 10 As shown, the fitting curve is an exponential function curve; the noise fitting result is as follows Figure 11 As shown. Noise is added to the simulated image data to complete the noise matching work and obtain the coding map after noise matching. Then the coding map is quickly reconstructed in one stage. The reconstruction result is shown as follows. Figure 12 As shown in the figure, the upper part is the target test image, and the lower part is the basic structure and some details of the image after the first stage of reconstruction. Figure 13Part (a) of the training is based on the target FFTConv-UNet model, and two-stage reconstruction is performed to obtain the final reconstructed image as shown in Figure 13 As shown in part (b) of the figure, during model training, the AdamW optimizer was used, the initial learning rate was 0.0002, the betas parameter was set to (0.9, 0.999), and the weight decay was set to 0.01.

[0110] This embodiment of the present invention utilizes a novel noise matching method, requiring only 16 grayscale images to obtain a noise distribution close to that of a real image. Due to a lack of real data, existing technologies typically use simulated data for model training. This embodiment of the present invention can match simulated data to data close to that of real-world capture, addressing the lack of real data in Fresnel aperture-coded imaging.

[0111] Table 1

[0112]

[0113]

[0114] Secondly, the embodiment of the present invention adopts a two-stage reconstruction method (a one-stage BP algorithm and a two-stage FFTConv-UNet model) to restore the image, and proposes a new network structure. The image reconstruction quality is better than the existing technology (such as the image reconstruction method using TwIST, ADMM, UNet, LinkNet, SUNet, ATTUNet technology). Figure 14 As shown in Figure 1, the image quality indicators PSNR (Peak Signal-to-Noise Ratio, reflecting the noise in the reconstructed image) and SSIM (Structural Similarity Index, reflecting the detail in the reconstructed image) are shown in each image. A larger value indicates higher reconstruction quality. Table 1 summarizes the average image reconstruction indicators. This shows that the image reconstruction quality of the method of the present invention is superior to that of the existing technology.

[0115] An embodiment of the present invention further provides a Fresnel aperture coded imaging device based on two-stage reconstruction and noise matching, which can implement the above-mentioned Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching. The device includes:

[0116] The first module is used to perform a convolution operation on the pre-processed real image and the Fresnel aperture to obtain a simulated image;

[0117] The second module is used to perform brightness conversion and noise fitting operations on the simulated image to obtain a coding image;

[0118] The third module is used to perform back-propagation reconstruction on the coding image to obtain an initial reconstructed image;

[0119] The fourth module is used to build the target FFTConv-UNet model based on the fast Fourier transform module and convolutional neural network;

[0120] The fifth module is used to input the initial reconstructed image into the target FFTConv-UNet model to obtain a target reconstructed image.

[0121] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0122] An embodiment of the present invention further provides an electronic device comprising a processor and a memory, wherein the memory stores a computer program. When the processor executes the computer program, the method for Fresnel aperture coded imaging based on two-stage reconstruction and noise matching is implemented. The electronic device can be any intelligent terminal, such as a tablet computer or an in-vehicle computer.

[0123] It can be understood that the contents of the above method embodiments are applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0124] refer to Figure 15 , Figure 15 The hardware structure of an electronic device according to another embodiment is shown. The electronic device includes:

[0125] The processor 601 may be implemented as a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute relevant programs to implement the technical solutions provided by the embodiments of the present invention.

[0126] The memory 602 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 602 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 602 and is called by the processor 601 to execute the Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching according to the embodiment of the present invention.

[0127] Input / output interface 603, used to implement information input and output;

[0128] Communication interface 604, used to implement communication interaction between this device and other devices, which can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WiFi, Bluetooth, etc.);

[0129] Bus 605 , which transmits information between various components of the device (e.g., processor 601 , memory 602 , input / output interface 603 , and communication interface 604 );

[0130] The processor 601 , the memory 602 , the input / output interface 603 and the communication interface 604 are connected to each other in communication within the device via a bus 605 .

[0131] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching is implemented.

[0132] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiment, the functions specifically implemented by the present storage medium embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0133] An embodiment of the present invention further provides a computer program product or computer program, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching.

[0134] In summary, the Fresnel aperture coded imaging method and apparatus based on two-stage reconstruction and noise matching according to the embodiments of the present invention have the following advantages:

[0135] 1. The embodiment of the present invention adopts a noise matching method. Only 16 grayscale images need to be taken to obtain a noise distribution close to the real captured image. By preprocessing the simulated data, its noise distribution is close to the real data, which solves the problem of lack of real data in Fresnel aperture coded imaging during network training.

[0136] 2. This embodiment of the present invention adopts a two-stage reconstruction method, combining the advantages of iterative reconstruction and deep learning techniques. In the first stage, the back-propagation (BP) algorithm is used to quickly restore the basic structure of the image. In the second stage, the forward convolution of the FFT-Block (Fast Fourier Transform) module is innovatively combined with the UNet (named for its U-like network structure) codec framework to construct a new network structure, FFTConv-UNet, which improves the image reconstruction quality.

[0137] In some optional embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided in an exemplary manner for the purpose of providing a more comprehensive understanding of the technology. The disclosed method is not limited to the operation and logic flow presented herein. Optional embodiments are contemplated in which the order of the various operations is changed and the sub-operations described as a part of a larger operation are performed independently.

[0138] Furthermore, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise indicated, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It will also be understood that a detailed discussion of the actual implementation of each module is not necessary for understanding the present invention. More specifically, given the properties, functions, and internal relationships of the various functional modules in the devices disclosed herein, the actual implementation of the module will be understood within the ordinary skill of an engineer. Therefore, a person skilled in the art using ordinary skill will be able to implement the present invention set forth in the claims without undue experimentation. It will also be understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0139] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0140] The logic and / or steps represented in the flowcharts or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing the logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (e.g., a computer-based system, a system including a processor, or other system that can fetch and execute instructions from an instruction execution system, apparatus, or device). For purposes of this specification, a "computer-readable medium" can be any device that can contain, store, communicate, propagate, or transport a program for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0141] More specific examples (a non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic devices), a portable computer disk cartridge (magnetic devices), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disc read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering, or processing in another suitable manner as necessary, and then stored in a computer memory.

[0142] It should be understood that various parts of the present invention can be implemented using hardware, software, firmware, or a combination thereof. In the above-described embodiments, multiple steps or methods can be implemented using software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented using hardware, as in another embodiment, any one of the following technologies known in the art or a combination thereof can be used: a discrete logic circuit having a logic gate circuit for implementing a logic function on a data signal, an application-specific integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0143] Throughout this specification, reference to terms such as "one embodiment," "some embodiments," "examples," "specific examples," or "some examples" means that a specific feature, structure, material, or characteristic described in conjunction with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples.

[0144] While embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions, and variations may be made to the embodiments without departing from the principles and spirit of the invention, and that the scope of the invention is defined by the claims and their equivalents.

[0145] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching, characterized in that: The following steps are involved: Perform convolution operation on the pre-processed real image and the Fresnel aperture to obtain a simulated image; Performing brightness conversion and noise fitting operations on the simulated image to obtain a coding image; Performing back-propagation reconstruction on the coding image to obtain an initial reconstructed image; Construct the target FFTConv-UNet model based on the fast Fourier transform module and convolutional neural network; The initial reconstructed image is input into the target FFTConv-UNet model to obtain a target reconstructed image.

2. The Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching according to claim 1, characterized in that: Before performing a convolution operation on the pre-processed real image and the Fresnel aperture to obtain a simulated image, the following steps are included: A preprocessing operation of magnification and black filling is performed on the real image to obtain a preprocessed real image.

3. The Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching according to claim 1, characterized in that: The method of performing brightness conversion and noise fitting operations on the simulated image to obtain a coding image comprises the following steps: Collecting an n-level grayscale original image of the real image through a sensor; Constructing an exponential function according to the grayscale original image and the real image; Obtaining a first mean and a first variance of the grayscale original image; Performing linear fitting on the first mean and the first variance to obtain Gaussian noise; According to the exponential function and the Gaussian noise, brightness conversion and noise fitting operations are performed on the simulation image to obtain the coding image.

4. The Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching according to claim 1, characterized in that: The coding image is back-propagated and reconstructed to obtain an initial reconstructed image, and the formula used includes: Where, O R represents the initial reconstructed image; O represents the real image; I represents the coding image; H -1 represents the inverse of the transfer function of Fresnel aperture coding in the frequency domain; represents the Hadamard product; F{·} represents the Fourier transform operation; F -1 {·} denotes an inverse Fourier transform operation.

5. The Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching according to claim 1, characterized in that: The target FFTConv-UNet model is constructed according to the fast Fourier transform module and the convolutional neural network, including the following steps: Using an encoder-decoder structure, the fast Fourier transform module is integrated with the convolutional neural network to obtain an initial FFTConv-UNet model; Construct the target loss function based on pixel loss, Laplace loss and fast Fourier transform loss; The initial FFTConv-UNet model is trained according to the coding graph and the target loss function to obtain the target FFTConv-UNet model.

6. The Fresnel aperture coded imaging method based on two-stage reconstruction and noise matching according to claim 5, characterized in that: The target loss function is constructed based on pixel loss, Laplace loss and fast Fourier transform loss. The formula used includes: L=L piexl +λ F L FFT +λ L L laplacian Where, Among them, L represents the target loss function; L piexl Indicates pixel loss; L FFT represents the Laplace loss; L laplacian represents the fast Fourier transform loss; λ F represents the weight of Laplace loss; λ L represents the weight of the fast Fourier transform loss; O represents the real image; O R represents the initial reconstructed image; N represents the total number of pixels in the image; F(·) represents the Fourier transform operation; ‖·‖2 represents the L2 norm operation; ‖·‖1 represents the L1 norm operation; represents the Laplace operator.

7. A Fresnel aperture coded imaging device based on two-stage reconstruction and noise matching, characterized in that: include: The first module is used to perform a convolution operation on the pre-processed real image and the Fresnel aperture to obtain a simulated image; The second module is used to perform brightness conversion and noise fitting operations on the simulated image to obtain a coding image; The third module is used to perform back-propagation reconstruction on the coding image to obtain an initial reconstructed image; The fourth module is used to build the target FFTConv-UNet model based on the fast Fourier transform module and convolutional neural network; The fifth module is used to input the initial reconstructed image into the target FFTConv-UNet model to obtain a target reconstructed image.

8. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images