Low-resolution ghost imaging image reconstruction method and device based on hierarchical network

Abstract

The present invention discloses a method and device for reconstructing low-resolution ghost imaging images based on a hierarchical network, comprising the following steps: generating a ghost imaging image from an initial image through simulation; inputting the ghost imaging image to be reconstructed as input into a pre-trained multi-level network model, wherein the multi-level network model processes the ghost imaging image to be reconstructed and outputs a reconstructed clear image; wherein the construction of the multi-level network model comprises: constructing a multi-level network, introducing feedback connections in the encoders of each level of the network; and training based on the ghost imaging image, the true value image corresponding to the ghost imaging image, and a perceptual loss function to obtain the multi-level network model. Compared with existing ghost imaging reconstruction methods based on deep learning, the present invention can restore a clean image with a clearer target structure and sharper edges, solving the problem of unclear and slow ghost imaging image restoration at low sampling rates.

Description

Technical Field

[0001] The present invention relates to a low-resolution ghost imaging image reconstruction method and device based on a hierarchical network, and belongs to the field of ghost imaging image processing. Background Art

[0002] Unlike traditional optical imaging, ghost imaging is based on high-order correlation between the two optical arms of an imaging device. Computational ghost imaging (CGI) is a method for obtaining both image and spatial information of an unknown target by illuminating it with a series of random binary patterns generated by a spatial light modulator (SLM). For each projected pattern, a conventional photodiode records the intensity of light reflected back from the target plane. By correlating the measured intensity with the corresponding projected pattern, the target image is reconstructed. A drawback of CGI is that it requires a large number of acquisitions to produce high-quality images, which increases imaging time and cost. Despite the emergence of schemes primarily based on basic scanning methods, CGI using random patterns remains popular in many applications due to its simple model and ease of deployment. Due to its low cost, robustness to noise and scattering, and ability to operate over a long spectral range, CGI has been widely used in many fields.

[0003] The emergence of ghost imaging reconstruction methods in the following years, combined with compressed sensing, has significantly improved the quality of computational ghost imaging. However, this only works at high sampling rates. At sampling rates below 25%, the method still fails to achieve good results. In the field of ghost imaging, the sampling rate is directly related to the speed and quality of computational ghost imaging.

[0004] How to obtain clearer ghost imaging reconstruction images faster at low sampling rates has become a research focus in academia. Summary of the Invention

[0005] The purpose of the present invention is to propose a low-resolution ghost imaging image reconstruction method and device based on a hierarchical network. The problem of ghost imaging image reconstruction at low resolution is a difficult problem. It is difficult to reconstruct a blurred ghost imaging image into a clear, noise-free image, and the dark light during imaging also has a great impact on the reconstruction and restoration of the target. The present invention can restore the target image more accurately and quickly under dark light and low sampling rates, solving the problem that the compressed sensing ghost imaging reconstruction method cannot restore a clear image under low sampling rates.

[0006] To achieve the above-mentioned object, in a first aspect, the present invention provides a low-resolution ghost imaging image reconstruction method based on a hierarchical network, comprising the following steps: generating a ghost imaging image from an initial image through simulation;

[0007] The ghost image to be reconstructed is used as input to a pre-trained multi-level network model. The multi-level network model processes the ghost image to be reconstructed and outputs a reconstructed clear image.

[0008] The construction of the multi-layer network model includes: constructing a multi-layer network and introducing feedback connections in the encoder of each layer of the network;

[0009] Based on the ghosted images, the true images corresponding to the ghosted images and the perceptual loss function, a multi-level network model is obtained for training.

[0010] Furthermore, the construction of the perceptual loss function includes the following steps:

[0011] Preprocess the simulated ghost imaging images and their corresponding true value images, and construct an image group;

[0012] Extracting feature values ​​of images based on image groups;

[0013] A perceptual loss function is constructed based on the extracted image feature values.

[0014] Furthermore, constructing the perceptual loss function includes the following steps:

[0015] L prec =L x +10 -3 L gen

[0016] Among them, L prec is the total loss, L x is the content loss, L gen It’s about fighting losses.

[0017] Furthermore, extracting the feature value of the image based on the image group includes the following steps:

[0018] Obtain two identical ghost images, and construct a three-channel color image based on the two identical ghost images;

[0019] The three-channel color image is input into the pre-trained VGG16 model to extract image feature values.

[0020] Furthermore, the multi-level network uses a method of aggregating multi-scale features to input the sliced ​​image into the network and perform noise decomposition through the slices;

[0021] The multi-layer network uses the feature information learned by the bottom-layer encoder and decoder as prior information for the upper-layer encoder and decoder.

[0022] In a second aspect, the present invention provides a low-resolution ghost imaging image reconstruction device based on a hierarchical network, comprising a processor and a storage medium;

[0023] The storage medium is used to store instructions;

[0024] The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 5.

[0025] Beneficial effects

[0026] Compared with the prior art, the present invention has the following technical effects:

[0027] (1) The present invention obtains a network architecture that is effective for ghost imaging image reconstruction tasks through training and learning, thereby solving the problem of slow reconstruction speed in ghost imaging reconstruction tasks.

[0028] (2) The present invention uses perceptual loss, so that both high-level global information and low-level detail information can be learned by the network, overcoming the problem that ordinary loss functions will cause the loss of high-frequency information of the image in the field of image reconstruction, resulting in blurred generated images.

[0029] (3) The present invention introduces a closed-loop concept into the network, which enables the network to reconstruct ghost images into clear images and better estimate the underlying downsampling kernel, thereby reducing some model space. Through continuous learning in this closed-loop, better reconstruction results are achieved.

[0030] (4) This invention uses a strategy of aggregating multi-scale features to feed the sliced ​​image into the network. Through the slices, the noise is better decomposed, resulting in a clearer reconstruction result. Especially at low sampling rates, the reconstruction effect exceeds that of other similar methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a model overall framework diagram of a low-resolution ghost imaging image reconstruction method based on a hierarchical network provided in Example 1 of the present invention;

[0032] Figure 2 A schematic diagram of a closed-loop module in a low-resolution ghost imaging image reconstruction method based on a hierarchical network provided in Example 1 of the present invention;

[0033] Figure 3 is the true value of the mnist dataset;

[0034] Figure 4 is the ghost imaging image after simulation;

[0035] Figure 5 This is the ghost imaging reconstruction result diagram of the present invention;

[0036] Figure 6 These are the reconstruction results of ghost imaging images of different digital simulations under the MNIST dataset of the present invention.

[0037] Specific implementation content

[0038] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the scope of protection of the present invention.

[0039] Example 1:

[0040] Embodiment 1 of the present invention provides a low-resolution ghost imaging image reconstruction method based on a hierarchical network, which includes the following three contents: First, using a typical U-net network, introducing jump connections in the encoder and decoder of the same layer, and introducing a feedback mechanism in the middle to achieve the effect of dual regression, so that the entire mapping forms a closed loop. The network can reconstruct ghost imaging images into clear images and reduce some model space. Through continuous learning of this closed loop, better reconstruction effects are obtained; second, since the high-level information (content and global structure) of ghost imaging images is as important as the underlying detail information, but many previous networks rarely pay attention to it, perceptual loss is introduced to compare the features obtained by convolution of the real image with the features obtained by convolution of the generated image, so that the content and global structure are continuously approached. In this way, both high-level global information and low-level detail information can be learned by the network, overcoming the problem of missing high-frequency information of images caused by mean square error loss (MSE loss) in the field of image reconstruction; thirdly, since the essence of ghost imaging images is sparse true values ​​scattered in dense noise, a multi-stage method is proposed. Using the strategy of aggregating multi-scale features, the sliced ​​images are input into the network. Through slicing, the noise is better decomposed, and the feature information learned by the bottom encoder and decoder is used as the prior information of the upper encoder and decoder. The ghost imaging image is then input into the network, thereby generating rich context and spatially accurate output, and finally reconstructing the ghost imaging image. Finally, we conducted experiments by simulating ghost imaging images using the MNIST handwritten digit dataset, and the results proved the effectiveness of the invented technical solution. The experimental results are shown in the figure. Figure 3-Figure 6 , this embodiment is a result diagram at a sampling rate of 3%.

[0041] Specific methods such as Figures 1 to 6 As shown, the following steps are included:

[0042] Step 1: Generate a ghost image from the clean image through simulation, and preprocess it and the corresponding clean true value image into an image in the universal format of 128*128 pixels.

[0043] The original dataset used in this paper is the MNIST dataset of handwritten digits, which includes various handwritten digits from "0" to "9". The original dataset is 28*28 pixels in size, so after simulating it into ghost imaging data, it needs to be resized and formatted to make it suitable for network training.

[0044] Step 2: Input the image group obtained in step 1 above into the VGG16 network to extract image features for the establishment of the loss function. The specific operation method is as follows:

[0045] Since the VGG16 network takes color RGB images as input, but the ghost imaging image is a single-channel grayscale image, the ghost imaging image is copied twice to form a three-channel RGB image with color values ​​of (128, 128, 3) to adapt to the network. The processed image is then passed through the pre-trained VGG16 model for feature extraction.

[0046] In the present invention, the open source trained VGG16 model is used to make the image extraction of this part more universal.

[0047] Step 3: Based on the extracted eigenvalues, calculate the perceptual loss. The specific calculation method is as follows:

[0048] L prec =L x +a·L gen

[0049] Among them, L prec is the total loss, L x is the content loss, L gen is the adversarial loss. For the dataset in this invention, a=10 -3 In this invention, mean square error (mse loss) is used as content loss, and the difference in image features from the VGG16 network is used as adversarial loss.

[0050] The first item L x The formula is:

[0051] L x =(x i -y i ) 2

[0052] Among them, x i ,y i are the values ​​of the pixels corresponding to the input image and the network output image respectively.

[0053] The second item L gen The formula is:

[0054]

[0055] Where X is the input image and Y is the output image of the network proposed in this invention. Represents the feature map obtained after the jth convolutional layer and before the i-th maximum pooling layer in the VGG16 network. Is the mapping function of the VGG16 network. W i,j and H i,j Describes the dimensions of the expected feature map of the VGG16 network. x and y are the specific pixels of the dimensions.

[0056] Step 4: Build a multi-layer network, which is a network constructed by three sub-networks. In each sub-network, a feedback connection is introduced between the encoder and decoder of each layer to better learn the information of each layer. The overall architecture of the multi-layer network is as follows: Figure 1 shown.

[0057] The schematic diagram of closed-loop feedback connection is as follows Figure 2 As shown, let the outputs of the encoder and decoder at the same layer be X1 and X2 respectively, then after the output of the weight processing layer, we get M out As a result of feature learning, we continue to learn:

[0058] M out =(1-α)V(V T V) -1 V T X i +αX2

[0059] Wherein, V is a matrix composed of M basis vectors of X1, X2. In the present invention, let M be 16, (·) T It means to find the transposed matrix of the current matrix. α is a weight parameter, which is set to 0.5 in this embodiment.

[0060] Since the essence of ghost images is sparse true values ​​inserted in dense noise, this network is a multi-stage method that uses a strategy of aggregating multi-scale features to input sliced ​​images into the network. Through slicing, the noise is better decomposed, and the feature information learned by the underlying encoder and decoder is used as the prior information for the upper-layer encoder and decoder. The ghost image is then input into the network, thereby generating rich context and spatially accurate output, and ultimately reconstructing the ghost image.

[0061] Step 5: Use the ghost image as input, the corresponding true value image as the label, and the perceptual loss as the loss function for training to obtain a trained multi-level network model.

[0062] This example uses the Pytorch library in the Python environment on a GeForce GTX 2080 Ti processor. In this invention, the Adam optimizer and ReLU are used as the activation function. During the training phase, the number of iterations is set to 500 and the learning rate is set to 10. -5 . The step size is set to 1.

[0063] Step 6: Take the ghost image with similar structure that needs to be reconstructed as input, and output the reconstructed clear image through the network.

[0064] Example 2:

[0065] A second embodiment of the present invention provides a low-resolution ghost imaging image reconstruction device based on a hierarchical network, comprising a processor and a storage medium;

[0066] The storage medium is used to store instructions;

[0067] The processor is configured to operate according to the instructions to execute any one step of the hierarchical network-based low-resolution ghost imaging image reconstruction method in embodiment one.

[0068] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0069] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the steps in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0070] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0071] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 The steps for the function specified in one or more boxes.

[0072] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A low-resolution ghost imaging image reconstruction method based on a hierarchical network, characterized by: The method comprises the following steps: generating a ghost imaging image by simulating an initial image; The ghost image to be reconstructed is used as input to a pre-trained multi-level network model. The multi-level network model processes the ghost image to be reconstructed and outputs a reconstructed clear image. The construction of the multi-level network model includes: constructing a multi-level network and introducing feedback connections in the encoder of each level of the network; A multi-layer network model is obtained by training based on the ghosted image, the true image corresponding to the ghosted image, and the perceptual loss function; The construction of the perceptual loss function includes the following steps: Preprocess the simulated ghost imaging images and their corresponding true value images, and construct an image group; Extracting feature values ​​of images based on image groups; Construct a perceptual loss function based on the extracted image feature values; Constructing a perceptual loss function involves the following steps: ； in, is the overall loss, is the content loss, It is to fight against loss; Content loss The formula is: ,in and are the values ​​of the pixels corresponding to the input image and the network output image respectively; Fighting Losses The formula is: ； in, is the input image, is the output image of the network, the feature map obtained after the jth convolutional layer and before the i-th maximum pooling layer in the VGG16 network, is the mapping function of the VGG16 network, and Describes the dimensions of the expected feature map of the VGG16 network, and It is the specific pixel of dimension.

2. The low-resolution ghost imaging image reconstruction method based on a hierarchical network according to claim 1, characterized in that: Extracting the feature values ​​of an image based on an image group includes the following steps: Obtain two identical ghost images, and construct a three-channel color image based on the two identical ghost images; The three-channel color image is input into the pre-trained VGG16 model to extract image feature values.

3. The low-resolution ghost imaging image reconstruction method based on a hierarchical network according to claim 1, characterized in that: The multi-level network uses a method of aggregating multi-scale features to input the sliced ​​image into the network and perform noise decomposition through the slices; The multi-layer network uses the feature information learned by the bottom-layer encoder and decoder as prior information for the upper-layer encoder and decoder.

4. A low-resolution ghost imaging image reconstruction device based on a hierarchical network, characterized by: including processors and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to execute the steps of the method according to any one of claims 1 to 3.

Images