A Fourier single-pixel imaging reconstruction method based on coordination of time domain and frequency domain

Abstract

The present invention discloses a Fourier single-pixel imaging reconstruction method based on coordination of time domain and frequency domain, the method includes an acquisition process and a reconstruction process, wherein: the acquisition process is to encode the measured target using a Fourier basis pattern under a set sampling mode, and then obtain its light intensity value through a single-pixel detector, and obtain the Fourier coefficient of the corresponding spatial frequency according to the light intensity value, so as to obtain the under-sampled Fourier spectrum of the target. After the target is acquired through the experimental device, a reconstruction method based on coordination of time domain and frequency domain is used to reconstruct the target image from the acquired under-sampled Fourier spectrum. The reconstruction method of the present invention uses a neural network to coordinate time domain information and frequency domain information to reconstruct the under-sampled spectrum, which can solve the problem of quality degradation under the condition of traditional Fourier single-pixel undersampling, thereby improving the efficiency of single-pixel imaging.

Description

Technical Field

[0001] The invention belongs to the field of optoelectronic imaging, and relates to a Fourier single-pixel imaging reconstruction method, and specifically to a Fourier single-pixel imaging reconstruction method based on coordination of time domain and frequency domain. Background Art

[0002] As a type of imaging technique for single-pixel imaging, Fourier single-pixel imaging technology is one of the technologies that has received much attention due to its high imaging quality and the advantage of Fourier spectrum sparsity that allows undersampling. In order to improve imaging efficiency, undersampling is usually used to reduce the number of measurements, thereby reducing data acquisition time. However, since undersampling only collects low-frequency coefficients and ignores high-frequency coefficients, the reconstructed image has an obvious ringing effect and lacks detailed information. The lower the sampling rate, the more obvious the degradation of the reconstructed image. Summary of the invention

[0003] In order to improve the imaging quality of Fourier single pixel imaging under undersampling conditions, the present invention provides a Fourier single pixel imaging reconstruction method based on coordination of time domain and frequency domain. The method uses a neural network to coordinate time domain information and frequency domain information, and directly reconstructs a high-quality image from the undersampling spectrum.

[0004] The objective of the present invention is achieved through the following technical solutions:

[0005] A Fourier single-pixel imaging reconstruction method based on coordination of time domain and frequency domain includes an acquisition process and a reconstruction process. The specific steps are as follows:

[0006] Step 1: Collection process

[0007] Step 1: Computer generated Fourier basis pattern

[0008] Step 1 and 2, setting a sampling mode, using a spatial light modulator to load a related Fourier basis pattern, wherein: the spatial light modulator can be a spatial light modulator (SLM) or a digital micromirror device (DMD);

[0009] Step 13: Use a single pixel detector to obtain the spatial frequency of the target (f x ,f y ) of modulated light intensity, thereby obtaining a spatial frequency of (f x ,f y )’s Fourier spectrum T(f x ,f y ), wherein: the method for obtaining the spectrum of the corresponding spatial frequency may adopt a four-step phase shift method or a three-step phase shift method, and the single pixel imaging technology may adopt a front-end modulation technology and a back-end modulation technology;

[0010] Step 14: Repeat the operation of step 13 until all sampling points are measured, and finally obtain the target under-sampled spectrum T;

[0011] Step 2: Reconstruction process

[0012] The target image is directly restored from the undersampled spectrum T through the reconstruction process, including the following steps:

[0013] Step 21: Construct a data set. According to the principle of the acquisition process, simulate under-sampled Fourier single-pixel imaging measurement is performed on the image of the data set to obtain an under-sampled spectrum data set Y of the data set.

[0014] Step 22: Prepare the supervised label of the data set, perform full sampling on the images of the data set to obtain the frequency domain label, i.e., the fully sampled Fourier spectrum Y1, perform inverse Fourier transform on the fully sampled Fourier spectrum Y1, and obtain the time domain label, i.e., the fully sampled time domain image Y2;

[0015] Step 2: Design and build a reconstruction network.

[0016] Step 24: Use the under-sampled data set Y obtained in step 21 to train the reconstruction network and obtain the best model;

[0017] Step 25: Use the best model trained in step 24, input the undersampled spectrum collected by the experimental device into the best model, and obtain the reconstruction result of Fourier single-pixel imaging.

[0018] Compared with the prior art, the present invention has the following advantages:

[0019] 1. The reconstruction method of the present invention is a Fourier single-pixel imaging image reconstruction method based on a deep neural network. The main framework of the network adopts a residual network, and the undersampled Fourier spectrum is input into the network model. The network extracts and reconstructs the input frequency domain data. In the reconstruction network, the fully sampled frequency domain data and the time domain image cooperate to supervise the Fourier spectrum and reconstructed image reconstructed by the network, and finally obtain a high-quality Fourier single-pixel imaging reconstruction result.

[0020] 2. The reconstruction method of the present invention utilizes a neural network to coordinate time domain information and frequency domain information to reconstruct the undersampled spectrum, which can solve the problem of quality degradation under traditional Fourier single-pixel undersampling conditions, thereby improving single-pixel imaging efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 The flowchart is a workflow of the Fourier single pixel imaging reconstruction method based on the coordination of time domain and frequency domain;

[0022] Figure 2 Reconstruct the network structure diagram for the present invention. DETAILED DESCRIPTION

[0023] The technical solution of the present invention is further described below in conjunction with the accompanying drawings, but is not limited thereto. Any modification or equivalent replacement of the technical solution of the present invention without departing from the spirit and scope of the technical solution of the present invention should be included in the protection scope of the present invention.

[0024] The present invention provides a Fourier single-pixel imaging reconstruction method based on the coordination of time domain and frequency domain, the method includes an acquisition process and a reconstruction process, wherein: the acquisition process is to encode the target under test using the Fourier basis pattern in a set sampling mode, and then obtain its light intensity value through a single-pixel detector, and obtain the Fourier coefficient of the corresponding spatial frequency according to the light intensity value, thereby obtaining the under-sampled Fourier spectrum of the target. After the target is acquired through the experimental device, a reconstruction method based on the coordination of time domain and frequency domain is used to reconstruct the target image from the acquired under-sampled Fourier spectrum. Figure 1 As shown, the specific steps include:

[0025] Step 1: Collection process

[0026] Step 1: Computer generated Fourier basis pattern It is expressed as:

[0027]

[0028] Where a is the average intensity that determines the average brightness of the illumination pattern, b is the amplitude of the modulation pattern, and x and y represent plane coordinates. is the phase, f x and f y are the spatial frequencies corresponding to the plane coordinates;

[0029] Step 1 and 2: According to the set sampling method, assuming that the image size is M×N, the sampling rate is S, the sampling times of the four-step phase shift rule is 4×M×N×S, and the sampling times of the three-step phase shift rule is 3×M×N×S, and the spatial light modulator (SLM) or digital micromirror device (DMD) is used to load the relevant Fourier basis pattern;

[0030] Step 13: Get the corresponding spatial frequency (f x ,f y )’s Fourier spectrum T(f x ,f y ), according to the four-step phase shift method, the spatial frequency (f x ,f y ) and four Fourier basis patterns with initial phases of 0, π / 2, π, and 3π / 2 are irradiated onto the object or the reflected light of the object is irradiated onto the DMD. These four Fourier basis patterns are recorded as P0, P π / 2 , Pπ and P 3π / 2 , the single pixel detector collects signals in turn to obtain light intensity values ​​D0, D π / 2 , D π and D 3π / 2 According to the four light intensity values ​​measured by the single pixel detector, the frequency (f x ,f y )’s Fourier spectrum T(f x ,f y ) is expressed as:

[0031] T(f x ,f y )=[D0(f x ,f y )-D π (f x ,f y )]+j[D 3π / 2 (f x ,f y )-D π / 2 (f x ,f y )]

[0032] Or use the three-step phase shift method to shift the spatial frequency (f x ,f y ) and three Fourier basis patterns with initial phases of 0, 2π / 3 and 4π / 3 are irradiated onto the object or the reflected light of the object is irradiated onto the DMD. These three Fourier basis patterns are recorded as P0, P 2π / 3 and P 4π / 3 , the single pixel detector collects signals in turn to obtain light intensity values ​​D0, D 2π / 3 and D 4π / 3 According to the three light intensity values ​​measured by the single pixel detector, the frequency (f x ,f y )’s Fourier spectrum T(f x ,f y ) is expressed as:

[0033]

[0034] Step 14: Repeat the operation of step 13 until all sampling points are measured, and finally obtain the target under-sampled spectrum T;

[0035] Step 2: Reconstruction process

[0036] The target image is directly restored from the undersampled spectrum T through the reconstruction process, including the following steps:

[0037] Step 21: Construct a data set. According to the principle of the acquisition process, simulate under-sampled Fourier single-pixel imaging measurement is performed on the image of the data set to obtain an under-sampled spectrum data set Y of the data set.

[0038] Step 22: Prepare the supervised label of the data set, perform full sampling on the images of the data set to obtain the frequency domain label, i.e., the fully sampled Fourier spectrum Y1, perform inverse Fourier transform on the fully sampled Fourier spectrum Y1, and obtain the time domain label, i.e., the fully sampled time domain image Y2;

[0039] Step 2: Design and build a reconstruction network. The main architecture of the reconstruction network adopts a residual network (ResNet), which consists of two convolutional layers, four residual blocks (ResBlock) and an inverse Fourier transform layer (IFFT);

[0040] Step 24: Use the undersampled data set Y obtained in step 21 to train the reconstruction network. The training of the reconstruction network uses the coordination of frequency domain loss and time domain loss to achieve high-quality reconstruction from the undersampled spectrum to the Fourier single-pixel imaging result. The loss function L includes the frequency domain loss L1 and the time domain loss L2, as shown in the following formula:

[0041]

[0042] Among them, f represents the reconstruction network, f conv2 Represents the Conv2 layer of the reconstructed network; according to the chain rule, the Adam optimizer is used to continuously optimize the loss function until the network converges and the loss no longer decreases. During the network training process, the model with the smallest loss function is continuously saved. When the training is completed, the best model is obtained;

[0043] Step 25: Use the best model trained in step 24, input the undersampled spectrum collected by the experimental device into the best model, and obtain the reconstruction result of Fourier single-pixel imaging.

[0044] Example:

[0045] In this embodiment, the Fourier single pixel imaging reconstruction method based on the coordination of time domain and frequency domain includes an acquisition process and a reconstruction process, and the specific steps are as follows:

[0046] Step 1: Collection process

[0047] Step 1: Computer generated Fourier basis pattern It is expressed as:

[0048]

[0049] Where a is the average intensity that determines the average brightness of the illumination pattern, b is the amplitude of the modulation pattern, and x and y represent plane coordinates. is the phase, fx and f y are the spatial frequencies of the corresponding plane coordinates respectively.

[0050] Step 1 and 2: According to the set sampling method, the image size is 256×256, the sampling rate is 1%, the sampling times are 2622 using the four-step phase shift rule, and the digital micromirror device (DMD) is loaded with the relevant Fourier basis pattern.

[0051] Step 13: Get the corresponding spatial frequency (f x ,f y )’s Fourier spectrum T(f x ,f y ), according to the four-step phase shift method, the spatial frequency (f x ,f y ) and four Fourier basis patterns with initial phases of 0, π / 2, π, and 3π / 2 are irradiated onto the object or the reflected light of the object is irradiated onto the DMD. These four Fourier basis patterns are recorded as P0, P π / 2 , P π and P 3π / 2 , the single pixel detector collects signals in turn to obtain light intensity values ​​D0, D π / 2 , D π and D 3π / 2 According to the four light intensity values ​​measured by the single pixel detector, the frequency (f x ,f y )’s Fourier spectrum T(f x ,f y ) is expressed as:

[0052] T(f x ,f y )=[D0(f x ,f y )-D π (f x ,f y )]+j[D 3π / 2 (f x ,f y )-D π / 2 (f x ,f y )].

[0053] Step 14: Repeat the operation of step 13 until all sampling points are measured, and finally obtain the target under-sampled spectrum T.

[0054] Step 2: Reconstruction process

[0055] After the acquisition process is completed, the under-sampled spectrum of the target is obtained, and the image of the target is directly restored from the under-sampled spectrum T through the reconstruction process, including the following steps:

[0056] Step 21: Construct a data set with a size of 10,000 images. According to the principle of the acquisition process, simulate the under-sampled Fourier single-pixel imaging measurement of the images of the data set to obtain the under-sampled spectrum data set Y of the data set.

[0057] Step 22: Create supervised labels for the dataset:

[0058] The images of the data set are fully sampled to obtain frequency domain labels, that is, the fully sampled Fourier spectrum Y1, and the fully sampled Fourier spectrum Y1 is inversely Fourier transformed to obtain time domain labels, that is, the fully sampled time domain image Y2.

[0059] Step 2 and 3: Design and build a reconstruction network. The reconstruction network structure diagram is as follows: Figure 2 As shown in the figure, the main architecture adopts the residual network (ResNet), which consists of two convolutional layers, four residual blocks (ResBlock) and an inverse Fourier transform layer (IFFT). The first convolutional layer is used as the initial feature extractor to extract the shallow features of the undersampled spectrum data. The four residual blocks form a feature transformation network to further extract high-level features. The last convolutional layer maps the extracted high-level features to frequency domain data, and finally uses a Fourier transform layer to transform the frequency domain data into time domain image information. In the figure, n represents the number of convolution kernels, k represents the convolution kernel size of the 2D convolution, and s represents the step size of the convolution. The residual block mainly consists of two convolutional layers and two Batchnorm2d layers. The function of the Batchnorm2d layer is normalization. The input feature map is connected through a residual connection to add the results after two convolutions. Finally, a relu activation function is used to normalize the pixels in the added feature map to 0-1.

[0060] Step 24: Use the under-sampled data set Y obtained in step 21 to train the reconstruction network. The loss function L includes the frequency domain loss L1 and the time domain loss L2, as shown in the following formula:

[0061]

[0062] Among them, f represents the reconstruction network, f conv2 Represents the Conv2 layer of the reconstructed network; according to the chain rule, the Adam optimizer is used to continuously optimize the loss function until the network converges and the loss no longer decreases. During the network training process, the model with the smallest loss function is continuously saved. When the training is completed, the best model is obtained;

[0063] Step 25: Use the best model trained in step 24, input the undersampled spectrum collected by the experimental device into the best model, and obtain the reconstruction result of Fourier single-pixel imaging.

Claims

1. A Fourier single-pixel imaging reconstruction method based on coordination of time domain and frequency domain, characterized in that The method includes an acquisition process and a reconstruction process, and the specific steps are as follows: Step 1: Collection process Step 1: Computer generated Fourier basis pattern Step 1 and 2: Set the sampling mode and use the spatial light modulator to load the relevant Fourier basis pattern; Step 13: Use a single pixel detector to obtain the spatial frequency of the target (f x ,f y ) of modulated light intensity, thereby obtaining a spatial frequency of (f x ,f y )’s Fourier spectrum T(f x ,f y ); Step 14: Repeat the operation of step 13 until all sampling points are measured, and finally obtain the target under-sampled spectrum T; Step 2: Reconstruction process The target image is directly restored from the undersampled spectrum T through the reconstruction process, including the following steps: Step 21: Construct a data set. According to the principle of the acquisition process, simulate under-sampled Fourier single-pixel imaging measurement is performed on the image of the data set to obtain an under-sampled spectrum data set Y of the data set. Step 22: Prepare the supervised label of the data set, perform full sampling on the images of the data set to obtain the frequency domain label, i.e., the fully sampled Fourier spectrum Y1, perform inverse Fourier transform on the fully sampled Fourier spectrum Y1, and obtain the time domain label, i.e., the fully sampled time domain image Y2; Step 2 and 3, design and build a reconstruction network. The main architecture of the reconstruction network adopts a residual network, which consists of two convolutional layers, four residual blocks and an inverse Fourier transform layer. The first convolutional layer is used as an initial feature extractor to extract shallow features of undersampled spectrum data. The four residual blocks form a feature transformation network to further extract high-level features. The last convolutional layer maps the extracted high-level features to frequency domain data. Finally, a Fourier transform layer is used to transform the frequency domain data into time domain image information. Step 24: Use the under-sampled data set Y obtained in step 21 to train the reconstruction network and obtain the best model. The specific steps are as follows: The reconstruction network is trained using the undersampled data set Y obtained in step 2.

1. The training of the reconstruction network uses the coordination of frequency domain loss and time domain loss to achieve high-quality reconstruction from the undersampled spectrum to the Fourier single-pixel imaging result. The loss function L includes the frequency domain loss L1 and the time domain loss L2, as shown in the following formula: Among them, f represents the reconstruction network, f conv2 Represents the Conv2 layer of the reconstructed network; according to the chain rule, the Adam optimizer is used to continuously optimize the loss function until the network converges and the loss no longer decreases. During the network training process, the model with the smallest loss function is continuously saved. When the training is completed, the best model is obtained; Step 25: Use the best model trained in step 24, input the undersampled spectrum collected by the experimental device into the best model, and obtain the reconstruction result of Fourier single-pixel imaging.

2. The Fourier single pixel imaging reconstruction method based on coordination of time domain and frequency domain according to claim 1 is characterized in that In the steps 1 and 2, the spatial light modulator is a spatial light modulator or a digital micromirror device.

3. The Fourier single pixel imaging reconstruction method based on coordination of time domain and frequency domain according to claim 1 is characterized in that In the steps 1 and 3, the method for obtaining the spectrum of the corresponding spatial frequency adopts a four-step phase shift method or a three-step phase shift method.

4. The Fourier single pixel imaging reconstruction method based on coordination of time domain and frequency domain according to claim 3 is characterized in that When the spectrum method adopts the four-step phase shift method, the specific steps of steps one and three are as follows: Get the corresponding spatial frequency (f x ,f y )’s Fourier spectrum T(f x ,f y ), according to the four-step phase shift method, the spatial frequency (f x ,f y ) and four Fourier basis patterns with initial phases of 0, π / 2, π, and 3π / 2 are irradiated onto the object or the reflected light of the object is irradiated onto the DMD. These four Fourier basis patterns are recorded as P0, P π / 2 , P π and P 3π / 2 , the single pixel detector collects signals in turn to obtain light intensity values ​​D0, D π / 2 , D π and D 3π / 2 According to the four light intensity values ​​measured by the single pixel detector, the frequency (f x ,f y )’s Fourier spectrum T(f x ,f y ) is expressed as: T(f x ,f y )=[D0(f x ,f y )-D π (f x ,f y )]+j[D 3π / 2 (f x ,f y )-D π / 2 (f x ,f y )]。 5. The Fourier single pixel imaging reconstruction method based on coordination of time domain and frequency domain according to claim 3 is characterized in that When the spectrum method adopts the three-step phase shift method, the specific steps of steps one and three are as follows: Get the corresponding spatial frequency (f x ,f y )’s Fourier spectrum T(f x ,f y ), the spatial frequency (f x ,f y ) and three Fourier basis patterns with initial phases of 0, 2π / 3 and 4π / 3 are irradiated onto the object or the reflected light of the object is irradiated onto the DMD. These three Fourier basis patterns are recorded as P0, P 2π / 3 and P 4π / 3 , the single pixel detector collects signals in turn to obtain light intensity values ​​D0, D 2π / 3 and D 4π / 3 According to the three light intensity values ​​measured by the single pixel detector, the frequency (f x ,f y )’s Fourier spectrum T(f x ,f y ) is expressed as:

6. The Fourier single pixel imaging reconstruction method based on coordination of time domain and frequency domain according to claim 1 is characterized in that In the steps 1 and 3, the single pixel imaging technology adopts the front-end modulation technology or the back-end modulation technology.

Images