Self-adaptive undersampling single-pixel imaging method based on deep learning
Through the adaptive undersampling method based on deep learning, the Fourier undersampling strategy is optimized using the generative adversarial network, the problem of undersampling ignores medium and high-frequency information in single-pixel imaging technology is solved, high-quality image reconstruction is achieved, and imaging efficiency and contrast are improved.
Patent Information
- Application Number
- CN202510280272.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-03-11
- Publication Date
- 2025-06-27
AI Technical Summary
The existing single-pixel imaging technology ignores medium and high frequency information during undersampling, resulting in a decrease in image quality, especially the reduction in contrast and insufficient detailed reproduction capabilities.
Adaptive undersampling method based on deep learning is adopted, and Fourier undersampling strategy is optimized by using generative adversarial networks. Generative adversarial network models are trained to generate real-time sampling templates to complete high-quality image reconstruction.
It improves the contrast and detail reproduction capabilities of the image, improves the imaging efficiency and quality, and has high adaptability and generalization capabilities.
Smart Images
Figure CN120219618A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to an optimization method for single-pixel imaging, and specifically relates to an adaptive undersampling single-pixel imaging method based on deep learning. Background Art
[0002] In recent years, single-pixel imaging technology based on ghost imaging systems has gradually become a research hotspot. The core advantage of this technology is that it does not require the use of a spatial resolution detector, and only uses a single-point optoelectronic detector to achieve efficient image acquisition. These characteristics enable single-pixel imaging technology to show broad application potential in many fields such as three-dimensional imaging, radar detection imaging, multispectral imaging, terahertz imaging, and real-time imaging. Since natural images are usually sparse in the Fourier domain, specifically manifested as most of the energy of the image spectrum concentrated in the low-frequency part, undersampling can be performed on this part, significantly reducing the number of samplings while ensuring a certain quality of the reconstructed image and improving the imaging efficiency.
[0003] While the undersampling strategy improves the imaging efficiency, it also causes the problem of image quality degradation. Specifically, undersampling only obtains the low-frequency information of the image and ignores the mid- and high-frequencies. This behavior of truncating the high-frequency components is equivalent to two-dimensional ideal low-pass filtering, which will cause oscillation phenomena during the time-frequency domain conversion. This phenomenon is manifested in the spatial domain as the ringing effect near sharp edges and the appearance of circular pseudo-edges. Therefore, simply undersampling the low-frequency reduces the image contrast and affects the reproduction of image details. Moreover, most of the current sampling optimization algorithms estimate the spatial frequencies where high-frequency information exists based on experience, lacking the direct utilization of real high-frequency components, and the final optimization results are not excellent, still lacking in the ability to retain image details. Summary of the Invention
[0004] The present invention proposes an adaptive undersampling single-pixel imaging method based on deep learning, which uses a deep learning method based on a generative adversarial network model to optimize the Fourier undersampling strategy and improve the Fourier single-pixel imaging efficiency.
[0005] The technical solution for implementing the present invention is as follows: An adaptive undersampling single-pixel imaging method based on deep learning, comprising the following steps:
[0006] Step 1: Using Fourier single-pixel imaging technology, calculate the real undersampling template;
[0007] Step 2: Perform undersampling preprocessing on the natural image dataset;
[0008] Step 3: Use the Fourier spectrum after low-frequency undersampling as training data to train the generative adversarial network model;
[0009] Step 4: Use a single-pixel imaging system to collect the low-frequency spectrum of the target image, then input it into the trained generative adversarial network model to obtain a real-time sampling template. Use the real-time sampling template to complete all samplings, and use the sampling results to reconstruct a high-quality image.
[0010] Preferably, using the Fourier single-pixel imaging technology, the specific process of calculating the real undersampling template is as follows:
[0011] Step 1.1: Generate Fourier basis using the frequency domain method;
[0012] Step 1.2: Calculate the Fourier coefficients obtained by projecting the Fourier basis of different spatial frequencies using the three-step phase-shift method;
[0013] Step 1.3: According to the calculation in Step 1.2, obtain the complete Fourier coefficient matrix of the reconstructed image to acquire the real undersampling template.
[0014] Preferably, set the size of the image to be reconstructed as M×N, and generate an all-zero matrix of M×N representing the Fourier spectrum of the Fourier basis pattern to be generated. Set the value of any spatial frequency point (u0, v0) in the all-zero matrix to Then perform a two-dimensional inverse Fourier transform on the all-zero matrix, take the real part of the result, and obtain the Fourier basis pattern corresponding to the spatial frequency Specifically expressed as:
[0015]
[0016] where real{·} is the operation of taking the real part, F -1 {} represents the inverse Fourier transform, (u, v) is the spatial frequency, is the initial phase.
[0017] Preferably, in Step 1.2, using the three-step phase-shift method to calculate the Fourier coefficients obtained by projecting the Fourier basis of different spatial frequencies is specifically as follows:
[0018]
[0019] where I(x, y) is the image, is the measurement value obtained using the Fourier basis pattern The initial phases are respectively set to 0, 2π / 3, 4π / 3 to obtain D0, D 2π / 3 、D 4π / 3 , and F(u, v) is the Fourier coefficient corresponding to the spatial frequency obtained by using the three-step phase-shift.
[0020] Preferably, the specific method for performing undersampling preprocessing on the natural image dataset is as follows:
[0021] Step 2.1: Design a low-frequency undersampling template Specifically: Generate a all-zero matrix of M×N, set the sampling rate to β′, calculate the number of samples as M×N×β′, divide a circular area with the center of the matrix as the center, set the points within the circular area to 1, the quantity is the same as the number of samples, and the rest are set to 0 to obtain the low-frequency undersampling template M;
[0022] Step 2.2: Select a set number of natural images and reshape the size of the natural images to M×N;
[0023] Step 2.3: Perform Fourier transform on the reshaped natural image x in Step 2.2 to obtain the Fourier spectrum of the training image Shift the zero frequency of the Fourier spectrum and multiply it with the low-frequency undersampling template to obtain the Fourier spectrum I after low-frequency undersampling B .
[0024] Preferably, the generative adversarial network includes a generator and a discriminator;
[0025] The generator adopts a U-net network structure, and the input data of the generator is the low-frequency undersampled spectrum I obtained in Step 2.3 B , and perform backbone feature extraction through the Encoder part of the generator. The Encoder part includes three backbone feature extraction processes. Each feature extraction first passes through two same convolutional layers to obtain multi-channel features, and then performs downsampling to obtain a feature set; the deepest feature set obtained after three backbone feature extractions is input to the Decoder part of the generator for enhanced feature extraction. The Decoder part of the generator includes three enhanced feature extraction processes. The enhanced feature extraction process upsamples the feature set and adds a skip connection process during the upsampling process. Specifically, it combines the multi-channel features extracted by the corresponding depth of the Encoder part with the result obtained after upsampling to obtain an enhanced feature set, and then restores the features through a convolutional layer; the final output of the generator is the sampling template I M ;
[0026] The discriminator adopts a PatchGAN structure, and the input of the PatchGAN is O M and O G , O M and O G are the undersampled images obtained by using I M and I G as sampling strategies and through Fourier reconstruction respectively. The discriminator uses a convolutional layer to map O M and O G to an N×N matrix, and the elements in the N×N matrix represent O M and OG The evaluation value of a certain area is calculated, and the average value of the evaluation index is used as the final output of the discriminator D.
[0027] Preferably, by training the generative adversarial network model and training the performance of the generator in the network, the total loss function L is minimized. The total loss function L is expressed as:
[0028] L = L G + L D
[0029] In the formula, L G is the generator loss function, and L D is the discriminator loss function;
[0030] The generator loss function L G is optimized by calculating the prediction deviation between the sampled template I M output by the generator and the real undersampled template I G Specifically:
[0031] L G = L adv + L b
[0032] Among them, L b is the prediction loss of the generator, and L adv is the generative adversarial loss of the generator;
[0033] The discriminator loss function L D is specifically:
[0034]
[0035] Compared with the prior art, the significant advantages of the present invention are: (1) The present invention has high self - adaptability and generalization ability. The network of the present invention models the correlation between the spatial domain and the frequency domain through a deep learning network, can be more widely applicable to different types of natural images, and provides a better sampling strategy, especially with outstanding effects on targets with rich details. (2) The present invention attempts to fit the quantitative relationship between the time - domain information and the spectrum through a deep learning network, thereby improving the accuracy of sampling optimization. This data - driven strategy can capture the key details in the image more accurately compared with the traditional experience - based sampling optimization method, improving the imaging efficiency and quality.
[0036] The following further describes the present invention in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS
[0037] Figure 1 is the generative adversarial network framework and algorithm flow designed by the present invention.
[0038] Figure 2 Partial images of the dataset used throughout the present invention.
[0039] Figure 3 Generator network structure designed for the present invention.
[0040] Figure 4 Discriminator network structure designed for the present invention.
[0041] Figure 5 Schematic diagram of the performance of the optimal network model of the present invention.
[0042] Figure 6 Experimental setup of the passive single-pixel imaging system. Detailed implementation manners
[0043] As Figure 1 shown, the concept of the present invention is: an adaptive undersampling single-pixel imaging method based on deep learning, which uses a deep learning method based on a generative adversarial network model to optimize the Fourier undersampling strategy and improve the Fourier single-pixel imaging efficiency. The information of natural images is sparse in the Fourier domain, and images can be efficiently reconstructed through undersampling. Natural images are used and a series of preprocessings are performed on them as the dataset for network training. A generative adversarial network model is designed, and a generator and a discriminator are constructed. The model is trained, and according to the generator loss and the discriminator loss, the accuracy of the sampling template generated by the network is trained. During this process, the calculation process is optimized to ensure the differentiability of each processing process. Finally, the network model parameters that can adaptively design the undersampling template are obtained, and the corresponding sampling template is generated. The specific steps are as follows:
[0044] Step 1: Calculate the true undersampling template by using the principle of the Fourier single-pixel imaging system.
[0045] Step 1.1: Generate Fourier bases using the frequency domain method. The specific process is as follows: Set the size of the image to be reconstructed as M×N, and generate an all-zero matrix of M×N representing the Fourier spectrum of the Fourier basis pattern to be generated. Set the value of any spatial frequency point (u0, v0) in the all-zero matrix as Then perform a two-dimensional inverse Fourier transform on the all-zero matrix, and take the real part of the result to obtain the Fourier basis pattern corresponding to the spatial frequency The specific formula for generating the Fourier basis is:
[0046]
[0047] where real{·} is the operation of taking the real part, F -1 {} represents the Fourier inverse transform, (u, v) is the spatial frequency, is the initial phase.
[0048] Step 1.2: Calculate the Fourier coefficients obtained from the projections of Fourier bases with different spatial frequencies using the three-step phase-shifting method, specifically as follows:
[0049]
[0050] where \(I(x, y)\) is the image, is the measured value obtained using the Fourier basis pattern The initial phases are set to 0, \(2\pi / 3\), and \(4\pi / 3\) respectively to obtain \(D_0\), \(D\) 2π / 3 and \(D\) 4π / 3 , and \(F(u, v)\) is the Fourier coefficient corresponding to the spatial frequency obtained using the three-step phase-shifting;
[0051] Step 1.3: Calculate the complete Fourier coefficient matrix of the reconstructed image according to Step 1.2. Set the sampling rate to \(\beta\), calculate the number of sampling points as \(M\times N\times\beta\) according to the sampling rate, perform intensity sorting on the coefficient matrix according to the magnitudes of the Fourier coefficients, take the first part of the coefficients as the sampling points, set the corresponding spatial frequencies to 1, and set the rest to 0. The obtained undersampling template is used as the true undersampling template \(I\) G for calculating the prediction loss of the generator during network training.
[0052] Step 2: Perform undersampling preprocessing on the natural image dataset.
[0053] Step 2.1: Natural images exhibit sparsity in the Fourier domain, and the key information is concentrated near the zero frequency. According to this characteristic, design a low-frequency undersampling template to preprocess the images, specifically as follows: Generate an all-zero matrix of size \(M\times N\), set the sampling rate to \(\beta'(\beta>\beta')\), calculate the number of samples as \(M\times N\times\beta'\), divide a circular region centered at the matrix center, set the points within this region to 1, and the number of them is the same as the number of samples, and set the rest to 0 to obtain the low-frequency undersampling template \(M\).
[0054] Step 2.2: Select 20,000 natural images from ILSVRC2012, reshape the sizes of these images to \(M\times N\), and some pictures are as shown in Figure 2 ;
[0055] Step 2.3: Perform Fourier transform on the training image \(x\) processed in Step 2.2 to obtain the Fourier spectrum of the training image \(x\). Then, after shifting the zero frequency of the Fourier spectrum and multiplying it by the low-frequency undersampling template , corresponding to the processing at the beginning of the Figure 1 algorithm flow. The Fourier spectrum \(I\) B after low-frequency undersampling is calculated by the following formula:
[0056] I B = M·F{x}
[0057] At this time, I B The data distribution in it is as follows: the middle circular area is the spectrum, and the periphery is 0.
[0058] Step 3: Use the Fourier spectrum after low-frequency undersampling as training data to train the generative adversarial network model, where the generative adversarial network model is as Figure 1 shown.
[0059] The generative adversarial network consists of a generator and a discriminator.
[0060] In a further embodiment, a U-net network structure is used as the generator to deeply extract image features. The U-net network is roughly a U-shaped structure, and its structure is as Figure 3 shown. The input data of the network is the low-frequency undersampled spectrum I B obtained in Step 2.3. After passing through the Encoder part for backbone feature extraction, the Encoder part contains three backbone feature extraction processes. Each time of feature extraction first passes through two same convolutional layers to obtain multi-channel features, and then performs downsampling. The downsampling part is composed of a double convolutional layer and a max pooling layer (Maxpool). After downsampling, a feature set is obtained. The deepest feature set obtained after three backbone feature extractions will be input into the Decoder part for enhanced feature extraction. This part also contains three enhanced feature extraction processes. The enhanced feature extraction process will upsample the feature set and add a skip connection process during the upsampling process. Specifically, the multi-channel features extracted by the corresponding depth of the Encoder part are combined with the result obtained after upsampling to obtain an enhanced feature set, and then the features are restored through a convolutional layer; the final output of the generator is the sampling template I M ;
[0061] In a further embodiment, a PatchGAN structure is used as the discriminator. The input of the PatchGAN is O M and O G , O M and O G are the undersampled images O M and I G obtained after using I M and I G as sampling strategies and performing Fourier reconstruction. The discriminator uses convolutional layers to map O M and O G into an N×N matrix, and the specific structure is as Figure 4 shown. The elements in the N×N matrix represent O M and O GThe evaluation value of a certain area in [the text] is used, and finally the average value of the evaluation index is calculated as the final output of the discriminator D.
[0062] By training the generative adversarial network model, the performance of the generator in the network is trained to minimize the total loss function L, expressed as:
[0063] L = L G + L D
[0064] In the formula, L G is the generator loss function, and L D is the discriminator loss function.
[0065] The generator loss function L G is optimized by calculating the prediction deviation between the sampled template I M output by the generator and the real undersampled template I G , specifically as follows:
[0066] L G = L adv + L b
[0067] Among them, L b is the prediction loss of the generator, and L adv is the generative adversarial loss of the generator;
[0068] The discriminator loss function L D is specifically as follows:
[0069]
[0070] Train to obtain the network parameter model.
[0071] Step 4: Use the single-pixel imaging system to collect the low-frequency spectrum of the target image, then input it into the trained generative adversarial network model to obtain the real-time sampling template, use the real-time sampling template to complete all samplings, and use the sampling results to reconstruct the high-quality image.
[0072] Step 4.1: Generate the base pattern of the spatial frequency corresponding to the sampling points in the sampling template according to the low-frequency undersampling template designed in Step 2.3 and project it onto the target image. Use the detector to receive the optical signal, and the value of the optical signal is used as the Fourier coefficient of the corresponding spatial frequency to obtain the low-frequency undersampled spectrum I B , and the experimental device is as Figure 6 shown.
[0073] Step 4.2: Input the low-frequency undersampled spectrum obtained in Step 4.1 into the generative adversarial network model to obtain the sampling template I E。
[0074] Step 4.3: According to the sampling template I E Continue with target sampling to obtain a Fourier coefficient matrix with a sampling rate of β. Through inverse Fourier transform, a high-quality target image can be reconstructed.
[0075] Verify the performance of the undersampling template generated by the generative adversarial network model of the present invention, corresponding to Figure 5 。
[0076] Use the key spectrum coverage rate as the evaluation index for the sampling template generated by the generator. Assume that the total number of sampling points of the sampling template I M generated by the generator is N. Take the modulus of all Fourier coefficients of the original image and sort them in descending order. After sorting, record the coordinates of the first N components and mark them to obtain the sampling template I G sorted by intensity, which represents the position of the key spectrum. Define S cov as the overlapping region of the sampling points in I M and I G . The number of sampling points included in S cov is N cov . Define the key spectrum coverage rate η, which is expressed as:
[0077]
[0078] Input the test set pictures into the network model to obtain the optimized sampling template, as shown in Figure 5 (c). Figure 5 (a) is the test set image, Figure 5 (b) is the true undersampling template at a certain sampling rate, Figure 5 (d) is the spectrum coverage map, and calculate the key spectrum coverage rate to verify the performance of the network model.
[0079] Finally, obtain the network model for designing the adaptive undersampling template and the predicted undersampling template.
[0080] The present invention proposes a key spectrum adaptive undersampling method for Fourier single-pixel imaging based on deep learning, which will be an undersampling Fourier single-pixel imaging method with adaptive ability, generalization ability, and high efficiency.
Claims
1. An adaptive undersampling single-pixel imaging method based on deep learning, characterized in that: The steps include: Step 1: Calculate the true under-sampling template using Fourier single-pixel imaging technology; Step 2: Perform undersampling preprocessing on the natural image dataset; Step 3: Use the Fourier spectrum after low-frequency undersampling as training data to train the generative adversarial network model; Step 4: Use a single-pixel imaging system to capture the low-frequency spectrum of the target image, and then input it into the trained generative adversarial network model to obtain a real-time sampling template. Use the real-time sampling template to complete all sampling, and use the sampling results to reconstruct a high-quality image.
2. The method for adaptive undersampling single pixel imaging based on deep learning according to claim 1, characterized in that: Using Fourier single pixel imaging technology, the specific process of calculating the true under-sampling template is as follows: Step 1.1: Generate Fourier basis using frequency domain method; Step 1.2: Use the three-step phase shift method to calculate the Fourier coefficients obtained by Fourier basis projection at different spatial frequencies; Step 1.3: Calculate the complete Fourier coefficient matrix of the reconstructed image according to step 1.2 to obtain the true under-sampling template.
3. The method for adaptive undersampling single pixel imaging based on deep learning according to claim 2, characterized in that: Set the image size to be reconstructed to M×N and generate an M×N all-zero matrix represents the Fourier spectrum of the Fourier basis pattern to be generated, and sets the value of any spatial frequency point (u0, v0) in the all-zero matrix to Then perform a two-dimensional inverse Fourier transform on the all-zero matrix and take the real part of the result to obtain the Fourier basis pattern corresponding to the spatial frequency Specifically expressed as: Among them, real{·} is the real part operation, F -1 {} represents the inverse Fourier transform, (u,v) is the spatial frequency, The initial phase.
4. The method for adaptive undersampling single pixel imaging based on deep learning according to claim 3, characterized in that: Step 1.2 uses the three-step phase shift method to calculate the Fourier coefficients obtained by Fourier basis projection at different spatial frequencies: Among them, I(x,y) is the image, To use the Fourier basis pattern The measured value, initial phase Set them to 0, 2π / 3, and 4π / 3 respectively, and get D0, D 2π3 , D 4π3 , F(u,v) is the Fourier coefficient of the corresponding spatial frequency obtained using three-step phase shift.
5. The method for adaptive undersampling single pixel imaging based on deep learning according to claim 1, characterized in that: The specific method for undersampling preprocessing of natural image datasets is: Step 2.1: Design low-frequency undersampling template Specifically, an M×N all-zero matrix is generated, the sampling rate is set to β′, the number of samples is calculated to be M×N×β′, a circular area is divided with the center of the matrix as the center, the points in the circular area are set to 1, the number is consistent with the number of samples, and the rest are set to 0, and a low-frequency under-sampling template M is obtained; Step 2.2: Select a set number of natural images and reshape the natural images into M×N; Step 2.3: Perform Fourier transform on the natural image x reshaped in step 2.2 to obtain the Fourier spectrum of the training image Fourier spectrum After zero frequency shift and low frequency undersampling template Multiply to obtain the Fourier spectrum I after low-frequency undersampling B .
6. The method for adaptive undersampling single pixel imaging based on deep learning according to claim 1, characterized in that: The generative adversarial network includes a generator and a discriminator; The generator adopts the U-net network structure, and the input data of the generator is the low-frequency under-sampled spectrum I obtained in step 2.3 B , the main feature is extracted through the generator Encoder part. The Encoder part includes three main feature extraction processes. Each feature extraction first passes through the same convolution layer twice to obtain multi-channel features, and then downsamples to obtain a feature set; the deepest feature set obtained after three main feature extractions is input into the generator Decoder part for enhanced feature extraction. The generator Decoder part includes three enhanced feature extraction processes. The enhanced feature extraction process upsamples the feature set and adds a jump connection process during the upsampling process, specifically combining the multi-channel features extracted by the Encoder part of the corresponding depth with the results obtained after upsampling to obtain an enhanced feature set, and then restores the features through the convolution layer; The final output of the generator is sampling template I M ; The discriminator adopts the PatchGAN structure, and the input of PatchGAN is O M and O G , O M and O G Use I M and I G As the sampling strategy and the under-sampled image obtained after Fourier reconstruction, the discriminator uses a convolutional layer to convert O M and O G Mapped to an N×N matrix, the elements in the N×N matrix represent O M and O G The evaluation value of a certain area in the image is obtained, and the average value of the evaluation index is calculated as the final output of the discriminator D.
7. The method for adaptive undersampling single pixel imaging based on deep learning according to claim 6, characterized in that: By training the generative adversarial network model, the performance of the generator in the training network is minimized, and the total loss function L is expressed as: L=L G +L D Where, L G is the generator loss function, L D is the discriminator loss function; Generator loss function L G The sample template I output by the calculation generator M Compared with the real under-sampling template I G The prediction deviation between them is calculated and the generator is optimized, specifically: L G =L adv +L b Among them, L b is the prediction loss of the generator, L adv Generative adversarial loss for the generator; Discriminator loss function L D Specifically: