A method and apparatus for single-pixel imaging combining supervised and unsupervised learning

Abstract

The present application relates to a kind of single-pixel imaging method and device of supervised and unsupervised learning combination, belong to the field of computational photography technology.The method designs a kind of single-pixel one-step imaging network based on Transformer, can directly from undersampled single-pixel measurement value reconstruct target image, without the step of solving approximate result, while using the way of supervised learning and unsupervised learning combination to train the network, guarantee the strong generalization of method.Through the combination of mean square loss function and total variation loss function, prevent network from appearing overfitting problem, guarantee the reconstruction performance of method.The device includes data acquisition submodule, supervised learning submodule, unsupervised learning submodule and reconstruction submodule.The present application can effectively capture the long-range dependence between single-pixel measurement value, can realize high-quality single-pixel imaging under low sampling.

Description

Technical Field

[0001] This invention relates to a single-pixel imaging method and apparatus that combines supervised and unsupervised learning, specifically to a single-pixel imaging method and apparatus based on a Transformer-based one-step imaging network and an imaging network training framework that combines supervised and unsupervised learning, belonging to the field of computational imaging technology. Background Technology

[0002] With the development of science and technology, the demand for low-sampling, low-cost, and high-resolution imaging technologies is constantly increasing. Single-pixel imaging, as a novel imaging technology, has attracted widespread attention from researchers both domestically and internationally. Compared to traditional array imaging using area detectors, single-pixel imaging technology uses a single-pixel detector without spatial resolution to reconstruct two-dimensional / three-dimensional images, making it highly suitable for situations where area detectors cannot image or are too expensive. Furthermore, because single-pixel imaging technology detects the total energy of light reflected or transmitted from an object, it is also suitable for low-light imaging. However, single-pixel imaging sacrifices time in acquiring spatial information, resulting in a certain gap in imaging efficiency and quality compared to traditional array imaging technologies.

[0003] To improve imaging efficiency, model-based single-pixel imaging methods combine convex optimization theory with various hand-designed image priors for undersampling single-pixel imaging. However, these hand-designed image priors are often insufficient to represent all the properties of real-world images, resulting in poor imaging results. Furthermore, this imaging method often requires multiple iterative calculations, leading to high algorithm complexity and significant computational resource consumption.

[0004] With the continuous development of deep learning, it has been applied to single-pixel imaging technology in recent years. Many studies have demonstrated that deep learning algorithms can maintain high single-pixel image reconstruction quality while reducing sampling. However, existing deep learning-based single-pixel imaging methods all employ convolutional neural networks for undersampling. These neural networks excel at extracting local features but neglect long-range dependencies between measurements, thus limiting their imaging performance. Furthermore, most deep learning-based single-pixel imaging methods require two steps for undersampling: first, a model-based single-pixel imaging method reconstructs a noisy approximation; then, this approximation is fed into a deep learning network to obtain a high-quality reconstructed image. This two-step method ignores the physical imaging model of a single pixel and introduces unnecessary noise during the approximation process. Moreover, the first step increases reconstruction time and reduces efficiency. In addition, existing deep learning-based single-pixel imaging methods suffer from poor generalization; if the sampling matrix changes, the learned imaging network becomes ineffective, requiring retraining. This significantly limits the practical application of deep learning-based single-pixel imaging methods.

[0005] In conclusion, there is an urgent need for a high-quality, efficient, and versatile single-pixel imaging technology. Summary of the Invention

[0006] The purpose of this invention is to address the shortcomings and deficiencies of existing technologies by creatively proposing a single-pixel imaging method and apparatus that combines supervised and unsupervised learning to achieve high-quality, efficient, and highly generalizable single-pixel imaging.

[0007] This invention proposes for the first time a Transformer-based single-pixel one-step imaging network that can directly reconstruct the target image from undersampled single-pixel measurements without requiring the step of solving for approximate results. Secondly, this invention proposes a combined supervised and unsupervised learning approach to train the proposed single-pixel one-step imaging network, ensuring strong generalization of the method. By constructing a joint unsupervised loss function combining the mean square loss function and the total variation loss function, overfitting is prevented, guaranteeing the reconstruction performance of the proposed method.

[0008] The present invention is achieved using the following technical solution.

[0009] A single-pixel imaging method combining supervised and unsupervised learning includes the following steps:

[0010] Step 1: Based on the principle of single-pixel imaging, construct a supervised single-pixel reconstruction training dataset.

[0011] Specifically, the single-pixel reconstruction training dataset includes: single-pixel measurements and the target scene corresponding to the measurements.

[0012] The principle of single-pixel imaging is expressed as follows:

[0013] I = Px

[0014] Where x represents the target scene, P represents the sampling matrix, and I represents the single-pixel measurement value.

[0015] Step 2: Construct a Transformer-based single-pixel one-step imaging network and train the network using the dataset constructed in Step 1.

[0016] The Transformer-based single-pixel one-step imaging network includes a feature transformation module, a shallow feature extraction module, a Transformer module, and an image reconstruction module. The feature transformation module converts one-dimensional single-pixel measurements into two-dimensional feature maps. The shallow feature extraction module extracts shallow features from the two-dimensional feature maps. The Transformer module extracts deep features, and the image reconstruction module reconstructs the target scene from the extracted features.

[0017] This invention uses the dataset constructed in step 1 to train the network, enabling the network to learn general prior knowledge of single-pixel reconstruction. In this stage, the training loss function L... sd Represented as:

[0018]

[0019] in, The proposed single-pixel one-step imaging network has parameters w. sd The subscript sd indicates the supervised learning phase; (I i x i Let I be the i-th data point in the constructed supervised dataset. i For single-pixel measurements, x i The target scene corresponding to the measured value; N is the total number of data in the dataset.

[0020] Step 3: Construct the optical path based on the single-pixel imaging principle.

[0021] The target scene is encoded and modulated using a sampling matrix P, and the corresponding undersampled one-dimensional measurement light signal is acquired by a single-pixel detector.

[0022] Step 4: Construct the loss function for the unsupervised learning stage.

[0023] To prevent overfitting, the loss function in the unsupervised learning stage is composed of both mean square loss (MSE loss) and total variation loss (TV loss).

[0024] Wherein, mean square loss function Represented as:

[0025]

[0026] Among them, I ′ These are the single-pixel measurements obtained by a single-pixel detector; This is a single-pixel one-step imaging network with parameters w. ud The subscript ud indicates the unsupervised learning stage.

[0027] Total variational loss function Represented as:

[0028]

[0029] in, For differential operators; I ′ These are the single-pixel measurements obtained by a single-pixel detector; This is a single-pixel one-step imaging network with parameters w. ud The subscript ud indicates the unsupervised learning stage.

[0030] The loss function L in the unsupervised stage ud Represented as:

[0031]

[0032] Where λ is the weighting factor.

[0033] Step 5: Use the network parameters learned in the supervised learning stage as the initial values ​​for the network parameters in the unsupervised learning stage. Input the single-pixel measurement values ​​obtained by the single-pixel detector into the network, and optimize the network using the unsupervised loss function to obtain a high-quality reconstructed target scene corresponding to the measurement values.

[0034] Furthermore, the present invention proposes an apparatus for implementing the method.

[0035] A single-pixel imaging device combining supervised and unsupervised learning includes a data acquisition submodule, a supervised learning submodule, an unsupervised learning submodule, and a reconstruction submodule.

[0036] The data acquisition submodule uses a sampling matrix to encode and modulate the target scene based on the single-pixel imaging principle, and obtains the corresponding single-pixel measurement value through a single-pixel detector.

[0037] The supervised learning submodule is used to perform initial training on the proposed Transformer-based single-pixel one-step imaging network, enabling the network to learn general prior knowledge of single-pixel reconstruction.

[0038] The unsupervised learning submodule is used to further optimize the network parameter values ​​obtained in the supervised learning stage based on the collected measurement values, so that the network learns prior knowledge of single-pixel reconstruction for the target scene.

[0039] The reconstruction submodule uses the network trained by the unsupervised learning submodule to reconstruct the target scene.

[0040] The connections between the above systems are as follows:

[0041] The outputs of the data acquisition submodule and the supervised learning submodule are connected to the input of the unsupervised learning submodule; the output of the unsupervised learning submodule is connected to the input of the reconstruction submodule.

[0042] Beneficial effects

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] 1. This invention can effectively capture the long-range dependence between single-pixel measurements, thereby enabling high-quality single-pixel imaging at low sampling rates.

[0045] 2. This invention effectively addresses the poor generalization problem of previous deep learning-based single-pixel imaging methods by combining supervised and unsupervised learning. When the actual sampling matrix changes, the optimization of the network during the unsupervised learning stage ensures the reconstruction quality of the target scene without requiring re-supervised learning of the imaging network.

[0046] 3. This invention ensures improved reconstruction performance and prevents overfitting in the imaging network by using a combination of mean square loss function and total variation loss function in the unsupervised stage. Attached Figure Description

[0047] Figure 1 This is a schematic flowchart of the method of the present invention;

[0048] Figure 2 This is a schematic diagram of the structure of the Transformer-based single-pixel one-step imaging network proposed in this invention;

[0049] Figure 3 This is a schematic diagram of the imaging network training method that combines supervised and unsupervised learning proposed in this invention;

[0050] Figure 4 This is a simulation result diagram according to an embodiment of the present invention;

[0051] Figure 5 To account for changes in the actual sampling matrix, a simulation result diagram according to an embodiment of the present invention is shown.

[0052] Figure 6 To account for different loss functions used in the unsupervised learning stage, a simulation performance comparison chart of an embodiment of the present invention is provided.

[0053] Figure 7 This is a schematic diagram of the structure of the device of the present invention. Detailed Implementation

[0054] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0055] like Figure 1 As shown, a single-pixel imaging method combining supervised and unsupervised learning is proposed. By using a Transformer-based one-step imaging network, the long-range dependencies between single-pixel measurements can be better captured, thus obtaining high-quality reconstructed images under low sampling conditions. The joint training of the imaging network with supervised and unsupervised learning can ensure the strong generalization of the proposed method, making it suitable for practical applications. The establishment of a joint unsupervised loss function of mean square loss function and total variation loss function can prevent overfitting of the imaging network and ensure reconstruction performance.

[0056] Includes the following steps:

[0057] Step 1: Based on the principle of single-pixel imaging, construct a supervised single-pixel reconstruction training dataset.

[0058] Step 2: Construct a Transformer-based single-pixel one-step imaging network and train the network using the dataset constructed in Step 1.

[0059] Step 3: Construct the optical path based on the single-pixel imaging principle. Encode and modulate the target scene using a sampling matrix, and acquire the corresponding undersampled one-dimensional measurement optical signal through a single-pixel detector.

[0060] Step 4: Construct the loss function for the unsupervised learning stage.

[0061] Step 5: Use the network parameters learned in the supervised learning stage as the initial values ​​of the network parameters in the unsupervised stage. Input the single-pixel measurement values ​​obtained by the single-pixel detector into the network, and optimize the network using the unsupervised loss function to obtain a high-quality reconstructed target image corresponding to the measurement values.

[0062] Furthermore, in embodiments of this application, a supervised single-pixel reconstruction training dataset is constructed based on the single-pixel imaging principle. The single-pixel reconstruction training dataset includes: single-pixel measurements and the corresponding target images. The single-pixel imaging principle is expressed as follows:

[0063] I = Px

[0064] Where x represents the target scene, P represents the sampling matrix, and I represents the single-pixel measurement value.

[0065] Furthermore, in the embodiments of this application, a Transformer-based single-pixel one-step imaging network is constructed, the structure of which is as follows: Figure 2 As shown, the network is trained using the supervised dataset described above, enabling it to learn general prior knowledge of single-pixel reconstruction. The loss function L is trained during this stage. sd Represented as:

[0066]

[0067] in, The proposed single-pixel one-step imaging network has parameters w. sd The subscript sd indicates the supervised learning phase; (I i x i ) represents the i-th data point in the constructed supervised dataset; I i This is a single-pixel measurement; x i The target scene corresponding to the measured value. N is the total number of data in the dataset.

[0068] Furthermore, in the embodiments of this application, an optical path is constructed based on the single-pixel imaging principle. The target scene is encoded and modulated using a sampling matrix P, and the corresponding undersampled one-dimensional measurement optical signal is acquired by a single-pixel detector.

[0069] Furthermore, in the embodiments of this application, the mean square loss function (MSE loss) and the total variation loss function (TV loss) are used to jointly construct the unsupervised loss function.

[0070] The mean squared loss function is expressed as:

[0071]

[0072] Among them, I ′ f is the single-pixel measurement value obtained by the single-pixel detector. wud The proposed single-pixel one-step imaging network has parameters w. ud The subscript ud indicates the unsupervised learning stage.

[0073] The total variational loss function is expressed as:

[0074]

[0075] in, For differential operators; I ′ These are the single-pixel measurements obtained by a single-pixel detector; The proposed single-pixel one-step imaging network has parameters w. ud The subscript ud indicates the unsupervised learning stage.

[0076] The loss function for the unsupervised stage is expressed as:

[0077]

[0078] Where λ is the weighting factor.

[0079] Furthermore, for the single-pixel measurement values ​​obtained from the single-pixel detector, the network parameters learned in the supervised learning stage are used as the initial values ​​of the network parameters in the unsupervised learning stage. The single-pixel measurement values ​​are input into the network, and the network is optimized using the unsupervised loss function, thereby obtaining a high-quality reconstructed target image corresponding to the measurement values.

[0080] Figure 3 This is a schematic diagram illustrating the imaging network training method of an embodiment of this application. By training on a large supervised single-pixel reconstruction training dataset and optimizing single-pixel measurements of the target scene, the proposed Transformer-based one-step single-pixel imaging network learns general prior knowledge of single-pixel reconstruction and essential information about single-pixel reconstruction of the target scene. Compared with other methods, it has better reconstruction results and strong generalization ability.

[0081] Comparison and verification

[0082] To verify the effectiveness of the method in this embodiment of the invention, it is compared with other existing single-pixel imaging methods. In this comparison, a "Cameraman" image is used as the target scene, with a resolution of 64×64. The sampling matrix is ​​an optimized binary sampling matrix, and the sampling rate is 30% (i.e., obtaining approximately 64×64×0.3≈1229 single-pixel measurements). The reconstruction result of the target scene is as follows. Figure 4 As shown, it can be observed that the proposed Transformer-based single-pixel one-step imaging network can reconstruct clearer and more visually appealing target scene images compared to other single-pixel imaging networks. Furthermore, it can be seen that the proposed combined supervised and unsupervised learning approach achieves better imaging results compared to training the network using only supervised learning.

[0083] To verify the generalization ability of the method in this embodiment, the proposed method is compared with other existing deep learning-based single-pixel imaging methods. In this comparison, a "Cameraman" image is used as the target scene, with a resolution of 64×64. The sampling matrix is ​​an optimized binary sampling matrix with a sampling rate of 30%. Random Gaussian white noise (standard deviation of 0.4) is added to the binary sampling matrix of the target scene. The resulting target scene reconstruction is shown below. Figure 5 As shown, when the actual sampling matrix changes, the imaging network using only supervised learning experiences varying degrees of performance degradation. To restore the original imaging performance, it is necessary to retrain the imaging network using a new single-pixel reconstruction training dataset generated with the changed sampling matrix. However, the method in this embodiment of the invention, by jointly training the network using supervised and unsupervised learning, eliminates the need for retraining when the sampling matrix changes. The proposed method solves the problem of poor generalization in previous deep learning-based single-pixel imaging methods and is suitable for practical single-pixel imaging applications.

[0084] To verify the effectiveness of the unsupervised learning stage loss function constructed by the method in this embodiment of the invention, different loss functions are considered for the unsupervised learning stage of the proposed method, and the reconstruction performance is compared. The results are as follows: Figure 6 As shown in the figure, the target scene is a "Cameraman" image with a resolution of 64×64 and a sampling rate of 30%. It can be seen that using only the mean square loss function as the loss function in the unsupervised stage leads to overfitting to noise, meaning the reconstruction performance initially increases but then decreases. However, using a combined mean square loss function and a total variation loss function as the unsupervised loss function ensures improved reconstruction performance and prevents overfitting in the imaging network.

[0085] To achieve the above method, this application also proposes a single-pixel imaging device that combines supervised and unsupervised learning, such as... Figure 7 As shown, it includes a data acquisition submodule, a supervised learning submodule, an unsupervised learning submodule, and a reconstruction submodule.

[0086] The data acquisition submodule uses a sampling matrix to encode and modulate the target scene based on the single-pixel imaging principle, and obtains the corresponding single-pixel measurement value through a single-pixel detector.

[0087] The supervised learning submodule is used to perform initial training on the proposed Transformer-based single-pixel one-step imaging network, enabling the network to learn general prior knowledge of single-pixel reconstruction.

[0088] The unsupervised learning submodule is used to further optimize the network parameter values ​​obtained in the supervised learning stage based on the collected measurement values, so that the network learns prior knowledge of single-pixel reconstruction for the target scene.

[0089] The reconstruction submodule utilizes the network trained by the unsupervised learning submodule to perform high-quality reconstruction of the target image.

[0090] The above detailed description further illustrates the purpose, technical solution, and beneficial effects of the invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A single-pixel imaging method combining supervised and unsupervised learning, characterized in that, Includes the following steps: Step 1: Based on the principle of single-pixel imaging, construct a supervised single-pixel reconstruction training dataset, including: single-pixel measurement values ​​and the target scene corresponding to the measurement values; Step 2: Construct a Transformer-based single-pixel one-step imaging network and train the network using the dataset constructed in Step 1; The Transformer-based single-pixel one-step imaging network includes a feature transformation module, a shallow feature extraction module, a Transformer module, and an image reconstruction module. The feature transformation module is used to transform one-dimensional single-pixel measurements into two-dimensional feature maps. The shallow feature extraction module is used to extract shallow features from the two-dimensional feature maps. The Transformer module is used to extract deep features. The image reconstruction module is used to reconstruct the target scene from the extracted features. Step 3: Construct the optical path based on the single-pixel imaging principle; Step 4: Construct the loss function for the unsupervised learning stage, including the mean square loss function and the total variation loss function; In step 4, the mean squared loss function Represented as: in, These are the single-pixel measurements obtained by a single-pixel detector; This is a single-pixel one-step imaging network with the following parameters: subscript Indicates the unsupervised learning stage The sampling matrix; Total variational loss function Represented as: in, It is a differential operator; These are the single-pixel measurements obtained by a single-pixel detector; This is a single-pixel one-step imaging network with the following parameters: subscript This indicates the unsupervised learning stage; Loss function in the unsupervised stage Represented as: in, As a weighting factor; Step 5: Use the network parameters learned in the supervised learning stage as the initial values ​​of the network parameters in the unsupervised learning stage; input the single-pixel measurement values ​​obtained by the single-pixel detector into the network, optimize the network using the unsupervised loss function, and obtain the reconstructed target scene corresponding to the measurement values.

2. The single-pixel imaging method combining supervised and unsupervised learning as described in claim 1, characterized in that, In step 1, the principle of single-pixel imaging is as follows: in, For the target scenario, For the sampling matrix, These are measurements for a single pixel.

3. The single-pixel imaging method combining supervised and unsupervised learning as described in claim 1, characterized in that, In step 2, when training the single-pixel one-step imaging network, the training loss function is... Represented as: in, The proposed single-pixel one-step imaging network has the following parameters: subscript Indicates the supervised learning phase; , ) is the first in the constructed supervised dataset One data point, This is a single-pixel measurement. The target scene corresponding to the measured value; This represents the total number of data points in the dataset.

4. The single-pixel imaging method combining supervised and unsupervised learning as described in claim 1, characterized in that, In step 3, the target scene is encoded and modulated using a sampling matrix, and the corresponding undersampled one-dimensional measurement light signal is acquired by a single-pixel detector.

5. A single-pixel imaging device for implementing the combined supervised and unsupervised learning of claim 1, characterized in that, It includes a data acquisition submodule, a supervised learning submodule, an unsupervised learning submodule, and a reconstruction submodule; Among them, the data acquisition submodule uses the sampling matrix to encode and modulate the target scene based on the single-pixel imaging principle, and obtains the corresponding single-pixel measurement value through the single-pixel detector; The supervised learning submodule is used to perform preliminary training on the proposed Transformer-based single-pixel one-step imaging network, enabling the network to learn general prior knowledge of single-pixel reconstruction. The unsupervised learning submodule is used to further optimize the network parameter values ​​obtained in the supervised learning stage based on the collected measurement values, so that the network learns prior knowledge of single-pixel reconstruction for the target scene; The reconstruction submodule uses the network trained by the unsupervised learning submodule to reconstruct the target scene; The connections between modules are as follows: The outputs of the data acquisition submodule and the supervised learning submodule are connected to the input of the unsupervised learning submodule; the output of the unsupervised learning submodule is connected to the input of the reconstruction submodule.

Images