A multi-sensor acquisition single-pixel imaging method based on deep learning
By employing multi-sensor acquisition and deep learning methods, and utilizing self-attention conditional generative adversarial networks to reconstruct high-resolution images, the image quality problem of single-pixel imaging at low sampling rates is solved, improving imaging quality and speed, and expanding the application areas.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- GUILIN UNIV OF ELECTRONIC TECH
- Filing Date
- 2022-07-11
- Publication Date
- 2026-05-29
AI Technical Summary
Single-pixel imaging technology has insufficient image quality when reconstructing images at low sampling rates, and existing reconstruction methods have room for improvement in both image quality and speed.
A deep learning-based multi-sensor acquisition method is adopted, which uses an LED array to generate different lighting patterns. Combined with a self-attention conditional generative adversarial network, the image is reconstructed by acquiring data from multiple sensors, including feature unfolding, self-attention feature extraction and conditional generative adversarial network training, and finally a high-resolution image is generated.
It improves the image reconstruction quality and speed of single-pixel imaging, and expands the application scope of single-pixel imaging, especially its application potential in fields such as 3D imaging, terahertz imaging and remote sensing imaging.
Smart Images
Figure CN115187765B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer imaging, specifically a method for improving single-pixel imaging quality based on a self-attention conditional generative adversarial network. Background Technology
[0002] Single-pixel imaging is a novel imaging technology. Compared to traditional charge-coupled devices (CCD) and complementary metal-oxide-semiconductor (CMOS) cameras, this technology illuminates the target scene with different lighting patterns and uses a light intensity sensor to simultaneously acquire the reflected light intensity to obtain spatial information. The acquired light intensity sequence is then used to reconstruct the image using algorithms. Single-pixel imaging has unique advantages, especially in imaging beyond the visible light wavelength. Given these advantages, single-pixel imaging has been applied in fields such as 3D imaging, terahertz imaging, remote sensing imaging, and biomedical imaging.
[0003] Improving image quality at low sampling rates is one of the challenges in single-pixel imaging, leading to the development of various reconstruction methods to enhance image quality. Regarding illumination patterns, binary illumination patterns based on Fast Fourier Transform (FFT) have been proposed. These involve binarying the Fourier Transform basis and loading it into a digital micromirror device to generate the illumination pattern. Furthermore, illumination patterns based on the Hadamard matrix have also been proposed due to its orthogonality and ease of binary conversion. In terms of reconstruction algorithms, two-step iterative threshold shrinkage reconstruction algorithms and total variational enhancement Lagrange alternating direction algorithms have been developed, both of which significantly improve both image quality and imaging speed. Summary of the Invention
[0004] The purpose of this invention is to provide a multi-sensor acquisition method for single-pixel imaging based on deep learning. This method uses multiple light intensity sensors to measure and image the images separately, and then uses deep learning to generate a high-resolution image from the multiple images, so that single-pixel imaging can have a wider range of applications.
[0005] The technical solution to achieve the objective of this invention is:
[0006] A deep learning-based multi-sensor single-pixel imaging method includes the following steps:
[0007] (1) Use an LED array panel to generate different lighting images to illuminate the scene. Select appropriate angles to place light intensity sensors at different positions. The light intensity sensors at different positions are required to be within a sphere of a certain radius. Quantitative data of the light intensity reflected back from different positions are obtained.
[0008] (2) Process the quantized data at all locations, arrange the data at each location into a column vector, and reconstruct the image from all the obtained column vectors using the single-pixel imaging method.
[0009] (3) Expand the features of each image, input the expanded features into the self-attention module, and obtain the most effective features for high-quality imaging for each image through iterative training of the model.
[0010] (4) Use a convolutional network to fuse the features and input them into a conditional generative adversarial network for iterative training to obtain a high-resolution image.
[0011] The method for illuminating a scene with different lighting patterns and collecting quantitative data of light intensity as described in step (1) includes the following steps:
[0012] 1) The lighting pattern is generated using four LED array panels, each 1 meter long and 1 meter wide, with each panel containing 25 independently controllable LED beads.
[0013] 2) The LED array panel is mounted on the bracket base. The bracket base supports adjustment of the arrangement of the LED array panel, which can be arranged in a straight line, a sun shape, or a field shape. The bracket base also supports adjustment of the tilt angle, horizontal rotation, and vertical rotation of the LED array panel.
[0014] 3) The LED array is automatically controlled by a LabVIEW program, which independently controls each LED bead. While controlling the generation of different illumination patterns, light intensity sensors at different positions synchronously acquire the reflected light intensity. The light intensity sensors can convert the acquired light intensity into digital signals. Through a series of illumination patterns, the sensors at each position will obtain a series of measurement values.
[0015] The method for arranging the data into column vectors and reconstructing the image in step (2) includes the following steps:
[0016] 1) First, sort the collected quantized data in chronological order, with the data collected by each sensor arranged into a column vector;
[0017] 2) An image I can be sparsely represented under a certain basis Ψ, I = Ψθ, where θ is the coefficient of the image under that basis, and only a small part of its elements are non-zero. Ψ can be selected from wavelet basis, Fourier basis, or discrete cosine basis. Given the sparse basis, the sparse vector of the image can be found, and the image can be reconstructed.
[0018] Furthermore, the specific principle of image reconstruction in step 2) is:
[0019]
[0020] Where ||θ||0 is the 0-norm, representing the number of non-zero elements in the vector, Φ is the sampling basis, Ψ is the sparse basis, and s is the sampled vector; since combinatorial optimization of the 0-norm is an NP-hard problem, the 1-norm or a non-convex regularization function is usually used as a relaxation of the 0-norm to construct the objective equation for solution:
[0021]
[0022] After solving for the sparsity coefficients, the final reconstructed image is
[0023] The method for extracting features using self-attention described in step (3) includes the following steps:
[0024] 1) The image set P = {I1, I2, ..., I...} obtained by the single-pixel imaging method c}, I i For the image reconstructed from the i-th sensor, all images are expanded and fed into the self-attention module to extract features:
[0025]
[0026] in, For self-attention feature extraction networks;
[0027] 2) Based on the extracted features, the features are weighted and fused. The weights are generated by a weight learning network, which consists of convolutional layers and the SoftPlus activation function. The final fused features are:
[0028]
[0029] Furthermore, the specific method for extracting features using the self-attention module in step 1) is as follows:
[0030] f(x) = W f x, g(x) = W g x, h(x) = W h x,
[0031] s kj =f(x) k ) T g(x j ),
[0032]
[0033]
[0034] Where x represents the feature unfolding of the image, W (·) Let β represent the weight matrix. j，kThis indicates the degree of association between the j-th composite region and the content of the k-th image.
[0035] The method for reconstructing an image using a conditional generative adversarial network, as described in step (4), includes the following steps:
[0036] 1) A conditional generative adversarial network consists of a generator and a discriminator. After obtaining the fused feature map, the feature map is input into the generator to generate the target image. The generated image is input into the discriminator to calculate the loss. The network is trained iteratively until the loss converges and then training stops.
[0037] 2) Inputting images generated by different position sensors into the trained generator will produce a high-resolution image.
[0038] The advantages of this invention are: it uses a self-attention conditional generative adversarial network to reconstruct images, which not only fully considers global features during the reconstruction process, but also extracts and fuses the most effective features for reconstructing high-resolution images through self-attention; for a trained network, images can be reconstructed relatively quickly by inputting data collected by sensors, and in some specific fields, it can provide a technical basis for reconstructing high-resolution images. Attached Figure Description
[0039] Figure 1 This is a structural schematic diagram of the LED lighting method provided in an embodiment of the present invention;
[0040] Figure 2 This is a schematic diagram of the structure of a network training model provided in an embodiment of the present invention;
[0041] Figure 3 This is a schematic flowchart of a single-pixel imaging method using multiple sensors provided in an embodiment of the present invention. Detailed Implementation
[0042] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0043] This invention first collects data. Due to the rapid switching characteristic of LEDs, an LED array is used to generate rapidly changing lighting patterns to illuminate the scene, and sensors simultaneously collect the light intensity reflected back from the scene. Sensors at different locations capture images independently; a portion of the data is used to train a self-attention generative adversarial network (GAN), and another portion is used to test the trained generator. The trained GAN is then input into the image set generated from single-pixel imaging to reconstruct a high-quality image.
[0044] The deep learning-based multi-sensor single-pixel imaging method of the present invention comprises the following steps:
[0045] S101. Use an LED array panel to generate different lighting patterns to illuminate the scene. Select appropriate angles and place light intensity sensors at different positions, requiring that the light intensity sensors at different positions are within a sphere of a certain radius, and obtain quantified data of the reflected light intensity at different positions; specifically including the following steps:
[0046] S1011. The lighting pattern is generated using four LED array panels, each panel being 1 meter long and 1 meter wide, with each panel containing 25 independently controllable LED beads.
[0047] S1012. The LED array panel is mounted on the bracket base. The bracket base supports adjustment of the arrangement of the LED array panel, which can be arranged in a straight line, a square, or a grid. The bracket base also supports adjustment of the tilt angle, horizontal rotation, and vertical rotation of the LED array panel.
[0048] The S1013 LED array is automatically controlled via LabVIEW, enabling independent control of each individual LED. While generating different illumination patterns, light intensity sensors at different locations simultaneously acquire the reflected light intensity, which is then converted from analog to digital. Through a series of illumination patterns, the sensors at each location will obtain a series of measurement values.
[0049] S102. Process the quantized data at all locations, arranging the data at each location into a column vector, and reconstructing the image from all the obtained column vectors using a single-pixel imaging method; specifically including the following steps:
[0050] S1021. First, sort the collected quantized data in chronological order, with the data collected by each sensor arranged into a column vector.
[0051] S1022. An image I can be sparsely represented under a certain basis Ψ, where I = Ψθ, and θ is the coefficient of the image under that basis, with only a small portion of its elements being non-zero. Ψ can be selected from wavelet basis, Fourier basis, discrete cosine basis, etc. Given the sparse basis, finding the sparse vector of the image allows for the reconstruction of the image.
[0052] S103. Expand the features of each image, input the expanded features into the self-attention module, and obtain the most effective features for high-quality imaging for each image through iterative training of the model; specifically including the following steps:
[0053] S1031. The image set P = {I1, I2, ..., I...} obtained according to the single-pixel imaging method. c}, I i For the image reconstructed from the i-th sensor, all images are expanded and fed into the self-attention module to extract features:
[0054]
[0055] in, For self-attention feature extraction networks;
[0056] S1032. Based on the extracted features, the features are weighted and fused. The weights are generated by a weight learning network, which consists of convolutional layers and the SoftPlus activation function. The final fused features are:
[0057]
[0058] S104. A convolutional network is used to fuse the features, and the results are fed into a conditional generative adversarial network for iterative training and imaging, ultimately yielding a high-resolution image. This includes the following specific steps:
[0059] S1041. The conditional generative adversarial network consists of a generator and a discriminator. After obtaining the fused feature map, the feature map is input into the generator to generate the target image. The generated image is input into the discriminator to calculate the loss. The network is trained iteratively until the loss converges and then training stops.
[0060] S1042. Inputting images generated by different position sensors into the trained generator will yield a high-resolution image.
Claims
1. A multi-sensor acquisition method for single-pixel imaging based on deep learning, characterized by: It includes the following steps: (1) Use an LED array panel to generate different illumination images to irradiate the scene. Select appropriate angles to arrange light intensity sensors at different positions. It is required that the light intensity sensors at different positions are within a sphere with a certain radius, and obtain the quantization data of the light intensity reflected back at different positions; (2) Process the quantization data at all positions. Arrange the data at each position into a column vector, and use the single-pixel imaging method to reconstruct the images for all the obtained column vectors respectively; (3) Expand the features of each image, input the expanded features into the self-attention module, and obtain the features that are most effective for high-quality imaging for each image through iterative training of the model; (4) Use a convolutional network to fuse the features and input them into the conditional generative adversarial network for iterative training for imaging, and finally obtain a high-resolution image.
2. The multi-sensor acquisition single-pixel imaging method based on deep learning according to claim 1, characterized in that: The method of irradiating the scene with the different illumination patterns in step (1) and collecting the quantization data of the light intensity includes the following steps: 1) Four LED array panels are used to generate the illumination pattern. The panel length is 1 meter and the width is 1 meter. Each panel contains 25 independently controllable LED beads; 2) The LED array panel is installed on the bracket base. The bracket base supports adjusting the arrangement mode of the LED array panel, which can be arranged in a "one" shape, a "day" shape or a "field" shape; the bracket base supports adjusting the pitch angle, horizontal rotation and vertical rotation of the LED array panel; 3) The LED array is automatically controlled through the LabView program to independently control each LED bead. While controlling the generation of different irradiation patterns, the light intensity sensors at different positions synchronously obtain the reflected light intensity. The light intensity sensors can perform analog-to-digital conversion on the collected light intensity; through a series of irradiation patterns, each position sensor will obtain a series of measurement values.
3. The multi-sensor acquisition single-pixel imaging method based on deep learning according to claim 1, characterized in that: The method of arranging into column vectors and reconstructing images in step (2) includes the following steps: 1) First, sort the collected quantization data in chronological order, and arrange the data collected by each sensor into a column vector respectively; 2) An image Can be in a certain set of bases Sparse representation below, , These are the coefficients of the image under this basis, and only a small portion of their elements are non-zero. By selecting wavelet basis, Fourier basis, and discrete cosine basis, and given the sparse basis, the sparse vectors of the image can be found, and the image can be reconstructed.
4. The multi-sensor acquisition single-pixel imaging method based on deep learning according to claim 3, characterized in that: Step 2) The method of image reconstruction is: ； in The zero norm represents the number of non-zero elements in a vector. As the sampling basis, It is a sparse base. The vector obtained from sampling; since combinatorial optimization of the 0-norm is an NP-hard problem, a 1-norm or non-convex regularization function is usually used as a relaxation of the 0-norm to construct the objective equation for solution: ； After solving for the sparsity coefficients, the final reconstructed image is .
5. The multi-sensor acquisition single-pixel imaging method based on deep learning according to claim 1, characterized in that: The method of extracting features using self-attention in step (3) includes the following steps: 1) Image set obtained based on single-pixel imaging method , For the first The images reconstructed from each sensor are expanded and then fed into a self-attention module to extract features. ； in, For self-attention feature extraction networks; 2) Based on the extracted features, the features are weighted and fused. The weights are generated by a weight learning network, which consists of convolutional layers and the SoftPlus activation function. The final fused features are: .
6. The deep learning-based multi-sensor acquisition single-pixel imaging method according to claim 5, characterized in that: Step 1) The method of extracting features by the self-attention module is: ； in This represents the feature expansion of the image. Represents the weight matrix. Indicates the first The first synthetic region and the first The degree of correlation between the content of each image.
7. The multi-sensor acquisition single-pixel imaging method based on deep learning according to claim 1, characterized in that: The method of reconstructing images using the conditional generative adversarial network in step (4) includes the following steps: 1) The conditional generative adversarial network consists of a generator and a discriminator. After obtaining the fused feature map, input the feature map into the generator to generate the target image, input the generated image into the discriminator to calculate the loss, and perform iterative training on the network until the loss converges and then stop training; 2) Input the images generated by the sensors at different positions into the trained generator to obtain a high-resolution image.