Single-photon single-pixel hyperspectral camera covering visible to near-infrared band

Abstract

The application provides a single-photon single-pixel full-spectrum camera covering a visible to near-infrared band, and belongs to the field of single-photon multi-spectral imaging; solves the problems of image detail loss, spectral proportion imbalance and limited spectral resolution in the prior art; the camera is composed of a multi-spectral light emitting module, a single-photon single-pixel imaging module and a computing processing module which can be modularly integrated; the emitting module integrates multiple independently modulated pulsed lasers to realize multi-band active illumination; the imaging module uses a digital micromirror device to load a compressed sensing mask to modulate target reflected light, and records photon events in the mask; the computing processing module extracts mask measurement values based on discrete photon events, reconstructs full-spectrum and photon counting images, and realizes detail enhancement and spectral prediction in combination with a correlation optimization neural network; the application can realize high-precision imaging and spectral restoration in an extremely weak light scene, and is suitable for real-time full-spectrum imaging of a portable, unmanned and embedded platform.

Description

Single-photon single-pixel full-spectrum camera covering the visible to near-infrared bands Technical Field

[0001] This application relates to the field of single-photon multispectral imaging technology, and in particular to a single-photon single-pixel full-spectrum camera covering the visible to near-infrared bands. Background Technology

[0002] Single-pixel multispectral imaging systems are a novel optical imaging technology that integrates spatial modulation and computational reconstruction strategies. They possess significant adaptability and development potential in applications such as remote sensing in complex environments, environmental monitoring, and cultural heritage protection. This system extends traditional single-band grayscale imaging to multispectral imaging through a single-point detector combined with a spectral separation strategy, enabling simultaneous measurement of targets across multiple spectral channels. Compared to traditional imaging methods, this approach significantly enhances spectral dimensionality and information acquisition capabilities, and possesses tunable imaging capabilities across a wide spectral range, accurately resolving the characteristic spectral responses of different materials in various bands.

[0003] However, in extreme environments such as extremely low light or long-range target detection, single-pixel multispectral imaging systems often face technical challenges such as low signal-to-noise ratio and limited reconstruction quality, urgently requiring detection schemes with higher sensitivity to improve imaging performance. Single-photon detectors, with their high sensitivity to extremely weak light signals, provide crucial support for building high-performance low-light imaging systems, significantly enhancing their image acquisition capabilities under extremely low light conditions. However, the high randomness of the single-photon detection process causes the measurement data to follow a typical Poisson distribution, introducing significant random noise during image reconstruction and severely affecting the imaging quality of single-photon single-pixel multispectral systems.

[0004] Current mainstream methods have limited spectral coverage and often employ independent optimization and denoising of images across different spectral bands, failing to fully exploit the structural correlations and intensity ratios between bands. This leads to problems such as lost details in dark areas, inconsistencies in intensity ratios between bands, structural distortion, and blurred edges during reconstruction. Furthermore, insufficient spectral coverage, particularly limited response in the near-infrared band beyond the visible light spectrum, makes it difficult to capture rich material reflection and absorption characteristics, restricting the detection capabilities for low-reflectivity objects, low-light scenes, and high-background-noise environments. However, in practical applications such as multispectral fusion recognition, neglecting the potential physical coupling characteristics and statistical correlations between bands directly impacts the accuracy and robustness of subsequent target detection and material recognition tasks. Summary of the Invention

[0005] To address the technical challenges of preserving image detail and intensity ratios across bands, limited spectral resolution, and insufficient spectral coverage in existing single-photon multispectral imaging processes, this application proposes a single-photon single-pixel full-spectrum camera covering the visible to near-infrared bands. This camera possesses wide spectral coverage capabilities and achieves efficient fusion and joint optimization reconstruction of multispectral information. While accurately restoring image details and spectral intensity ratios, it fully leverages the advantages of wide spectral coverage in multi-source information fusion and material difference characterization.

[0006] The technical solution adopted in this application is: a single-photon single-pixel full-spectrum camera covering the visible to near-infrared bands, including a multispectral emission module, a single-photon single-pixel imaging module and a computing processing module. The multispectral emission module is used to output pulsed lasers in multiple bands, covering the visible to near-infrared bands, and by independently modulating each band, it gives each band a repetition frequency that does not interfere with each other, thereby realizing the combined output of multi-channel frequency domain encoded beams.

[0007] The single-photon single-pixel imaging module is used to receive the reflected light from the target and perform spatial modulation. It controls the optical path through a modulation mask and works with a single-photon detector to obtain the photon arrival time series, thereby achieving high-sensitivity image sampling.

[0008] The computational processing module includes a multispectral information demodulation module, an image reconstruction module, and a multispectral image association optimization and full-spectrum prediction neural network module. The multispectral information demodulation module performs frequency domain transformation on the photon arrival time data, extracts the fundamental frequency and higher harmonic amplitude of each frequency channel, and generates mask measurement values. The image reconstruction module combines the mask measurement values ​​and mask information to generate preliminary multi-channel multispectral images and photon counting images based on linear reconstruction methods. The multispectral image association optimization and full-spectrum prediction neural network module fuses and optimizes the multispectral reconstructed image and photon counting image, outputting a high-quality full-spectrum image and corresponding spectral prediction results.

[0009] Furthermore, the multispectral optical emission module includes pulsed lasers of different wavelengths, an optical fiber combining module, a beam expander and collimator, and a signal generator. The signal generator is used to apply a modulation signal with a set repetition frequency to each pulsed laser. After the multi-modulated lasers are combined through the optical fiber combining module, they are transmitted uniformly to the beam expander and collimator.

[0010] Furthermore, the repetition frequency of the pulsed laser is modulated using the coprime fundamental frequency method to ensure that the fundamental frequency and its preceding harmonics of each channel do not overlap in the spectrum, thereby achieving frequency domain separation of optical information in different bands.

[0011] Furthermore, the single-photon single-pixel imaging module includes an imaging lens, a digital micromirror device, a lens group, a single-photon detector, and a time-to-digital converter. The imaging lens images the target reflected light onto the modulation area of ​​the digital micromirror device. The digital micromirror device performs two-dimensional modulation on the incident image according to a pre-loaded modulation mask. The modulated beam is focused onto the receiving surface of the single-photon detector through the lens group. Whenever the single-photon detector detects a photon event, it outputs an electrical pulse signal. This electrical pulse signal is then marked with its precise arrival time by the time-to-digital converter and recorded on the time axis to form single-photon counting time series data.

[0012] Furthermore, the multispectral information demodulation module uses discrete photon Fourier frequency domain transform to perform frequency domain analysis on the photon arrival time, and extracts the amplitude of each repetition frequency and its preceding harmonics as the modulation mask measurement value.

[0013] Furthermore, the frequency scanning range of the discrete photon Fourier frequency domain transform covers the repetition frequency range of the pulsed laser; when extracting higher-order harmonic components, the frequency scanning range is extended according to the harmonic order, so that the frequency scanning range covers the corresponding higher-order harmonic frequencies, thereby achieving effective acquisition and analysis of higher-frequency components.

[0014] Furthermore, the image reconstruction module employs imaging methods including but not limited to correlated ghost imaging, Fourier imaging, and TVAL3 imaging. It combines modulation mask measurements, photon counts, and compressed sensing masks to perform image inversion, outputting multi-channel multispectral images and photon count images, respectively.

[0015] Furthermore, the multispectral image association optimization and full-spectrum prediction neural network module takes the reconstructed multi-channel multispectral image and photon counting image as joint inputs, and outputs the optimized high-quality full-spectral image and the continuous spectral prediction result of each pixel, which is used to restore the full-spectral information of the scene under test.

[0016] Furthermore, the multispectral image association optimization and full-spectrum prediction neural network module adopts an encoder-decoder structure design. The encoding stage consists of multiple sets of convolutional encoding sub-modules with the same structure as the input channels but without shared parameters. These sub-modules extract features from the multi-channel joint input image. Each set of convolutional encoding sub-modules includes a convolutional layer, a normalization layer, and a nonlinear activation function. A max-pooling layer is connected after the convolutional encoding sub-module to achieve spatial downsampling and semantic compression. At the end of the encoding stage, deep feature maps from all channels are fused, and channel attention and spatial attention mechanisms are introduced for global adaptive modeling to optimize the correlation between channels and the saliency distribution of spatial regions. The fused deep feature maps are input to the decoding module, which uses layer-by-layer upsampling and skip connections to fuse high-level and low-level features, enhancing edge details and structural restoration capabilities, and finally outputting a high-quality full-spectrum image and continuous spectral prediction results for each pixel. On the other hand, the spectral prediction module is input to the spectral prediction module, which includes a flattening layer, a fully connected layer, a nonlinear activation layer, and a second fully connected layer. This module compresses and fits the pixel-level joint features, and finally outputs continuous spectral prediction results.

[0017] A single-photon single-pixel full-spectrum imaging method covering the visible to near-infrared band, employing the aforementioned single-photon single-pixel full-spectrum camera covering the visible to near-infrared band, includes the following steps:

[0018] Step 1: The multispectral light emission module modulates a multichannel pulsed laser covering visible and near-infrared light into different repetition frequencies to irradiate the target;

[0019] Step 2: The target reflected light is imaged by the single-photon single-pixel imaging module and the photon statistical sequence of the unit modulation mask is extracted;

[0020] Step 3: The multispectral information demodulation module performs discrete photon Fourier frequency domain transform on the photon statistical sequence of the modulation mask to obtain the modulation mask measurement value;

[0021] Step 4: The image reconstruction module reconstructs the modulation mask measurement values, photon counts, and modulation mask into multi-channel multispectral images and photon count images, respectively;

[0022] Step 5: The multispectral image association optimization and full-spectrum prediction neural network module takes the multi-channel multispectral image and photon counting image as input, and outputs the optimized high-quality full-spectrum image and the continuous spectrum prediction result of each pixel.

[0023] The advantages of this application over the prior art are as follows:

[0024] 1. This application makes full use of the high measurement bandwidth of the photon detection system. By using the coprime fundamental frequency method to encode the repetition frequency of the multispectral pulsed laser, it realizes multi-channel frequency domain modulation and spectral information separation covering the 450nm to 1064nm band range. This effectively avoids frequency aliasing between bands, significantly improves spectral separation capability and measurement accuracy, and lays the foundation for high-throughput multispectral data acquisition.

[0025] 2. This application constructs a neural network model with channel attention and spatial attention mechanisms to perform deep fusion and joint optimization of multi-channel multispectral images and photon counting images, which significantly improves the ability to restore image details and the consistency of intensity between bands; the brightness difference between output images is below the threshold that can be distinguished by the human eye, ensuring the naturalness of color and the visual uniformity across bands; the color depth resolution of single-band images reaches 42 gray levels, which can support high-precision structural restoration and multi-band image fusion.

[0026] 3. This application adopts a modular structure design, in which each module can be deployed independently, flexibly replaced or expanded, and has the ability to separate software and hardware. It is convenient to quickly integrate and customize functions on different platforms (such as portable devices, unmanned systems, and embedded terminals). The neural network part supports deployment on GPU, FPGA or embedded platforms, has edge computing capabilities, meets the needs of real-time spectral prediction and image reconstruction on site, and has good industrial adaptability and promotion potential. Attached Figure Description

[0027] The following description, in conjunction with the accompanying drawings, further illustrates this application:

[0028] Figure 1 is a schematic diagram of the overall system structure provided in an embodiment of this application;

[0029] Figure 2 is a schematic diagram of the structure of the multispectral emission module provided in an embodiment of this application;

[0030] Figure 3 is a schematic diagram of the structure of the multispectral image association optimization and spectral prediction neural network provided in the embodiment of this application;

[0031] Figure 4 is a line graph showing the imaging results of fresh leaves and its spectral prediction provided in the embodiments of this application;

[0032] Figure 5 is a line graph showing the imaging results of dead leaves and their spectral predictions provided in the embodiments of this application.

[0033] Figure 6 is a line graph showing the image brightness difference and standard deviation under different single measurement unit pixel photon numbers provided in the embodiments of this application;

[0034] In the diagram: 101 is a multispectral light emission module, 102 is a beam expander and collimator, 103 is an imaging lens, 104 is a digital micromirror device, 105 is a lens group, 106 is a single-photon detector, 107 is a computing processing module, 201 is a pulsed laser, and 202 is an optical fiber combiner module. Detailed Implementation

[0035] As shown in Figures 1 to 6, this application provides a single-photon single-pixel full-spectrum camera covering the visible to near-infrared bands. It adopts a highly modular structural design, with each functional module possessing excellent hardware and software decoupling capabilities. It can be independently deployed, flexibly expanded, or have its functions replaced according to application requirements, and is suitable for integration with various hardware platforms. The camera mainly includes a multispectral emission module 101, a single-photon single-pixel imaging module, and a computational processing module 107. The multispectral emission module 101 is used to modulate pulsed lasers of different wavelengths into different repetition frequencies and then emit them in a common path. The single-photon single-pixel imaging module is used to perform two-dimensional modulation of the target scene and record photon arrival events generated by single-photon detectors 106 within different modulation masks. The computational processing module 107 is used to process, reconstruct, and jointly optimize the photon arrival events to obtain optimized multispectral images and corresponding spectral prediction results.

[0036] The multispectral emission module 101 integrates multiple independently modulated pulsed lasers 201 covering visible to near-infrared light, a signal generator, an fiber optic combiner module 202, and a beam expander / collimator 102. The signal generator provides electrical modulation signals to each pulsed laser 201 to control the repetition frequency of the pulsed lasers, achieving frequency domain encoding of the multi-channel pulsed lasers. Upon receiving the modulation signal from the signal generator, each pulsed laser 201 outputs nanosecond pulses with a set repetition frequency, ensuring that each laser has independent time control capabilities. The multi-modulated lasers are combined via the fiber optic combiner module 202 and then uniformly transmitted to the beam expander / collimator 102. The beam expander / collimator 102 expands the diameter and collimates the optical axis of the combined beam, ensuring good parallelism and a suitable illumination area. This meets the requirement for uniform illumination of large field-of-view target areas, improving the spatial consistency and edge quality of the system imaging. The sub-components in the multispectral light emission module 101 are connected via SMA / BNC signal lines, which is scalable, adaptable to multiple types of light source systems, and can achieve flexible band configuration.

[0037] To effectively distinguish the information carried by each band under co-path propagation conditions and ensure the orthogonality between the spectra of each laser channel and the non-interference of modulation information, a coprime fundamental frequency method is used to modulate the laser pulse repetition frequency. This method sets a coprime repetition frequency for each band of pulsed laser 201, ensuring that the fundamental frequency and its preceding harmonic components of each channel do not overlap in the spectrum. This avoids mutual interference and aliasing between modulated repetition frequencies, ensuring the accuracy and independence of subsequent frequency domain demodulation, and providing a clear spectral division basis for subsequent frequency domain demodulation and image reconstruction.

[0038] As shown in Figure 2, the multispectral optical emission module 101 includes six independently modulated pulsed lasers 201 with different wavelengths: 450nm, 520nm, 635nm, 750nm, 905nm, and 1064nm. Each laser has independent modulation capability, and a modulation signal with a specific repetition frequency is applied to it through a connected signal generator, modulating it to 8.2MHz, 8.6MHz, 8.8MHz, 9.0MHz, 9.2MHz, and 9.8MHz respectively. The signal generator supports multiple independent outputs, enabling precise control of the repetition frequency and pulse parameters of each pulsed laser 201, thereby achieving frequency division and control of the multi-band laser source. This setup ensures that the fundamental frequency and its first five harmonics of each laser channel do not overlap, thus achieving independent frequency domain encoding of the multi-channel signal. The modulated six pulsed lasers are combined by a 6-1 beam combiner to form a composite beam, which then enters the beam expander and collimator 102.

[0039] As shown in Figure 1, the single-photon single-pixel imaging module specifically includes an imaging lens 103, a digital micromirror device (DMM) 104, a lens group 105, a single-photon detector 106, and a time-to-digital converter. The imaging lens 103 images the reflected light from the target onto the modulation area of ​​the DMM 104. The DMM 104 performs two-dimensional modulation of the incident image according to a pre-loaded modulation mask: the pixel units marked as "1" in the mask deflect the reflected light to a fixed +12° direction, while the remaining pixels deflect it to a -12° direction. Subsequently, the beam at the +12° direction is focused by a converging lens onto the receiving surface of the single-photon detector 106. Whenever the single-photon detector 106 detects a photon event, it outputs an electrical pulse signal. This signal is then marked with its precise arrival time by the time-to-digital converter and recorded on the time axis, forming single-photon counting time-series data.

[0040] The computational processing module 107 includes a multispectral information demodulation module, an image reconstruction module, and a multispectral image correlation optimization and full-spectrum prediction neural network module. The multispectral information demodulation module performs piecewise Fourier transform on the recorded photon event data, extracting the intensity information of the fundamental frequency and higher harmonics as measurement values ​​under a mask. Specifically, this module utilizes a discrete photon Fourier frequency domain transform method to amplify the measured spectrum to 8MHz-10MHz, with a frequency accuracy of 1kHz. Since the information carried by the fundamental frequency cannot fully express the complete waveform of the pulse, higher-order frequency components are extracted at its higher harmonics to enhance harmonic information.

[0041] The multispectral information demodulation module receives data from the time-to-digital converter and divides the entire time series into several segments based on the mask number and the repetition frequency parameters of the pulsed laser 201. A discrete photonic Fourier frequency domain transform is applied to each segment, with a frequency scanning range of 8MHz to 10MHz and a frequency accuracy of 1kHz. The frequency domain transform uses a weighted Fourier sum in the following form:

[0042] ;

[0043] in, This represents the arrival time of the nth photon. Let H represent the fundamental frequency corresponding to the h-th laser band, H be the total number of bands, K be the extracted harmonic order, and N be the total number of photons under the current mask. Through the above frequency domain transformation, the system can extract the amplitude of the fundamental frequency component and its first 5 harmonic components corresponding to each laser band, which are used as the modulation mask measurement values ​​under different spectral channels for subsequent image reconstruction. At the same time, the total number of photons N recorded under each modulation mask is used as the measurement value of the photon counting channel to assist in multi-channel image fusion and spectral prediction.

[0044] The image reconstruction module is based on the principle of correlated ghost imaging, combining modulation mask measurements, photon counting, and the modulation mask to reconstruct multispectral and photon-count images, respectively. It uses modulation mask measurements extracted in the frequency domain and the corresponding two-dimensional modulation mask. Linear reconstruction is performed. The correlated ghost imaging method has the advantages of simple form and high reconstruction efficiency, and is particularly suitable for single-pixel imaging scenarios with limited photon counts and low signal-to-noise ratio.

[0045] The reconstruction formulas for 6-channel multispectral images and photon number integral images are as follows:

[0046] ;

[0047] in, This represents the measurement value under the i-th modulation mask. The mean of all measurements. This represents the energy normalization factor of the i-th mask. The normalized mean of all masks, The total number of measurements is represented. This reconstruction method is a typical linear non-iterative algorithm, which is computationally simple, easy to parallelize, and suitable for real-time imaging applications. However, this system is also open-source, allowing for flexible replacement of reconstruction algorithms. In addition to associated ghost imaging, the TVAL3 reconstruction algorithm based on sparse priors, the least squares method, the regularized inversion method, and the deep learning-based reconstruction network can also be introduced to adapt to different imaging requirements and computational conditions.

[0048] The multispectral image association optimization and full-spectrum prediction neural network module takes multi-channel multispectral images and corresponding photon count images as joint inputs, and outputs optimized high-quality full-spectral images and continuous spectral prediction results for each pixel. The related computation process can be deployed through heterogeneous computing platforms such as GPUs and FPGAs, or embedded edge computing modules, to meet real-time processing requirements on-site.

[0049] The input to the multispectral image association optimization and full-spectrum prediction neural network module is the reconstructed 6-channel 2D multispectral image and the corresponding band photon count image, for a total of 7 input channels. The network adopts an encoder-decoder structure design, in which the encoding stage consists of 7 sets of structurally identical but parameter-distributed convolutional coding sub-modules, which process the information of the 7-channel input image respectively, to achieve local feature extraction and cross-channel structural modeling between different bands.

[0050] In the network encoding stage, the input image with a total of 7 channels is processed through 4 sets of convolutional blocks consisting of convolutional layers, normalization layers, and activation functions for feature extraction. Each set of convolutional blocks is followed by a max pooling layer to achieve spatial downsampling of the image (gradual compression of spatial resolution). At the very end of the encoding stage, the deep feature maps of all channels are fused, and channel attention and spatial attention mechanisms are introduced to perform global adaptive optimization modeling on the fused feature map, dynamically adjusting the importance of different channel features and spatial regions, thereby improving the high-level representation's ability to perceive key structures and inter-spectral correlations. The fused optimized features are then fed into the decoding stage for gradual upsampling and into the spectral prediction module.

[0051] During the decoding phase, the network upsamples the compressed deep features of the encoding part step by step. Each upsampled feature map is combined with the 7-channel features output by the corresponding encoding layer through skip connections to achieve the fusion of multi-level information. This process fuses the low-order features from the encoding process with the currently upsampled features, effectively restoring spatial details and edge textures, and finally outputting a high-quality 6-channel optimized multispectral image for the restoration and structural enhancement of realistic reflection images.

[0052] The spectral prediction module consists of a flattening layer, a fully connected layer, a nonlinear activation layer, and a second fully connected layer. It is used to perform dimensionality compression and spectral shape fitting on the joint feature vector of each pixel, and finally outputs the prediction result of the continuous spectral distribution at each pixel location, realizing spectral reconstruction and fitting estimation from finite discrete bands to continuous spectrum.

[0053] Lightweight strategies can be introduced into the design of neural networks, including but not limited to depthwise separable convolution, group convolution, feature channel pruning, and model quantization, to reduce parameter size and computational complexity.

[0054] The computing processing module 107 supports deployment based on GPU, FPGA or embedded lightweight computing platforms, and has the ability to perform real-time reconstruction and spectral prediction on site, which facilitates its integration into portable terminals, unmanned systems or industrial inspection platforms.

[0055] To ensure stable and reliable spectral prediction capabilities, this application employs a pre-trained data-driven strategy for training the neural network. The pre-training data consists of multiple real or simulated samples with known continuous spectra, effectively guiding the network to learn cross-band mapping relationships and chromatographic structure patterns. Regarding the loss function design, a combination of cosine similarity loss and mean square error (MSE) loss is introduced as a joint optimization objective, considering both spectral direction similarity and numerical intensity consistency. The cosine similarity loss measures the angular consistency between the predicted and true spectra, helping to maintain the consistency of spectral shape trends; while the MSE loss constrains the accurate fitting of spectral amplitudes. Joint optimization of both enhances the network's sensitivity and adaptability to both spectral structure and numerical accuracy. This training strategy effectively improves the network's prediction robustness under low photon counts, ensuring reconstruction and prediction performance under extreme imaging conditions.

[0056] The validity of this application is verified through experiments below:

[0057] To verify the effectiveness and practical application potential of the camera proposed in this application in distinguishing and predicting spectral information that is indistinguishable to the naked eye, a multispectral single-photon imaging and spectral prediction experimental platform was built, and comparative experiments were conducted. Fresh leaves and withered leaves that had been naturally dried for 24 hours were selected as imaging targets. Both types of samples appear green to the naked eye under visible light illumination, with minimal visual difference, making them difficult to distinguish using traditional color imaging methods.

[0058] Figures 4 and 5 show the single-photon single-pixel multispectral imaging results of fresh and withered leaves at different wavelengths (450nm, 520nm, 635nm, 750nm, 905nm, 1064nm), and the comparison between the neural network-optimized imaging results and the images after channel fusion, to visually demonstrate the optimization effect. The results show that the neural network-optimized image integrates structural information from different wavelengths with detailed features provided by the photon count map, enabling clearer recovery of the texture details of the leaf surface and effectively enhancing the expression of leaf veins and tissue structure. Especially under low photon count conditions, the neural network significantly improves the structural fidelity of the image, demonstrating good robustness in low-light imaging.

[0059] As shown in the full-spectrum predictions at the bottom of Figures 4 and 5, fresh leaves and dead leaves exhibit significant differences in the intensity distribution of their reflectance spectral intensity in the visible to near-infrared bands. Fresh leaves show a marked increase in reflectance in the near-infrared band above 750 nm, which is related to the selective absorption and scattering of different wavelengths of light by water, chlorophyll, and cell structure in living leaves. In contrast, dead leaves show a significant decrease in reflectance in the near-infrared region due to water loss and changes in their internal structure, resulting in an overall shift in their spectral curve and a tendency to flatten out.

[0060] To evaluate the image output fidelity of this application, the brightness difference and pixel standard deviation of the image were further calculated under different single-measurement unit pixel photon numbers, as shown in Figure 6. The results show that as the single-measurement unit pixel photon number increases, both the image brightness difference and pixel standard deviation gradually tend to saturate. Specifically, the image brightness difference reaches a saturation point when the photon number reaches 5 × 10⁻⁶. -4 The brightness has stabilized, indicating a strong overall brightness reconstruction capability. However, the pixel standard deviation, being more sensitive to local noise fluctuations, shows an increase in photon count to approximately 8 × 10⁻⁶. -4 It gradually approaches saturation.

[0061] The experimental results above demonstrate that the camera proposed in this application not only possesses high-fidelity multi-band image reconstruction capabilities, but also effectively extracts and predicts spectral structure information, exhibiting excellent distinguishing ability for targets that are visually consistent but spectrally heterogeneous. This proves its potential application value in scenarios such as high-sensitivity multispectral recognition, plant health status assessment, and material composition identification.

[0062] The camera proposed in this application fully leverages the randomness of single-photon arrival time and the broadband detection advantages provided by high temporal resolution to achieve synchronous compressed acquisition and collaborative reconstruction of multispectral information in the same imaging process, thereby improving data utilization efficiency and image quality. During image reconstruction, a neural network model with adaptive optimization capabilities is proposed to fuse multi-band image information, achieving complementary enhancement between spectra and precise registration of color ratios, and possessing automatic prediction capabilities for spectral distribution. Even under extremely low photon count conditions, this camera can still achieve high-precision multispectral image reconstruction and spectral restoration, significantly improving imaging quality and spectral resolution.

[0063] In one specific embodiment, this application can also provide a single-photon single-pixel full-spectrum imaging method covering the visible to near-infrared band based on the above-mentioned camera, including the following steps:

[0064] Step 1: The multispectral light emission module 101 modulates a multichannel pulsed laser covering visible and near-infrared light into different repetition frequencies and then irradiates the target.

[0065] Step 2: The target reflected light is imaged by the single-photon single-pixel imaging module and the photon statistical sequence of the unit modulation mask is extracted;

[0066] Step 2.1: The imaging lens 103 images the target reflected light onto the modulation area of ​​the digital micromirror device 104;

[0067] Step 2.2: The digital micromirror device 104 multiplies the image with the loaded compressed sensing mask and reflects the portion of the modulation mask that is 1 at +12°.

[0068] Step 2.3: After the lens group 105 adjusts the +12° reflected light, it converges the light onto the target surface of the single-photon detector 106;

[0069] Step 2.4: The single-photon detector 106 outputs an electrical pulse signal after receiving a photon;

[0070] Step 2.5: The electrical pulse signal is marked on the time axis as a photon event by the time-to-digital converter.

[0071] Step 3: The multispectral information demodulation module performs discrete photon Fourier frequency domain transformation on the photon statistical sequence of the modulation mask to obtain the modulation mask measurement value.

[0072] Step 4: The image reconstruction module uses the principle of correlated ghost imaging to reconstruct the multispectral image and the photon count image from the modulation mask measurement value, photon count, and modulation mask, respectively.

[0073] Step 5: The multispectral image association optimization and full-spectrum prediction neural network module takes the multi-channel multispectral image and photon counting image as input, and outputs the optimized high-quality full-spectrum image and the continuous spectrum prediction result of each pixel.

[0074] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A single-photon single-pixel full-spectrum camera covering the visible to near-infrared band, characterized in that: The system comprises a multispectral emission module, a single-photon single-pixel imaging module, and a computational processing module. The multispectral emission module outputs pulsed laser light across multiple wavelengths, covering the visible to near-infrared range. It independently modulates each wavelength to achieve a unique repetition frequency, enabling the combined output of multi-channel frequency-domain encoded beams. The single-photon single-pixel imaging module receives reflected light from the target and performs spatial modulation. It controls the optical path through a modulation mask and, in conjunction with a single-photon detector, acquires the photon arrival time sequence, achieving high-sensitivity image sampling. The computational processing module includes a multispectral information demodulation module, an image reconstruction module, and a multispectral image association optimization and full-spectrum prediction neural network module. The multispectral information demodulation module performs frequency domain transformation on the photon arrival time data, extracting the fundamental frequency and higher harmonic amplitudes of each frequency channel to generate mask measurement values. The image reconstruction module combines the mask measurement values ​​and mask information to generate preliminary multi-channel multispectral images and photon counting images based on a linear reconstruction method. The multispectral image association optimization and full-spectrum prediction neural network module is used to fuse and optimize multispectral reconstructed images and photon counting images, and output high-quality full-spectrum images and corresponding spectral prediction results.

2. A single-photon single-pixel full-spectrum camera covering the visible to near-infrared band according to claim 1, characterized in that: The multispectral optical emission module includes pulsed lasers of different wavelengths, an optical fiber combining module, a beam expander and collimator, and a signal generator. The signal generator is used to apply a modulation signal with a set repetition frequency to each pulsed laser. After the multi-modulated laser beams are combined by the fiber optic combiner module, they are transmitted to the beam expander and collimator.

3. A single-photon single-pixel full-spectrum camera covering the visible to near-infrared band according to claim 2, characterized in that: The repetition frequency of the pulsed laser is modulated using the coprime fundamental frequency method to ensure that the fundamental frequency and its preceding harmonics of each channel do not overlap in the spectrum, thereby achieving frequency domain separation of optical information in different bands.

4. A single-photon single-pixel full-spectrum camera covering the visible to near-infrared band according to claim 1, characterized in that: The single-photon single-pixel imaging module includes an imaging lens, a digital micromirror device (DMM), a lens group, a single-photon detector, and a time-to-digital converter. The imaging lens images the target reflected light onto the modulation area of ​​the DMM. The DMM performs two-dimensional modulation of the incident image according to a pre-loaded modulation mask. The modulated beam is focused onto the receiving surface of the single-photon detector by the lens group. Whenever the single-photon detector detects a photon event, it outputs an electrical pulse signal. This electrical pulse signal is then marked with its precise arrival time by the time-to-digital converter and recorded on the time axis to form single-photon counting time series data.

5. A single-photon single-pixel full-spectrum camera covering the visible to near-infrared band according to claim 1, characterized in that: The multispectral information demodulation module uses discrete photon Fourier frequency domain transform to perform frequency domain analysis on photon arrival time, and extracts the amplitude of each repetition frequency and its preceding harmonics as the measurement value of the modulation mask.

6. A single-photon single-pixel full-spectrum camera covering the visible to near-infrared band according to claim 5, characterized in that: The frequency scanning range of the discrete photonic Fourier frequency domain transform covers the repetition frequency range of the pulsed laser; when extracting higher-order harmonic components, the frequency scanning range is extended according to the harmonic order so that the frequency scanning range covers the corresponding higher-order harmonic frequencies, thereby achieving effective acquisition and analysis of higher-frequency components.

7. A single-photon single-pixel full-spectrum camera covering the visible to near-infrared band according to claim 1, characterized in that: The image reconstruction module employs imaging methods including but not limited to correlated ghost imaging, Fourier imaging, and TVAL3 imaging. It combines modulation mask measurements, photon counts, and compressed sensing masks to perform image inversion, outputting multi-channel multispectral images and photon count images, respectively.

8. A single-photon single-pixel full-spectrum camera covering the visible to near-infrared band according to claim 7, characterized in that: The multispectral image association optimization and full-spectrum prediction neural network module takes the reconstructed multi-channel multispectral image and photon counting image as joint inputs, and outputs the optimized high-quality full-spectral image and the continuous spectral prediction result of each pixel, which is used to restore the full-spectral information of the scene under test.

9. A single-photon single-pixel full-spectrum camera covering the visible to near-infrared band according to claim 8, characterized in that: The multispectral image association optimization and full-spectrum prediction neural network module adopts an encoder-decoder structure design. The encoding stage consists of multiple sets of convolutional encoding sub-modules with the same structure as the input channels but without shared parameters. These sub-modules extract features from the multi-channel joint input image. Each set of convolutional encoding sub-modules includes a convolutional layer, a normalization layer, and a nonlinear activation function. A max-pooling layer is connected after the convolutional encoding sub-module to achieve spatial downsampling and semantic compression. At the end of the encoding stage, deep feature maps from all channels are fused, and channel attention and spatial attention mechanisms are introduced for global adaptive modeling to optimize the correlation between channels and the saliency distribution of spatial regions. The fused deep feature maps are input to the decoding module, which uses layer-by-layer upsampling and skip connections to fuse high-level and low-level features, enhancing edge details and structural restoration capabilities, and finally outputting a high-quality full-spectrum image and continuous spectral prediction results for each pixel. On the other hand, the spectral prediction module is input to the spectral prediction module, which includes a flattening layer, a fully connected layer, a nonlinear activation layer, and a second fully connected layer. This module compresses and fits the pixel-level joint features, and finally outputs continuous spectral prediction results.

10. A single-photon single-pixel full-spectrum imaging method covering the visible to near-infrared band, characterized in that: The single-photon single-pixel full-spectrum camera covering the visible to near-infrared bands as described in any one of claims 1-9 includes the following steps: Step 1: A multispectral light emission module modulates a multi-channel pulsed laser covering visible and near-infrared light to different repetition frequencies and then illuminates the target; Step 2: The target reflected light is imaged by a single-photon single-pixel imaging module, and the photon statistical sequence of a unit modulation mask is extracted; Step 3: A multispectral information demodulation module performs discrete photon Fourier frequency domain transformation on the photon statistical sequence of the modulation mask to obtain the modulation mask measurement value; Step 4: An image reconstruction module reconstructs the modulation mask measurement value, photon count, and modulation mask into a multi-channel multispectral image and a photon count image, respectively; Step 5: A multispectral image association optimization and full-spectrum prediction neural network module takes the multi-channel multispectral image and the photon count image as input and outputs an optimized high-quality full-spectrum image and a continuous spectral prediction result for each pixel.

Images