Deep learning noise suppression reconstruction method applied to quantum image sensor

Abstract

The invention discloses a deep learning noise suppression reconstruction method applied to a quantum image sensor, and relates to the field of image processing, and the method comprises the steps: firstly, carrying out the calculation of a block and an image of a three-dimensional frame plane sequence, and facilitating the improvement of the suppression capability of Poisson noise through precise unbiased variance stabilization transformation; secondly, the method is based on a network framework fusing Transform and CNN, the framework serves as a core denoising device and can deeply search key information in image denoising in a linear and nonlinear component combination mode, and the adaptability of the network to the denoising capacity of various scenes is enhanced through the sub-network parallel technology; and finally, using accurate unbiased variance to stabilize inverse transformation, using maximum likelihood estimation to restore a real photon number, and starting from a QIS imaging principle, further suppressing noise in a reconstructed image. According to the method provided by the invention, the reconstructed image noise of the QIS can be remarkably suppressed, and the method has the potential of being applied to a customized neural network processor chip, so that the method can obtain a relatively high operation speed.

Description

Technical Field

[0001] The present invention relates to the field of image processing, and in particular to a deep learning noise suppression and reconstruction method applied to a quantum image sensor. Background Art

[0002] Quantum image sensor (QIS) is a single-photon sensing image sensor. Compared with traditional CMOS image sensors, QIS has higher frame rate, higher resolution, lower noise, and has time and space oversampling characteristics. The output of the sensor is a three-dimensional frame plane sequence. The value of each pixel in the sequence is single-bit data, representing the presence or absence of photons. Reconstructing the three-dimensional frame plane sequence can significantly improve the image acquisition quality in special scenes. QIS has broad application prospects in the fields of high-resolution imaging, extremely weak light imaging, and high dynamic range imaging.

[0003] The temporal and spatial oversampling characteristics of QIS make the data in the form of a three-dimensional frame plane sequence, so it is necessary to design a special image reconstruction algorithm to reconstruct the three-dimensional frame plane sequence into a two-dimensional image to suppress the noise of the reconstructed image and improve the quality of the reconstructed image. At present, most image reconstruction algorithms are based on mathematical estimation methods for noise suppression, such as estimating the true number of photons through maximum likelihood estimation or maximum a posteriori estimation. However, this method based on mathematical estimation requires multiple iterative calculations and has limited noise suppression capabilities for the reconstructed image. In order to improve the noise suppression level of QIS reconstructed images, a deep learning-based method is considered to suppress the noise of the reconstructed image. Summary of the invention

[0004] The present invention aims to propose a reconstructed image noise suppression method based on deep learning. The method mainly adopts a deep learning algorithm that combines accurate unbiased variance stable transformation with fusion Transformer and convolutional neural network (CNN) framework. It can not only significantly improve the reconstruction quality of QIS and effectively suppress the reconstructed image noise, but also can be applied to customized neural network processor chips to improve the running speed of the reconstruction algorithm.

[0005] To achieve the above object, the present invention provides the following technical solutions:

[0006] The deep learning noise suppression reconstruction method applied to quantum image sensor includes the following steps:

[0007] (1) Input a 3D frame plane sequence;

[0008] (2) summing the three-dimensional frame plane sequence according to the temporal and spatial oversampling rate to form blocks and images;

[0009] (3) Perform variance stabilization transform on blocks and images;

[0010] (4) Apply a network framework that integrates Transformer and CNN;

[0011] (5) Perform accurate unbiased and variance-stable inverse transformation on the output image;

[0012] (1) Obtain a noise-suppressed reconstructed image.

[0013] Furthermore, the three-dimensional frame plane sequence is divided into many "blocks" according to time-space oversampling, and the sum of the sequences within the block is called a "block sum image".

[0014] Furthermore, in the step (2), in the step of summing the three-dimensional frame plane sequence according to the time-space oversampling rate to form blocks and images, the input three-dimensional frame plane sequence is first divided into non-overlapping time blocks with the same number of frames according to the time oversampling rate, and then each time block is divided into non-overlapping spatial blocks with the same size according to the spatial oversampling rate, and finally the binary numbers in each block are added to form blocks and images.

[0015] Furthermore, in the step (3), in the step of performing variance-stabilized transformation on blocks and images, a variance-stabilized Anscombe transform is adopted, that is, a transformation formula is applied to each original data point in the blocks and images generated in the previous step, so that the transformed data is closer to a Gaussian distribution with a constant variance.

[0016] Furthermore, in the step (4), in the step of applying the network framework integrating Transformer and CNN, the variance-stabilized image is used as the input of the network framework. The network framework integrates Transformer and CNN, complements the advantages and limitations of the two, and utilizes the global information capture capability of Transformer and the local feature extraction capability of CNN to achieve better image noise suppression effect; in the network framework structure, sub-network parallel technology is also adopted, and the cross-features in the sub-network interact with each other, and the relevant information between pixels is widely searched, thereby improving the denoising adaptability of the network in different scenarios.

[0017] Furthermore, in the step (5), in the step of performing an accurate unbiased variance-stable inverse transformation on the output image, an accurate unbiased variance-stable inverse transformation is used to restore the data to the original scale, that is, the Anscombe inverse transformation formula is first applied to each point in the network output image to make the data closer to the form of the original Poisson distribution; after the unbiased variance-stable Anscombe inverse transformation, the maximum likelihood estimation method is used to restore the true number of photons, and this accurate unbiased variance-stable inverse transformation is used to further suppress Poisson noise, thereby enhancing the suppression effect on the reconstructed image noise.

[0018] Compared with the prior art, the beneficial effects of the present invention are as follows: first, the blocks and images of the three-dimensional frame plane sequence are calculated, so that the ability to suppress Poisson noise can be enhanced by using accurate and unbiased variance-stabilized transformation; second, the method is based on a network framework that integrates Transformer and CNN. As a core denoiser, the framework can deeply search for key information in image denoising by combining linear and nonlinear components, and the sub-network parallel technology enhances the adaptability of the network to denoising in various scenes; finally, accurate and unbiased variance-stabilized inverse transformation is used, and the maximum likelihood estimation is used to restore the real number of photons, and the noise in the reconstructed image is further suppressed based on the QIS imaging principle. The proposed method can significantly suppress the reconstructed image noise of QIS, and the method has the potential to be applied to customized neural network processor chips, so that the method can achieve a faster running speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Flowchart for implementing noise suppression for QIS reconstructed images based on deep learning.

[0020] Figure 2 The 3D frame plane sequence consists of blocks and image flow charts.

[0021] Figure 3 Flowchart of the exact unbiased variance-stabilizing transformation.

[0022] Figure 4 A rough schematic diagram of the network framework that combines Transformer and CNN.

[0023] Figure 5 Detailed schematic diagram of the QIS reconstructed image noise suppression method based on deep learning. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0025] In the embodiment of the present invention, a deep learning noise suppression reconstruction method applied to a quantum image sensor is used to suppress the noise of a QIS reconstructed image and improve the image reconstruction quality. The present invention aims to propose a method for realizing noise suppression of a QIS reconstructed image based on deep learning. The steps are as follows: input a three-dimensional frame plane sequence → sum the three-dimensional frame plane sequence according to the time and space oversampling rate to form blocks and images → perform variance-stabilized transformation on the blocks and images → apply a network framework that integrates Transformer and CNN → perform accurate unbiased variance-stabilized inverse transformation on the output image → obtain a noise-suppressed reconstructed image. Figure 1 The process of achieving noise suppression in QIS reconstructed images based on deep learning is demonstrated.

[0026] Since the output data of QIS is a 3D frame plane sequence, according to the temporal and spatial oversampling rate, the 3D frame plane sequence needs to be divided into non-overlapping time blocks with the same number of frames according to the temporal oversampling rate, and each time block needs to be divided into non-overlapping spatial blocks with the same size according to the spatial oversampling rate. In each temporal and spatial oversampling block, the binary data is added to obtain the block and image. The specific process is as follows: Figure 2 As shown, after the blocks and images are formed, the subsequent accurate unbiased variance stable transformation can be implemented. This step is the preprocessing of the QIS output data.

[0027] Figure 3 The process of accurate unbiased variance stabilizing transformation is shown:

[0028] (1) Perform variance-stabilized Anscombe transform on blocks and images to transform Poisson noise into Gaussian noise. n , whose Anscombe transform can be expressed as:

[0029]

[0030] Where L is the size of the temporal and spatial oversampling rate.

[0031] (2) Embedded denoiser is a network framework that integrates Transformer and CNN.

[0032] (3) (3) Perform unbiased variance-stable inverse transform, for the output image Z n , its unbiased variance-stable inverse transform can be expressed as:

[0033]

[0034] (4) After the unbiased variance-stable inverse transformation, an accurate and unbiased variance-stable inverse transformation is achieved, that is, the true number of photons is restored using maximum likelihood estimation. The QIS data satisfies the Poisson distribution:

[0035]

[0036] where s n Representing all binary pixels when forming blocks and images, the maximum likelihood estimation is performed on the blocks and images, and the estimation result of the real number of photons can be expressed as:

[0037]

[0038] Where K is the time oversampling rate, α is a constant parameter related to the sensor, Ψ -1 q is the incomplete gamma function, which can be expressed as:

[0039]

[0040] Where q is the quantization threshold of QIS, which is usually taken as 1. This accurate and unbiased variance-stabilized transformation is embedded in the network denoiser and uses maximum likelihood estimation to restore the true number of photons. This structure is suitable for QIS image reconstruction and has a significant suppressive effect on reconstructed image noise.

[0041] The network framework that combines Transformer and CNN is embedded in the process of accurate unbiased and variance-stable transformation as a denoiser.

[0042] Figure 4 A rough schematic diagram of the network framework is shown. In this network framework, a deep denoising network module is used to perform a deep search on the image structure information after the variance-stabilized transformation. The structure is mainly composed of three structures: convolutional layer, activation function, and Transformer. The convolutional layer converts the given noisy image into linear features, the nonlinear activation function is used to extract richer features, and the Transformer explores the relationship between different blocks of the image to enhance the denoising performance of complex scenes. In addition to the deep denoising network module, the network framework also cascades a denoising robustness enhancement network module, in which two parallel sub-networks interact with each other to obtain complementary features from different angles to enhance the robustness of the obtained denoising model and enhance the adaptability of the denoiser in different scenarios. The standard mean square error loss function is used when training the network, which can be expressed as:

[0043]

[0044] The network framework adopting this structure has the ability to deeply learn noise information and can adapt to different complex scenarios. It has a significant noise suppression effect and is suitable as a denoiser embedded in a precise unbiased variance stable transformation structure.

[0045] The detailed implementation of the proposed method is shown in Figure 5 shown.

[0046] It will be apparent to those skilled in the art that the invention is not limited to the details of the exemplary embodiments described above and that the invention can be implemented in other specific forms without departing from the spirit or essential features of the invention. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the invention is defined by the appended claims rather than the foregoing description, and it is intended that all variations falling within the meaning and scope of the equivalent elements of the claims be included in the invention. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

[0047] In addition, it should be understood that although the present specification is described according to implementation modes, not every implementation mode contains only one independent technical solution. This description of the specification is only for the sake of clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment may also be appropriately combined to form other implementation modes that can be understood by those skilled in the art.

Claims

1. Deep learning noise suppression and reconstruction method applied to quantum image sensors, It is characterized in that The steps include: (1) Input a 3D frame plane sequence; (2) summing the three-dimensional frame plane sequence according to the temporal and spatial oversampling rate to form blocks and images; (3) Perform variance stabilization transform on blocks and images; (4) Apply a network framework that integrates Transformer and CNN; (5) Perform accurate unbiased and variance-stable inverse transformation on the output image; (6) Obtain a noise-suppressed reconstructed image.

2. The deep learning noise suppression and reconstruction method for quantum image sensor according to claim 1, It is characterized in that The three-dimensional frame plane sequence is divided into many "blocks" after time-space oversampling, and the sum of the sequences within the block is called a "block sum image".

3. The deep learning noise suppression reconstruction method for quantum image sensor according to claim 2, It is characterized in that In the step (2), in the step of summing the three-dimensional frame plane sequence according to the time and space oversampling rate to form blocks and images, the input three-dimensional frame plane sequence is first divided into non-overlapping time blocks with the same number of frames according to the time oversampling rate, and then each time block is divided into non-overlapping space blocks with the same size according to the space oversampling rate, and finally the binary numbers in each block are added to form blocks and images.

4. The deep learning noise suppression and reconstruction method for quantum image sensor according to claim 3, It is characterized in that In the step (3), in the step of performing variance-stabilized transformation on blocks and images, a variance-stabilized Anscombe transformation is adopted, that is, a transformation formula is applied to each original data point in the blocks and images generated in the previous step, so that the transformed data is closer to a Gaussian distribution with a constant variance.

5. The deep learning noise suppression and reconstruction method for quantum image sensor according to claim 4, It is characterized in that In the step (4), in the step of applying the network framework integrating Transformer and CNN, the variance-stabilized image is used as the input of the network framework. The network framework integrates Transformer and CNN, complements the advantages and limitations of the two, and utilizes the global information capture capability of Transformer and the local feature extraction capability of CNN to achieve better image noise suppression effect. In the network framework structure, sub-network parallel technology is also adopted, and the cross-features in the sub-network interact with each other, and the relevant information between pixels is widely searched, thereby improving the denoising adaptability of the network in different scenarios.

6. The deep learning noise suppression reconstruction method for quantum image sensor according to claim 5, It is characterized in that In the step (5), in the step of performing an accurate unbiased variance-stabilized inverse transformation on the output image, an accurate unbiased variance-stabilized inverse transformation is used to restore the data to the original scale, that is, the Anscombe inverse transformation formula is first applied to each point in the network output image to make the data closer to the form of the original Poisson distribution; after the unbiased variance-stabilized Anscombe inverse transformation, the maximum likelihood estimation method is used to restore the true number of photons, and this accurate unbiased variance-stabilized inverse transformation is used to further suppress Poisson noise, thereby enhancing the suppression effect on the reconstructed image noise.

Images