A method for on-chip compressive encoding acquisition of scene information based on compressive sensing

Abstract

The present invention discloses a method for on-chip compressed coding acquisition of scene information based on compressive sensing. The specific steps are as follows: (1) Perform photonization processing on the original image to construct a model of the image under the transient photon distribution; (2) Compressively acquire the photonization model of the image; (3) According to the dead time effect of the sensor and the register bit number, eliminate the invalid photons and construct the training data pairs of the acquired data and the original image; (4) Input the training data pairs into the neural network to train the network model; (5) Calculate the error between the reconstructed image and the real image, and update the network parameters by backpropagation of the error; (6) Repeat steps (4) to (5) until the error is small, complete the training of the neural network, and output the reconstructed image. This method can achieve on-chip compressed coding acquisition of scene information, improve the sensor fill factor, and has a good noise suppression effect, enabling high-quality acquisition and reconstruction of scene information.

Description

Technical Field

[0001] The present invention relates to the field of computational photography, and particularly to a method for on-chip compressed encoding acquisition of scene information based on compressive sensing. Background Art

[0002] Performing high-speed imaging of scene information is a challenging problem in the field of computational photography. From capturing bullets to capturing the propagation process of light in a scene, the continuous improvement and breakthrough of time-dimensional imaging capabilities have made imaging tasks that were previously impossible possible, and high-speed imaging technology has received increasing attention.

[0003] Single photons can be detected by photodetectors with inherently high gain. These photodetectors include photomultiplier tubes, microchannel plate photomultiplier tubes, and single-photon avalanche photodiodes, etc. Among them, the single-photon avalanche diode combined with time-correlated imaging technology can achieve high-speed imaging with picosecond-level time resolution. This imaging technology has zero readout noise, high sensitivity, and low cost. However, the single-photon avalanche diode requires a quenching circuit for actual use, its fill factor is insufficient, and the detection efficiency is limited, which has certain limitations in practical applications.

[0004] Therefore, how to design an effective scene information acquisition scheme, solve the problem of insufficient fill factor of the single-photon avalanche diode, and perform high-quality reconstruction of the acquired scene information is a research hotspot today. Summary of the Invention

[0005] Aiming at the above-mentioned defects existing in the actual scene information acquisition of the existing single-photon avalanche diode, the purpose of the present invention is to propose a method for on-chip compressed encoding acquisition of scene information based on compressive sensing.

[0006] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0007] A method for on-chip compressed encoding acquisition of scene information based on compressive sensing includes the following steps:

[0008] Step 1, perform photonization processing on the original image data x to construct a model x' of the image under the photon transient distribution;

[0009] Step 2, perform compressed acquisition on the photon transient distribution model x' obtained in Step 1 according to the connection distribution of the register and the sensor photosensitive element to obtain the data y' after compressed acquisition;

[0010] Step 3: According to the dead time effect of the sensor and the number of bits of the register, eliminate the photons that are saturated and invalid due to the dead time effect and the register bit number in the data y' collected after compression in Step 2, and obtain a photonization model that conforms to compressive sampling in the real scene. The number of photons in the photonization model is the data y after compressive sampling, and construct a training data pair in one-to-one correspondence between the original image data x and the data y;

[0011] Step 4: Input the training data pair obtained in Step 3 into the convolutional neural network and train the network model;

[0012] Step 5: Calculate the error between the reconstructed image and the real image through the loss function, and update the weight parameters and bias parameters of the network through the backpropagation of the error;

[0013] Step 6: Repeat Steps 4-5 to train the convolutional neural network until the error between the reconstructed image and the real image is small, complete the training of the neural network, and output the reconstructed image of compressive sampling.

[0014] Furthermore, in Step 1, the specific steps for photonizing the original image data x are as follows:

[0015] Step 11: For a pixel point with a pixel value of x in the original image data x 11 generate a random number from 0 to 1, and repeat this operation until the generated random number t 0 is less than the value of the negative exponential distribution with a rate parameter of x 11 at the random number t 0 The value of t 0 is the time interval for photon generation; add the time count t to the time interval t for photon generation 0 and set the value of the time count t + t 0 at this time from 0 to 1, indicating that there is a photon at this moment;

[0016] Step 12: Repeat the operation in Step 11 until the time position reaches the exposure time length of the sensor, and a photon transient distribution model at the pixel point with a pixel value of x 11 can be obtained;

[0017] Step 13: Repeat the operations in Steps 11 and 12 for each pixel point of the original image data x, and then obtain the model x' of the entire original image under the photon transient distribution.

[0018] Furthermore, in Step 2, the specific steps for compressing and collecting the photon transient distribution model x' are as follows:

[0019] Step 21: Divide the photon transient distribution model x′ of the original image into blocks according to the size of the sensor array. According to the theory of compressive sensing, four different connection methods between the register and the sensor are adopted. The connection method requirements are as follows: (1) Each pixel point is connected to the register at least once in the four connection methods. (2) Only half of the sensor pixel points are connected to the register in each connection method, and the connection points are randomly distributed. Collect the pixel points belonging to the same sensor array.

[0020] Step 22: Arrange the photon transient distribution model x′ corresponding to the 8 pixel points connected to the register on the sensor array in the order of the time when each photon reaches the sensor, that is, obtain the compressed acquisition data under one connection method. The 4 data obtained by the four connection methods between the register and the sensor are a set of compressed acquisition data, that is, the compressed acquisition data of the array.

[0021] Step 23: Repeat the operation of Step 22 for each block pixel array of the photon transient distribution model x′ of the original image, and the compressed acquisition data y′ can be obtained.

[0022] Further, in Step 3, photons with an adjacent time interval less than the dead time in the compressed acquisition data y′ are eliminated, and photons exceeding the register bit number are eliminated.

[0023] Based on the single-photon avalanche diode for scene information acquisition, the present invention uses the theory of compressive sensing to compress and acquire scene information, and solves the negative impact brought by the insufficient fill factor of the single-photon avalanche diode. At the same time, in the stage of reconstructing the compressed acquisition information, a training set is made by imitating the mathematical model in actual information acquisition, and a network model corresponding to the single-photon avalanche diode in this method is trained, and the original scene image with higher quality can be reconstructed through the compressed acquisition data. Compared with the prior art, this method can achieve better results both in the efficiency of scene information acquisition and the quality of the obtained scene image, and can effectively cope with the challenges brought by the insufficient fill factor of the single-photon avalanche diode. Description of the Drawings

[0024] Figure 1 It is a flow chart of the method of the present invention;

[0025] Figure 2 It is a schematic diagram of the connection between the single-photon avalanche diode register and the sensor in the embodiment, where (a), (b), (c), and (d) represent four different connection methods;

[0026] Figure 3 It is a schematic diagram of the image reconstruction network in the embodiment;

[0027] Figure 4 It is a schematic diagram of the convolutional residual network in the embodiment. Detailed implementation manners

[0028] The present invention will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The described embodiments are exemplary and are intended to explain the present invention, and should not be construed as a limitation to the present invention.

[0029] Referring to Figure 1 , a method for on-chip compression encoding acquisition of scene information based on compressive sensing in this embodiment is as follows:

[0030] Step 1: Photonize each pixel point of the original image data x(32×32). Taking one pixel value of x 11 as an example of the pixel point:

[0031] Step 11: Generate a random number from 0 to 1, and repeat this operation until the generated random number t 0 is less than the value of the negative exponential distribution with the rate parameter of x 11 at the random number t 0 . The value of t 0 is the time interval for photon generation. Add the time interval t 0 for photon generation to the time count t, and set the value of the time count t + t 0 from 0 to 1, indicating that there is a photon at this moment;

[0032] Step 12: Repeat Step 11 until the time count is greater than or equal to the sensor exposure time. In this example, the sensor exposure time is 0.001 s;

[0033] Step 13: Repeat Step 11 and Step 12 for each pixel point of the original image x to obtain the photon transient distribution model x′ of the original image.

[0034] Step 2: Divide the photonized transient distribution model x′ of the original image into blocks according to the size of the sensor array. Figure 2In this embodiment, there are 4 connection distributions designed for the register and the sensor photosensitive element based on the compressed sensing theory. The requirements for the connection distribution are as follows: (1) Each pixel point is connected to the register at least once in the 4 connection methods. (2) Only half of the sensor pixel points are connected to the register in each connection method, and the connection points are randomly distributed. The pixel points belonging to the same sensor array are collected. Among them, the points marked "1" in the figure represent the points where the pixel points are connected to the register, that is, the collected points, and the unmarked points indicate that they are not collected. The photonization models of the pixel points connected to the register are sorted according to the time series. In a 4×4 photosensitive array, 4 compressed acquisition data can be obtained through 4 connection methods. Repeat the operation of step 2 for each block after the image is segmented, and the data y′(1×256) of the photonization transient distribution model of the original image after compressed sampling can be obtained, realizing 25% compressed sampling.

[0035] Step 3, according to the dead time effect t of the sensor dead and considering the limitation of the register bit number, eliminate the photons that fail due to the dead time effect and register bit number saturation during the compressed acquisition process, obtain the data y that conforms to the sampling in the real scenario, and construct a training data pair in one-to-one correspondence between the real image x and the data y after transient model compressed sampling. In this example, the dead time t of the sensor dead = 2×10 -9 s, and the register bit number is 12bit.

[0036] Step 4, input the training data pair obtained in step 3 into the convolutional neural network to train the network model. In this example, the neural network used is as Figure 3 and Figure 4 shown, and the specific structure is as follows:

[0037] First, perform batch normalization operation on the input data, and then input it into the first layer of the neural network: the linear fully connected layer. The number of input layer nodes is the dimension 256 of the signal after compressed sampling, and the number of output nodes is the dimension 1024 of the original image, realizing the upsampling of the image; then pass through 3 groups of residual modules of (Conv2d+BN+ReLU)-(Conv2d+BN+ReLU)-(Conv2d+BN+ReLU) to further reconstruct the details of the image. The size of the convolution kernel used is 3×3, and zero padding operation is performed for each layer of convolution. In addition, the change of the tensor channel number in the residual block is 64, 32, 1, and finally the reconstructed image x out .

[0038] Step 5, calculate the error between the reconstructed image and the real image through the loss function MSE, and update the weight parameter w and bias parameter b of the network through the reverse transmission of the error. The optimizer for updating the network parameters by the reverse transmission of the error is Adam.

[0039] Step 6: Repeat Steps 4-5 to train the neural network until the error between the reconstructed image and the real image is small, complete the training of the neural network, and output the reconstructed image.

Claims

1. A method for on-chip compressed encoding acquisition of scene information based on compressive sensing, characterized in that, it includes the following steps: Step 1, perform photonization processing on the original image data x to construct a model x' of the image under the photon transient distribution; Step 2, according to the connection distribution of the register and the sensor photosensitive element, perform compressive acquisition on the photon transient distribution model x' obtained in Step 1 to obtain the data y' after compressive acquisition; the specific steps of its compressive acquisition are: Step 21, divide the photon transient distribution model x' of the original image into blocks according to the size of the sensor array. According to the compressive sensing theory, adopt 4 different connection methods between the register and the sensor. The connection method requirements are: (1) each pixel point is connected to the register at least once in the 4 connection methods, (2) only half of the sensor pixel points are connected to the register in each connection method, and the connection points are randomly distributed; collect the pixel points belonging to the same sensor array; Step 22, arrange the corresponding photon transient distribution model x' of the 8 pixel points connected to the register on the sensor array in the order of the time when each photon reaches the sensor, that is, obtain the compressive acquisition data under one connection method. The 4 data obtained by the 4 connection methods between the register and the sensor are a set of compressive acquisition data, that is, the compressive acquisition data of this array; Step 23, repeat the operation of Step 22 for each block pixel array of the photon transient distribution model x' of the original image, and then the data y' after compressive acquisition can be obtained; Step 3, according to the dead time effect of the sensor and the register bit number, eliminate the photons in the data y' after compressive acquisition in Step 2 that are saturated and invalid due to the dead time effect and the register bit number, and obtain a photonization model that conforms to the compressive sampling in the real scene. The number of photons in the photonization model is the data y after compressive sampling, and construct a training data pair in one-to-one correspondence between the original image data x and the data y; Step 4, input the training data pair obtained in Step 3 into the convolutional neural network and train the convolutional neural network; Step 5, calculate the error between the reconstructed image and the real image through the loss function, and update the weight parameters and bias parameters of the network through the reverse transmission of the error; Step 6, repeat Steps 4-5 to train the convolutional neural network until the error between the reconstructed image and the real image is less than the threshold, complete the training of the neural network, and output the reconstructed image of the compressive sampling.

2. A method for on-chip compressed encoding acquisition of scene information based on compressive sensing according to Claim 1, characterized in that, in the said Step 1, the specific steps of performing photonization processing on the original image data x are: Step 11, for a pixel value x of the original image data x 11 pixel point, generate a random number from 0 to 1, and repeat this operation until the generated random number t 0 is less than the negative exponential distribution with a rate parameter of x 11 at the random number t 0 value, t 0 is the time interval for photon generation; add the time count t to the time interval t for photon generation 0 , and set the value of the time count t + t 0 at this time from 0 to 1, indicating that there is a photon at this moment; Step 12, repeat the operation in Step 11 until the time position reaches the exposure time length of the sensor, and a photon transient distribution model of the pixel points with pixel value x can be obtained. 11 ​ Step 13, repeat the operations of Step 11 and Step 12 for each pixel point of the original image data x, and then the model x' of the entire original image under the photon transient distribution can be obtained.

3. A method for on-chip compressed encoding acquisition of scene information based on compressive sensing according to Claim 1, characterized in that, in the said Step 3, eliminate the photons in the data y' after compressive acquisition whose adjacent time intervals are less than the dead time, and eliminate the photons exceeding the register bit number.

Images