SPAD active imaging data compression method oriented to extremely low illumination

Abstract

The invention discloses an ultra-low illumination-oriented SPAD active imaging data compression method, which comprises the following steps of: performing wavelet decomposition on histogram data of a single pixel in a time-frequency domain based on wavelet transform to obtain low-frequency data of each pixel after wavelet decomposition; by taking each pixel as a center, performing non-maximum suppression and data enhancement on the low-frequency data after the wavelet decomposition of the current pixel by using adjacent pixels to obtain processed compressed data; customizing different Gaussian kernel parameters for each pixel according to the possibility that each pixel is located at the boundary, and performing Gaussian filtering on the processed compressed data of each pixel to obtain filtered data; and performing depth estimation on the filtered data to obtain a final depth image. According to the method, the depth reconstruction performance of the laser pulse can be improved by utilizing the multi-resolution characteristic of wavelet transform, the space-time correlation of signal photons and the smoothness of Gaussian filtering.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of single photon imaging, and particularly relates to a SPAD active imaging data compression method for extremely low illumination. BACKGROUND

[0002] A laser radar (LiDAR) system uses a single photon avalanche diode (SPAD) as a core detector, and its working principle can be summarized as "emission-reflection-reception-processing" four key links. The system first emits a nanosecond pulsed laser beam (typical wavelength 905nm or 1550nm) through a laser, and when the laser encounters a target object, a reflection signal is generated. The SPAD detector can capture extremely weak photon echoes due to its single photon sensitivity, and convert a single photon event into a detectable electrical signal through an avalanche effect.

[0003] In terms of signal processing, the system accumulates reflection data of multiple laser pulses to generate a time histogram. The horizontal axis of the histogram represents a time bin, and each time bin corresponds to a specific distance interval; the vertical axis records photon counts, and the reflection signal of the target object forms a significant peak in the time bin corresponding to the distance, while the ambient light noise is uniformly distributed. By identifying the peak position in the time histogram, the system can calculate the target distance with a picosecond time resolution, and the typical ranging accuracy can reach centimeter level.

[0004] Compared with a traditional photomultiplier tube (PMT), the SPAD detector has two major advantages: first, its CMOS process compatibility supports large-scale array integration, which can realize high-resolution three-dimensional imaging; second, through time-correlated single photon counting (TCSPC) technology, it can effectively extract signals in an extremely low light intensity environment.

[0005] However, a super large scale SPAD array requires extremely high bandwidth when working: conservatively, a 30 frame / s megapixel SPAD array will generate extremely high data rates: In addition, in extreme cases such as extremely low photon count level and signal-to-background ratio, its depth reconstruction is easily affected by ambient light, resulting in poor depth reconstruction performance. SUMMARY

[0006] In order to solve the above problems in the prior art, the application provides a SPAD active imaging data compression method for extremely low illumination, which utilizes the multi-resolution characteristics of wavelet transform, the space-time correlation of signal photons, and the smoothing ability of Gaussian filtering to improve the performance of depth reconstruction under low photon count level. The technical problems to be solved by the application are solved by the following technical solutions:

[0007] This invention provides a SPAD active imaging data compression method for extremely low illumination, comprising:

[0008] S1: Based on wavelet transform, the histogram data of a single pixel is decomposed in the time-frequency domain to obtain the low-frequency data of each pixel after wavelet decomposition.

[0009] S2: Centered on each pixel, use neighboring pixels to perform non-maximum suppression and data augmentation on the low-frequency data after wavelet decomposition of the current pixel to obtain the compressed data after processing each pixel;

[0010] S3: Customize different Gaussian kernel parameters for each pixel based on the probability that each pixel is located on the boundary, and perform Gaussian filtering on the compressed data after processing each pixel to obtain the filtered data of each pixel;

[0011] S4: Perform depth estimation on the filtered data to obtain the final depth image.

[0012] In one embodiment of the present invention, S1 includes:

[0013] S1.1: Obtain the low-frequency filter coefficients {a1, a2, ..., a...} of the wavelet based on the type of wavelet used. n}, where n represents the number of low-frequency filter coefficients;

[0014] S1.2: Construct the first-level coding matrix based on the low-frequency filter coefficients:

[0015]

[0016] Among them, C k,i This represents the element value in the k-th row and i-th column of the first-level coding matrix; the number of rows in the first-level coding matrix is ​​equal to the number of time bins of the histogram corresponding to the histogram data, which is equal to the number of data points M of the histogram data;

[0017] S1.3: Multiply the first-level coding matrix with the histogram data to obtain the discrete data after first-level wavelet decomposition;

[0018] S1.4: Construct the coding matrix of the next level wavelet decomposition based on the low-frequency filter coefficients, and combine the discrete data after the previous level wavelet decomposition with the coding matrix of the next level wavelet decomposition to obtain the discrete data after the next level wavelet decomposition, wherein the number of rows in the coding matrix of the next level wavelet decomposition is equal to the number of bits in the discrete data after the previous level wavelet decomposition.

[0019] S1.5: Based on the predetermined wavelet decomposition level and the constructed multi-level coding matrix, obtain the time-frequency data of each pixel after multi-level wavelet decomposition.

[0020] In one embodiment of the present application, the S2 comprises:

[0021] S2.1: determining a neighborhood range of the current pixel with the current pixel as the center, sequentially adding data at corresponding positions in wavelet-decomposed time-frequency data of all pixels in the neighborhood range to obtain summed data of the current pixel;

[0022] S2.2: setting a window length and a step size of a sliding window, performing a sliding window operation on the summed data of the current pixel, adding data in each window of the sliding window, retaining data indexes in a window with a maximum addition result, and obtaining compressed data of the current pixel after sliding window processing;

[0023] S2.3: performing steps S2.1 and S2.2 for each pixel to obtain compressed data of each pixel after processing.

[0024] In one embodiment of the present application, the S2.1 comprises:

[0025] The 7x7 pixel array around the current pixel is taken as the neighborhood range with the current pixel as the center;

[0026] The first data element of the wavelet-decomposed time-frequency data of each pixel in the 7x7 pixel array is added, the second data element is added, the third data element is added, and so on until the last data element is added, and the data after the corresponding addition is taken as the summed data of the current pixel.

[0027] In one embodiment of the present application, the window length of the sliding window is 7 and the step size is 1.

[0028] In one embodiment of the present application, the S3 comprises:

[0029] S3.1: determining a neighborhood range of the current pixel with a fixed distance with the current pixel as the center, obtaining a peak value index of each pixel in the neighborhood range, and obtaining a highest frequency value and a second highest frequency value of occurrence of the peak value index of all pixels;

[0030] S3.2: obtaining a variance corresponding to the current pixel according to the highest frequency value and the second highest frequency value, and obtaining a Gaussian kernel corresponding to the current pixel according to the variance, wherein the Gaussian kernel is a two-dimensional data, each data element of which is a Gaussian kernel weight, and the relationship between the variance and the highest frequency value and the second highest frequency value is:

[0031]

[0032] wherein p peak represents the highest frequency value of occurrence of the peak value index, psubpeak denotes the second highest frequency value where the peak index occurs, and denotes a positive correlation;

[0033] S3.3: performing a smoothing filtering operation on the current pixel using the obtained Gaussian kernel, to obtain smoothed filtered data:

[0034]

[0035] wherein Gq denotes the weight of the Gaussian kernel of the qth pixel in the neighborhood range of the current pixel p, denotes the compressed data obtained by intercepting the qth pixel in the neighborhood range of the current pixel p in step S2, denotes the smoothed filtered data of the current pixel p, denotes the neighborhood range.

[0036] In an embodiment of the present application, the S4 comprises:

[0037] performing depth estimation on the filtered data using zero-mean normalized cross-correlation, to obtain a final depth image.

[0038] Another aspect of the present application provides a storage medium having stored therein a computer program for performing the steps of the SPAD active imaging data compression method for extremely low illumination in any of the above embodiments.

[0039] Still another aspect of the present application provides an electronic device comprising a memory having stored therein a computer program and a processor, wherein the processor, when invoking the computer program in the memory, implements the steps of the SPAD active imaging data compression method for extremely low illumination in any of the above embodiments.

[0040] Compared with the prior art, the present application has the following beneficial effects:

[0041] The present application provides a SPAD active imaging data compression method for extremely low illumination, which utilizes the multi-resolution characteristics of wavelet transform to encode (wavelet transform) the original histogram data of each pixel, to preliminarily realize data rate reduction and noise suppression; further generates a more compact data representation and effectively suppresses noise through the spatial correlation of adjacent pixels in the time-frequency domain; finally, performs smoothing processing on the compressed data using adaptive Gaussian filtering, so that the data bandwidth can be compressed by two orders of magnitude at a low photon counting level while having extremely high depth reconstruction performance.

[0042] The present application will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1is a flowchart of a SPAD active imaging data compression method for extremely low illumination provided by an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to further illustrate the technical means and effects taken by the present application to achieve the predetermined object, a SPAD active imaging data compression method for extremely low illumination according to the present application is described in detail below in combination with the drawings and specific embodiments.

[0045] The foregoing and other technical contents, features and effects of the present application can be clearly presented in the specific embodiment description below in combination with the drawings. Through the description of the specific embodiments, the technical means and effects taken by the present application to achieve the predetermined object can be understood more deeply and specifically. However, the attached drawings are provided for reference and illustration only, and are not intended to limit the technical solutions of the present application.

[0046] It should be noted that, in this document, relational terms such as first and second, and the like, are used solely to distinguish one entity or action from another entity or action, without necessarily requiring or implying any actual such relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without more limitations, an element defined by an expression "comprising a... " does not exclude the existence of additional identical elements in the process, method, article, or apparatus that includes the element.

[0047] Embodiment one

[0048] Please see Figure 1 , Figure 1 is a flowchart of a SPAD active imaging data compression method for extremely low illumination provided by an embodiment of the present application. The method comprises the following steps:

[0049] S1: based on wavelet transform, wavelet decomposition of the histogram data of a single pixel in time-frequency domain is performed to obtain low-frequency data after wavelet decomposition of each pixel.

[0050] The histogram is used to count the time delay distribution of laser pulses from emission to reception from a laser radar, reflecting the depth and intensity information of the target object. The horizontal axis of the histogram represents a time bin, and each time bin corresponds to a specific time interval; the vertical axis records the photon count. Thus, the histogram data of a single pixel can be regarded as one-dimensional discrete data, where each data corresponds to the number of photons in the current time bin, which can be represented as x={x1,x2,…,x m ,…,x M}, wherein x m represents the number of photons in the mth time bin, and M represents the total number of time bins.

[0051] Specifically, the step S1 of the embodiment includes the following steps:

[0052] S1.1: obtaining low-frequency filter coefficients {a1, a2, …, an} of a wavelet according to the type of the wavelet used. n}, wherein n represents the number of low-frequency filter coefficients.

[0053] Specifically, the type of the wavelet to be used for wavelet decomposition is determined according to actual needs, and after the type of the wavelet is obtained, the low-frequency filter coefficients {a1, a2, …, an} corresponding to the type of the wavelet are obtained. n}.

[0054] S1.2: constructing a first-level encoding matrix according to the low-frequency filter coefficients:

[0055]

[0056] wherein C k,i represents the element value in the kth row and the ith column of the first-level encoding matrix; the number of rows of the first-level encoding matrix is equal to the number of time bins in the histogram corresponding to the histogram data, that is, equal to the data number M of the histogram data.

[0057] In practice, the histogram data usually includes 1024 data elements, and in the embodiment, for the sake of simplicity of description, it is assumed that the histogram data is six bits, (11, 12, 13, 14, 15, 16), and it is assumed that the low-frequency filter coefficients of the selected wavelet are (2, 3, 4), that is, a1=2, a2=3, a3=4, and then the first-level encoding matrix constructed according to the formula (1) is:

[0058]

[0059] The data in each column of the first-level encoding matrix is the data in the previous column shifted down by two units. Specifically, the first-level encoding matrix is composed of the low-frequency filter coefficients and 0, the first column is composed of the last two bits of the low-frequency filter coefficients and 0, the second column is the low-frequency filter coefficients of the wavelet shifted down by two units based on the first column, the third column is the low-frequency filter coefficients of the wavelet shifted down by two units based on the second column, and so on until the previous column of the all-0 column, that is, the all-0 column is not included in the matrix.

[0060] S1.3: multiplying the first-level encoding matrix with the histogram data to obtain the data after the first-level wavelet decomposition.

[0061] Since the high frequency part of the laser pulse point cloud data usually corresponds to noise, only the low frequency part obtained after wavelet transform is reserved. For a one-dimensional discrete data with a length of M, the low frequency component of the first level wavelet decomposition is the result of twice down-sampling after the convolution of the low frequency filter coefficient and the one-dimensional discrete data. The result is equivalent to the result obtained by convolution with a step of two, which can be converted into a multiplication of the encoding matrix and the one-dimensional discrete data.

[0062] Taking the above assumed data as an example, the data after the first level wavelet decomposition is:

[0063]

[0064] S1.4: Construct the encoding matrix for the next level wavelet decomposition according to the low frequency filter coefficient, and multiply the discrete data after the previous level wavelet decomposition with the encoding matrix for the next level wavelet decomposition to obtain the discrete data after the next level wavelet decomposition, wherein the number of rows of the encoding matrix for the next level wavelet decomposition is equal to the number of bits of the discrete data after the previous level wavelet decomposition.

[0065] In this embodiment, the histogram data is decomposed at multiple levels, so after obtaining the discrete data after the first level wavelet decomposition, the second level wavelet decomposition is continued. First, a second level encoding matrix for the second level wavelet decomposition needs to be constructed. The form of the second level encoding matrix is the same as that of the first level encoding matrix, but the number of rows and the number of columns are different. The number of rows of the second level encoding matrix is equal to the number of bits of the discrete data after the first level wavelet decomposition. Taking the above assumed data as an example, the discrete data after the first level wavelet decomposition is (81, 119, 137, 32), and the number of data bits is 4. Therefore, the number of rows of the second level encoding matrix is 4. According to formula (1), the second level encoding matrix can be obtained as:

[0066]

[0067] The calculation process of the second level wavelet decomposition can be represented as:

[0068]

[0069] S1.5: According to the pre-determined wavelet decomposition level and the constructed multi-level encoding matrix, the time-frequency data after the multi-level wavelet decomposition of each pixel is obtained.

[0070] Similarly, the encoding matrix for the third level wavelet decomposition is constructed according to step S1.4. The original one-dimensional discrete data (histogram data) is multiplied with the first level encoding matrix to obtain the result of the first level wavelet decomposition. The result of the first level wavelet decomposition is multiplied with the second level encoding matrix to obtain the result of the second level wavelet decomposition. The result of the second level wavelet decomposition is multiplied with the third level encoding matrix to obtain the result of the third level wavelet decomposition, and so on.

[0071] In actual application, assuming that three-level wavelet decomposition needs to be performed, a first-level encoding matrix, a second-level encoding matrix and a third-level encoding matrix are first constructed, and then the first-level encoding matrix, the second-level encoding matrix and the third-level encoding matrix are multiplied to obtain a total encoding matrix; the histogram data is multiplied by the total encoding matrix to obtain time-frequency data of the current pixel after multi-level wavelet decomposition.

[0072] Continuing with the above-mentioned assumed data as an example, three-level wavelet decomposition can be expressed as:

[0073]

[0074] The above steps are performed for the histogram data of each pixel, and then low-frequency data of each pixel after multi-level wavelet decomposition can be obtained, which is still a set of one-dimensional discrete data at this time.

[0075] For the case of performing multi-level wavelet decomposition on the histogram data and retaining the low-frequency data (low-frequency components), since each level of low-frequency data is the result obtained by wavelet decomposition of the previous level, the encoding matrix required for h-level wavelet decomposition is the product of multiple single-level encoding matrices constructed by formula (1) in step S1.2, and the number of rows of each encoding matrix corresponds to the number of data bits of the result obtained by the previous level of wavelet decomposition. The total encoding matrix corresponding to multi-level wavelet decomposition is denoted as C h , and the low-frequency data after multi-level wavelet decomposition is , which can be obtained by the following formula:

[0076]

[0077] where x = {x1, x2, …, x m , x M} represents the histogram data, and in this embodiment, h = 2 or 3, i.e., the number of levels of wavelet decomposition is 2 or 3. Exemplarily, if the original histogram data includes 1024 data elements and the low-frequency filter coefficient includes 12-bit data elements, the data after three-level wavelet decomposition is discrete data including 137-bit data elements.

[0078] S2: Centering on each pixel, the low-frequency data of the current pixel after wavelet decomposition is subjected to non-maximum suppression and data enhancement by using adjacent pixels to obtain processed compressed data of each pixel.

[0079] Specifically, S2 of the present embodiment includes the following steps:

[0080] S2.1: Centering on the current pixel, the neighborhood range of the current pixel is determined at a fixed distance, and the data at the corresponding positions in the time-frequency data of all pixels after wavelet decomposition in the neighborhood range are sequentially added to obtain the summed data of the current pixel.

[0081] Exemplarily, taking the current pixel as the center, a 7x7 pixel array around the current pixel is taken as the neighborhood range. Assuming that the data of each pixel after wavelet decomposition in step S1 is discrete data including 137 data elements, the first data element of the 7x7 pixel wavelet decomposed time-frequency data is added, the second data element is added, the third data element is added, the fourth data element is added, and the last data element is added, and a discrete data including 137 data elements is obtained after the corresponding addition, which is taken as the data of the current pixel after summation.

[0082] S2.2: Set the window length and step length of the sliding window, and perform sliding window operation on the data of the current pixel after summation, add the data in each sliding window, retain the data index in the window with the maximum addition result, and obtain the compressed data of the current pixel after sliding window processing.

[0083] Specifically, the window length of the sliding window is set to 7 and the step length is set to 1. As described above, assuming that the data of the current pixel after summation in step S2.1 is discrete data including 137 data elements, represented as {y1, y2, y3,..., y137}, the sliding window operation is performed on the discrete data, and the data in each sliding window is added, and a group of data {y1+y2+y3+y4+y5+y6+y7, y2+y3+y4+y5+y6+y7+y8,..., y 137 131 132 133 134 135 136 137} is obtained, and the size of each data after addition is determined. Assuming that the value of y2+y3+y4+y5+y6+y7+y8 is the largest, the data index in the current window is retained, that is, the data indexes corresponding to the values y2, y3, y4, y5, y6, y7, and y8 are retained, and other data is regarded as noise and discarded, and the compressed data {y2, y3, y4, y5, y6, y7, y8} of the current pixel after processing and the data index corresponding to each data are obtained.

[0084] Similarly, the processing of steps S2.1 and S2.2 is performed on each pixel, and the compressed data of each pixel after processing is obtained.

[0085] ​​​​​​​The sliding window operation of this step can be equivalent to performing a convolution operation on the summation result and a one-dimensional data with all 1s and a length equal to the sliding window length, to obtain the window index corresponding to the maximum result. For the current pixel to be processed, only the data within the index range is retained, and the data outside the index range is discarded as noise. The sliding window operation process can be represented as:

[0086]

[0087] where i represents the data index of the current pixel, i.e., the position of the data; i0 represents the data index start position corresponding to the window with the maximum summation value; W represents the window length of the sliding window; N p represents the neighborhood range of the pixel to be processed p, represents the final data obtained by the current pixel after being cut off, i.e., the compressed data of the current pixel after sliding window processing, represents the time-frequency data after wavelet decomposition of the data with index i in the qth pixel in the neighborhood range, represents the time-frequency data after wavelet decomposition of the data with index i in the pth pixel in the neighborhood range.

[0088] S3: Customizing different Gaussian kernel parameters for each pixel according to the possibility of each pixel being located at the boundary, and performing Gaussian filtering on the compressed data of each pixel after processing, to obtain the filtered data of each pixel.

[0089] Specifically, step S3 of this embodiment includes:

[0090] S3.1: Determining the neighborhood range of the current pixel with the current pixel as the center and a fixed distance, obtaining the peak value index of each pixel in the neighborhood range, and obtaining the highest frequency value and the second highest frequency value of the peak value index of all pixels.

[0091] S3.2: Obtaining the variance corresponding to the current pixel according to the highest frequency value and the second highest frequency value, and obtaining the Gaussian kernel corresponding to the current pixel according to the variance.

[0092] For natural scenes and artificial scenes, there is spatial correlation between adjacent pixels - adjacent pixels can belong to the same object, thus having similar depth. In addition, it can be approximately considered that the peak index of the original time histogram is positively correlated with the pixel depth value, and after wavelet transform, this correlation is still retained. In view of this, the peak index of all pixels in the fixed neighborhood of the current pixel to be processed is taken, and the highest frequency value and the second highest frequency value of the index are used to approximately obtain the possibility of the pixel located at the boundary, and the Gaussian kernel with smaller variance is applied to the pixel with high possibility of being located at the boundary, and the Gaussian kernel with larger variance is applied to the pixel with low possibility of being located at the boundary, wherein the Gaussian kernel is a two-dimensional data, and each data element is a Gaussian kernel weight, and the relationship between the variance and the highest frequency value and the second highest frequency value is as follows:

[0093]

[0094] Wherein, ρ peak represents the highest frequency of the peak index, ρ subpeak represents the second highest frequency of the peak index, and α represents the positive correlation.

[0095] S3.2: Smooth filtering operation is performed on the current pixel using the obtained Gaussian kernel, and smoothed data is obtained:

[0096]

[0097] Wherein, Gq represents the weight of the Gaussian kernel of the qth pixel in the neighborhood range of the current pixel p, represents the compressed data obtained by intercepting the qth pixel in the neighborhood range of the current pixel p in step S2, represents the smoothed data of the current pixel p, represents the neighborhood range.

[0098] S4: Depth estimation is performed on the filtered data to obtain a final depth image.

[0099] In this embodiment, zero-mean normalization cross-correlation (ZNCC) is used to perform depth estimation on the filtered data to obtain a final depth image. The principle of depth estimation is as follows:

[0100]

[0101] Wherein, represents the time shift required for depth estimation, represents the depth (c represents the speed of light), C represents the total coding matrix designed based on wavelet, and C hdenotes the result of the convolution of each row of the total encoding matrix C with the system impulse response function, denotes the data of the i-th column of C h denotes the data of the i-th column of C

[0102] The SPAD active imaging data compression method for extremely low illumination proposed in the application, compared with other methods, utilizes the multi-resolution characteristics of wavelet transform, the spatial correlation brought by the structure of the scene itself, and the smoothing ability of Gaussian filtering, to improve the performance of deep reconstruction in extremely challenging scenes, i.e. low photon counting level. At the same time, it can also be combined with other deep reconstruction methods to further improve the performance.

[0103] The effectiveness of the method of the present application is verified by selecting real collected SPAD histogram data and representative Middlebury dataset in the field. Specifically, simulated histogram data with signal-to-background photon ratios of 2:50 and 10:2 are generated based on eight scenes of the Middlebury dataset, and the deep reconstruction performance of the method of the present application is verified under compression ratios of 128 and 78.Please refer to Table 1, which shows the root mean square error and mean absolute error (mm) of the proposed method and the existing ML method (log-matched filter), the Shin method [D. Shin, A. Kirmani, V. K. Goyal, and J. H. Shapiro, “Photon-efficient computational 3-d and reflectivity imaging with single-photondetectors,” IEEE Transactions on Computational Imaging, vol. 1, no. 2, pp. 112-125, 2015.], the UA method [J. Rapp and V. K. Goyal, “A few photons among many: Unmixing signal and noise for photon-efficient active imaging,” IEEE Transactions on Computational Imaging, vol. 3, no. 3, pp. 445-459, 2017.] (iterative algorithm, high computational complexity), the Lindell method [D. B. Lindell, M. O’Toole, and G. Wetzstein, “Single-photon 3d imaging with deep sensor fusion.” ACM Trans. Graph., vol. 37, no. 4, p. 113, 2018] (neural network method, computationally intensive), and the SSPI method [G. Yao, Y. Chen, C. Jiang, Y. Xuan, X. Hu, Y. Liu, and Y. Pan, “Dynamic single-photon 3d imaging with a sparsity-based neural network,” Optics Express, vol. 30, no. 21, pp. 37 323-37 340, 2022.] (neural network method, computationally intensive) under the eight scenes of the Middlebury simulated dataset, where the average signal photon number and background photon number are 2 and 50, respectively. The results show that the proposed method has lower average error in depth reconstruction than other methods under lower computational complexity, and better depth reconstruction performance is obtained.

[0104] Table 1. Performance comparison results of the method provided by the embodiment of the present application compared with other methods

[0105]

[0106] The application provides a SPAD active imaging data compression method for extremely low illumination, which encodes (wavelet transform) original histogram data of each pixel by using the multi-resolution characteristics of wavelet transform, preliminarily realizes data rate reduction and noise suppression, further generates more compact data representation and effectively suppresses noise through the spatial correlation of adjacent pixels in the time-frequency domain, and finally performs smoothing processing on the compressed data by using adaptive Gaussian filtering, so that the data bandwidth can be compressed by two orders of magnitude at a low photon counting level while having extremely high depth reconstruction performance.

[0107] Another embodiment of the application provides a storage medium in which a computer program is stored, the computer program being used to execute the steps of the SPAD active imaging data compression method for extremely low illumination described in the above embodiments. In another aspect of the application, an electronic device is provided, which includes a memory and a processor, the memory storing a computer program, and the processor calling the computer program in the memory to realize the steps of the SPAD active imaging data compression method for extremely low illumination described in the above embodiments. Specifically, the integrated modules realized in the form of software functional modules described above can be stored in a computer-readable storage medium. The software functional modules described above are stored in a storage medium, including a plurality of instructions for enabling an electronic device (which can be a personal computer, a server, or a network device, etc.) or a processor to execute part of the steps of the method described in the embodiments of the application. The aforementioned storage medium includes a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program code storage media.

[0108] The above is a further detailed description of the application in combination with specific preferred embodiments, and the specific implementation of the application cannot be limited to these descriptions. For ordinary skilled persons in the technical field to which the application belongs, some simple deductions or replacements can be made without departing from the concept of the application, and all of them should be regarded as falling within the protection scope of the application.

Claims

1. A SPAD active imaging data compression method for extremely low illumination, characterized in that, The method comprises the following steps: S1: wavelet decomposition is performed on the histogram data of a single pixel in a time-frequency domain based on wavelet transformation, and low-frequency data after wavelet decomposition of each pixel is obtained; S2: non-maximum suppression and data enhancement are performed on the low-frequency data after wavelet decomposition of the current pixel by using adjacent pixels as the center of each pixel, and compressed data after processing of each pixel is obtained; S3: different Gaussian kernel parameters are customized for each pixel according to the possibility of each pixel being located at a boundary, and Gaussian filtering is performed on the compressed data after processing of each pixel, and filtered data of each pixel is obtained; S4: depth estimation is performed on the filtered data, and a final depth image is obtained.

2. The extremely low-illumination-oriented SPAD active imaging data compression method according to claim 1, wherein, The S1 comprises: S1.1: Obtain low-frequency filter coefficients {a1, a2,..., an} of the wavelet according to the type of the wavelet used, where n represents the number of low-frequency filter coefficients. n}, where n represents the number of low-frequency filter coefficients. S1.2: a first-level encoding matrix is constructed according to the low-frequency filter coefficient; wherein C k,i represents the element value of the kth row and ith column in the first-level encoding matrix; the number of rows of the first-level encoding matrix is equal to the number of time bins of the histogram corresponding to the histogram data, that is, equal to the data number M of the histogram data. S1.3: the first-level encoding matrix is multiplied with the histogram data, and discrete data after one-level wavelet decomposition is obtained; S1.4: an encoding matrix for next-level wavelet decomposition is constructed according to the low-frequency filter coefficient, and discrete data after one-level wavelet decomposition is multiplied with the encoding matrix for next-level wavelet decomposition, and discrete data after next-level wavelet decomposition is obtained, wherein the number of rows of the encoding matrix for next-level wavelet decomposition is equal to the number of bits of the discrete data after one-level wavelet decomposition; S1.5: time-frequency data after multi-level wavelet decomposition of each pixel is obtained according to a predetermined wavelet decomposition level and the constructed multi-level encoding matrix.

3. The extremely low-illumination-oriented SPAD active imaging data compression method according to claim 2, characterized in that, The S2 comprises: S2.1: a neighborhood range of the current pixel is determined with the current pixel as the center, data at corresponding positions in time-frequency data after wavelet decomposition of all pixels in the neighborhood range are sequentially added, and summed data of the current pixel is obtained; S2.2: a window length and a step length of a sliding window are set, the summed data of the current pixel is subjected to sliding window operation, data in a window with a maximum addition result is retained by adding data in each sliding window, and compressed data after sliding window processing of the current pixel is obtained; S2.3: steps S2.1 and S2.2 are performed for each pixel, and compressed data after processing of each pixel is obtained.

4. The extremely low-illumination-oriented SPAD active imaging data compression method according to claim 3, characterized in that, The S2.1 comprises: The current pixel is taken as the center, and a 7*7 pixel array around the current pixel is taken as the neighborhood range; first data elements of time-frequency data after wavelet decomposition of each pixel in the 7*7 pixel array are added, second data elements are added, third data elements are added, and the like is added until last data elements are added, and the data after addition is taken as summed data of the current pixel.

5. The extremely low-illumination-oriented SPAD active imaging data compression method according to claim 3, characterized in that, The window length of the sliding window is 7, and the step length is 1.

6. The extremely low-illumination-oriented SPAD active imaging data compression method according to claim 1, wherein, The S3 comprises: S3.1: a neighborhood range of the current pixel is determined with a fixed distance as the center, peak value indexes of each pixel in the neighborhood range are obtained, and a highest frequency value and a second highest frequency value of all peak value indexes are obtained; S3.2: obtaining a variance corresponding to the current pixel according to the highest frequency value and the second highest frequency value, and obtaining a Gaussian kernel corresponding to the current pixel according to the variance, wherein the Gaussian kernel is a two-dimensional data, each data element of which is a Gaussian kernel weight, and the variance and the highest frequency value and the second highest frequency value satisfy the following relationship: where ρ peak represents the highest frequency value of peak index occurrence, ρ subpeak represents the second highest frequency value of peak index occurrence, and α represents a positive correlation; S3.3: performing a smoothing filtering operation on the current pixel by using the obtained Gaussian kernel, to obtain filtered data: wherein Gq represents a weight of a Gaussian kernel of the qth pixel in the neighborhood range of the current pixel p, denotes the compressed data obtained by intercepting the qth pixel in the neighborhood range of the current pixel p in step S2, denotes the data of the current pixel p after the smoothing filtering, denotes the neighborhood range.

7. The extremely low-illumination-oriented SPAD active imaging data compression method according to claim 1, wherein, The S4 comprises: performing depth estimation on the filtered data by using zero-mean normalized cross-correlation, to obtain a final depth image.

8. A storage medium, characterized by The storage medium stores a computer program, and the computer program is used for executing the steps of the SPAD active imaging data compression method for extremely low illumination in any one of claims 1 to 7.

9. An electronic device, comprising: The device comprises a memory and a processor, the memory stores a computer program, and the processor invokes the computer program in the memory to implement the steps of the SPAD active imaging data compression method for extremely low illumination in any one of claims 1 to 7.

Images