Image deblurring method applied to quantum image sensor

Abstract

The present application relates to image sensor, image processing technical field, for putting forward new image reconstruction method, obtains the image of removing motion blur, for this, the technical scheme that the present application adopts is, the image removing motion blur method for quantum image sensor, the block and image of input binary image sequence composition, patch-based optical flow algorithm realizes the alignment of block and image, based on the merging algorithm of incident photon number change with time realizes binary sequence merging, obtains the image of removing motion blur, wherein, the combination of multiple binary image sequences is called "block", the spatial window of pixel is called "patch", the sum of binary sequence in block is called "block and image". The present application is mainly applied to the design and manufacture occasion of quantum image sensor.

Description

Technical Field

[0001] This invention relates to the fields of image sensors and image processing technology, and specifically to an image de-motion blurring method applied to quantum image sensors. Background Technology

[0002] Quantum image sensors (QIS) possess three main characteristics: single-photon counting, spatial oversampling, and temporal oversampling. Compared to traditional CMOS image sensors, QIS exhibits lower readout noise, a wider dynamic range, a higher frame rate, and lower dark current, thus significantly improving image acquisition quality in many challenging environments. QIS holds broad application prospects in fields such as high-resolution imaging, target tracking and recognition, low-light imaging, large dynamic range imaging, and high-speed photography.

[0003] Due to the temporal and spatial oversampling characteristics of QIS, a large amount of 3D bitstream is generated, requiring specific image reconstruction algorithms to reconstruct the bitstream into an image. Currently, most image reconstruction algorithms are only suitable for reconstructing static scenes, such as directly merging binary bitstreams to obtain a reconstructed image. However, when capturing moving objects or scenes and performing image processing, using traditional image reconstruction algorithms will result in severe motion blur in the reconstructed image. Summary of the Invention

[0004] To overcome the shortcomings of existing technologies, this invention aims to propose a new image reconstruction method to obtain motion-blurred images. The technical solution adopted by this invention is an image motion-blurring method applied to quantum image sensors. This method assembles input binary image sequences into blocks and images, aligns the blocks and images using a patch-based optical flow algorithm, and merges the binary sequences using a merging algorithm based on the time-varying incident photon count to obtain a motion-blurred image. Here, multiple binary image sequences combined together are called a "block," the spatial window of a pixel is called a "patch," and the sum of the binary sequences within a block is called the "block and image."

[0005] In the steps of assembling blocks and images from a binary image sequence, the input binary image sequence is first divided into non-overlapping time blocks with the same number of frames. The number of frames is determined based on the illumination level and motion changes; the lower the illumination level and the more drastic the motion changes, the more frames are in the time block. Secondly, the block at the center of the sequence is used as a reference block, and the other blocks are used as auxiliary blocks. Finally, the binary image sequences in each block are added together to form the block and the image.

[0006] In the step of aligning blocks and images using the patch-based optical flow algorithm, a hierarchical patch alignment method is first adopted. That is, the number of pyramid layers is determined according to the resolution of the input image. In the nearest neighbor spatial domain of each pyramid layer, the minimum mean absolute error (L1 matching error) between the auxiliary block and the reference block is calculated. The purpose is to align the auxiliary block with the reference block and generate a coarse optical flow vector. Secondly, global regularization is performed at the optimal level of the pyramid to refine the patch alignment result. Finally, linear interpolation is performed on the coarse optical flow vector to obtain a fine optical flow vector. The fine optical flow vector is then mapped to a binary image sequence to obtain an aligned binary image sequence.

[0007] In the steps of merging binary sequences based on the merging algorithm of incident photon number changing over time, the aligned binary sequences are first divided into different time blocks to determine the reference block and auxiliary block. Secondly, based on the change of photon number within the time block over time, the variance of the sum of pixels at the same position is compared with the threshold. Different merging methods are used according to the comparison results to obtain the merged image. Finally, the merged image is subjected to variance-stabilized Anscombe transform and 3D block matching filtering algorithm BM3D for denoising, and the nonlinear response is inverted to obtain a linear image without motion blur and noise.

[0008] A patch-based optical flow algorithm implements a detailed block and image alignment process:

[0009] (1) Determine the number of pyramid layers based on the resolution of the input binary image; the higher the resolution, the more pyramid layers there are.

[0010] (2) For a pixel index (i,j), the index is expanded using a patch to [iM,(i+1)M-1]×[jM,(j+1)M-1]. Find the minimum motion vector (u,v) that minimizes the L1 matching error in the following equation:

[0011]

[0012] Where M×M is the size of the patch used, S aux As an auxiliary block, S ref For reference block;

[0013] (3) Spatial regularization is achieved at the finest level of the pyramid by minimizing the energy in the following equation:

[0014]

[0015] in It is the spatial domain of pixel indices i,j, E d For matching error, The regularization term, also known as the smoothing factor, is used to represent the smooth variation of optical flow across the entire image. H×W is the spatial resolution of the input image. This is achieved by minimizing the Charbonnier loss. As a differentiable substitute for L1 loss, the optical flow vector of inter-block motion roughness is obtained;

[0016] (4) Linear interpolation is performed on the coarse optical flow vector to obtain the fine optical flow vector for inter-frame motion, and the fine optical flow vector is mapped to the binary image sequence to achieve alignment of the binary image sequence.

[0017] The detailed process of merging binary sequences based on the merging algorithm of incident photon number changing over time is as follows: First, the aligned binary sequence is divided into different time blocks, the size of which can be... Figure 2 The differences are identified, and reference and auxiliary blocks are determined. Next, the sum of pixels at the same position in the time block is calculated, and the variance of the sum is calculated. If the variance is less than a set threshold T, the pixel at that position is considered stationary. The stationary region is averaged as the merging result for that position. The specific formula is as follows:

[0018]

[0019] in, The merged pixel values, G is the size of the time block, and S is the value of the merged pixel value. ref The pixel value of the reference block is used. If the variance is greater than the set threshold, it indicates that the alignment step may have failed, and the pixel at that position has significant movement. Positions that differ significantly from the reference block are assigned lower weights, and the pixel value of that position in the reference block is used to replace the pixel value of the auxiliary block, while maintaining the original weights. A weighted average of all matching pixels is then used as the merging result for that position, as shown in the following formula:

[0020]

[0021] Here, ω(x,y) represents the weights. This method can also expand pixels into patches, allowing multiple patches to be computed in parallel, which can reduce computation time.

[0022] Finally, Anscombe transform and BM3D denoising are performed on the merged image, and the nonlinear response is inverted to obtain a linear image that is free of motion blur and noise.

[0023] The features and beneficial effects of this invention are:

[0024] This invention proposes a method for de-motion blurring of QIS images based on image alignment and merging. This method primarily employs a patch-based optical flow algorithm and a merging algorithm based on the time-varying incident photon count. This not only significantly improves the reconstruction effect of QIS for moving scenes and effectively achieves de-motion blurring, but also reduces computation time and computational load, making it easy to implement in hardware.

[0025] The method proposed in this invention has several advantages. First, the resulting blocks and images have a higher signal-to-noise ratio than a single binary image, allowing for image alignment using traditional optical flow methods. Second, the patch-based optical flow algorithm, through pyramid layering and linear interpolation, further refines the alignment results of the blocks and images, thereby improving motion blur removal. Finally, a merging algorithm based on the time-varying incident photon count is used. This algorithm compares the variance of the sum of pixels at the same location with a threshold based on the time-varying photon count, and uses different merging methods depending on the comparison results. This approach yields better merging results while reducing computation time and computational load. Attached image description:

[0026] Figure 1 Flowchart for motion blur removal from quantum image sensors based on image alignment and merging.

[0027] Figure 2 Binary image sequence composition blocks and image flowchart.

[0028] Figure 3 Block and image alignment flowchart.

[0029] Figure 4 Flowchart of merging binary sequences.

[0030] Figure 5 A schematic diagram of a merging algorithm based on the change of the number of incident photons over time. Detailed Implementation

[0031] This invention relates to the field of image processing, and particularly to a patch-based optical flow algorithm and a merging method based on the time-varying number of incident photons, thereby achieving motion blur removal from quantum image sensors. Specifically, it relates to a reconstruction method for quantum image sensors.

[0032] When photographing moving objects or scenes and performing image processing, using traditional image reconstruction algorithms will result in severely motion-blurred images. To solve this image quality problem, the binary bitstreams must first be aligned and then merged, thus obtaining a motion-blurred image.

[0033] This invention primarily employs a patch-based optical flow algorithm and a merging algorithm based on the time-varying incident photon count, effectively improving the reconstruction quality of QIS for moving scenes. The technical solution adopted in this invention is as follows: input a binary image sequence → assemble the binary image sequence into blocks and an image → align the blocks and images using a patch-based optical flow algorithm → merge the binary sequences using a merging algorithm based on the time-varying incident photon count → obtain a de-blurred image. Here, multiple binary image sequences combined together are called a "block," the spatial window of pixels is called a "patch," and the sum of the binary sequences within a block is called the "block and image."

[0034] In the steps of assembling blocks and images from a binary image sequence, the input binary image sequence is first divided into non-overlapping time blocks with the same number of frames. The number of frames is determined based on the illumination level and motion changes; the lower the illumination level and the more drastic the motion changes, the more frames are in the time block. Secondly, the block at the center of the sequence is used as a reference block, and the other blocks are used as auxiliary blocks. Finally, the binary image sequences in each block are added together to form the block and the image.

[0035] In the step of aligning blocks and images using the patch-based optical flow algorithm, a hierarchical patch alignment method is first adopted. That is, the number of pyramid layers is determined according to the resolution of the input image. In the nearest neighbor spatial domain of each pyramid layer, the L1 matching error between the auxiliary block and the reference block is calculated. The purpose is to align the auxiliary block with the reference block and generate a coarse optical flow vector. Secondly, global regularization is performed at the optimal level of the pyramid to refine the patch alignment result. Finally, linear interpolation is performed on the coarse optical flow vector to obtain a fine optical flow vector. The fine optical flow vector is then mapped to a binary image sequence to obtain an aligned binary image sequence.

[0036] In the steps of merging binary sequences based on the merging algorithm of incident photon number changing over time, the aligned binary sequences are first divided into different time blocks to determine the reference block and auxiliary block. Secondly, based on the change of photon number within the time block over time, the variance of the sum of pixels at the same position is compared with the threshold. Different merging methods are used according to the comparison results to obtain the merged image. Finally, the merged image is subjected to Anscombe transform and BM3D denoising, and the nonlinear response is inverted to obtain a linear image without motion blur and noise.

[0037] The present invention will be further described in detail below with reference to the accompanying drawings and specific examples.

[0038] To overcome the motion blur problem in QIS image reconstruction during moving scenes, this invention aims to propose a method for removing motion blur from QIS images based on image alignment and merging. The steps are as follows: Input a binary image sequence → assemble the binary image sequence into blocks and images → align the blocks and images using a patch-based optical flow algorithm → merge the binary sequences using a merging algorithm based on the time-varying incident photon count → obtain the de-blurred image. Figure 1 The process of achieving motion blur removal in QIS images based on image alignment and merging is demonstrated.

[0039] Because binary images have extremely low signal-to-noise ratios, the constant brightness assumption in traditional optical flow algorithms does not hold for observed random binary signals. Therefore, traditional optical flow algorithms cannot be used for binary images. To address the high frame rate of QIS, the entire image sequence can first be divided into non-overlapping time blocks with the same number of frames. The number of frames is determined by the illumination level and motion changes: the lower the illumination level and the more drastic the motion changes, the more frames are in the time block. Second, the block at the center of the sequence is used as a reference block, and the other blocks are used as auxiliary blocks. Finally, all binary image sequences within each block are summed to form the block and the image. The specific process is as follows: Figure 2 As shown. Because blocks and images have a higher signal-to-noise ratio than a single binary image, image alignment can be performed using traditional optical flow methods.

[0040] Figure 3 The process of aligning blocks and images using a patch-based optical flow algorithm is demonstrated: (1) Determine the number of pyramid layers based on the resolution of the input binary image; the higher the resolution, the more pyramid layers there are. (2) For a pixel index (i,j), use a patch to expand the index to [iM,(i+1)M-1]×[jM,(j+1)M-1], and find the minimum motion vector (u,v) that minimizes the L1 matching error in the following formula.

[0041]

[0042] Where M×M is the size of the patch used, S aux As an auxiliary block, S ref For reference block. (3) Spatial regularization is achieved at the finest level of the pyramid by minimizing the energy in the following formula.

[0043]

[0044] in It is the spatial domain of pixel indices i,j, E d For matching error, Here, H×W is the regularization term, also known as the smoothing factor, used to represent the smooth variation of optical flow across the entire image, where H×W is the spatial resolution of the input image. In practice, this is achieved by minimizing the Charbonnier loss. As a differentiable substitute for L1 loss, a coarse optical flow vector for inter-block motion can be obtained. (4) Linear interpolation is performed on the coarse optical flow vector to obtain a fine optical flow vector for inter-frame motion, and the fine optical flow vector is mapped to a binary image sequence to achieve alignment of the binary image sequence. Since the amount of motion within each block is very small, the possibility of alignment error is reduced, so this mapping can effectively remove motion blur.

[0045] Figure 4 This paper demonstrates the merging process of binary sequences based on a merging algorithm that considers the change in the number of incident photons over time. Taking into account the characteristic that the number of incident photons does not change along a specific transmission direction of the binary sequence, but the number of incident photons from a moving object changes during the exposure time, the following steps are taken. First, the aligned binary sequence is divided into different time blocks, the size of which can be [missing information - likely a number]. Figure 2 The differences are identified, and reference and auxiliary blocks are determined. Next, the sum of pixels at the same position in the time block is calculated, and the variance of the sum is also calculated. If the variance is less than a set threshold T, the pixel at that position is considered stationary. The static area is averaged as the merging result for that position, which can reduce noise in static areas. The specific formula is as follows:

[0046]

[0047] in, The merged pixel values, G is the size of the time block, and S is the value of the merged pixel value. ref The pixel value is the reference block value. If the variance is greater than the set threshold, it indicates that the alignment step may have failed, and the pixel at that location has significant movement. Locations that differ significantly from the reference block are assigned lower weights, and the pixel value of that location in the reference block is used instead of the pixel value of the auxiliary block, while maintaining the original weights. A weighted average of all matching pixels is taken as the merging result for that location, which can avoid artifacts in dynamic regions. The specific formula is as follows:

[0048]

[0049] Where ω(x,y) represents the weights. This method can also expand pixels into patches, with multiple patches computed in parallel, reducing computation time. Finally, Anscombe transform and BM3D denoising are performed on the merged image, and the nonlinear response is inverted to obtain a linear image free of motion blur and noise. Detailed implementation methods are as follows... Figure 5 As shown.

[0050] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for de-motion blurring of images applied to quantum image sensors, characterized in that, The input binary image sequence is grouped into blocks and images. A patch-based optical flow algorithm aligns the blocks and images, and a merging algorithm based on the time-varying incident photon number merges the binary sequences to obtain a de-blurred image. Multiple binary image sequences combined together are called a "block," a pixel spatial window is called a "patch," and the sum of the binary sequences within a block is called the "block and image." In the step of aligning blocks and images using the patch-based optical flow algorithm, a hierarchical patch alignment method is first adopted. That is, the number of pyramid layers is determined according to the resolution of the input image. In the nearest neighbor spatial domain of each pyramid layer, the minimum mean absolute error (L1 matching error) between the auxiliary block and the reference block is calculated. The purpose is to align the auxiliary block with the reference block and generate a coarse optical flow vector. Secondly, global regularization is performed at the optimal level of the pyramid to refine the patch alignment result. Finally, linear interpolation is performed on the coarse optical flow vector to obtain a fine optical flow vector, and the fine optical flow vector is mapped to a binary image sequence to obtain an aligned binary image sequence. In the steps of merging binary sequences based on the merging algorithm of incident photon number changing over time, the aligned binary sequences are first divided into different time blocks to determine the reference block and auxiliary block. Secondly, based on the change of photon number within the time block over time, the variance of the sum of pixels at the same position is compared with the threshold. Different merging methods are used according to the comparison results to obtain the merged image. Finally, the merged image is subjected to variance-stabilized Anscombe transform and 3D block matching filtering algorithm BM3D for denoising, and the nonlinear response is inverted to obtain a linear image without motion blur and noise.

2. The image de-motion blurring method applied to quantum image sensors as described in claim 1, characterized in that, In the steps of assembling blocks and images from a binary image sequence, the input binary image sequence is first divided into non-overlapping time blocks with the same number of frames. The number of frames is determined based on the illumination level and motion changes; the lower the illumination level and the more drastic the motion changes, the more frames are in the time block. Secondly, the time block at the center of the sequence is used as a reference block, and the other time blocks are used as auxiliary time blocks. Finally, the binary image sequences in each time block are added together to form the time block and the image.

3. The image de-motion blurring method applied to quantum image sensors as described in claim 1, characterized in that, A patch-based optical flow algorithm implements a detailed block and image alignment process: (1) Determine the number of pyramid layers based on the resolution of the input binary image; the higher the resolution, the more pyramid layers there are. (2) For a pixel index Use patch to expand index for Find the minimum motion vector To minimize the L1 matching error: in, To determine the size of the patch to use, For auxiliary blocks, Reference block; (3) Spatial regularization is achieved at the finest level of the pyramid by minimizing the energy in the following equation: in It is a pixel index spatial domain, For matching error, This is the regularization term, also known as the smoothing factor, used to represent the smooth variation of optical flow across the entire image. Given the spatial resolution of the input image, minimize the Charbonnier loss. As a differentiable substitute for L1 loss, the optical flow vector of inter-block motion roughness is obtained; (4) Linear interpolation is performed on the coarse optical flow vector to obtain the fine optical flow vector for inter-frame motion, and the fine optical flow vector is mapped to the binary image sequence to achieve alignment of the binary image sequence.

4. The image de-motion blurring method applied to quantum image sensors as described in claim 1, characterized in that, The detailed process of merging binary sequences based on a merging algorithm that considers the time-varying incident photon number is as follows: First, the aligned binary sequence is divided into different time blocks, and a reference block and an auxiliary block are determined. Second, the sum of pixels at the same position in the time block is calculated, and the variance of the sum is calculated. If the variance is less than a set threshold T, the pixel at that position is considered to be in a static state. The static region is averaged as the merging result for that position. The specific formula is as follows: in, The merged pixel values ​​are represented by G, where G is the size of the time block. The pixel value of the reference block is used. If the variance is greater than the set threshold, it indicates that the alignment step has failed and there is significant movement in the pixel at that position. Positions that differ greatly from the reference block are assigned low weights, and the pixel value of that position in the reference block is used to replace the pixel value of the auxiliary block, while maintaining the original weights. A weighted average of all matching pixels is then used as the merging result for that position, as shown in the following formula: in, Using weights, this method can also expand pixels into patches, with multiple patches being calculated in parallel, which can reduce computation time; Finally, Anscombe transform and BM3D denoising are performed on the merged image, and the nonlinear response is inverted to obtain a linear image that is free of motion blur and noise.

Images