Imaging system and method based on low-light sub-pixel dislocation sampling

Abstract

The invention discloses an imaging system and an imaging method based on low-light sub-pixel dislocation sampling, which are used for solving the technical problem that the image resolution is easy to degrade due to under-sampling during the sampling of the existing image enhanced ICCD or ICMOS camera. The imaging system comprises a sampling unit, a rotary table and an upper computer, wherein a TDI-ICMOS camera in the lighting unit is formed by coupling an image intensifier and a TDI-CMOS detector, n rows of photosensitive channels which are distributed in parallel are arranged in the TDI-CMOS detector, and every two adjacent rows of photosensitive channels are sequentially staggered by # imgabs 0 # pixels in the row direction and are sequentially spaced by # imgabs 1 # pixels in the TDI push-broom direction; the rotary table is used for installing a camera, and the upper computer carries out fusion enhancement on images. According to the TDI-ICMOS camera, n low-light push-broom images with sub-pixel relative displacement can be obtained only by using a push-broom imaging mode under the cooperation of the turntable, so that a foundation is laid for fusing, enhancing and improving the signal-to-noise ratio and MTF (Modulation Transfer Function).

Description

Technical Field

[0001] The invention relates to a low-light-level remote sensing imaging system, and in particular to an imaging system and method based on low-light-level sub-pixel staggered sampling. Background Art

[0002] At present, high-resolution earth observation uses satellites as observation platforms and mainly uses optical imaging systems to obtain large-scale, high-precision and multi-level surface space information. It is a modern strategic high-tech means for a country to master a series of major issues such as resource status and environmental monitoring. It plays a significant role in national economic construction and national defense security, and has become a technical field that countries around the world are vigorously developing and competing fiercely. However, current high-resolution visible light remote sensing imaging at home and abroad is limited by the response sensitivity caused by the photosensitivity mechanism of visible light detectors. High signal-to-noise ratio images can generally only be obtained between 9 am and 4 pm when the ground illumination reaches hundreds to tens of thousands of lux. There is still a lack of effective means for high-resolution imaging under low-light conditions such as dawn and dusk and night when the ground illumination is far below 100 lux.

[0003] Twilight conditions refer to lighting conditions when the ground illuminance is less than 0.01 lux. Under typical twilight conditions such as a full moon and a quarter moon, the radiance entering the optical system is 1E-4 W / m 2 / sr and 1E-5W / m 2 / sr level, in order to perform high-resolution visible optical remote sensing imaging under such weak brightness conditions, the higher the spatial resolution, the fewer photons reaching the detector pixels, and the greater the difficulty in achieving high signal-to-noise ratio imaging. Therefore, to achieve low-light-level high-resolution imaging with high signal-to-noise ratio, firstly, the optical system is required to use a large relative aperture to collect as many sparse photon streams as possible and focus them on the detector surface; secondly, the detector is required to have a sufficiently high sensitivity to ensure effective detection under weak light signals; finally, there must be a sufficiently long integration time to achieve the accumulation of weak signals. However, it should be pointed out that, on the one hand, the satellite platform capability determines the upper limit of the relative aperture of the payload, and on the other hand, the speed-to-height ratio matching required by the low-orbit push-broom working mode also determines the upper limit of the imaging integration time. Therefore, high-sensitivity low-light detectors have become the key to restricting the realization of high-resolution and high-signal-to-noise ratio imaging in low-light remote sensing.

[0004] At present, the CCD or CMOS cameras working in orbit at home and abroad show serious lack of sensitivity during night imaging. Most of them can only realize light imaging, but cannot obtain the details of the dark area outside the light area. Therefore, image-intensified ICCD or ICMOS cameras (cameras formed by coupling image intensifiers with CCD or CMOS) with weak signal front-end amplification capabilities should be used to improve sensitivity, thereby improving the imaging signal-to-noise ratio. Image-intensified cameras can not only greatly improve the sensitivity of weak signal detection through the multiplication function of the image tube, but also will not amplify the dark current noise and readout noise of the CCD or CMOS itself. In addition, they also have functions such as automatic suppression of overload noise, automatic brightness control and automatic protection from strong light, so they can take into account both high sensitivity and large dynamic range, and are particularly suitable for weak light imaging.

[0005] However, even with image-intensified cameras, it is still a challenge to achieve the high signal-to-noise ratio imaging of more than 20 dB required for daytime remote sensing applications when facing the lighting conditions of the full moon and quarter moon, which are several orders of magnitude different from those during the day. Calculations show that to achieve high-resolution visible light low-light remote sensing imaging with detailed inspection-level resolution requirements, the full moon signal-to-noise ratio at an orbital altitude of 500 km is generally only about 10 dB, and the quarter moon signal-to-noise ratio will only be a few dB. Therefore, even if image-intensified cameras are used as the core to achieve low-light remote sensing imaging, how to further improve the sensitivity and imaging quality of low-light imaging is still a key issue that needs to be solved urgently.

[0006] At present, researchers have found that under the condition of limited aperture of space optical cameras, increasing the equivalent integration time is the most effective way to achieve high-sensitivity detection and high signal-to-noise ratio imaging. For low-orbit push-scan imaging, the matching of the speed-to-height ratio determines the row integration time of the detector. Therefore, to increase the equivalent integration time, the use of multiple rows of detectors for synchronous imaging and digital superposition is a way to achieve equivalent long integration. Studies have shown that compared with detectors such as EMCCD and sCMOS that can also perform weak light imaging, the method of multi-frame superposition to increase the equivalent integration time and thus improve the signal-to-noise ratio is particularly suitable for image-intensified ICMOS or ICCD detectors. Therefore, the use of a multi-channel detector with a special structure can achieve the purpose of multi-frame superposition to improve the imaging signal-to-noise ratio. However, whether it is an ICCD or ICMOS camera, the special process control in the coupling process between the image tube and the CCD or CMOS detector is prone to image blur, that is, the degradation of the MTF (modulation transfer function), which causes the imaging system to have undersampling and resolution degradation. Summary of the invention

[0007] The purpose of the present invention is to solve the technical problem that the image resolution of existing image-intensified ICCD or ICMOS cameras is easily degraded due to undersampling during sampling, and to provide an imaging system and method based on low-light sub-pixel staggered sampling.

[0008] To achieve the above objectives, the technical solutions provided by the present invention are:

[0009] An imaging system based on low-light sub-pixel staggered sampling, which is special in that it includes a sampling unit, a turntable and a host computer;

[0010] The light collection unit is a low-light camera, including a TDI-ICMOS camera and a low-light lens arranged at the front end of the TDI-ICMOS camera; the low-light lens is used to collect sparse photon flow under low-light conditions and transmit it to the TDI-ICMOS camera;

[0011] The TDI-ICMOS camera is formed by coupling an image intensifier with a TDI-CMOS detector, and is used to convert the received sparse photon stream into a corresponding image signal; the TDI-CMOS detector is provided with n rows of parallel distributed photosensitive channels, 2≤n≤4; two adjacent rows of photosensitive channels are staggered in the row direction. pixels, spaced in sequence in the TDI push-scan direction pixels, where D represents the physical distance between two adjacent rows of photosensitive channels;

[0012] The turntable is used to install the low-light camera, and the corresponding rotation angular velocity is set according to the line frequency of the TDI-ICMOS camera;

[0013] The host computer is connected to the output end of the TDI-ICMOS camera, and is used for receiving the image signal output by the TDI-ICMOS camera and performing image fusion enhancement, thereby realizing imaging.

[0014] Furthermore, the host computer receives the image signal output by the TDI-ICMOS camera through an image acquisition card.

[0015] Furthermore, the interior of the micro-light lens is a total reflection configuration, a refraction configuration or a catadioptric configuration.

[0016] At the same time, the present invention also provides an imaging method based on low-light-level sub-pixel staggered sampling, which adopts the above-mentioned imaging system based on low-light-level sub-pixel staggered sampling, and includes the following steps:

[0017] [1] A sparse photon stream from a target scene under low-light conditions is collected by a low-light lens; the low-light conditions refer to the brightness conditions at dawn and dusk, full moon or quarter moon;

[0018] [2] A TDI-ICMOS camera is used to acquire n low-light-level images by TDI push-scanning, and the images are transmitted to a host computer; a TDI-CMOS detector of the TDI-ICMOS camera is provided with n rows of parallel photosensitive channels, 2≤n≤4; and there are overlapping portions between any two adjacent images in the n low-light-level images;

[0019] 【3】The received low-light images are fused and post-processed by the host computer to achieve low-light sub-pixel staggered sampling imaging of the target scene.

[0020] Furthermore, step [2] is specifically as follows:

[0021] 【2a】Pre-set the row integration time and integration level of each row of photosensitive channels in the TDI-CMOS detector;

[0022] 【2b】The corresponding line frequency is calculated based on the line integration time, and then the rotational angular velocity ω of the turntable is calculated;

[0023] 【2c】Reset the turntable and set the start position and rotation angle of the turntable according to the scanning range of the target scene;

[0024] [2d] Start the TDI-ICMOS camera and the turntable, and the turntable starts to rotate according to the rotation angular velocity ω calculated in step [2b]. At the same time, the TDI-ICMOS camera acquires n low-light images through TDI push scanning;

[0025] [2e] When the turntable reaches the rotation angle set in step [2c], it stops rotating. At the same time, the TDI-ICMOS camera also stops imaging and transmits the obtained low-light image to the host computer.

[0026] Furthermore, step [3] is specifically as follows:

[0027] 【3a】The first low-light image received by the host computer is defined as the reference image. The remaining images are shifted by (n-1)×(D+N) pixels in sequence in the opposite direction of the push-scanning direction based on the reference image to achieve cropping and alignment of n low-light images, where D represents the physical interval between two adjacent rows of photosensitive channels, and N represents the integral level of each row of photosensitive channels.

[0028] [3b] performing dark level correction processing on the cropped and aligned low-light images respectively; the dark level correction processing refers to deducting the dark current signal corresponding to the row integration time;

[0029] 【3c】Perform noise reduction filtering on the low-light image after dark level correction;

[0030] [3d] Taking the reference image as a reference again, the sub-pixel relative displacements of the remaining low-light images relative to the reference image in the row direction of the photosensitive channel and the push-scan direction are calculated, thereby constructing the sub-pixel relative displacement matrix M;

[0031] [3e] Based on the sub-pixel relative displacement matrix M constructed in step [3d], align the remaining low-light images with the reference image respectively;

[0032] 【3f】Enhance each registered low-light image separately, and then perform noise reduction filtering on each enhanced low-light image;

[0033] [3g] Based on step [3f], preprocess each low-light image separately, and then perform fusion enhancement processing on each low-light image; the preprocessing refers to first performing mirror boundary extension processing and then performing windowing processing;

[0034] 【3h】The low-light image after fusion enhancement processing is subjected to noise reduction filtering, MTF enhancement, and noise reduction filtering again in sequence, thereby achieving low-light sub-pixel staggered sampling imaging of the target scene.

[0035] Furthermore, in step [3g], the specific process of the fusion enhancement processing is:

[0036] 【3g1】The pre-processed low-light images are superimposed and averaged to obtain a mean image, and then the mean image is super-resolution-enlarged to 2 times to obtain the initial estimate X of the fused estimated image;

[0037] 【3g2】Define the initialization iteration number of the outer loop as t, the maximum iteration number as T, and the fusion estimation image of the tth iteration as X t , the cumulative matrix of the tth iteration is G t , where G t The dimension of G is the same as that of the fused estimated image, and let t = 1, G t =0,X t =X;

[0038] 【3g3】Use the backward iterative back-projection algorithm for the inner loop to obtain the cumulative matrix G of the tth iteration t ;

[0039] 【3g4】For the fusion estimation image X t Regularized filtering is performed, and then the product of it and the regularization factor β is used as the regularization term Norm of the tth iteration t ; At the same time, the accumulation matrix G t The product of the gradient iteration factor λ is used as the update term ΔX for this iteration. t ; Then through the regularization term Norm t and the update term ΔXt For the fusion estimation image X t Update to get the updated fusion estimated image X t* , where X t* =X t -ΔX t +Norm t ;

[0040] 【3g5】Judge whether t=T is true. If true, obtain the fused enhanced image X' final Otherwise, let X t =X t* , G t =0, t=t+1, and return to execute step [3g3].

[0041] Furthermore, in step [3c], the noise reduction filter adopts a median filter algorithm;

[0042] In step [3f], the enhancement processing adopts a multi-scale Retinex algorithm or a deep learning low-light image enhancement algorithm; the noise reduction filtering adopts a median filtering algorithm;

[0043] In step [3h], the noise reduction filter adopts the non-local mean NLM denoising algorithm; the MTF enhancement adopts the accelerated vector extrapolation RL algorithm.

[0044] Furthermore, in step [3g1], the super-resolution amplification adopts a single-frame amplification algorithm based on Fourier transform.

[0045] Furthermore, in step [3g3], the inner loop is performed using the backward iterative back-projection algorithm as follows:

[0046] 【3g31】Define the number of initialization iterations as k, the maximum number of iterations as n, and set k = 1;

[0047] 【3g32】Initialize the intermediate matrix temp1,…,temp n-1 ;

[0048] 【3g33】Based on the sub-pixel displacement vector of the kth row in the sub-pixel displacement matrix M obtained in step 【3d】, merge the fused estimated image with the kth image I k Then, the disk filter psf and the downsampling factor factor are used to blur and downsample the fused estimated image after the alignment, and the low-resolution simulated image g corresponding to the kth image in this iteration is obtained. k ;

[0049] 【3g34】Calculate low-resolution simulated image g k With the kth image I kThe difference between them is used to obtain the matrix temp1;

[0050] 【3g35】Through temp2=G t +temp1 and G t =temp2 to G t Make updates;

[0051] 【3g35】Judge whether k=n is true. If true, the cumulative matrix G of this iteration is obtained. t ; Otherwise, let k=k+1 and execute step [3g33].

[0052] Compared with the prior art, the present invention has the following beneficial technical effects:

[0053] 1. The imaging system based on low-light-level sub-pixel staggered sampling proposed in the present invention adopts an image-intensified TDI-ICMOS camera coupled by an image intensifier and a TDI-CMOS detector, wherein n rows of parallel photosensitive channels are arranged inside the TDI-CMOS detector. Therefore, during sampling, by using a push-scan imaging mode with the cooperation of a turntable, n low-light-level images with sub-pixel relative displacement can be obtained, thereby laying a foundation for improving the signal-to-noise ratio and the transfer function MTF by fusion enhancement.

[0054] 2. The imaging method based on low-light-level sub-pixel staggered sampling proposed in the present invention takes the backward iterative back-projection algorithm as the framework and relies on the random characteristics of the noise between multiple sub-pixel staggered sampling images and the reasonable use of the minimum root mean square criterion in the fusion process. It can not only solve the low signal-to-noise ratio problem of low-light-level imaging, but also simultaneously improve the reduction of spatial resolution and transfer function MTF caused by the complex structure and assembly process of the image intensifier and the multiple photoelectric and electro-optical conversions therein, thereby enhancing the camera's ability to obtain higher quality images under low-light conditions.

[0055] 3. In the imaging method based on low-light-level sub-pixel staggered sampling proposed in the present invention, the backward iterative back-projection algorithm is used as the starting point, and the low-light-level image enhancement algorithm is combined with the sub-pixel reconstruction image. The high-frequency information is highlighted through the enhancement algorithm, thereby improving the fusion quality.

[0056] 4. In the imaging method based on low-light sub-pixel staggered sampling proposed in the present invention, the initial estimate of the high-resolution fused image is constructed by median filtering, dark level correction, pixel-level superposition and Fourier transform-based single-frame super-resolution amplification of the multiple registered low-light images, providing a better starting point for subsequent fusion.

[0057] 5. In the imaging method based on low-light sub-pixel staggered sampling proposed in the present invention, the preprocessing operation of anti-mirror boundary and windowing processing is combined, and the high-frequency information iteratively accumulated in the fusion is synchronously suppressed using the gradient as a clue, thereby effectively weakening the ringing effect at the strong contrast boundary.

[0058] 6. In the imaging method based on low-light-level sub-pixel staggered sampling proposed in the present invention, MTF enhancement is performed on the high-resolution low-light-level image after fusion enhancement, thereby eliminating the MTF degradation introduced by the TDI-ICMOS detector development process and further improving the low-light-level image quality.

[0059] 7. The imaging method based on low-light sub-pixel staggered sampling proposed in the present invention improves the signal-to-noise ratio of low-light imaging through three steps as a whole: first, dark level correction and median filtering realize the first signal-to-noise ratio improvement of the original low-light image; secondly, the superposition of sub-pixel staggered sampling low-light images after cropping, alignment and registration brings about the second improvement of the signal-to-noise ratio; finally, high-resolution fusion enhancement of multiple sub-pixel staggered sampling low-light images brings about the third improvement of the signal-to-noise ratio through non-local mean denoising. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 It is a schematic diagram of the overall structure of an embodiment of an imaging system based on low-light-level sub-pixel staggered sampling according to the present invention;

[0061] Figure 2a It is a structural schematic diagram of an image intensifier in an embodiment of an imaging system based on low-light-level sub-pixel staggered sampling of the present invention;

[0062] Figure 2b It is a schematic diagram of the structure of each photosensitive channel inside the TDI-CMOS detector in an embodiment of the imaging system based on low-light sub-pixel staggered sampling of the present invention.

[0063] Figure 3 It is a position relationship diagram of the low-light-level images output by each row of photosensitive channels during push-scan imaging by the TDI-CMOS detector in an imaging system embodiment based on low-light-level sub-pixel staggered sampling of the present invention.

[0064] Figure 4 It is a schematic diagram of the flow of fusion enhancement and post-processing in an embodiment of an imaging method based on low-light-level sub-pixel staggered sampling of the present invention;

[0065] Figure 5a It is the original low-light image without fusion enhancement.

[0066] Figure 5b It is a low-light-level image after fusion and enhancement based on the imaging method of the low-light-level sub-pixel staggered sampling of the present invention.

[0067] The reference numerals are as follows:

[0068] 1-turntable, 2-host computer, 3-low-light camera, 4-photocathode, 5-microchannel plate, 6-fluorescent screen, 7-tube shell. DETAILED DESCRIPTION

[0069] In order to make the purpose, advantages and features of the present invention clearer, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the scope of protection of the present invention.

[0070] At present, there are few reports on combining sub-pixel staggered sampling imaging technology with ICCD or ICMOS cameras for low-light imaging, and it is even more challenging to achieve high-resolution and high-signal-to-noise ratio low-light remote sensing imaging at the level of detailed inspection. In fact, while using multi-channel structure detectors to achieve multi-frame superposition to improve the signal-to-noise ratio, by introducing sub-pixel relative displacement between multiple rows of detectors, sub-pixel technology can be used to fuse and enhance the push-broom images output by multiple rows of detectors with sub-pixel relative displacement relationships to eliminate undersampling, thereby improving the system MTF. However, sub-pixel technology requires a special detector structure to ensure it. Although sub-pixel technology itself is highly mature, it is not born for night remote sensing. Therefore, the use of this technology still cannot achieve high-resolution and high-signal-to-noise ratio imaging at the level of detailed inspection under extremely dim lighting conditions such as full moon and 1 / 4 moon.

[0071] Aiming at the demand for high-quality optical image acquisition in nighttime low-light level remote sensing, the present invention proposes an imaging system based on low-light level sub-pixel staggered sampling and a supporting imaging method, aiming to solve the problem of high signal-to-noise ratio and high MTF imaging under extremely dim lighting conditions such as a full moon and a quarter moon.

[0072] Reference Figure 1 This embodiment provides an imaging system based on low-light-level sub-pixel staggered sampling, including a sampling unit, a turntable 1 and a host computer 2.

[0073] The light collection unit is a low-light camera 3, including a TDI-ICMOS camera and a low-light lens arranged at the front end of the TDI-ICMOS camera; the low-light lens is used to collect sparse photon flows under low-light conditions and transmit them to the TDI-ICMOS camera. The low-light lens can be configured as a total reflection type, a refraction type, or a catadioptric type. The configuration of the low-light lens used is related to the potential application field of the technology. For example, in the field of low-light remote sensing, in order to achieve high-resolution low-light remote sensing while taking into account reasonable optical and mechanical weight and volume constraints, the main configuration is a total reflection type or a catadioptric type. However, no matter which configuration is used, in order to improve the collection efficiency of sparse photon flows under low-light conditions, the low-light lens should adopt a relatively large relative aperture to improve the photon collection capability.

[0074] The TDI-ICMOS camera is composed of an image intensifier coupled with a TDI-CMOS detector, which is used to convert the received sparse photon stream into the corresponding image signal. Among them, the TDI-CMOS detector is equipped with n parallel distributed photosensitive channels, 2≤n≤4; two adjacent rows of photosensitive channels are staggered in the row direction. pixels, spaced in sequence in the TDI push-scan direction pixels, where D represents the physical distance between two adjacent rows of photosensitive channels.

[0075] The TDI-ICMOS detector is the core of low-light imaging at night, and the fiber panel is used as a relay optical element to couple the image intensifier with the TDI-CMOS detector. As an electric vacuum device, the working principle of the TDI-ICMOS camera is as follows: the photocathode of the image intensifier receives the target photon and emits photoelectrons, which are closely focused on the microchannel plate. The microchannel plate (MCP component) multiplies the electrons, and after the electrons are multiplied, they are accelerated by the high voltage of the fluorescent screen and bombard the fluorescent screen to form an enhanced optical image. The fiber panel transmits the enhanced image to the TDI-CMOS chip, and the TDI-CMOS device converts the optical signal into an electrical signal. Since the microchannel plate can provide up to 10 6 Therefore, the TDI-ICMOS camera is particularly suitable for low-light detection and imaging.

[0076] Turntable 1 is used to install the low-light camera 3 and set the corresponding rotation angular velocity according to the line frequency of the TDI-ICMOS camera. Turntable 1 has a high-precision interface for installing the TDI-ICMOS camera. First, the push-scan line frequency is calculated according to the resolution and focal length of the TDI-ICMOS camera, and the optimal rotation angular velocity of turntable 1 is calculated to ensure the charge transfer matching during the push-scan imaging process. During the push-scan imaging process, the low-light lens first collects the sparse photon flow and focuses it on the cathode of the TDI-ICMOS detector, and then the back-end circuit reads out the image, and finally converts it into a digital image and sends it to the host computer 2.

[0077] The host computer 2 is connected to the output end of the TDI-ICMOS camera. The host computer 2 receives the image signal output by the TDI-ICMOS camera through the image acquisition card, and implements fusion enhancement to achieve imaging.

[0078] Figure 2a An exploded diagram of the structure of the image intensifier is given. It can be seen from the diagram that the photocathode 4, the microchannel plate 5 and the fluorescent screen 6 of the image intensifier are wrapped and nested by the tube shell 7 to form a complete device. Figure 2b The schematic diagram of the TDI-CMOS detector structure coupled with the image intensifier is given. Figure 2bIt can be seen that, unlike conventional TDI-CMOS detectors, the TDI-CMOS detector used in the present invention has 4 rows of parallel photosensitive channels, each row of photosensitive channels is sequentially displaced by 1 / 4 pixel in the detector row direction, and is also sequentially displaced by D+1 / 4 pixel in the TDI push-scan direction. Wherein, D represents the physical interval between two adjacent rows of detectors, which is the number of pixels converted according to the pixel size. Therefore, Figure 2b The 4-channel TDI-CMOS detector shown can synchronously obtain 4 push-scan images with sub-pixel staggered sampling because each row of photosensitive channels has a fixed sub-pixel phase relationship, thereby providing hardware guarantee for sub-pixel staggered sampling fusion enhancement.

[0079] In the push-scan imaging process, with the first row of photosensitive channels as the reference, the physical spacing between the photosensitive channels makes the four images have a fixed D+N pixel offset in sequence along the push-scan direction (where N represents the integral order of the photosensitive channel), as shown in Figure 3 As shown in the figure, the effect of stitching the images obtained by the four channels into one image after the push-scan imaging of the four-channel detector is given. It is obvious that the position of the same target in different channels changes according to the fixed number of pixels. Therefore, before the subsequent fusion enhancement is implemented, the second image, the third image and the fourth image are shifted by 1×(D+N) pixels, 2×(D+N) pixels and 3×(D+N) pixels in the reverse push-scan direction with the first row of detector images as reference, so that the four images are aligned to the theoretical sub-pixel through cropping.

[0080] On this basis, this embodiment also provides an imaging method based on low-light sub-pixel staggered sampling. This method is based on the framework of the classic iterative back projection algorithm IBP (Iterative Back Projection). It is essentially a constrained optimization problem. Based on multiple frames of low-resolution measured images, the simulated low-resolution image generated by the forward imaging model and the measured low-resolution image are used to achieve the minimum root mean square error to find an optimal high-resolution fused image. Among them, according to the characteristics of low-light imaging, improvements are made to the preprocessing process, the initial estimation of the high-resolution fused image, etc., thus producing the low-light sub-pixel staggered sampling imaging method proposed by the present invention.

[0081] The following takes a 4-channel (ie, 4 photosensitive channels) TDI-CMOS detector as an example to explain the specific steps of the method in detail.

[0082] [1] The sparse photon stream from the target scene under low light conditions is collected by a low light lens. The low light conditions in this embodiment refer to the brightness conditions at dusk, full moon or quarter moon.

[0083] [2] A TDI-ICMOS camera is used to acquire four low-light images by TDI push scanning, and there is an overlap between two adjacent images in the four low-light images. Then, the TDI-ICMOS camera transmits the four low-light images to the host computer 2. The specific steps are as follows:

[0084] 【2a】Pre-set the row integration time Integration T and the integration level N of each row of the photosensitive channel in the TDI-CMOS detector.

[0085] 【2b】Calculate the corresponding row frequency v according to the row integration time IntegrationT, Then, according to the pixel size p of the detector and the focal length f of the low-light lens, the rotation angular velocity ω of the turntable 1 is calculated.

[0086] [2c] Reset the turntable 1 and set the start position and rotation angle θ of the turntable according to the scanning range of the target scene;

[0087] [2d] Start the TDI-ICMOS camera and turntable 1 through the ground inspection equipment, and turntable 1 starts to rotate according to the angular velocity ω calculated in step [2b]. At the same time, the TDI-ICMOS camera starts push-scan imaging through TDI push-scanning to obtain 4 low-light images.

[0088] [2e] When the turntable 1 reaches the rotation angle θ set in step [2c], it stops rotating, and the TDI-ICMOS camera also stops imaging, and transmits the obtained 4 low-light images to the host computer 2.

[0089] 【3】The received low-light images are fused and post-processed by the host computer 2, thereby realizing low-light sub-pixel staggered sampling imaging of the target scene. Figure 4 The specific process of fusion or post-processing in this embodiment is as follows:

[0090] [3a] The first low-light image among the four low-light images received by the host computer 2 is defined as the reference image, and based on the reference image, the second low-light image is shifted by 1×(D+N) pixels in the opposite direction of the push-sweep direction, the third low-light image is shifted by 2×(D+N) pixels in the opposite direction of the push-sweep direction, and the fourth low-light image is shifted by 3×(D+N) pixels in the opposite direction of the push-sweep direction to achieve cropping and alignment of the four low-light images, where D represents the physical interval between two adjacent rows of photosensitive channels, and N represents the integration level of each row of photosensitive channels.

[0091] 【3b】The host computer 2 performs dark level correction on the four cropped and aligned low-light images respectively.

[0092] The dark level correction process described in this embodiment refers to the deduction of the dark current signal corresponding to the row integration time. Since low-light images are more susceptible to the unevenness of dark current response, it is necessary to perform dark level correction on the obtained low-light images. Among them, dark level correction can be achieved by two methods. If the detector has an overscan area, and the dark current accumulation time in the overscan area is the same as the push-scan imaging time, then the dark pixels corresponding to the overscan area in each column of the push-scan image can be extracted and averaged as the dark level grayscale and deducted from each column of the image to complete the dark level correction. If the detector does not have an overscan area, firstly, a long-term push-scan imaging is performed without exposure, and multiple dark field images are obtained and averaged to obtain a standard dark field image. Then, the standard dark field image is converted according to the actual push-scan imaging time to obtain a dark level correction image corresponding to the actual push-scan imaging. Finally, the dark level correction image is deducted from the push-scan image to complete the dark level correction.

[0093] 【3c】The median filter algorithm is used to perform noise reduction filtering on the low-light-level image after dark level correction. Since the TDI-ICMOS detector needs to work at a higher electron multiplication gain, and the thermal noise caused by the dark emission of the cathode of the image tube will also be amplified together with the signal photons, the original low-light-level image needs to be filtered through a median filter to eliminate the influence of local abnormal thermal noise.

[0094] [3d] Taking the reference image as the benchmark again, the sub-pixel relative displacement of the remaining low-light images in the row direction of the photosensitive channel and the push-scan direction relative to the reference image is calculated, where the sub-pixel relative displacement is obtained using the upsampling phase correlation algorithm, thereby constructing a sub-pixel relative displacement matrix M with a matrix dimension of 4×2, where the displacement includes both x and y directions, so the dimension is 2, and the matrix row is the number of images. The matrix dimension of the four low-light images is 4×2.

[0095] For sub-pixel imaging technology, to achieve the fusion of multiple images, the sub-pixel relative displacement between each image must first be obtained. In the present invention, since the four rows of parallel detectors of the TDI-ICMOS detector have a known relative phase relationship in the detector row direction and the push-scan direction, the sub-pixel relative displacement matrix between each image is theoretically known. However, in practice, considering that there may be external disturbances during the push-scan imaging process, when the same scene target is push-scanned by different rows of detectors at different times, the phase relationship between adjacent images may deviate from the theoretical value. Therefore, for the sake of safety, the upsampling phase correlation algorithm is used to obtain high-precision sub-pixel image displacement to create conditions for fusion.

[0096] [3e] Based on the sub-pixel relative displacement matrix M constructed in step [3d], the remaining low-light images are registered with the reference image.

[0097] 【3f】Each registered low-light image is enhanced, and then each enhanced low-light image is subjected to noise reduction filtering. The enhancement process uses a multi-scale Retinex algorithm or a deep learning low-light image enhancement algorithm; the noise reduction filtering process uses a median filtering algorithm to eliminate the influence of local abnormal thermal noise.

[0098] [3g] Based on step [3f], each low-light image is preprocessed separately, and then each low-light image is fused and enhanced. The preprocessing here refers to first performing a mirror boundary extension process, then performing a windowing process, and then constructing an image data set I containing the four preprocessed low-light images.

[0099] Taking into account the characteristics of low-light images, this embodiment creatively combines the low-light image enhancement algorithm with the sub-pixel low-light image fusion in the fusion enhancement part of the proposed algorithm, and uses the image enhancement algorithm to solve the problem of difficulty in information extraction caused by low illumination and low contrast of low-light images, which can provide a guarantee for the accurate extraction of high-frequency information in the fusion algorithm. In the present invention, after the original low-light image is cropped, aligned, dark level corrected, and median filtered, a multi-scale Retinex algorithm can be used to achieve adaptive low-light image enhancement, or a deep learning low-light image enhancement algorithm can be used to obtain an enhanced low-light image.

[0100] In this embodiment, the specific process of fusion enhancement processing is as follows:

[0101] [3g1] The four pre-processed low-light images are superimposed and averaged to obtain a mean image, and then the mean image is super-resolution magnified to 2 times to obtain the initial estimate X of the fused estimation image. The super-resolution magnification described here adopts a single-frame magnification algorithm based on Fourier transform.

[0102] The core mechanism of high-resolution image fusion is to take the initial estimation of the high-resolution fused image as the starting point, and the minimum root mean square error between the simulated degraded low-resolution image and the low-resolution measured image as the objective function, and obtain the desired high-resolution fused image through optimization and iterative calculation.

[0103] Sub-pixel misaligned sampling image fusion based on iterative back-projection framework first needs to generate an initial estimate of a high-resolution fused image. Usually, any image in the data set is selected and enlarged by 2 times as the initial estimate of the high-resolution fused image. Here, considering the characteristics of low-light imaging, a new method for generating an initial estimate of a high-resolution fused image is proposed. First, the four sub-pixel misaligned sampling low-light images that have been trimmed, aligned, dark-level corrected, and median filtered are superimposed and averaged to obtain a mean image, and then a single-frame enlargement algorithm based on Fourier transform interpolation is used to obtain an initial estimate of the high-resolution fused image. The average of the four images can reduce the noise variance, and the single-frame enlargement algorithm based on Fourier transform interpolation can itself achieve a certain ratio of resolution improvement. Therefore, the initial estimate of the high-resolution fused image obtained by combining the two methods provides a good signal-to-noise ratio and resolution starting point for subsequent fusion enhancement.

[0104] 【3g2】Define the initialization iteration number of the outer loop as t, the maximum iteration number as T, and the fusion estimation image of the tth iteration as X t , the cumulative matrix of the tth iteration is G t , where G t The dimension of the fused estimated image X t The dimensions are the same, and let t = 1, G t =0,X t =X.

[0105] 【3g3】The host computer 2 uses the backward iterative back-projection algorithm to perform the inner loop and obtain the cumulative matrix G of the tth iteration t , specifically:

[0106] 【3g31】Define the number of initialization iterations as k, the maximum number of iterations as 4, and set k = 1;

[0107] 【3g32】Initialize the intermediate matrices temp1, temp2, temp3;

[0108] 【3g33】Based on the sub-pixel displacement vector of the kth row in the sub-pixel displacement matrix M obtained in step 【3d】, the host computer 2 combines the fused estimated image with the kth image I k Then, the disk filter psf and the downsampling factor factor are used to blur and downsample the fused estimated image after the alignment, and the low-resolution simulated image g corresponding to the kth image in this iteration is obtained. k ;

[0109] 【3g34】 Host computer 2 calculates simulated image g k With the kth image I k The difference between temp1=g k -I k, thus obtaining the matrix temp1; then, after amplifying temp1 by 2 times, filtering it with the edge enhancement high frequency enhancement filter H, aligning the filtering result with X according to the sub-pixel displacement vector of the kth row in the sub-pixel displacement matrix M, obtaining the matrix temp2;

[0110] 【3g35】Through temp3=G t +temp2 and G t =temp3 vs. G t Make updates;

[0111] 【3g35】Judge whether k=4 is true. If so, the cumulative matrix G of this iteration is obtained. t ; Otherwise, let k = k + 1 and execute step [3g33] until the cumulative matrix G of this iteration is obtained t , thus ending the inner loop.

[0112] 【3g4】Estimated image X through fusion of 2 pairs of host computers t Regularized filtering is performed, and then the product of it and the regularization factor β is used as the regularization term Norm of the tth iteration t ; At the same time, the accumulation matrix G t The product of the gradient iteration factor λ is used as the update term ΔX for this iteration. t ; Then through the regularization term Norm t and the update term ΔX t For the fusion estimation image X t Update to get the updated fusion estimated image X t* , where X t* =X t -ΔX t +Norm t .

[0113] 【3g5】The host computer 2 determines whether t=T is established. If so, the fused enhanced image X' is obtained. final Otherwise, let X t =X t* , G t =0, t=t+1, and execute step [3g3].

[0114] The simulated imaging process of generating a low-resolution degraded image from the initial estimate of the high-resolution fused image mainly includes three steps: alignment, blurring and downsampling. The current estimate of the high-resolution fused image is first aligned with the current low-resolution observed image after sub-pixel translation according to the sub-pixel displacement matrix; then the disk low-pass filter is used as the system degradation function to simulate the blurring effect of the optical lens. The radius parameter of the disk low-pass filter should be equal to the size of the optical diffuse spot determined by the relative aperture of the low-light lens used, thereby ensuring compliance with the physical laws of imaging; then the blurred and degraded high-resolution fused image is downsampled by 2 times to obtain a low-resolution simulated image.

[0115] After obtaining the simulated low-resolution image, the difference between the low-resolution simulated image and the low-resolution measured image is reversely mapped to the high-resolution space through upsampling. This process mainly includes two steps: upsampling and high-frequency enhancement. First, the difference between the low-resolution simulated image and the low-resolution measured image is upsampled and amplified by 2 times, and then filtered using a Gaussian-Laplacian filter as an edge enhancement high-frequency enhancement filter and accumulated into an intermediate matrix until all low-resolution measured images have been traversed, thereby obtaining the desired update matrix required to update the high-resolution fused image.

[0116] After obtaining the update matrix required to update the high-resolution fusion image, first filter the current estimate of the high-resolution fusion image through a negative Gaussian-Laplacian filter and multiply it with the regularization factor β to generate a regularization term that suppresses noise amplification. Then multiply the update matrix required to update the high-resolution fusion image with the gradient iteration factor λ as the new update matrix. Finally, calculate the current high-resolution fusion image minus the update matrix plus the regularization term to obtain the high-resolution fusion image obtained by the current iteration. This process continues until the number of iterations reaches the upper limit, at which time the required high-resolution fusion image is obtained.

[0117] It should be noted that in the high-resolution fusion process, due to the forward and reverse Fourier transforms involved, ringing may occur at high-contrast edges. Therefore, on the one hand, the more traditional anti-mirror boundary method and windowing method are used to process the low-light image before fusion, and on the other hand, an innovative method is used in the fusion process to further weaken and eliminate the ringing effect. Ringing often occurs in edge areas with strong contrast, so the high-frequency edge gradient can be used as a clue to perform local low-pass filtering on the update matrix containing high-frequency information to suppress the ringing, thereby reducing its impact on the fused image and improving the effect of the fused image.

[0118] 【3h】 Host computer 2 pairs of fusion enhanced images X' finalDe-noising filtering, MTF enhancement, and de-noising filtering are performed in sequence to achieve low-light sub-pixel staggered sampling imaging of the target scene and obtain a high-resolution fusion enhanced image X final .

[0119] When executing this step, both denoising filters are processed using the fast non-local mean NLM denoising algorithm, and the MTF enhancement uses the improved accelerated vector extrapolation RL algorithm.

[0120] After the fusion enhancement is completed, post-processing is required to produce the final low-light image with improved signal-to-noise ratio and MTF. On the one hand, there must be residual noise amplification in the fusion enhancement process, so noise reduction filtering should be performed again after obtaining the fusion enhanced image; on the other hand, due to the regularization processing and low-pass filtering of the high-frequency information update matrix during the fusion enhancement process, the edges of the fused image will be blurred and degraded, so the MTF compensation technology should be used to improve the contrast of the fusion enhanced image and obtain better subjective visual effects. Here, the noise reduction filter uses the non-local mean NLM algorithm, and the MTF compensation uses the improved acceleration vector extrapolation RL algorithm. Both algorithms are mature and reliable and have been tested to have good applicability to the fusion enhancement algorithm for sub-pixel dislocation.

[0121] In summary, the low-light-level sub-pixel staggered sampling imaging system and the supporting imaging method proposed in the present invention are introduced, and finally an outdoor imaging test is used to prove the applicability of the low-light-level sub-pixel staggered sampling imaging system and the method proposed in the present invention under low-light conditions. Among them, the lens focal length of the adopted TDI-ICMOS low-light camera 3 is 144mm, the relative aperture is 1:9, the pixel size is 14um, the imaging distance is about 3km, the imaging time is about 1 am on the eighth day of the first lunar month, and the imaging target is the village at the foot of the mountain.

[0122] refer to Figure 5a and Figure 5b , gives the results of the field imaging test, where Figure 5a This is an image of a channel, which has not been processed. It can be seen that the low contrast and low brightness make it difficult to distinguish the layers and vegetation information of the distant mountains. Figure 5b The image is obtained by fusion and enhancement of the 4-channel low-light sub-pixel staggered sampling image using the imaging method proposed in the present invention. Figure 5b ,The image quality improvement effect of this figure is very obvious, which once again proves the effectiveness of the imaging method.

[0123] Compared with the prior art, the greatest innovation of the present invention is that it combines the sub-pixel staggered sampling technology with the low-light imaging technology based on the image-intensified TDI-ICMOS low-light detector. It takes the multi-channel TDI-ICMOS detector as the starting point and utilizes the sub-pixel staggered sampling relationship between each row of detectors in the TDI push-scan direction and the detector row direction. Through the fusion enhancement of multi-channel sub-pixel staggered sampling low-light images, the dual purposes of high signal-to-noise ratio detection of weak light and high-resolution imaging can be achieved.

[0124] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the present invention.

Claims

1. An imaging system based on low-light sub-pixel staggered sampling, characterized by: It includes a sampling unit, a turntable (1) and a host computer (2); The light collection unit is a low-light camera (3), comprising a TDI-ICMOS camera and a low-light lens arranged at the front end of the TDI-ICMOS camera; the low-light lens is used to collect sparse photon flow under low-light conditions and transmit it to the TDI-ICMOS camera; The TDI-ICMOS camera is formed by coupling an image intensifier with a TDI-CMOS detector, and is used to convert the received sparse photon stream into a corresponding image signal; the TDI-CMOS detector is provided with n rows of parallel distributed photosensitive channels, 2≤n≤4; two adjacent rows of photosensitive channels are staggered in the row direction. pixels, spaced in sequence in the TDI push-scan direction pixels, where D represents the physical distance between two adjacent rows of photosensitive channels; The turntable (1) is used to install the low-light camera (3), and to set a corresponding rotation angular velocity according to the line frequency of the TDI-ICMOS camera; The host computer (2) is connected to the output end of the TDI-ICMOS camera and is used for receiving the image signal output by the TDI-ICMOS camera and performing image fusion enhancement, thereby realizing imaging.

2. The imaging system based on low-light-level sub-pixel staggered sampling according to claim 1, characterized in that: The host computer (2) receives the image signal output by the TDI-ICMOS camera through an image acquisition card.

3. The imaging system based on low-light-level sub-pixel staggered sampling according to claim 1 or 2, characterized in that: The interior of the micro-light lens is a total reflection configuration, a refraction configuration or a catadioptric configuration.

4. An imaging method based on low-light sub-pixel staggered sampling, characterized in that: The imaging system based on low-light-level sub-pixel staggered sampling according to any one of claims 1 to 3 comprises the following steps: [1] A sparse photon stream from a target scene under low-light conditions is collected by a low-light lens; the low-light conditions refer to the brightness conditions at dawn and dusk, full moon or quarter moon; [2] Using a TDI-ICMOS camera to acquire n low-light-level images by TDI push-scanning, and transmitting the images to a host computer (2); a TDI-CMOS detector of the TDI-ICMOS camera is provided with n rows of parallel photosensitive channels, 2≤n≤4; and there is an overlap between any two adjacent images in the n low-light-level images; 【3】The received low-light-level images are fused and post-processed by the host computer (2), thereby realizing low-light-level sub-pixel staggered sampling imaging of the target scene.

5. The imaging method based on low-light-level sub-pixel staggered sampling according to claim 4, characterized in that: Step [2] is as follows: 【2a】Pre-set the row integration time and integration level of each row of photosensitive channels in the TDI-CMOS detector; 【2b】The corresponding line frequency is calculated based on the line integration time, and then the rotational angular velocity ω of the turntable (1) is calculated; [2c] Reset the turntable (1), and set the starting position and rotation angle of the turntable (1) according to the scanning range of the target scene; [2d] Start the TDI-ICMOS camera and the turntable (1), and the turntable (1) starts to rotate according to the rotation angular velocity ω calculated in step [2b]. At the same time, the TDI-ICMOS camera acquires n low-light images by TDI push scanning; [2e] When the turntable (1) reaches the rotation angle set in step [2c], it stops rotating, and the TDI-ICMOS camera also stops imaging and transmits the obtained low-light image to the host computer (2).

6. The imaging method based on low-light-level sub-pixel staggered sampling according to claim 4 or 5, characterized in that: Step [3] is as follows: 【3a】The first low-light image received by the host computer (2) is defined as the reference image. The remaining images are shifted in sequence by (n-1)×(D+N) pixels in the opposite direction of the push-scanning direction based on the reference image to achieve cropping and alignment of n low-light images, where D represents the physical interval between two adjacent rows of photosensitive channels, and N represents the integration level of each row of photosensitive channels; [3b] performing dark level correction processing on the cropped and aligned low-light images respectively; the dark level correction processing refers to deducting the dark current signal corresponding to the row integration time; 【3c】Perform noise reduction filtering on the low-light image after dark level correction; [3d] Taking the reference image as a reference again, the sub-pixel relative displacements of the remaining low-light images relative to the reference image in the row direction of the photosensitive channel and the push-scan direction are calculated, thereby constructing a sub-pixel relative displacement matrix M; [3e] Based on the sub-pixel relative displacement matrix M constructed in step [3d], align the remaining low-light images with the reference image respectively; 【3f】Enhance each registered low-light image separately, and then perform noise reduction filtering on each enhanced low-light image; [3g] Based on step [3f], preprocess each low-light image separately, and then perform fusion enhancement processing on each low-light image; the preprocessing refers to first performing mirror boundary extension processing and then performing windowing processing; 【3h】The low-light image after fusion enhancement processing is subjected to noise reduction filtering, MTF enhancement, and noise reduction filtering again in sequence, thereby realizing low-light sub-pixel staggered sampling imaging of the target scene.

7. The imaging method based on low-light-level sub-pixel staggered sampling according to claim 6, characterized in that: In step [3g], the specific process of the fusion enhancement processing is as follows: 【3g1】The pre-processed low-light images are superimposed and averaged to obtain a mean image, and then the mean image is super-resolution-enlarged to 2 times to obtain the initial estimate X of the fused estimated image; 【3g2】Define the initialization iteration number of the outer loop as t, the maximum iteration number as T, and the fusion estimation image of the tth iteration as X t , the cumulative matrix of the tth iteration is G t , where G t The dimension of G is the same as that of the fused estimated image, and let t = 1, G t =0,X t =X; 【3g3】Use the backward iterative back-projection algorithm for the inner loop to obtain the cumulative matrix G of the tth iteration t ; 【3g4】For the fusion estimation image X t Regularized filtering is performed, and then the product of it and the regularization factor β is used as the regularization term Norm of the tth iteration t ; At the same time, the accumulation matrix G t The product of the gradient iteration factor λ is used as the update term ΔX for this iteration. t ; Then through the regularization term Norm t and the update term ΔX t For the fusion estimation image X t Update to get the updated fusion estimated image X t* , where X t* =X t -ΔX t +Norm t ; 【3g5】Judge whether t=T is true. If true, obtain the fused enhanced image X' final Otherwise, let X t =X t* , G t =0, t=t+1, return to step [3g3].

8. The imaging method based on low-light-level sub-pixel staggered sampling according to claim 7, characterized in that: Step [3c], the noise reduction filter adopts a median filter algorithm; In step [3f], the enhancement processing adopts a multi-scale Retinex algorithm or a deep learning low-light image enhancement algorithm; the noise reduction filtering adopts a median filtering algorithm; In step [3h], the noise reduction filter adopts the non-local mean NLM denoising algorithm; the MTF enhancement adopts the accelerated vector extrapolation RL algorithm.

9. The imaging method based on low-light-level sub-pixel staggered sampling according to claim 8, characterized in that: In step [3g1], the super-resolution amplification adopts a single-frame amplification algorithm based on Fourier transform.

10. The imaging method based on low-light-level sub-pixel staggered sampling according to claim 7, characterized in that: In step [3g3], the inner loop is performed using the backward iterative back-projection algorithm as follows: 【3g31】Define the number of initialization iterations as k, the maximum number of iterations as n, and set k = 1; 【3g32】Initialize the intermediate matrix temp1,…,temp n-1 ; 【3g33】Based on the sub-pixel displacement vector of the kth row in the sub-pixel displacement matrix M obtained in step 【3d】, merge the fused estimated image with the kth image I k Then, the disk filter psf and the downsampling factor factor are used to blur and downsample the fused estimated image after the alignment, and the low-resolution simulated image g corresponding to the kth image in this iteration is obtained. k ; 【3g34】Calculate low-resolution simulated image g k With the kth image I k The difference between them is used to obtain the matrix temp1; 【3g35】Through temp2=G t +temp1 and G t =temp2 to G t Make updates; 【3g36】Judge whether k=n is true. If true, the cumulative matrix G of this iteration is obtained. t ; Otherwise, set k=k+1 and return to step [3g33].

Images