Imaging system and method based on micro-light sub-pixel misregistration sampling

Abstract

The application discloses an imaging system and method based on micro-light sub-pixel misregistration sampling, and is used for solving the technical problem that the existing image intensifier ICCD or ICMOS camera is prone to image resolution degradation due to under-sampling when sampling. The imaging system comprises a sampling unit, a turntable and an upper computer; wherein the TDI-ICMOS camera in the light sampling unit is coupled by an image intensifier and a TDI-CMOS detector, the TDI-CMOS detector is internally provided with n rows of parallel distributed light sensing channels, and adjacent two rows of light sensing channels are misregistered by one pixel in the row direction and are spaced by one pixel in the TDI push scanning direction; the turntable is used for installing the camera, and the upper computer is used for image fusion enhancement. The TDI-ICMOS camera only needs to use the push scanning imaging mode under the cooperation of the turntable, so that n micro-light push scanning images with sub-pixel relative displacement can be obtained, thereby laying a foundation for improving the signal-to-noise ratio and MTF of fusion enhancement.

Description

TECHNICAL FIELD

[0001] The present application relates to a micro-light remote sensing imaging system, in particular to an imaging system and method based on micro-light sub-pixel misregistration sampling. BACKGROUND

[0002] At present, high-resolution earth observation takes satellite as observation platform, and mainly uses optical imaging system to realize acquisition of large range, high precision and multi-level earth surface space information, which is a modern strategic high-tech means for a country to master resource situation, implement environmental monitoring and a series of major issues, and has a great role in national economic construction and national defense security, and has become a technical field that is vigorously developed and fiercely competed by countries in the world. However, at present, high-resolution visible light remote sensing imaging at home and abroad is limited by the response sensitivity caused by the photosensitive mechanism of the visible light detector, and high signal-to-noise ratio images can generally be obtained only when the ground illuminance reaches several hundred to several ten thousand lux from 9 o'clock in the morning to 4 o'clock in the afternoon, and there is still a lack of effective means for high-resolution imaging under micro-light conditions such as dawn and dusk, night and the like, whose ground illuminance is far lower than 100 lux.

[0003] Micro-light condition refers to the light condition when the ground illuminance is less than 0.01 lux, wherein the radiance entering the optical system under the typical micro-light condition of full moon and 1 / 4 moon is 1E-4 W / m 2 / sr and 1E-5 W / m 2 / sr order of magnitude, and the higher the spatial resolution is, the fewer the photons reaching the detector pixel are, and the more difficult the realization of high signal-to-noise ratio imaging is. Therefore, to realize micro-light high signal-to-noise ratio high-resolution imaging, firstly, the optical system is required to adopt a large relative aperture to collect as many sparse photon streams as possible and focus them on the detector surface; secondly, the detector is required to have high enough sensitivity to ensure effective detection under weak light signal; and finally, long enough integration time is required to realize accumulation of weak signal. However, it is necessary to point out that, on the one hand, the satellite platform capacity determines the upper limit of the relative aperture of the load, and on the other hand, the speed-height ratio matching required by the low-orbit push-broom working mode also determines the upper limit of the imaging integration time, so the high-sensitivity micro-light detector becomes the key to restrict the realization of high-resolution high signal-to-noise ratio imaging of micro-light remote sensing.

[0004] Currently, CCD or CMOS cameras in orbit at home and abroad show serious lack of sensitivity when imaging at night, and most of them can only achieve light imaging and cannot obtain details in dark areas outside the light area. Therefore, image intensifier ICCD or ICMOS cameras (cameras formed by coupling image intensifiers with CCD or CMOS) with weak signal front-end amplification capability should be used to improve sensitivity and thus improve the signal-to-noise ratio of imaging. The image intensifier camera not only greatly improves the weak signal detection sensitivity through the multiplication function of the image tube, but also does not amplify the dark current noise and readout noise of the CCD or CMOS itself, in addition, it also has the functions of automatic overload noise suppression, automatic brightness control and strong light automatic protection, so it can balance high sensitivity and large dynamic range, and is particularly suitable for weak light imaging.

[0005] However, even if an image intensifier camera is used, it is still a challenge to achieve high signal-to-noise ratio imaging of more than 20dB required for daytime remote sensing applications when facing full moon and 1 / 4 moonlight conditions which differ from daytime conditions by several orders of magnitude. Calculation shows that to achieve high-resolution visible light micro-light remote sensing imaging with detailed level resolution requirements, the full moon signal-to-noise ratio at an orbital altitude of 500km is generally only about 10dB, and the 1 / 4 moon signal-to-noise ratio is only a few dB, so even if an image intensifier camera is used as the core to realize micro-light remote sensing imaging, how to further improve the sensitivity and imaging quality of weak light imaging is still a key problem to be solved.

[0006] Currently, researchers have found that under the condition of limited aperture of space optical cameras, increasing the equivalent integration time is the most effective way to realize high-sensitivity detection and high-signal-to-noise ratio imaging. For low-orbit push-broom imaging, the matching of speed-height ratio determines the line integration time of the detector, so to increase the equivalent integration time, using multiple line detectors to image synchronously and then performing digital superposition is a way to realize equivalent long integration. Studies have shown that compared with detectors such as EMCCD and sCMOS which can also perform weak light imaging, the way of increasing the equivalent integration time by multi-frame superposition to improve the signal-to-noise ratio is particularly suitable for image intensifier ICMOS or ICCD detectors. Therefore, using a multi-channel detector with a special structure can achieve the purpose of improving the signal-to-noise ratio of imaging by multi-frame superposition. However, whether it is an ICCD or an ICMOS camera, special process control during coupling of the image tube and the CCD or CMOS detector can easily cause image blurring, that is, MTF (modulation transfer function) degradation, thereby causing resolution degradation due to under-sampling of the imaging system. SUMMARY

[0007] The purpose of the present application is to solve the technical problem that the existing image intensifier ICCD or ICMOS camera is prone to image resolution degradation due to under-sampling during sampling, and to provide an imaging system and method based on micro-light sub-pixel misregistration sampling.

[0008] To achieve the above object, the technical solution provided by the present application is as follows:

[0009] An imaging system based on micro-light sub-pixel misregistration sampling, characterized in that it comprises a sampling unit, a turntable and a host computer.

[0010] The light sampling unit is a micro-light camera, which comprises a TDI-ICMOS camera and a micro-light lens arranged in front of the TDI-ICMOS camera; the micro-light lens is used to collect sparse photon streams under micro-light conditions and transmit them to the TDI-ICMOS camera.

[0011] The TDI-ICMOS camera is coupled by an image intensifier and a TDI-CMOS detector, and is used to convert the received sparse photon streams into corresponding image signals; the TDI-CMOS detector is internally provided with n rows of parallel distributed light-sensitive channels, 2≤n≤4; the adjacent two rows of light-sensitive channels are misregistered by one pixel in the row direction and are spaced by D pixels in the TDI push-scan direction, where D represents the physical spacing between the adjacent two rows of light-sensitive channels. The turntable is used to install the micro-light camera and set the corresponding rotational angular velocity according to the line frequency of the TDI-ICMOS camera.

[0012] The host computer is connected with the output end of the TDI-ICMOS camera, and is used to receive the image signals output by the TDI-ICMOS camera and perform image fusion enhancement, so as to realize imaging.

[0013] The host computer is connected with the output end of the TDI-ICMOS camera, and is used to receive the image signals output by the TDI-ICMOS camera and perform image fusion enhancement, so as to realize imaging.

[0014] Further, the host computer receives the image signals output by the TDI-ICMOS camera through an image acquisition card.

[0015] Further, the micro-light lens is internally configured in a total reflection type, a refraction type or a catadioptric type.

[0016] Meanwhile, the present application also provides an imaging method based on micro-light sub-pixel misregistration sampling, which adopts the above imaging system based on micro-light sub-pixel misregistration sampling and comprises the following steps:

[0017] 【1】Collecting sparse photon streams from a target scene under micro-light conditions through a micro-light lens; the micro-light conditions refer to the brightness conditions at dawn and dusk, full moon or 1 / 4 moon;

[0018] ​【2】TDI-ICMOS camera acquires n faint light images by TDI push-broom method and transmits them to the host computer; the TDI-CMOS detector of the TDI-ICMOS camera is internally provided with n rows of parallel distributed light sensing channels, 2≤n≤4; there is an overlapping part between any two adjacent faint light images in the n faint light images;

[0019] 【3】The host computer fuses and processes the received faint light images to realize faint light sub-pixel misregistration sampling imaging of the target scene.

[0020] Further, step 【2】 is specifically:

[0021] 【2a】The row integration time and integration order of each row of light sensing channels in the TDI-CMOS detector are set in advance;

[0022] 【2b】The corresponding row frequency is calculated according to the row integration time, and then the rotational angular velocity ω of the turntable is calculated;

[0023] 【2c】The turntable is reset, and the starting position and rotational angle of the turntable are set according to the scanning range of the target scene;

[0024] 【2d】The TDI-ICMOS camera and the turntable are started, and the turntable starts rotating according to the rotational angular velocity ω calculated in step 【2b】, at the same time, the TDI-ICMOS camera acquires n faint light images by TDI push-broom method;

[0025] 【2e】When the turntable reaches the rotational angle set in step 【2c】, it stops rotating, at the same time, the TDI-ICMOS camera also stops imaging, and the obtained faint light images are transmitted to the host computer.

[0026] Further, step 【3】 is specifically:

[0027] 【3a】The first faint light image received by the host computer is defined as a reference image, and the remaining images are sequentially shifted by (n-1)×(D+N) pixels in the opposite direction of the push-broom direction based on the reference image to realize the clipping alignment of the n faint light images, wherein D represents the physical interval between two adjacent rows of light sensing channels, and N represents the integration order of each row of light sensing channels;

[0028] 【3b】The faint light images after clipping alignment are respectively subjected to dark level correction processing; the dark level correction processing refers to deducting the dark current signal corresponding to the row integration time;

[0029] 【3c】The faint light images after dark level correction processing are subjected to noise reduction filtering processing;

[0030] [3d] Again, using the reference image as a reference, the sub-pixel relative displacements of the remaining low-light images relative to the reference image in the photosensitive channel row direction and the push-broom direction are calculated, thereby constructing the sub-pixel relative displacement matrix M.

[0031] [3e] Based on the sub-pixel relative displacement matrix M constructed in step [3d], register the remaining low-light images with the reference image respectively;

[0032] [3f] Enhance each registered low-light image separately, and then perform noise reduction filtering on each enhanced low-light image;

[0033] [3g] Based on step [3f], each low-light image is preprocessed separately, and then the low-light images are fused and enhanced; the preprocessing refers to first performing mirror boundary extension processing, and then performing windowing processing;

[0034] [3h] The low-light image after fusion enhancement is sequentially subjected to noise reduction filtering, MTF enhancement, and noise reduction filtering again to achieve low-light sub-pixel misaligned sampling imaging of the target scene.

[0035] Furthermore, in step [3g], the specific process of the fusion enhancement processing is as follows:

[0036] [3g1] The mean image is obtained by superimposing the preprocessed low-light images and averaging them. Then, the mean image is super-resolution magnified to 2 times to obtain the initial estimate X of the fused estimation image.

[0037] [3g2] Define the initial iteration count of the outer loop as t, the maximum iteration count as T, and the fusion estimation image of the t-th iteration as X. t The accumulation matrix in the t-th iteration is G. t Among them, G t The dimension of G is the same as the dimension of the fused estimated image, and let t=1, G t =0,X t =X;

[0038] 【3g3】The back-projection algorithm is used for the inner loop to obtain the accumulation matrix G of the t-th iteration. t ;

[0039] [3g4] Estimating the fusion image X t Perform regularization filtering, and then multiply it by the regularization factor β as the regularization term Norm for the t-th iteration. t Simultaneously, the accumulation matrix G t The product of the gradient iteration factor λ and the gradient iteration factor λ is used as the update term ΔX for this iteration. t Then, through the regularization term Norm t and update item ΔXt updating the fusion estimation image X t to obtain an updated fusion estimation image X t* wherein X t* = X t - ΔX t + Norm t ;

[0040] 【3g5】determine whether t = T is true, if true, obtain a fusion enhanced image X' final ; otherwise, let X t = X t* , G t = 0, t = t + 1, and return to execute step 【3g3】.

[0041] Further, in step 【3c】, the noise reduction filter uses a median filter algorithm;

[0042] In step 【3f】, the enhancement processing uses a multi-scale Retinex algorithm or a deep learning low-light image enhancement algorithm; the noise reduction filter uses a median filter algorithm;

[0043] In step 【3h】, the noise reduction filter uses a non-local mean NLM denoising algorithm; the MTF enhancement uses an accelerated vector extrapolation RL algorithm.

[0044] Further, in step 【3g1】, the super-resolution magnification uses a single-frame magnification algorithm based on Fourier transform.

[0045] Further, in step 【3g3】, the inner loop is performed using a back-iterative back-projection algorithm, which is specifically:

[0046] 【3g31】define an initial iteration number k and a maximum iteration number n, and let k = 1;

[0047] 【3g32】initialize intermediate matrices temp1, …, temp n-1 ;

[0048] 【3g33】based on the sub-pixel displacement vector of the kth row in the sub-pixel displacement matrix M obtained in step 【3d】, register the fusion estimation image with the kth image I k ; then use a disc filter psf and a down-sampling factor factor to sequentially perform blurring and down-sampling on the registered fusion estimation image, to obtain a low-resolution simulation image g k corresponding to the kth image in this iteration;

[0049] 【3g34】calculate the low-resolution simulation image g k and the kth image I kThe difference between them, so as to obtain the matrix temp1;

[0050] 【3g35】Update temp2=G t +temp1 and G t = temp2 to G t ;

[0051] 【3g35】Determine whether k=n is established, if so, obtain the accumulated matrix G t of this iteration; otherwise, let k=k+1, and execute step 【3g33】.

[0052] Compared with the prior art, the present application has the beneficial technical effects as follows:

[0053] 1、The imaging system based on micro-light sub-pixel misregistration sampling provided by the present application adopts an image intensifier TDI-ICMOS camera coupled by an image intensifier and a TDI-CMOS detector, wherein the TDI-CMOS detector is internally provided with n rows of parallel distributed light-sensitive channels, so that when sampling, n micro-light images with sub-pixel relative displacement can be obtained by using the push-broom imaging mode with the cooperation of a turntable, thereby laying a foundation for improving the signal-to-noise ratio and the transfer function MTF in the fusion enhancement.

[0054] 2、The imaging method based on micro-light sub-pixel misregistration sampling provided by the present application takes the backward iterative back-projection algorithm as a framework, relies on the random characteristics of the noise between multiple sub-pixel misregistration sampling images and the reasonable use of the minimum root mean square criterion in the fusion process, can solve the low signal-to-noise ratio problem of micro-light imaging, and can also simultaneously improve the problems of the reduction of spatial resolution and the transfer function MTF caused by the complex structure, assembly process and multiple photoelectric and electro-optical conversions of the image intensifier, thereby enhancing the ability of the camera to obtain higher quality images under micro-light conditions.

[0055] 3、In the imaging method based on micro-light sub-pixel misregistration sampling provided by the present application, the micro-light image enhancement algorithm is combined with the sub-pixel reconstructed image by taking the backward iterative back-projection algorithm as a starting point, and the high-frequency information is highlighted through the enhancement algorithm to further improve the fusion quality.

[0056] 4、In the imaging method based on micro-light sub-pixel misregistration sampling provided by the present application, the multiple sub-pixel misregistration sampling micro-light images after registration are subjected to median filtering, dark level correction, pixel-level superposition and single-frame super-resolution magnification based on Fourier transform to construct an initial estimate of a high-resolution fusion image, thereby providing a better starting point for subsequent fusion.

[0057] 5、The imaging method based on micro-light sub-pixel misregistration sampling provided by the present application, in combination with the pre-processing operation of anti-mirror image boundary and windowing processing, synchronously suppresses the high-frequency information accumulated in the fusion with gradient as a clue, effectively weakening the ringing effect at the strong contrast boundary.

[0058] 6、The imaging method based on micro-light sub-pixel misregistration sampling provided by the present application, in combination with the pre-processing operation of anti-mirror image boundary and windowing processing, synchronously suppresses the high-frequency information accumulated in the fusion with gradient as a clue, effectively weakening the ringing effect at the strong contrast boundary.

[0059] 7、The imaging method based on micro-light sub-pixel misregistration sampling provided by the present application, in combination with the pre-processing operation of anti-mirror image boundary and windowing processing, synchronously suppresses the high-frequency information accumulated in the fusion with gradient as a clue, effectively weakening the ringing effect at the strong contrast boundary. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 is the overall structure schematic diagram of the imaging system based on micro-light sub-pixel misregistration sampling of the present application;

[0061] Figure 2a is the structure schematic diagram of the image intensifier in the imaging system based on micro-light sub-pixel misregistration sampling of the present application;

[0062] Figure 2b is the structure schematic diagram of the internal light-sensitive channels of the TDI-CMOS detector in the imaging system based on micro-light sub-pixel misregistration sampling of the present application.

[0063] Figure 3 is the position relationship diagram of the micro-light image output by each row of light-sensitive channels of the TDI-CMOS detector in the imaging system based on micro-light sub-pixel misregistration sampling of the present application.

[0064] Figure 4 is the flowchart of the fusion enhancement and post-processing in the imaging method based on micro-light sub-pixel misregistration sampling of the present application;

[0065] Figure 5a is the original micro-light image without fusion enhancement.

[0066] Figure 5b is the micro-light image after fusion enhancement by the imaging method based on micro-light sub-pixel misregistration sampling of the present application.

[0067] The reference signs are as follows:

[0068] 1-rotating platform, 2-upper computer, 3-micro-light camera, 4-photo-cathode, 5-micro-channel plate, 6-fluorescent screen, 7-tube shell. DETAILED DESCRIPTION

[0069] In order to make the objects, advantages and features of the present application clearer, the following further describes the present application in combination with the drawings and specific embodiments. Those skilled in the art should understand that these embodiments are only used to explain the technical principles of the present application, and the purpose is not to limit the protection scope of the present application.

[0070] Currently, there are few reports on combining sub-pixel misregistration sampling imaging technology with ICCD or ICMOS cameras for micro-light imaging, and it is a great challenge to realize high-resolution and high-signal-to-noise ratio micro-light remote sensing imaging at the detailed level. In fact, while using a multi-channel structure detector to realize multi-frame superposition to improve the signal-to-noise ratio, the relative displacement of sub-pixels between multiple rows of detectors can be used to fuse and enhance the push-broom images output by the multiple rows of detectors with sub-pixel level relative displacement relationship, so as to improve the system MTF. However, sub-pixel technology requires special detector structure to ensure, although the maturity of sub-pixel technology is relatively high, but sub-pixel technology itself is not born for night remote sensing, therefore, using this technology still cannot realize high-resolution and high-signal-to-noise ratio imaging at the detailed level under the condition of full moon, 1 / 4 moon and other extreme dark and weak light conditions.

[0071] The present application aims at the demand of night micro-light remote sensing for high-quality optical image acquisition, and proposes an imaging system based on micro-light sub-pixel misregistration sampling and a matching imaging method, which aims to solve the problem of high-signal-to-noise ratio and high-MTF imaging under the condition of full moon, 1 / 4 moon and other extreme dark and weak light conditions.

[0072] REFERENCE Figure 1 The embodiment provides an imaging system based on micro-light sub-pixel misregistration sampling, which comprises a sampling unit, a rotating platform 1 and an upper computer 2.

[0073] The light collecting unit is a micro-light camera 3, which comprises a TDI-ICMOS camera and a micro-light lens arranged in front of the TDI-ICMOS camera; the micro-light lens is used for collecting sparse photon flow under micro-light conditions and transmitting the sparse photon flow to the TDI-ICMOS camera. The micro-light lens can be of a total reflection type, a refraction type or a catadioptric type. Which type of micro-light lens is adopted is related to the potential application field of the technology, for example, in the field of micro-light remote sensing, in order to realize high-resolution micro-light remote sensing while giving consideration to reasonable optical-mechanical weight and volume constraints, the total reflection type or the catadioptric type is mainly adopted. However, no matter which type is adopted, in order to improve the collection efficiency of sparse photon flow under micro-light conditions, the micro-light lens is preferably of a relatively large relative aperture to improve the photon collection capacity.

[0074] A TDI-ICMOS camera consists of an image intensifier coupled with a TDI-CMOS detector, used to convert the received sparse photon stream into a corresponding image signal. The TDI-CMOS detector contains n parallel photosensitive channels, where 2 ≤ n ≤ 4. Adjacent rows of photosensitive channels are staggered sequentially along the row direction. Each pixel is spaced sequentially along the TDI push-broom direction. Pixels, where D represents the physical spacing between two adjacent rows of photosensitive channels.

[0075] The TDI-ICMOS detector, as the core component for low-light imaging in nighttime, uses a fiber optic panel as a relay optical element to couple the image intensifier and the TDI-CMOS detector. The TDI-ICMOS camera, as an electro-vacuum device, works as follows: the photocathode of the image intensifier receives target photons and emits photoelectrons, which are then focused onto a microchannel plate (MCP). The MCP component multiplies the electrons, which are then accelerated by a high voltage and bombard the fluorescent screen to form an enhanced optical image. The fiber optic panel transmits the enhanced image to the TDI-CMOS chip, where the TDI-CMOS device converts the optical signal into an electrical signal. Because the microchannel plate can provide up to 10... 6 Due to electron multiplication, TDI-ICMOS cameras are particularly suitable for low-light detection and imaging.

[0076] Turntable 1 is used to mount the low-light camera 3, and its rotational angular velocity is set according to the line frequency of the TDI-ICMOS camera. Turntable 1 has a high-precision interface for mounting the TDI-ICMOS camera. First, the pushbroom line frequency is calculated based on the resolution and focal length of the TDI-ICMOS camera, and then the optimal rotational angular velocity of turntable 1 is calculated to ensure charge transfer matching during the pushbroom imaging process. During the pushbroom imaging process, the low-light lens first collects the sparse photon stream and focuses it onto the cathode of the TDI-ICMOS detector. Then, the back-end circuit reads out the image and finally converts it into a digital image before sending it to the host computer 2.

[0077] The host computer 2 is connected to the output terminal of the TDI-ICMOS camera. The host computer 2 receives the image signal output by the TDI-ICMOS camera through the image acquisition card, and performs fusion enhancement to achieve imaging.

[0078] Figure 2a An exploded view of the image intensifier structure is given. As can be seen from the figure, the photocathode 4, microchannel plate 5, and fluorescent screen 6 of the image intensifier are enclosed and nested by the tube shell 7 to form a complete device. Figure 2b A schematic diagram of a TDI-CMOS detector structure coupled to an image intensifier is given. Figure 2bIt can be seen that, unlike the conventional TDI-CMOS detector, the TDI-CMOS detector used in the application has 4 rows of parallel distributed light sensing channels, each row of light sensing channels is sequentially staggered by 1 / 4 pixel in the detector row direction, and is also sequentially staggered by D+1 / 4 pixel in the TDI push scanning direction. Among them, D represents the physical interval between the adjacent two rows of detectors, which is the pixel number converted according to the pixel size. Therefore, Figure 2b The 4-channel TDI-CMOS detector shown can synchronously obtain 4 push scanning images with sub-pixel staggered sampling due to the fixed sub-pixel phase relationship of each row of light sensing channels, thereby providing a hardware guarantee for sub-pixel staggered sampling fusion enhancement.

[0079] In the push scanning imaging process, with the first row of light sensing channels as the reference, the physical interval between each light sensing channel makes there be a fixed D+N pixel offset between the 4 images along the push scanning direction (where N represents the integration order of the light sensing channel), as shown in Figure 3 The effect of splicing the 4 images obtained by the 4-channel detector after push scanning imaging into one image is shown, and it can be seen that the positions of the same target in different channels change by a fixed number of pixels. Therefore, before implementing the subsequent fusion enhancement, the second image, the third image and the fourth image are sequentially translated by 1×(D+N) pixels, 2×(D+N) pixels and 3×(D+N) pixels in the reverse push scanning direction with the first row of detector image as the reference, so that the 4 images are aligned to within the theoretical sub-pixel through clipping.

[0080] On this basis, the embodiment also provides an imaging method based on low-light sub-pixel staggered sampling. The method is based on the classical iterative back projection algorithm IBP (Iterative Back Projection) framework, and is essentially a constrained optimization problem. Based on multiple low-resolution measured images, a simulated low-resolution image generated by a forward imaging model is used to find an optimal high-resolution fusion image by minimizing the root mean square error with the measured low-resolution image. Among them, according to the characteristics of weak light imaging, improvements are made in the preprocessing process, the initial estimation of the high-resolution fusion image, etc., thereby generating the low-light sub-pixel staggered sampling imaging method proposed in the application.

[0081] The specific steps of the method will be described in detail below taking a 4-channel (i.e. 4 light sensing channels) TDI-CMOS detector as an example.

[0082] 【1】Collect sparse photon streams from the target scene under low-light conditions through a low-light lens. The low-light conditions described in the embodiment refer to the brightness conditions at dawn, dusk, full moon or 1 / 4 moon.

[0083] 【2】TDI-ICMOS camera obtains four faint light images by TDI push-broom, and there is an overlap between every two adjacent faint light images, then the TDI-ICMOS camera transmits the four faint light images to the host computer 2, and the specific steps are as follows:

[0084] 【2a】The row integration time IntegrationT and the integration number N of each row of light-sensitive channels in the TDI-CMOS detector are set in advance.

[0085] 【2b】The corresponding row frequency v is calculated according to the row integration time IntegrationT, and the rotation angular velocity ω of the turntable 1 is calculated according to the pixel size p of the detector, the focal length f of the faint light lens,

[0086] 【2c】The turntable 1 is reset, and the starting position and the rotation angle θ of the turntable are set according to the scanning range of the target scene.

[0087] 【2d】The TDI-ICMOS camera and the turntable 1 are started by the ground detection equipment, then the turntable 1 starts to rotate according to the rotation angular velocity ω calculated in step 【2b】, and at the same time, the TDI-ICMOS camera starts to push-broom imaging by TDI push-broom to obtain four faint light images.

[0088] 【2e】When the turntable 1 reaches the rotation angle θ set in step 【2c】, it stops rotating, at the same time, the TDI-ICMOS camera also stops imaging, and transmits the obtained four faint light images to the host computer 2.

[0089] 【3】The host computer 2 fuses and processes the received faint light images to realize the faint light sub-pixel misregistration sampling imaging of the target scene. Figure 4 The fusion and processing process of the embodiment is as follows:

[0090] 【3a】The first faint light image in the four faint light images received by the host computer 2 is defined as a reference image, and the second faint light image is shifted by 1×(D+N) pixels in the opposite direction of the push-broom direction, the third faint light image is shifted by 2×(D+N) pixels in the opposite direction of the push-broom direction, and the fourth faint light image is shifted by 3×(D+N) pixels in the opposite direction of the push-broom direction, to realize the clipping and alignment of the four faint light images, wherein D represents the physical interval between two adjacent rows of light-sensitive channels, and N represents the integration number of each row of light-sensitive channels.

[0091] 【3b】The host computer 2 performs dark level correction processing on the four faint light images after clipping and alignment.

[0092] The dark level correction processing described in the embodiment refers to deducting the dark current signal corresponding to the line integration time. Since the low-light low-illumination image is more susceptible to the non-uniformity of the dark current response, the obtained low-light image needs to be corrected for the dark level. The dark level correction can be realized by two methods. If the detector has an overscan area and the dark current accumulation time in the overscan area is the same as the push-broom imaging time, the dark pixels in each column of the push-broom image corresponding to the overscan area can be extracted to obtain the average dark level gray scale and deduct it from each column of the image to complete the dark level correction. If the detector does not have an overscan area, first, a long-time push-broom imaging is performed without exposure, and then a plurality of dark field images are obtained, and the standard dark field image is obtained by averaging the plurality of dark field images. Then, the standard dark field image is scaled according to the actual push-broom imaging time to obtain a dark level correction image corresponding to the actual push-broom imaging. Finally, the dark level correction image is deducted from the push-broom image to complete the dark level correction.

[0093] 【3c】The low-light image after the dark level correction processing is subjected to a median filter algorithm for noise reduction filtering processing. Since the TDI-ICMOS detector needs to work at a relatively high electron multiplication gain, and the thermal noise caused by the dark emission of the photocathode will also be amplified together with the signal photons, the original low-light image needs to be subjected to median filtering to eliminate the influence of local abnormal thermal noise.

[0094] 【3d】Again, the reference image is taken as the reference to calculate the sub-pixel relative displacement of the remaining low-light images relative to the reference image in the line direction and the push-broom direction of the photosensitive channel, wherein the sub-pixel relative displacement is obtained by using an upsampling phase correlation algorithm, thereby constructing a sub-pixel relative displacement matrix M, and the matrix dimension is 4x2. Since the displacement includes x and y directions, the dimension is 2, and the matrix row is the number of images, so the matrix dimension of the four low-light images is 4x2.

[0095] For the sub-pixel imaging technology, to realize the fusion of multiple images, the sub-pixel relative displacement between images must be obtained. In the present application, since the four rows of parallel detectors of the TDI-ICMOS detector have a known relative phase correlation in the detector line direction and the push-broom direction, the sub-pixel relative displacement matrix between images is theoretically known. However, in practice, considering that there may be external disturbances during push-broom imaging, causing the phase correlation between adjacent images to deviate from the theoretical value when the same scene target is imaged by different rows of detectors at different times, therefore, for safety, an upsampling phase correlation algorithm is used to obtain high-precision sub-pixel image motion to create conditions for fusion.

[0096] 【3e】Based on the sub-pixel relative displacement matrix M constructed in step 【3d】, the remaining low-light images are respectively registered with the reference image.

[0097] [3f] Each registered low-light image is enhanced separately, and then each enhanced low-light image is subjected to noise reduction filtering. The enhancement process uses the multi-scale Retinex algorithm or a deep learning low-light image enhancement algorithm; the noise reduction filtering process uses the median filtering algorithm to eliminate the influence of local abnormal thermal noise.

[0098] [3g] Based on step [3f], each low-light image is preprocessed separately, and then the low-light images are fused and enhanced. The preprocessing mentioned here refers to first performing mirror boundary extension processing, and then performing windowing processing, thereby constructing an image dataset I containing the four preprocessed low-light images.

[0099] Considering the characteristics of low-light images, this embodiment creatively combines a low-light image enhancement algorithm with sub-pixel low-light image fusion in the proposed algorithm's fusion enhancement section. This utilizes the image enhancement algorithm to address the difficulty of information extraction caused by low illumination and low contrast in low-light images, ensuring accurate extraction of high-frequency information in the fusion algorithm. In this invention, after the original low-light image undergoes cropping, alignment, dark level correction, and median filtering, adaptive low-light image enhancement can be achieved using a multi-scale Retinex algorithm, or an enhanced low-light image can be obtained using a deep learning low-light image enhancement algorithm.

[0100] In this embodiment, the specific process of fusion enhancement processing is as follows:

[0101] [3g1] The four preprocessed low-light images are superimposed and averaged to obtain the mean image. Then, the mean image is super-resolution magnified to 2 times to obtain the initial estimate X of the fused estimation image. The super-resolution magnification mentioned here uses a single-frame magnification algorithm based on Fourier transform.

[0102] The core mechanism of high-resolution image fusion is to start with the initial estimation of the high-resolution fused image, and take the minimum root mean square error between the simulated degraded low-resolution image and the low-resolution measured image as the objective function. The desired high-resolution fused image is obtained through optimization and iterative calculation.

[0103] The sub-pixel misregistration sampling image fusion based on the iterative back-projection framework firstly needs to generate an initial estimation of the high-resolution fusion image. Generally, any one image in the data set is selected and enlarged by 2 times to be the initial estimation of the high-resolution fusion image. Here, considering the characteristics of low-light imaging, a new method for generating the initial estimation of the high-resolution fusion image is proposed. First, the mean image is obtained by superimposing and averaging the four sub-pixel misregistration sampling low-light images after the operations of cropping, alignment, dark level correction and median filtering, and then the initial estimation of the high-resolution fusion image is obtained by using the single-frame enlargement algorithm based on Fourier transform interpolation. The four times averaging can reduce the noise variance, and the single-frame enlargement algorithm based on Fourier transform interpolation itself can realize a certain resolution enhancement, so the initial estimation of the high-resolution fusion image obtained by combining the two methods can provide a better signal-to-noise ratio and resolution starting point for the subsequent fusion enhancement.

[0104] 【3g2】Define the initialization iteration number of the outer loop as t, the maximum iteration number as T, the fusion estimation image of the tth iteration as X t , and the accumulation matrix of the tth iteration as G t , where the dimension of G t is the same as that of the fusion estimation image X t , and let t = 1, G t = 0, and X t = X.

[0105] 【3g3】The host computer 2 performs inner loop by using the back-projection iteration algorithm to obtain the accumulation matrix G t of the tth iteration, which is specifically:

[0106] 【3g31】Define the initialization iteration number as k, the maximum iteration number as 4, and let k = 1;

[0107] 【3g32】Initialize the intermediate matrices temp1, temp2 and temp3;

[0108] 【3g33】The host computer 2 registers the fusion estimation image with the kth image I k based on the sub-pixel displacement vector of the kth row in the sub-pixel displacement matrix M obtained in step 【3d】; then the registered fusion estimation image is sequentially blurred and down-sampled by using the disc filter psf and the down-sampling factor factor to obtain the low-resolution simulation image g k corresponding to the kth image in this iteration;

[0109] 【3g34】The host computer 2 calculates the difference temp1 = g k -I k between the simulation image g k and the kth image I kThus, matrix temp1 is obtained; then temp1 is magnified by 2 times and filtered by edge enhancement high-frequency enhancement filter H. The filtering result is aligned with X according to the sub-pixel displacement vector in the k-th row of the sub-pixel displacement matrix M to obtain matrix temp2.

[0110] 【3g35】via temp3=G t +temp2 and G t =temp3 to G t Update;

[0111] 【3g35】Determine if k=4 is true. If true, then obtain the cumulative matrix G for this iteration. t Otherwise, let k = k + 1 and execute step [3g33] until the cumulative matrix G of this iteration is obtained. t This ends the inner loop.

[0112] [3g4] Estimating image X by fusing two pairs of images via the host computer. t Perform regularization filtering, and then multiply it by the regularization factor β as the regularization term Norm for the t-th iteration. t Simultaneously, the accumulation matrix G t The product of the gradient iteration factor λ and the gradient iteration factor λ is used as the update term ΔX for this iteration. t Then, through the regularization term Norm t and update item ΔX t For the fusion estimation image X t The image is updated to obtain the updated fusion estimation image X. t* , where X t* =X t -ΔX t +Norm t .

[0113] 【3g5】The host computer determines whether t=T holds true. If it does, the fused and enhanced image X' is obtained. final Otherwise, let X t =X t* G t =0, t=t+1, and execute step [3g3].

[0114] The simulation imaging process of generating low-resolution degraded images from the initial estimation of high-resolution fusion image mainly includes three steps: alignment, blurring and down-sampling. The current estimation of high-resolution fusion image is firstly aligned with the current low-resolution observation image respectively through sub-pixel translation according to the sub-pixel displacement matrix; then the blurring effect of optical lens is simulated by a disc low-pass filter as the system degradation function, the radius parameter of the disc low-pass filter should be equal to the size of the optical diffraction spot determined by the relative aperture of the employed low-light lens, thus ensuring the compliance with the imaging physical law; further, the high-resolution fusion image after blurring degradation is down-sampled by 2 to obtain the low-resolution simulation image.

[0115] After obtaining the simulated low-resolution image, the difference between the low-resolution simulation image and the low-resolution actual measurement image is mapped back to the high-resolution space through up-sampling, which mainly includes two steps: up-sampling and high-frequency enhancement. The difference between the low-resolution simulation image and the low-resolution actual measurement image is firstly up-sampled by 2, and then filtered by a Gaussian-Laplace filter as an edge-enhancing high-frequency enhancement filter and added to an intermediate matrix, until all low-resolution actual measurement images have been traversed, thus obtaining the update matrix required for the desired update high-resolution fusion image.

[0116] After obtaining the update matrix required for the update high-resolution fusion image, the current estimation of high-resolution fusion image is firstly filtered by a negative Gaussian-Laplace filter and multiplied by a regularization factor β to generate a regularization term that suppresses noise amplification, then the result of multiplying the update matrix required for the update high-resolution fusion image by a gradient iteration factor λ is taken as a new update matrix, and finally the result of subtracting the update matrix from the current high-resolution fusion image and then adding the regularization term is calculated, thus obtaining the high-resolution fusion image obtained by the current iteration. This process continues until the number of iterations reaches the upper limit, at which time the required high-resolution fusion image is obtained.

[0117] It should be noted that, in the high-resolution fusion process, since the forward and inverse Fourier transform is involved, ringing may occur at high-contrast edges, so on the one hand, the traditional anti-image boundary method and windowing method are used to process the low-light images before fusion, and on the other hand, an innovative way is used to further weaken and eliminate the ringing effect during the fusion process. Ringing often occurs in high-contrast edge regions, so the high-frequency edge gradient can be used as a clue to locally low-pass filter the update matrix containing high-frequency information to suppress ringing, thereby reducing its impact on the fusion image and improving the effect of the fusion image.

[0118] 【3h】The host computer 2 processes the fusion enhanced image X' finalThe noise reduction filter, MTF enhancement and noise reduction filter are sequentially implemented, so that the low-light sub-pixel misregistration sampling imaging of the target scene is realized, and a high-resolution fusion enhanced image X is obtained final .

[0119] When the step is performed, the fast non-local mean (NLM) denoising algorithm is used for the two times of noise reduction filtering, and the improved accelerated vector extrapolation (RL) algorithm is used for MTF enhancement.

[0120] After the fusion enhancement is completed, post-processing is still needed to generate the final low-light image with improved signal-to-noise ratio and MTF. On the one hand, there is inevitably residual noise amplification in the fusion enhancement process, so noise reduction filtering should be performed again after the fusion enhanced image is obtained. On the other hand, due to the regularization processing and low-pass filtering of the high-frequency information update matrix in the fusion enhancement process, the edges of the fused image will be blurred and degraded, so MTF compensation technology should be used to improve the contrast of the image after fusion enhancement, so as to obtain better subjective visual effect. Here, the non-local mean (NLM) algorithm is used for noise reduction filtering, and the improved accelerated vector extrapolation (RL) algorithm is used for MTF compensation. Both of these two algorithms are mature and reliable, and have been tested to have good applicability to the sub-pixel misregistration fusion enhancement algorithm.

[0121] In summary, the low-light sub-pixel misregistration sampling imaging system and the matching imaging method proposed in the present application are introduced, and finally the applicability of the low-light sub-pixel misregistration sampling imaging system and method proposed in the present application under low-light conditions is proved through an outdoor imaging test. Among them, the lens focal length of the TDI-ICMOS low-light camera 3 used is 144 mm, the relative aperture is 1:9, the pixel size is 14 um, the imaging distance is about 3 km, the imaging time is around 1 am on the eighth day of the lunar calendar, and the imaging target is a village at the foot of a mountain.

[0122] Reference Figure 5a and Figure 5b give the effect of the outdoor imaging test, wherein, Figure 5a is an image of a certain channel, which has not been processed. It can be seen that the low contrast and low brightness make it difficult to distinguish the levels and vegetation information of the distant mountains. And Figure 5b is the image obtained by using the imaging method proposed in the present application to fuse and enhance the four-channel low-light sub-pixel misregistration sampling image. Compared with Figure 5b Figure 5b , the image quality of the image is improved very obviously, which again proves the effectiveness of the imaging method.

[0123] Compared with the prior art, the biggest innovation of the present application is that the sub-pixel misalignment sampling technology is combined with the low-light imaging technology based on the image-enhanced TDI-ICMOS low-light detector, and the low-light weak light high signal-to-noise ratio detection and high-resolution imaging dual purposes can be realized by using the sub-pixel misalignment sampling relationship of each row of the detector in the TDI push scanning direction and the detector row direction, and through the fusion enhancement of the multi-channel sub-pixel misalignment sampling low-light image.

[0124] Finally, it should be noted that: the above examples are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing examples, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing examples, or make equivalent replacement for part or all of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the present application.

Claims

1. An imaging method based on micro-light sub-pixel misregistration sampling, characterized in that, An imaging system based on micro-light sub-pixel misregistration sampling is adopted, the imaging system comprises a sampling unit, a rotary table (1) and a host computer (2); the sampling unit is a micro-light camera (3), comprising a TDI-ICMOS camera and a micro-light lens arranged in front of the TDI-ICMOS camera; the micro-light lens is internally configured in a total reflection type, a refraction type or a catadioptric type, used for collecting sparse photon flow under micro-light conditions and transmitting to the TDI-ICMOS camera; the TDI-ICMOS camera is coupled by an image intensifier and a TDI-CMOS detector, used for converting the received sparse photon flow into corresponding image signals; the TDI-CMOS detector is internally provided with n rows of parallel distributed light-sensitive channels, 2≤n≤4; adjacent two rows of light-sensitive channels are misregistered by D pixels in the row direction and are spaced by D pixels in the TDI push-scan direction, wherein D represents the physical interval between the adjacent two rows of light-sensitive channels; the rotary table (1) is used for mounting the micro-light camera (3) and setting a corresponding rotary angular velocity according to the line frequency of the TDI-ICMOS camera; the host computer (2) is connected with the output end of the TDI-ICMOS camera, receives the image signals output by the TDI-ICMOS camera through an image acquisition card and performs image fusion enhancement, so as to realize imaging; the imaging method comprises the following steps: ​​ 【1】Collecting sparse photon stream from target scene under micro light condition through micro light lens; the micro light condition refers to the brightness condition at dawn and dusk, full moon or 1 / 4 moon; 【2】Acquiring n micro light images through TDI push scanning mode of TDI-ICMOS camera and transmitting them to host computer (2); the TDI-CMOS detector of the TDI-ICMOS camera is internally provided with n rows of parallel distributed light sensing channels, 2≤n≤4; there is an overlapping part between adjacent two of the n micro light images; 【3】Fusing and post-processing the received micro light images through the host computer (2), so as to realize micro light sub-pixel misregistration sampling imaging of the target scene, specifically: 【3a】Defining the first micro light image received by the host computer (2) as reference image, and sequentially shifting the rest of the images by (n-1)×(D+N) pixels in the reverse direction of the push scanning direction, so as to realize the clipping alignment of the n micro light images, wherein D represents the physical interval between adjacent two rows of light sensing channels, and N represents the integration order of each row of light sensing channels; 【3b】Respectively performing dark level correction processing on the micro light images after clipping alignment; the dark level correction processing refers to deducting the dark current signal corresponding to the row integration time; 【3c】Respectively performing noise reduction filtering processing on the micro light images after dark level correction processing; 【3d】Again taking the reference image as the benchmark, calculating the sub-pixel relative displacement of the rest of the micro light images relative to the reference image in the light sensing channel row direction and the push scanning direction, so as to construct a sub-pixel relative displacement matrix M; 【3e】Based on the sub-pixel relative displacement matrix M constructed in step 【3d】, registering the rest of the micro light images with the reference image respectively; 【3f】Respectively performing enhancement processing on the registered micro light images, and then performing noise reduction filtering processing on each of the enhanced micro light images; 【3g】On the basis of step 【3f】, respectively performing preprocessing on each of the micro light images, and then performing fusion enhancement processing on the micro light images; the preprocessing refers to first performing mirror boundary extension processing, and then performing windowing processing; the specific process of the fusion enhancement processing is: 【3g1】Stacking and averaging the preprocessed micro light images to obtain a mean image, and then performing super-resolution amplification to 2 times on the mean image to obtain an initial estimate X of the fusion estimate image; 【3g2】define the initialization iteration number of the outer loop as t, the maximum iteration number as T, the fusion estimation image of the tth iteration as X t , and the accumulation matrix of the tth iteration as G t , where the dimension of G t is the same as that of the fusion estimation image, and let t = 1, G t = 0, and X t = X; 【3g3】The back-projection algorithm is used for inner loop to obtain the accumulated matrix G of the tth iteration t ; 【3g4】updating the fused estimation image X t regularization filter, and then taking the product of the regularization factor β as the regularization term Norm t ; meanwhile, taking the product of the gradient iteration factor λ and the accumulated matrix G t as the update term ΔX t ; then updating the fused estimation image X t by the regularization term Norm t and the update term ΔX t , to obtain the updated fused estimation image X t* , wherein X t* = X t - ΔX t + Norm t ; 【3g5】Determine if t = T is true, if true, get fusion enhanced image X' final ; otherwise, let X t = X t* , G t = 0, t = t + 1, go back to step 【3g3】; 【3h】Respectively performing noise reduction filtering, MTF enhancement and again noise reduction filtering on the micro light images after fusion enhancement processing, so as to realize micro light sub-pixel misregistration sampling imaging of the target scene; the noise reduction filtering adopts non-local mean NLM denoising algorithm; the MTF enhancement adopts accelerated vector extrapolation RL algorithm.

2. The micro-light sub-pixel misregistration sampling based imaging method according to claim 1, characterized in that, Step 【2】 is specifically: 【2a】Pre-setting the row integration time and integration order of each row of light sensing channels in the TDI-CMOS detector; 【2b】Calculating the corresponding row frequency according to the row integration time, and then calculating the rotational angular velocity ω of the turntable (1); 【2c】Resetting the turntable (1), and setting the starting position and rotational angle of the turntable (1) according to the scanning range of the target scene; 【2d】Start the TDI-ICMOS camera and the turntable (1), the turntable (1) starts to rotate according to the rotation angular velocity ω calculated in step 【2b】, at the same time, the TDI-ICMOS camera acquires n faint light images by TDI push-broom method; 【2e】When the turntable (1) reaches the rotation angle set in step 【2c】, it stops rotating, at the same time, the TDI-ICMOS camera also stops imaging, and the obtained faint light images are transmitted to the host computer (2).

3. The faint light sub-pixel misregistration sampling based imaging method according to claim 1, characterized in that: In step 【3c】, the noise reduction filter adopts a median filter algorithm; In step 【3f】, the enhancement processing adopts a multi-scale Retinex algorithm or a deep learning faint light image enhancement algorithm; the noise reduction filter adopts a median filter algorithm.

4. The faint light sub-pixel misregistration sampling based imaging method according to claim 1, characterized in that: In step 【3g1】, the super-resolution magnification adopts a single-frame magnification algorithm based on Fourier transform.

5. The micro-light sub-pixel misregistration sampling based imaging method of claim 1, wherein, In step 【3g3】, the inner loop is performed by using a back-iterative back-projection algorithm, which is specifically: 【3g31】Define the initial iteration number as k, the maximum iteration number as n, and let k = 1; [3g32] initialize intermediate matrices temp1,..., temp n-1 ; 【3g33】Based on the sub-pixel displacement vector of the kth row in the sub-pixel displacement matrix M obtained in step 【3d】, the fusion estimation image is registered with the kth image I k The registered fusion estimation image is then blurred and down-sampled using the disc filter psf and the down-sampling factor factor to obtain a low-resolution simulated image g corresponding to the kth image in the current iteration k ; 【3g34】Computing a low resolution simulation image g k the difference between the kth image I k and the difference between the kth image I and the difference between the kth image I and the difference between the kth image I and the difference between the kth image I and the difference between the kth image I and the difference between the kth image I and the difference between the 【3g35】Update temp2 = G t + temp1 with G t = temp2 on G t ; 【3g36】If k = n is true, the accumulated matrix G of this iteration is obtained t ; Otherwise, let k = k + 1, and return to step 【3g33】.

Images