Night vision anti-blooming method for multi-region fusion of heterogeneous images

Abstract

The application is a kind of heterologous image multi-region fusion night vision anti-halation method, comprising the following steps: YUV transformation is carried out on the registered visible light image to obtain three components of brightness Y, chroma U and saturation V; high-brightness halation area alpha H , high-brightness useful information area alpha L and low-illumination area alpha B are divided; FNSCT decomposition is carried out to obtain the low-frequency sub-band and high-frequency sub-band of the brightness Y and infrared image; the low-frequency sub-band is subjected to multi-region fusion to obtain the fused low-frequency sub-band of alpha H , alpha L and alpha B ; the statistical matching feature strategy is adopted to obtain the fused high-frequency sub-band; the fused high-frequency sub-band and low-frequency sub-band are subjected to FNSCT reconstruction to obtain a new brightness component Y', and YUV inverse transformation is carried out on the original hue U and saturation V to output the fused image. The application effectively ensures the integrity of the useful information of the fused image in the night vision halation scene, and can meet the requirement of eliminating high-brightness halation while retaining high-brightness useful information and improving the visibility of the low-illumination area in the night vision halation scene.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of night vision anti-halation, and mainly relates to a kind of heterologous image multi-region fusion night vision anti-halation method. BACKGROUND

[0002] The halation phenomenon caused by the misuse of high beam at night is one of the important reasons for the occurrence of night road traffic accidents. The heterologous image fusion anti-halation method combines the advantages of rich color detail information of visible light image and halation-free of infrared image, and can obtain a halation-free and texture-rich fusion image, effectively solving the traffic hazards caused by night halation phenomenon.

[0003] The document "Infrared and Visible Light Image Fusion Anti-Halation System for Cars" proposes a YUV-wavelet transform image fusion anti-halation method, which fuses low-frequency subbands with a weighted average strategy. However, there is still a phenomenon of insufficient halation elimination, and the junction between halation and non-halation areas is not natural. The document "Improved IHS-Curvelet Transform Fusion Visible Light and Infrared Image Anti-Halation Method" designs a tangent low-frequency coefficient weight automatic adjustment strategy to fuse low-frequency subbands, and uses the modulus to fuse high-frequency subbands. The obtained fusion image has low key target saliency of vehicles, pedestrians, etc. The document "Night Vision Anti-Halation Algorithm Based on Different-Source Image Fusion Combining Visual Saliency with YUV-FNSCT" establishes a fusion weight based on an infrared saliency map, and the obtained fusion image has less color information preserved in the background area. The above documents all limit the participation of high-brightness information in fusion to eliminate high-brightness halation. However, the high-brightness information that affects driving safety at night not only includes car headlight halation and strong glare reflected from the road surface, but also includes important traffic sign information such as lane lines and zebra crossings. The existing heterologous image fusion anti-halation method does not distinguish between high-brightness halation and high-brightness useful information, and fuses high-brightness useful information according to the same rule, which leads to the mis-elimination of high-brightness useful information and causes new potential safety hazards for night driving. SUMMARY

[0004] The present application aims to provide a kind of heterologous image multi-region fusion night vision anti-halation method, to solve the problem that the existing heterologous image fusion anti-halation method uses the same rule to fuse high-brightness halation and high-brightness useful information, leading to the mis-elimination of high-brightness useful information.

[0005] To achieve the above-mentioned purpose, the technical solution adopted by the present application is a kind of heterologous image multi-region fusion night vision anti-halation method, comprising the following steps:

[0006] Step 1, registration of visible light and infrared images of the simultaneously collected night halo scene to obtain a pre-processed image with high time-space consistency;

[0007] Step 2, YUV color space transformation of the registered visible light image to obtain three components of luminance Y, chrominance U and saturation V;

[0008] Step 3, division of regions based on the visible light luminance component Y and the registered infrared image to obtain complete and accurate high-luminance halo region H , high-luminance useful information region L and low-illumination region B ;

[0009] Step 4, decomposition of the luminance component Y and the infrared image by fast non-subsample contourlet transform (FNSCT) to obtain low-frequency subband and high-frequency subband

[0010] Step 5, multi-region fusion of the low-frequency subband based on the region division result of Step 3 to obtain fusion low-frequency subbands of the high-luminance halo region, the high-luminance useful information region and the low-illumination region respectively

[0011] Step 6, fusion of the high-frequency subbands of the luminance component Y and the infrared image by statistical matching feature strategy to obtain a fused high-frequency component

[0012] Step 7, FNSCT reconstruction of the fused high-frequency and low-frequency components to obtain a new luminance component Y';

[0013] Step 8, YUV inverse transformation of the new luminance component Y' and the original hue U and saturation V components to output a fused image with complete high-luminance useful information, completely eliminated high-luminance halo information and low-illumination region conforming to human visual effect.

[0014] Further, in Step 3 above, the method for dividing regions is as follows:

[0015] Step (1), obtaining adaptive iterative threshold values T Y , T I for the luminance component Y and the registered infrared image respectively, and extracting high-luminance information V HB , I HB of the visible light and infrared images and high-luminance information V HB , I HB of the infrared image;

[0016] ​​Step (2), design a heterogeneous semantic segmentation model to identify V HB 、 HB Step (3), design a mask correlation algorithm based on Step (4), design a high-light halo area High-light useful information And infrared high-light useful information

[0017] Step (3), based on Design a mask correlation algorithm for fingerprint information matching, integrate the missed information while eliminating the misidentified information, and obtain the high-light halo area α H , high-light useful information area α L And low-illumination area α B .

[0018] Further, the specific steps of step 5 above are:

[0019] Step 5.1, using a low-frequency coefficient weight nonlinear adjustment strategy to fuse the high-light halo area α And of the visible light and infrared low-frequency sub-band H , obtain the fusion low-frequency sub-band of the halo area

[0020] Construct a nonlinear automatic adjustment strategy for the infrared low-frequency coefficient weight of the halo area :

[0021]

[0022] In the formula, C is a constraint factor; is the visible light low-frequency coefficient at point (x, y); is the infrared low-frequency coefficient weight corresponding to the halo critical point;

[0023] The mapped brightness low-frequency component is:

[0024]

[0025] In the formula, represents the low-frequency coefficient value of point projected to the interval [a, b];

[0026] The fusion low-frequency sub-band of the halo area is represented as:

[0027]

[0028] Step 5.2, based on the high-light useful information mutual mapping method, fuse the high-light useful information α And of the visible light and infrared low-frequency sub-band LFusion of the low-frequency sub-band of visible light and infrared light to obtain useful information

[0029] Highlight useful information at the post-fusion point (x, y) is expressed as:

[0030]

[0031] In the formula, respectively, the gray value of the visible light and infrared low-frequency sub-band in the region α L ;

[0032] Step 5.3, the low-illumination information fusion algorithm based on sparse representation fuses the low-illumination region α and of the visible light and infrared low-frequency sub-band to obtain the fusion low-frequency sub-band of the dark region B

[0033] Further, the specific steps of the above 5.3 include:

[0034] Step 5.3.1, obtaining a dictionary from a large number of patches of night vision halation images:

[0035] The dictionary learning model is expressed as:

[0036]

[0037] In the formula, ε>0 is a fault tolerance coefficient; is an unknown sparse vector; and D∈R n×m is an unknown dictionary to be learned.

[0038] Step 5.3.2, in a sliding window manner, the visible light and infrared low-frequency sub-band and are sequentially divided into multiple patches from the top left to the bottom right, and the visible light and infrared low-frequency patches at the i-th position are respectively expressed in the form of column vectors as

[0039] The mean of each vector is normalized to zero to obtain

[0040]

[0041]

[0042] In the formula, 1 represents a vector full of 1;

[0043] Step 5.3.3, according to the visible light and infrared low-frequency patches ​The obtained visible light, infrared sparse coefficient vectors are respectively

[0044]

[0045]

[0046] In the formula, D is a learned dictionary;

[0047] Step 5.3.4, merging by using a max-L1 rule And The obtained fusion sparse vector is

[0048]

[0049] The fusion result is

[0050]

[0051] In the formula, the mean value is:

[0052]

[0053] The above process is repeated for all source image patches, and all low-illumination pixel fusion vectors in the formula are obtained, that is Compared with the prior art, the application has the advantages and effects that:

[0054] 1. The heterogeneous image multi-region fusion method designed in the application can reasonably eliminate high-brightness halos while retaining useful high-brightness information, enrich the low-illumination region detail texture information, ensure the integrity of useful information of the fusion image in the night vision halo scene, and improve the overall visual effect.

[0055] 2. The partition fusion mechanism based on the importance degree of high-brightness information designed in the application can meet the needs of eliminating high-brightness halos while retaining useful high-brightness information in the night vision halo scene, and provides a guarantee for driving safety in the night halo scene.

[0056] 3. The application designs the fusion rules of the corresponding regions according to the fusion mechanism of different regions, adjusts the infrared low-frequency weight of the high-brightness halo region, mutually maps the high-brightness useful information, and represents the low-illumination information, improves the retention degree of the high-brightness useful information, improves the high-brightness halo elimination effect, and improves the visibility of the low-illumination region.

[0057] BRIEF DESCRIPTION OF DRAWINGS

[0058] Figure 1 ​is the visible light image of the night halo scene;

[0059] Figure 2 is the infrared image of the night halo scene;

[0060] Figure 3 is the fusion image obtained by YUV-Wavelet transform method;

[0061] Figure 4 is the fusion image obtained by improved IHS-Curvelet transform method;

[0062] Figure 5 is the fusion image obtained by YUV-FNSCT transform combined with visual saliency;

[0063] Figure 6 is the fusion image obtained by the present application;

[0064] Figure 7 is the workflow diagram of the present application. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical solutions and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application, and are not used to limit the present application.

[0066] Referring to Figure 1 and Figure 2 It can be seen that, in addition to high-luminance halos and high-luminance useful information, the visible light image of the night halo scene is almost impossible to observe low-illumination information, while the infrared image has no halos and rich background contours and textures, and the two have information complementarity. Referring to Figures 3 to 5 It can be seen that, in addition to high-luminance halos and high-luminance useful information, the visible light image of the night halo scene is almost impossible to observe low-illumination information, while the infrared image has no halos and rich background contours and textures, and the two have information complementarity. Referring to

[0067] The design idea of the present application is that: the visible light and infrared images are divided into high-luminance halo area, high-luminance useful information area and low-illumination area by high-luminance information recognition algorithm, and the fusion strategy of the corresponding area is designed according to the fusion mechanism of different areas. For the high-luminance halo area, a low-frequency coefficient weight nonlinear adjustment strategy is designed, and the infrared coefficient weight is dynamically adjusted according to the halo degree of the visible light image, so as to reasonably eliminate the halo; for the high-luminance useful information area, a high-luminance useful area pixel mutual mapping method is designed, which maps the high-luminance pixels of the visible light and infrared images to the fusion image, so as to completely retain the high-luminance useful information; for the low-illumination area, a low-illumination information fusion algorithm based on sparse representation is designed according to the low-rank nature of the pixels of the visible light and infrared images in the night low-illumination scene, so as to improve the visibility of the low-illumination area.

[0068] The present application provides a kind of heterologous image multi-region fusion night vision anti-blooming method, comprising the following steps: the visible light image and infrared image of night blooming scene are registered;YUV color space transformation is carried out to the registered visible light image, and luminance Y, chroma U and saturation V three components are obtained;High luminance blooming area α H , high luminance useful information area α L And low-illumination area α B Are obtained by using night vision blooming image high luminance information identification and segmentation method;Fast non-subsample contourlet transform (FNSCT) is used to decompose luminance component Y and infrared image, and visible light Y component, infrared image low-frequency sub-band And high-frequency sub-band Based on the region division result, the low-frequency sub-band Multi-region fusion is carried out, and the fusion low-frequency sub-band of high luminance blooming area, high luminance useful information area and low-illumination area is obtained Statistical matching feature strategy is used to fuse the high-frequency sub-band of luminance component Y and infrared image And The fused high-frequency component is obtained FNSCT reconstruction is carried out on the fused high-frequency component and low-frequency component , and new luminance component Y' is obtained;YUV inverse transformation is carried out on new luminance component Y' and original hue U and saturation V component, and the fusion image of high luminance useful information integrity, high luminance blooming information elimination thoroughly and low-illumination area meeting human eye visual effect is output.

[0069] Embodiment:

[0070] Referring to Figure 7 , the present application provides a kind of heterologous image multi-region fusion night vision anti-blooming method, and the method comprises the following steps:

[0071] Step 1, the visible light and infrared image of night blooming scene collected simultaneously are registered, and the preprocessed image with high time space consistency is obtained, and the specific steps used in the embodiment are as follows:

[0072] (1) the visible light image and infrared image of night blooming scene are read simultaneously using imread function;

[0073] (2) 4 pairs of feature points are selected from the two images using cpselect function, and saved in base_points and input_points;

[0074] (4) the transformation matrix is obtained by the feature points obtained and the transformation type used using cp2tform function;

[0075] (5) Apply the affine transformation using the imtransform function to the to-be-registered image with the transformation matrix;

[0076] (6) Crop the visible light picture to the same size as the infrared image using the imcrop function.

[0077] Step 2, YUV color space transformation is performed on the registered visible light image to obtain three components of brightness Y, chroma U and saturation V. The conversion formula from the RGB model to the YUV model is as follows:

[0078]

[0079] Step 3, the brightness component Y and the infrared image are divided into regions using the recognition and segmentation method of high brightness information of the night vision halo image, to obtain complete and accurate high brightness halo area α H , high brightness useful information area α L , and low illumination area α B . Specifically, the following steps are included:

[0080] Step 3.1, adaptive iterative threshold T Y , T I is calculated for the brightness component Y and the infrared image, respectively, and high brightness information V HB , I HB of the visible light and infrared images is extracted.

[0081] The adaptive iterative threshold T i+1 of the i+1th time is:

[0082] T i+1 = m (μ1+μ2) (2)

[0083] In the formula, m is an adaptive coefficient, μ1 and μ2 are the pixel mean values of the high brightness pixel area and the background area after thresholding T i , and are expressed as:

[0084]

[0085] In the formula, In(j) is the gray value of the jth pixel, N is the total number of pixels, N1 and N2 are the pixel numbers of the two areas, and N=N1+N2.

[0086] The initial threshold T0 takes the median value of the pixels of the image, and the iteration calculation is performed until the threshold no longer changes, and the latest threshold is the critical pixel threshold T. The high brightness information α HB of the image is extracted according to the critical pixel threshold T:

[0087]

[0088] Step 3.2, identifying highlight information V of the visible light image and the infrared image by using a heterogeneous semantic segmentation model HB 、 HB Step 3.2.1, performing down-sampling on the input visible light-infrared image by using a stack operation of convolution and pooling, constantly increasing the receptive field while extracting the convolution feature map U of the source image. In the convolution layer, the size of the convolution kernel is 3*3, the number of convolution kernels is 64, 128, 256, and 512, the activation function is Relu, and the size of the feature map U is expanded by using bilinear interpolation to obtain the up-sampling feature map V. Highlight useful information and infrared highlight useful information

[0089] Step 3.2.1, performing down-sampling on the input visible light-infrared image by using a stack operation of convolution and pooling, constantly increasing the receptive field while extracting the convolution feature map U of the source image. In the convolution layer, the size of the convolution kernel is 3*3, the number of convolution kernels is 64, 128, 256, and 512, the activation function is Relu, and the size of the feature map U is expanded by using bilinear interpolation to obtain the up-sampling feature map V.

[0090] Step 3.2.2, extracting the shallow position information of the target by using a channel attention module.

[0091] The convolution feature map U (size H*W*C) is subjected to global maximum pooling and average pooling to obtain two 1*1*C feature maps, the results of which are input into two weight-shared perceptrons MLP, the output features are added and subjected to an activation function to generate channel attention weights M C . Finally, the input feature map is multiplied by the channel attention weights to generate a channel attention feature map F', represented as:

[0092] F' = M c * U (5)

[0093] M c = σ [f2 (δ (f1 (Z avg )) ) + f2 (δ (f1 (Z max )) ) ] (6)

[0094] Z avg = AvgPool (U) (7)

[0095] Z max = MaxPool (U) (8)

[0096] In the formula, AvgPool represents global average pooling, which calculates the average value of pixels in each channel; MaxPool represents global maximum pooling, which retains the maximum value of pixels in each channel feature map; f1 is a fully connected layer with input channel C and output channel C / 16; f2 is a fully connected layer with input channel C / 16 and output channel C; δ is a ReLU function, and σ is a Sigmoid function.

[0097] Step 3.2.3, extracting the deep semantic information of the target by using a spatial attention module.

[0098] The channel attention feature map F' is taken as input, first average pooling and maximum pooling in the channel dimension are performed, two HxWxl feature maps are obtained, then the two feature maps are spliced and passed through a 7x7 convolution layer, and the result is subjected to a Sigmoid function to obtain a two-dimensional single-channel spatial attention weight M s .

[0099] Finally, the spatial attention weight M s is multiplied by the channel attention feature map F' to generate a feature map F" with both spatial and channel attention to suppress invalid features and focus on target regions only.

[0100] The new convolution feature map U' is represented as:

[0101] U' = F" = M s x F' (9)

[0102] M s = σ [f3 (Z AVG , Z MAX )] (10)

[0103] Z AVG = AVGPool (U) (11)

[0104] Z MAX = MAXPool (U) (12)

[0105] In the formula, AVGPool represents channel dimension average pooling, the corresponding pixel values of each channel feature map are added and averaged; MAXPool represents channel dimension maximum pooling, the maximum pixel value at the corresponding position of each channel feature map is retained; f3 is a convolution with a size of 7x7 and an output channel of 1.

[0106] The new convolution feature map U' is spliced with the corresponding up-sampling feature map V, and the up-sampling and feature fusion are repeated 4 times to complete the segmentation of high-brightness pixels in the visible light and infrared images, and the visible light high-brightness halo information is obtained Visible light high-brightness useful information and infrared high-brightness useful information

[0107] Step 3.3, based on the obtained visible light, infrared complementary high-brightness useful information The mask association algorithm based on fingerprint information matching is adopted to integrate the missed information and eliminate the misidentified information, and complete and accurate high-brightness halo area a H , high-brightness useful information area a L and low-illumination area a B .

[0108] Visible light, infrared image to be associated information F of the i-th connected domain i VI (x, y), F i IR (x, y) are respectively expressed as:

[0109]

[0110]

[0111] In the formula, are respectively the control fingerprints corresponding to the i-th connected domain of the visible light image and the infrared image; is expressed as a vector and information matching degree; τ is a matching threshold. Repeat the traversal of all connected domains, and obtain the high-brightness useful information F of the visible light image and the infrared image to be associated VI (x, y), F IR (x, y) and integrate, and obtain the high-brightness halo area α H , the high-brightness useful information area α L and the low-illumination area α B are respectively expressed as:

[0112] α L (x, y) = F VI (x, y) || F IR (x, y) (15)

[0113]

[0114]

[0115] In the formula, Y(x, y) is the pixel value of the visible light luminance component Y at the point (x, y).

[0116] Step 4, decompose the luminance component Y and the infrared image by using fast non-subsample contourlet transform (FNSCT), to obtain the low-frequency subband and the high-frequency subband

[0117] Step 4.1, the general formula of the low-frequency channel filter is:

[0118]

[0119] In the formula, indicates a low-pass filter in NSP; i indicates the number of FNSCT decomposition.

[0120] Step 4.2, the high-frequency channel filter set The general formula is:

[0121]

[0122] Where H(z) and U(z) are:

[0123]

[0124]

[0125] Where, H1(z 2m ) represents a high-pass filter; U ν (z 2i-m-1 ) represents a dual-channel fan filter, v is 0 or 1, and satisfies U1(z 2i-m-1 )=1–U0(z 2i-m-1 ); m = 0, 1, 2, ..., i-1; j is the number of directions of high-frequency decomposition, expressed as:

[0126]

[0127] Visible light and infrared images are decomposed into corresponding low-frequency sub-bands by R(z) and High frequency sub-band and

[0128] Step 5: Based on the regional division results of step 3, the low-frequency sub-band Perform multi-region fusion to obtain the fused low-frequency sub-bands of high-brightness halo area, high-brightness useful information area, and low-illumination area respectively.

[0129] Step 5.1: Use the nonlinear adjustment strategy of low-frequency coefficient weights to adjust the visible light and infrared low-frequency sub-bands and High brightness halo area α H Perform fusion to obtain the fused low-frequency sub-band in the halo area

[0130] In the high-brightness halo area, as the halo becomes stronger, the visible light low-frequency coefficient value becomes larger, the infrared low-frequency coefficient weight should be increased, and the visible light low-frequency coefficient weight should be reduced to better eliminate the halo; when gradually progressing to the halo center area, the corresponding infrared low-frequency coefficient weight should be large to completely eliminate the halo; at the halo critical point, in order to make the halo and the low-illuminance area connect naturally, and to prevent the halo from being eliminated excessively, a buffer zone should be reasonably set, and the infrared low-frequency coefficient of the low-illuminance area should be used as the benchmark for the infrared low-frequency coefficient weight of the halo area, and it should change slowly. According to the above idea, the infrared low-frequency coefficient weight of the halo area is constructed Nonlinear automatic adjustment strategy:

[0131]

[0132] where C is a constraint factor to adjust the vignetting elimination degree; is the visible low-frequency coefficient at point (x, y); is the infrared low-frequency coefficient weight corresponding to the vignetting critical point.

[0133] In order to make the infrared low-frequency weight smoothly transition from the high-brightness vignetting area to the low-illumination area, the luminance low-frequency component is mapped to the interval [a, b] based on the vignetting critical threshold. The mapped luminance low-frequency component is:

[0134]

[0135] where represents the low-frequency coefficient value of point projected to the interval [a, b].

[0136] The fusion low-frequency subband of the vignetting area is represented as:

[0137]

[0138] Step 5.2, the high-brightness useful information area α of the visible light and infrared low-frequency subbands is fused based on the high-brightness useful information mutual mapping method to obtain the fusion low-frequency subband of the useful information L .

[0139] In the high-brightness useful information area, the present application utilizes the complementary characteristics of the visible light and infrared images to take the modulus of the useful pixels of the visible light and infrared low-frequency subbands, and mutually map to the fusion image, so that the useful pixels in the fusion image are natural and complete.

[0140] The high-brightness useful information at point (x, y) after fusion is represented as:

[0141]

[0142] where are respectively the gray values of the visible light and infrared low-frequency subbands at point (x, y) in the area α L .

[0143] Step 5.3, the low-illumination information fusion algorithm based on sparse representation is used to fuse the low-illumination area α of the visible light and infrared low-frequency subbands to obtain the fusion low-frequency subband of the useful information BFusion is performed to obtain the fused low-frequency subband of the dark region

[0144] Step 5.3.1, obtaining a dictionary from patches of a large number of night-vision halo images. M training patches of size are rearranged into column vectors in R n , and each y i ∈R n is used to construct a training data set The dictionary learning model can be expressed as:

[0145]

[0146] In the formula, ε>0 is a fault tolerance coefficient; is an unknown sparse vector; and D∈R n×m is an unknown dictionary to be learned.

[0147] Step 5.3.2, in a sliding window manner, the visible light, infrared low-frequency subband and are sequentially divided into multiple patches from the top left to the bottom right, and the visible light, infrared low-frequency patches at the i-th position are represented in the form of column vectors as The mean of each vector is normalized to zero, and

[0148]

[0149]

[0150] In the formula, 1 represents a vector full of 1.

[0151] Step 5.3.3, according to the visible light, infrared low-frequency patches , the obtained visible light, infrared sparse coefficient vectors are

[0152]

[0153]

[0154] In the formula, D is the learned dictionary.

[0155] Step 5.3.4, using the "max-L1" rule to combine and to obtain the fused sparse vector:

[0156]

[0157] The fusion result is

[0158]

[0159] where the mean value is:

[0160]

[0161] Repeat the above process for all source image patches to obtain the fusion vector of all low-illumination pixels in , that is

[0162] Step 6, fuse the high-frequency sub-band of the luminance component Y and the infrared image using a statistical matching feature strategy and to obtain the fused high-frequency component

[0163] The high-frequency sub-band of the luminance component Y and the high-frequency sub-band of the infrared visual saliency image differ greatly, and the statistical matching feature strategy is used in the present application to enhance the texture detail information of the fusion image:

[0164]

[0165] wherein is the high-frequency coefficient of the fusion image.

[0166] Step 7, perform FNSCT reconstruction on the fused high-frequency and low-frequency components to obtain a new luminance component Y';

[0167] Design a reconstruction filter for each channel filter to form a reconstruction multi-channel filter bank r(z). Sum the convolution results of the fused high-frequency and low-frequency components with the corresponding reconstruction multi-channel filter to obtain a reconstructed image. The reconstructed new luminance component Y' is:

[0168]

[0169] Step 8, perform YUV inverse transformation on the new luminance component Y' and the original hue U and saturation V components to output a fusion image with complete high-brightness useful information, complete elimination of high-brightness halation information, and low-illumination area meeting the human eye visual effect. The conversion formula from the YUV model to the RGB model is:

[0170]

[0171] In order to verify the effectiveness of the present application in eliminating high-brightness halation while retaining high-brightness useful information, the visible light image and the infrared image of the collected night vision halation scene are subjected to image fusion experiments by using YUV-Wavelet transform, improved IHS-Curvelet transform, YUV-FNSCT transform and the method of the present application respectively, and the high-brightness halation area and the low-illumination area of the fused image are subjected to objective evaluation according to the method of the literature "Adaptive Partition Quality Evaluation of Night Vision Anti-halation Fused Image".

[0172] The halation elimination degree (D HE ) is used to evaluate the halation area, and the average gradient (AG), the spatial frequency (SF), the edge intensity (EI), the gray mean value (μ) and the edge retention degree (Q AB / F ) are used to evaluate the low-illumination area, wherein the greater D HE , the more complete the elimination of the image halation, the greater AG, the greater the change rate of the image detail contrast; the greater SF, the stronger the image spatial domain change; the greater EI, the more obvious the image edge detail, and the greater μ, the higher the brightness of the low-illumination area of the fused image; the greater Q AB / F , the better the retention of the edge of the original image.

[0173] In addition, the high-brightness useful information retention degree D LR of the present application is used to quantify the retention degree of the useful information in the fused image, which is represented by the ratio of the number of high-brightness pixels in the high-brightness useful information area of the fused image to the total number of pixels. The adaptive high-brightness information extraction threshold T Y of the visible light image is used to define the high-brightness pixels, and D LR is:

[0174]

[0175] In the formula, N GT is the total number of pixels in the high-brightness useful information area of the fused image corresponding to the Ground Truth; N FU {p(i,j)≥T Y} is the number of high-brightness pixels in the high-brightness useful information area of the fused image, and p(i,j) is the pixel value at point (i,j).

[0176] The visible light image (see Figure 1 ) and the infrared image (see Figure 2) can be seen that the visible light image has high brightness halation and road surface reflection, and part of the high brightness useful information (lane line) becomes obvious due to the halation; the infrared image has no halation, and the vehicle and pedestrian contour is obvious, and the details are rich. The objective evaluation indexes of the fusion images (see Figure 3 、 Figure 4 、 Figure 5 、 Figure 6 ) obtained by the YUV-Wavelet, improved IHS-Curvelet transform, YUV-FNSCT transform and the method are shown in Table 1.

[0177] It can be seen from Figure 3 that the YUV-Wavelet fusion image has good halation elimination effect, but the retention degree of high brightness useful information is low and the background contour is blurred. It can be seen from Figure 4 that the improved IHS-Curvelet transform retains part of the high brightness useful information, and the background texture is richer than that of the Wavelet transform, but the color information retention degree is low, and the target saliency is insufficient. It can be seen from Figure 5 that compared with the YUV-Wavelet and improved IHS-Curvelet transform, the fusion image obtained by the YUV-FNSCT algorithm has obvious improvement in background brightness, target saliency and image smoothness, and the halation is completely eliminated, but the high brightness useful information is not retained.

[0178] It can be seen from Figure 6 that the fusion image of the method completely and accurately retains the high brightness useful information, reasonably eliminates the halation, has moderate background brightness, and the key target is obvious, which is consistent with the human eye visual effect. Compared with Figure 3 , the fusion image of the method eliminates the fusion trace and retains the useful information; compared with Figure 4 , the fusion image of the method obviously improves the background brightness and target saliency, and the retained high brightness useful information is more complete; compared with Figure 5 , the fusion image of the method retains more rich color information, which is more consistent with the color sensitivity of the human eye vision, and the useful information is completely retained, which indicates that the method can eliminate the high brightness halation while retaining the high brightness useful information, and can effectively improve the visualization degree of the low illumination area.

[0179] Table 1 objective evaluation indexes of the heterologous fusion image

[0180]

[0181] It can be seen from Table 1 that in the high brightness area, the D HE index of the YUV-FNSCT algorithm is the highest, but its D LRThe reason is that the YUV-FNSCT algorithm fuses the high-brightness halo and the high-brightness useful information with the same rule, and the higher the halo elimination degree is, the lower the useful information retention degree is; and the D HE The index is only less than the YUV-FNSCT algorithm by 3.7%, but the D LR The reason is that the application partitions according to the useful degree of high-brightness information and designs the targeted fusion rule, so that the false elimination of the high-brightness useful information can be avoided. In the low-illumination area, the YUV-FNSCT algorithm has the highest μ index, the background is the brightest, and the application is the second with a difference of 3.8%, and the brightness is moderate; in addition to μ, the remaining indexes of the application are the highest, which indicates that the fusion algorithm of sparse representation can better retain the color, details and contour information of the source image.

[0182] From the above subjective and objective analysis, it can be seen that the multi-region fusion night anti-halo method for heterogeneous images provided by the application can reasonably eliminate the high-brightness halo information while retaining the high-brightness useful information, improve the target saliency and the color and texture information richness in the low-illumination area, and is more in line with the human visual effect, which is conducive to avoiding the potential safety hazards of night halo driving.

[0183] The above embodiments only exemplarily illustrate the principles and effects of the application, and for those skilled in the art, without departing from the concept of the application, a number of modifications and improvements can be made, which all belong to the protection scope of the application.

Claims

1. A night vision anti-halation method based on multi-region fusion of heterogeneous images, characterized in that: The following steps are involved: Step 1: register the visible light and infrared images of the night halo scene collected simultaneously to obtain a pre-processed image with high temporal and spatial consistency; Step 2: Perform YUV color space transformation on the registered visible light image to obtain three components: brightness Y, chroma U, and saturation V; Step 3: Based on the visible light brightness component Y and the infrared image, the high brightness information recognition and segmentation method of the night vision halo image is used to divide the area to obtain a complete and accurate high brightness halo area α H , high brightness useful information area α L and low illumination area α B ; Step 4: Use the fast non-subsampled contourlet transform FNSCT to decompose the brightness component Y and the infrared image to obtain the visible light Y component and the low-frequency subband of the infrared image. and high frequency sub-band Step 5: Based on the regional division results of step 3, the low-frequency sub-band Perform multi-region fusion to obtain the fused low-frequency sub-bands of high-brightness halo area, high-brightness useful information area, and low-illumination area respectively. Step 6: Use statistical matching feature strategy to fuse the brightness component Y with the high frequency sub-band of the infrared image and Get the fused high-frequency component Step 7: fusion of high and low frequency components Perform FNSCT reconstruction to obtain the new brightness component Y′; Step 8: Perform YUV inverse transformation on the new brightness component Y' and the original hue U and saturation V components to output a fused image with complete high-brightness useful information, complete elimination of high-brightness halo information, and low-light areas that meet the visual effects of the human eye.

2. The night vision anti-halation method for multi-region fusion of heterogeneous images according to claim 1 is characterized in that: In step 3, the method of dividing the area is: Step (1) Calculate the adaptive iterative threshold T for the brightness component Y and the registered infrared image respectively Y 、T I , extract high brightness information V of visible light and infrared images HB , I HB , high brightness information of infrared image V HB , I HB ; Step (2) Design a heterogeneous semantic segmentation model to identify V HB , I HB , obtain the high brightness halo area of ​​the visible light image High brightness useful information And infrared high brightness useful information Step (3) is based on Design a mask association algorithm for fingerprint information matching, integrate the missed information and eliminate the misidentified information, and obtain the high brightness halo area α H , high brightness useful information α L and low illumination area α B .

3. The night vision anti-halation method for multi-region fusion of heterogeneous images according to claim 1 or 2, characterized in that: The step 5 comprises the following steps: Step 5.1: Use the nonlinear adjustment strategy of low-frequency coefficient weights to adjust the visible light and infrared low-frequency sub-bands. and High brightness halo area α H Perform fusion to obtain the fused low-frequency sub-band in the halo area Step 5.2: Use the high brightness useful information mutual mapping method to map the visible light and infrared low frequency sub-bands. and High brightness useful information area α L Perform fusion to obtain the fused low-frequency sub-band with useful information High brightness useful information at the fused point (x, y) Expressed as: Where, Region α L In the figure, the grayscale value of the visible light and infrared low-frequency sub-bands at the point (x, y); Step 5.3: Low illumination information fusion algorithm based on sparse representation for visible light and infrared low frequency sub-bands and Low illumination area α B Perform fusion to obtain the fused low-frequency sub-band in the dark area 4. The night vision anti-halation method for multi-region fusion of heterogeneous images according to claim 3 is characterized in that: The specific steps of 5.3 include: Step 5.3.

1. Obtain a dictionary from a large number of patches of night vision halo images: The dictionary learning model is expressed as: Where, ε>0 is the fault tolerance coefficient; is an unknown sparse vector; and D∈R n×m is the unknown dictionary to be learned; Step 5.3.2: Use a sliding window to move the visible light and infrared low-frequency sub-bands and Multiple patches are segmented from the upper left to the lower right, and the visible light and infrared low-frequency patches at the i-th position are represented in the form of column vectors: Normalizing the mean of each vector to zero, we get Where 1 represents a vector of all 1s; Step 5.3.3: Based on visible light and infrared low-frequency patches The obtained visible light and infrared sparse coefficient vectors are Where D is the learned dictionary; Step 5.3.4: Merge using the "max-L1" rule and Get the fused sparse vector The fusion result is In the formula, the mean for: Repeat the above process for all source image patches to obtain All low-light pixel fusion vectors in , that is,

Images