Method for anti-glare night vision of heterologous image fusion based on low-frequency sequence generation

Abstract

The present invention relates to a method for fusing heterologous images and anti-glare in night vision based on low-frequency sequence generation, which is used to solve the problem of mis-elimination of important and useful information with high brightness in the existing heterologous image fusion anti-glare technology. It can effectively retain the useful information with high brightness and reasonably eliminate the glare information with high brightness, improving the integrity of the useful information in the fused image and the overall visual effect in the night vision glare scene. Through the constructed low-frequency sequence generation model, the present invention fuses the visible light brightness component and the low-frequency component of the infrared image into a sequence of brightness low-frequency component images with different degrees of glare elimination; designs a membership function for maximizing visual information, assigns a larger weight to the area where the brightness conforms to the human eye vision according to the illumination estimation result, adjusts the different brightness areas of the fused image to the range suitable for human eye observation, so that all high-brightness areas no longer have glare, which not only solves the problem of glare elimination, but also achieves the purpose of effectively retaining important information with high brightness.

Description

Technical Field

[0001] The present invention belongs to the technical field of night vision anti-glare, and mainly relates to a method for fusing heterologous images for night vision anti-glare based on low-frequency sequence generation. Background Art

[0002] According to statistics, traffic accidents caused by the abuse of high beams during night-time oncoming vehicle meetings, which lead to driver glare, account for nearly half of the total night-time traffic accidents. The night vision anti-glare technology of heterologous image fusion combines the advantages of no glare in infrared images and rich color and detail information in visible light images. The fused image has more thorough glare elimination, rich color and detail information, and can effectively improve the safety of night driving.

[0003] The literature "Automobile Anti-Glare System for Infrared and Visible Light Image Fusion" weighted and fused the low-frequency components obtained by wavelet transform, reducing glare interference. However, wavelet transform lacks the retention of edge information, the clarity of the fused image is poor, and with the adopted weighting strategy, glare information is still involved in the fusion, resulting in insufficient glare elimination in strong glare scenarios. The literature "Improved IHS-Curvelet Transform Fusion Method for Anti-Glare of Visible Light and Infrared Images" utilizes the anisotropic characteristics of Curvelet transform to improve the clarity of the fused image. By designing an automatic adjustment strategy for low-frequency coefficient weights, higher infrared low-frequency weights are assigned to the high-brightness regions of the fused image, and glare is eliminated more thoroughly. However, some important and useful high-brightness information is also eliminated, creating new traffic hazards while eliminating glare and being unfavorable for safe night driving of drivers. Summary of the Invention

[0004] The purpose of the present invention is to provide a method for fusing heterologous images for night vision anti-glare based on low-frequency sequence generation, which is used to solve the problem of mis-elimination of important and useful high-brightness information in the existing heterologous image fusion anti-glare technology, and can reasonably eliminate high-brightness glare information while retaining important and useful high-brightness information, improving the visual effect of the fused image.

[0005] To achieve the above purpose, the technical solution adopted by the present invention is: a method for fusing heterologous images for night vision anti-glare based on low-frequency sequence generation, including the following steps

[0006] Step 1: Register the visible light and infrared images of the night-time glare scene collected simultaneously to obtain preprocessed images;

[0007] Step 2: Perform YUV color space transformation on the preprocessed images to obtain three components: luminance Y, chrominance U, and saturation V;

[0008] Step 3: Perform Curvelet decomposition on the luminance component Y and the infrared image to obtain the low-frequency component L Y and high-frequency components Low-frequency component L of the infrared image IR and the high-frequency component

[0009] Step 4: Construct a low-frequency sequence generation model, and use the luminance low-frequency component L Y and the infrared low-frequency component L IR to fuse into luminance low-frequency sequences with different degrees of veiling glare elimination

[0010] Step 5: Adopt the strategy of taking the larger modulus value to fuse the luminance component Y and the high-frequency component of the infrared image and to obtain the fused high-frequency component

[0011] Step 6: Perform Curvelet reconstruction on each component in the low-frequency sequence respectively with the fused high-frequency component to obtain a set of anti-veiling glare luminance component sequences with different degrees of veiling glare elimination;

[0012] Step 7: Perform illumination estimation through convolution of the multi-scale Gaussian kernel function and the anti-veiling glare luminance component sequence to obtain the illumination component sequence {I (0) , I (1) ,..., I (n)};

[0013] Step 8: Design a membership function for maximizing visual information, and assign larger weights to the regions suitable for human eye observation according to the illumination component sequence {I (0) , I (1) ,..., I (n)}, so as to determine the weights of each component in the anti-veiling glare luminance component sequence participating in the fusion, and then perform weighted summation on the sequence to obtain the new luminance component Y';

[0014] Step 9: Perform YUV inverse transformation on the new luminance component Y' and the original hue U and saturation V components, and output the fused image.

[0015] Furthermore, the specific steps of the above Step 4 are

[0016] Step 4.1: Calculate the constraint factor for the nth iteration

[0017] Step 4.2: Calculate the veiling glare threshold β for the nth time n , and divide the low-frequency component L n into veiling glare and non-veiling glare regions through β Y ;

[0018] If the low-frequency component L Y at a certain point L Y (x, y) ≥ β n , then the pixel (x, y) is in the blooming area in the nth generated low-frequency component;

[0019] If L Y (x, y) < β n , then the pixel is in the non-blooming area;

[0020] Step 4.3. In the non-blooming area, design a pixel mean prior strategy and calculate the weight value of the infrared low-frequency coefficient in the nth iteration

[0021] Step 4.4. In the blooming area, design a non-linear infrared low-frequency weight adjustment strategy and calculate the weight value of the infrared low-frequency coefficient in the nth iteration

[0022] Step 4.5. Generate the nth fused low-frequency component

[0023]

[0024] In the formula, L IR (x, y) is the pixel value of the infrared low-frequency component at (x, y). ω IR (n) is the infrared low-frequency weight matrix in the nth iteration, expressed as:

[0025]

[0026] Step 4.6. Judge the constraint factor Whether it is satisfied

[0027] If not satisfied, repeat Steps 4.1 to 4.5 to generate a new low-frequency component;

[0028] If satisfied, stop the iteration and output the low-frequency sequence

[0029] Furthermore, the specific steps of the above Step 8 are

[0030] Step 8.1. Calculate the anti-blooming sequence The weight matrix W n (x, y) of the nth component in it is:

[0031]

[0032] Step 8.2. The new luminance component Y' is:

[0033]

[0034] In the formula, M represents the number of components of the anti-halos sequence Y AH .

[0035] Furthermore, in step 4.1 above, calculate the constraint factor of the nth iteration

[0036]

[0037] In the formula, N represents the number of pixels of the low-frequency luminance component L Y , N * represents the number of pixels in L Y that are greater than its pixel mean value.

[0038] Furthermore, in step 4.2 above, calculate the halos threshold β of the nth time n :

[0039]

[0040] In the formula, and respectively represent the pixel mean value and the maximum value of L Y , and is the initial constraint factor.

[0041] Furthermore, in step 4.3 above, in the non-halos region, calculate the weight value of the infrared low-frequency coefficient of the nth iteration

[0042]

[0043] In the formula, respectively represent the pixel mean values of the low-frequency luminance component L Y and the infrared low-frequency component L IR in the non-halos region, is the initial low-frequency coefficient weight value in the non-halos region.

[0044] Furthermore, the above takes 0.3.

[0045] Furthermore, in step 4.4 above, in the halos region, calculate the weight value of the infrared low-frequency coefficient of the nth iteration

[0046]

[0047] In the formula, p is the pixel value of the low-frequency luminance component L Y at (x, y), and r is the adjustment factor.

[0048] Taking the halos critical threshold as the benchmark, the low-frequency luminance component L YThe pixel values are mapped to the interval [a, b], and the low-frequency luminance component L Y ′(x, y) is as follows:

[0049]

[0050] Furthermore, the above r takes 75; the mapping interval is [0, 5].

[0051] Compared with the prior art, the advantages and effects of the present invention are as follows:

[0052] 1. The present invention can effectively retain the useful information of high luminance and reasonably eliminate the halation information of high luminance, improving the integrity of the useful information and the overall visual effect of the fused image in the night halation scene.

[0053] 2. The low-frequency sequence generation model constructed by the present invention can output image sequences with different degrees of halation information elimination and strong complementarity of useful information based on the low-frequency information of the luminance component, which is beneficial to improving the information richness of the fused image and eliminating halation. Among them, the designed iterative constraint factor can dynamically adjust the number of generated low-frequency sequences according to the overall luminance of the visible light image, reducing the number of generated sequences and improving the processing efficiency on the premise of ensuring the quality of the fused image; the designed low-frequency component halation threshold can generate different halation thresholds for each component of the low-frequency sequence, dynamically dividing the halation and non-halation regions, better including the range of the real halation critical point, and ensuring the natural connection between the halation and non-halation regions of the fused image; the designed pixel mean prior strategy can make the brighter components included in the non-halation region of the generated low-frequency component account for a larger proportion in the whole sequence, ensuring that the fused image has higher luminance and more significant effective information in the non-halation region; the designed non-linear infrared low-frequency weight adjustment strategy can timely adjust the change curve of the infrared low-frequency weight with the number of iterations, and the infrared low-frequency component weight is non-linearly adjusted with the luminance low-frequency component in the halation region, ensuring that the halation elimination degree of each component in the generated low-frequency sequence is different.

[0054] 3. The designed visual information maximization membership function of the present invention can assign larger weights to the regions suitable for human eye observation in the image sequence according to the illumination estimation result, ensuring the overall visual effect of the fused image. Description of the Drawings

[0055] Figure 1 is the visible light image of the night halation scene;

[0056] Figure 2 is the infrared image of the night halation scene;

[0057] Figure 3 is the fused image obtained by the YUV-Wavelet transform method;

[0058] Figure 4 is the fused image obtained by improving the IHS-Curvelet transform method;

[0059] Figure 5 is Figure 1 the clustering map of the visible light image;

[0060] Figure 6 is the workflow diagram of the low-frequency sequence generation model;

[0061] Figure 7 is the weight curve of the infrared low-frequency coefficient in the veiling glare area;

[0062] Figure 8 the veiling glare-resistant luminance component sequence;

[0063] Figure 9 the new luminance component;

[0064] Figure 10 is the membership function for maximizing visual information;

[0065] Figure 11 is the fused image obtained by the present invention;

[0066] Figure 12 is the workflow block diagram of the present invention. Specific embodiments

[0067] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0068] The present invention will be described in detail below with reference to the accompanying drawings and embodiments.

[0069] The design idea of the present invention is to fuse the visible light luminance component and the infrared image into an image sequence with different degrees of veiling glare elimination and different background luminances. By using the complementarity between the image sequences, through the constructed low-frequency sequence generation model, the visible light luminance component and the low-frequency component of the infrared image are fused into a luminance low-frequency component image sequence with different degrees of veiling glare elimination; a membership function for maximizing visual information is designed, and a larger weight is assigned to the area where the luminance conforms to the human eye vision according to the illumination estimation result, and different luminance areas of the fused image are adjusted to the range suitable for human eye observation, so that all high-luminance areas no longer have veiling glare, which not only solves the problem of veiling glare elimination but also achieves the purpose of effectively retaining important information in high-luminance areas.

[0070] See Figure 12 , the present invention provides a heterologous image fusion night vision veiling glare-resistant method based on low-frequency sequence generation, and the method includes the following steps:

[0071] Step 1: Register the visible light and infrared images of the night glow scene collected simultaneously to obtain preprocessed images, and the registered images have high spatio-temporal consistency;

[0072] Step 2: Perform YUV color space transformation on the preprocessed images to obtain three components: luminance Y, chrominance U, and saturation V. The conversion formula from the RGB model to the YUV model is as follows:

[0073]

[0074] Step 3: Perform Curvelet decomposition on the luminance component Y and the infrared image, that is:

[0075]

[0076] In the formula, f[t1,t2] represents the input, represents the Curvelet function with decomposition scale j, direction l, and position k. The low-frequency component L Y of the luminance component Y and the high-frequency component the low-frequency component L IR of the infrared image and the high-frequency component

[0077] Step 4: Construct a low-frequency sequence generation model, and fuse the luminance low-frequency component L Y and the infrared low-frequency component L IR into luminance low-frequency sequences with different degrees of glow elimination

[0078] First, divide the luminance low-frequency component L n into a glow and a non-glow region through the glow threshold β Y , and generate the low-frequency components of the corresponding regions according to the low-frequency weight adjustment strategies of the two regions. Then, weight them with the infrared low-frequency component L IR to generate the low-frequency component L F of the fused image. Then, judge whether the constraint conditions are met. If not, continue to iterate to generate new low-frequency components until the constraint conditions are met and the iteration stops, and output low-frequency sequences with different degrees of glow elimination. Its working process (see Figure 6 ) specifically includes the following 6 steps:

[0079] Step 4.1: Calculate the constraint factor for the nth iteration

[0080]

[0081] In the formula, N represents the number of pixels of the luminance low-frequency component L Y , N * represents LY The number of pixels in it that are greater than its pixel mean value.

[0082] As can be seen from Equation (3), it decreases as the number of iterations n increases. When the visible light image is brighter as a whole, N * is larger, then it is smaller, and the generated low-frequency component sequence is less. That is, when the visual effect of the visible light image is better, only a few low-frequency component sequences with different degrees of veiling glare elimination are needed to synthesize a high-quality image. On the contrary, when the image is darker as a whole, N * is smaller, then it is larger, and more images with different degrees of information content are often needed to synthesize a high-quality image. Therefore, more low-frequency component sequences with different degrees of veiling glare elimination need to be generated.

[0083] Step 4.2: Calculate the veiling glare threshold β at the nth time n , and divide the low-frequency component L n into veiling glare and non-veiling glare regions through β Y . β n is expressed as:

[0084]

[0085] In the formula, and respectively represent the pixel mean value and the maximum value of L Y , and is the initial constraint factor.

[0086] Since the pixel values at the junction of the veiling glare and non-veiling glare regions change continuously and there is no obvious boundary, the idea of selecting the veiling glare threshold in the present invention is: by iterating to make the constraint factor decrease, so that the veiling glare threshold of each generated low-frequency component is different, and then the range of region division is also different, which can better include the range of the true veiling glare critical point. As can be seen from Equation (4), β n and are in an inverse proportional relationship. As the number of iterations n increases, β decreases and n increases, and the area of the divided veiling glare region gradually becomes smaller.

[0087] If for a certain point L Y in the low-frequency component L Y (x, y) ≥ β n , then the pixel (x, y) is in the veiling glare region in the nth generated low-frequency component; on the contrary, if L Y (x, y) < β n , then the pixel is in the non-veiling glare region.

[0088] Step 4.3. In the non-halos area, design a pixel mean prior strategy to calculate the weight of the infrared low-frequency coefficient at the nth iteration It is expressed as:

[0089]

[0090] In the formula, respectively represent the pixel means of the low-frequency luminance component L Y and the infrared low-frequency component L IR in the non-halos area, is the initial low-frequency coefficient weight in the non-halos area.

[0091] In the generated low-frequency sequence, each low-frequency component contains different visible and infrared information in the non-halos area, and the overall trend is towards a brighter image. Generally, the higher the pixel mean of an image, the higher the weight of its participation in low-frequency fusion, and the better the visual effect of the fused image. However, determining the weight of the fused image only by the pixel mean will cause some local information loss. Therefore, the generated low-frequency sequence needs to contain low-frequency components with different weights, and the proportion of brighter components should be relatively large in the total number. Based on this idea, the present invention determines the brighter image by comparing the pixel means of the low-frequency luminance component L Y and the infrared low-frequency component L IR in the non-halos area, and makes the information of the brighter image contained in the subsequent generated low-frequency components gradually increase. After multiple calculations and optimizations, the present invention takes 0.3.

[0092] Step 4.4. In the halos area, design a non-linear infrared low-frequency weight adjustment strategy to calculate the weight of the infrared low-frequency coefficient at the nth iteration It is expressed as:

[0093]

[0094] In the formula, p is the pixel value of the low-frequency luminance component L Y at (x, y). r is an adjustment factor used to adjust the shape of the infrared low-frequency weight curve.

[0095] To ensure that the degree of halos elimination of each low-frequency component in the generated low-frequency sequence is different in the halos area, the basic idea of the low-frequency coefficient weight adjustment is: the weight of the infrared low-frequency component L IR increases with the increase of the pixel value of the low-frequency luminance component L Y to better eliminate halos; it changes gently at the halos critical point to establish a buffer area between the halos and non-halos areas and avoid the phenomenon of light and dark splitting; when approaching the center of the halos, L IRTake the larger value of the weight to completely eliminate the veiling glare. On the other hand, as the number of iterations changes, adjust the change curve of the infrared low-frequency weight in a timely manner to generate low-frequency components with different degrees of veiling glare elimination. Through actual tests, when r takes 75, the slope at each point of the curve meets the design requirements. When p is fixed, is proportional to , and as the number of iterations n increases, and both decrease.

[0096] In order to make the infrared low-frequency weight smoothly transition from the non-veiling glare area to the veiling glare area, based on the veiling glare critical threshold, map the pixel values of the luminance low-frequency component L Y to the interval [a, b]. The mapped luminance low-frequency component L Y ′(x, y) is:

[0097]

[0098] Through actual tests, the present invention selects the mapping interval as [0, 5], that is, a = 0, which is the veiling glare critical point; b = 5, which is the brightest pixel of L Y . From the change trend of (see Figure 7 ), it can be seen that the change trends of for different curves are consistent. The change is gentle at the veiling glare critical point, and it increases as L Y ′(x, y) increases. The value is larger when approaching the veiling glare center, indicating that in the fused low-frequency component, the higher the visible light luminance in the area, the higher the infrared low-frequency weight and the higher the veiling glare elimination degree; for a determined L Y ′(x, y), as the number of iterations n increases, decreases, and is also smaller, indicating that the veiling glare elimination degree of the generated low-frequency fusion component sequence decreases from large to small. Substitute the L Y ′(x, y) obtained from Equation (7) into p in Equation (6) to obtain the infrared low-frequency weight matrix in the veiling glare area at the nth iteration

[0099] Step 4.5. Generate the nth fused low-frequency component

[0100]

[0101] In the formula, L IR (x, y) is the pixel value of the infrared low-frequency component at (x, y). ω IR (n) is the infrared low-frequency weight matrix at the nth iteration, expressed as:

[0102]

[0103] Step 4.6, determine the constraint factor Whether it satisfies: If not, repeat Steps 4.1 to 4.5 to generate a new low-frequency component; if satisfied, stop the iteration and output the low-frequency sequence

[0104] Step 5, adopt the strategy of taking the larger modulus value to fuse the high-frequency components to obtain the fused high-frequency components Retain more and clearer texture detail information. The fused high-frequency components Can be expressed as:

[0105]

[0106] Step 6, for each component in the low-frequency sequence Perform Curvelet reconstruction with the fused high-frequency components To obtain a set of anti-halos luminance component sequences with different degrees of halo elimination The discrete Curvelet transform expression in the frequency domain is:

[0107]

[0108] In the formula, Represents the input in the frequency domain, Is the frequency domain Curvelet function.

[0109] Anti-halos luminance component sequence (See Figure 8 ) Among them, the components with a high degree of halo elimination have complete halo elimination, but the high-brightness useful information is miseliminated; the components with a low degree of halo elimination, although the halo elimination is insufficient, better retain the high-brightness useful information, and the visual effect in the dark areas affected by the halo is better.

[0110] Step 7, perform illumination estimation through the convolution of the multi-scale Gaussian kernel function and the anti-halos luminance component sequence To obtain the illumination component sequence {I (0) , I (1) ,..., I (n)}. The nth illumination component I (n) (x, y) is:

[0111]

[0112] In the formula, K is the number of scales, and G k (x, y) represents the kth scale Gaussian function, expressed as:

[0113]

[0114] Wherein, μ is the normalization constant, σ is the scale factor, and "*" represents the convolution operation. Considering the accuracy of illumination component extraction and the operation balance, the present invention takes K = 3, and the selected scale factors are σ = 50, 150, 200 respectively.

[0115] Step 8, refer to Figure 10 Design the membership function of visual information maximization, and assign greater weights to the regions suitable for human eye observation according to the illumination component sequence {I (0) , I (1) ,..., I (n)}, so as to determine the weights of each component in the anti-halos luminance component sequence participating in the fusion, and then perform weighted summation on the sequence to obtain the new luminance component Y'. Specifically, it includes 2 steps:

[0116] Step 8.1, calculate the weight matrix W of the nth component in the anti-halos sequence n (x, y) as:

[0117]

[0118] A simple definition of whether the exposure of a pixel point is good is the distance between the pixel and the middle value of the luminance component. The closer the pixel value is to the middle value, the more the exposure conforms to the human eye visual observation. The present invention designs a triangular membership function of visual information maximization, whose domain is [0, 255], the range is [0, 1], and it is symmetric about the middle value of the domain, and assigns greater weights to the pixel points whose pixel values are close to 128. The membership function is as Figure 8 shown.

[0119] Step 8.2, the new luminance component Y' is:

[0120]

[0121] Wherein, M represents the number of components of the anti-halos sequence Y AH .

[0122] The present invention utilizes the complementarity of information between components with different degrees of halo elimination, and assigns larger weights to the pixels that conform to the human eye visual observation in the anti-halos luminance components through the designed membership function of visual information maximization, so that each region of the synthesized new luminance component Y' (see Figure 9 ) is suitable for human eye observation.

[0123] Step 9: Perform YUV inverse transformation on the new luminance component Y' and the original hue U and saturation V components, and output a fused image that meets the visual effects of the human eye, such as complete high-brightness useful information, thorough elimination of high-brightness veiling glare information, and rich color detail information in dark areas. The conversion formula from the YUV model to the RGB model is as follows:

[0124]

[0125] The following gives a specific simulation example. The simulation conditions for this embodiment are: Windows 10 operating system and MATLAB software.

[0126] Main content: Register the visible light and infrared images of the night veiling glare scene; perform YUV color space transformation on the registered visible light image to obtain three components: luminance Y, chrominance U, and saturation V; then perform Curvelet decomposition on the luminance component Y and the infrared image to obtain their respective low-frequency and high-frequency components; generate a low-frequency luminance sequence with different degrees of veiling glare elimination through the low-frequency sequence generation model; adopt the strategy of taking the larger modulus value to obtain the fused high-frequency component; perform Curvelet reconstruction on each component in the low-frequency luminance sequence with the fused high-frequency component to obtain a sequence of luminance components with different degrees of veiling glare elimination; obtain the illumination component sequence corresponding to the anti-veiling glare luminance component sequence through illumination estimation; determine the weight of each component in the anti-veiling glare luminance component sequence participating in the fusion through the visual information maximization membership function, and then perform weighted summation on the sequence to obtain a new luminance component; perform YUV inverse transformation on the new luminance component Y' and the original hue U and saturation V components to obtain the fused image. The specific steps are as follows:

[0127] I. Image registration:

[0128] (1) Use the imread function to read the visible light image and infrared image of the night veiling glare scene;

[0129] (2) Use the cpselect function to select 4 pairs of feature points from the two images and save them in base_points and input_points;

[0130] (4) Use the cp2tform function to obtain the transformation matrix through the obtained feature points and the transformation type used;

[0131] (5) Use the imtransform function to apply the transformation matrix to the image to be registered for affine transformation;

[0132] (6) Use the imcrop function to crop the visible light picture to the same size as the infrared image.

[0133] II. YUV transformation:

[0134] Convert the visible light image from the RGB model to the YUV model according to formula (1) to obtain the luminance component Y, chrominance component U, and concentration component V of the visible light image.

[0135] III. Curvelet Decomposition:

[0136] Perform Curvelet decomposition on the luminance component Y and the infrared image respectively according to formula (2) to obtain the low-frequency components and high-frequency components of the luminance component Y and the infrared image respectively.

[0137] IV. Generation of Anti-Halos Luminance Component Sequence: According to the working process of the low-frequency sequence generation model, execute formulas (3) to (9) to obtain luminance low-frequency sequences with different degrees of halo elimination.

[0138] V. According to formula (10), the high-frequency components of the luminance component Y and the infrared image and are fused to obtain the fused high-frequency components.

[0139] VI. According to formula (11), each component of the luminance low-frequency sequence is Curvelet reconstructed with the fused high-frequency components to obtain the anti-halos luminance component sequence.

[0140] VII. Perform illumination estimation on the anti-halos luminance component sequence according to formulas (12) and (13) to obtain the illumination component sequence {I (0) , I (1) ,..., I (n)}.

[0141] VIII. Solve the weight matrix of each component of the anti-halos luminance component sequence {I (0) , I (1) ,..., I (n)} according to formula (14); synthesize the image sequence according to formula (15) to obtain the new luminance component Y'.

[0142] IX. Perform YUV inverse transformation on the new luminance component Y' and the original U and V components according to formula (16) to obtain the final fused image.

[0143] To verify the effectiveness of the present invention in retaining high-brightness useful information and eliminating high-brightness halation, for the visible light images and infrared images of the night vision halation scene collected, the YUV-Wavelet transform, the improved IHS-Curvelet transform and the method of the present invention are respectively used for the image fusion anti-halation experiment comparison, and the fusion images are objectively evaluated according to the method in the literature "Adaptive Partition Quality Evaluation of Night Vision Anti-halation Fusion Images", and the halation elimination degree (D HE ) is selected to evaluate the anti-halation effect of the halation area. The larger D HE is, the more thorough the image halation elimination is; the average gradient (AG), spatial frequency (SF), edge intensity (EI), gray mean (μ), edge retention degree (Q AB / F ) and other indicators are selected to evaluate the visual effect of the non-halation area. AG reflects the change rate of the detail contrast of the image. The larger the value, the more detail information the image in the non-halation area contains. SF reflects the change in the image spatial domain. The larger the value, the clearer the image. EI reflects the amplitude of the image edge gradient. The larger the value, the more obvious the image edge details are. μ represents the average gray value of the image. The larger the value, the higher the brightness of the non-halation area. Q AB / F reflects the degree of edge retention of the fusion image to the original image. The larger the value, the better the edge retention degree of the original image.

[0144] From the visible light image (see Figure 1 ) and infrared image (see Figure 2 ) of the night vision halation scene, it can be seen that the halation phenomenon in the visible light image is relatively serious, but some useful information such as lane lines becomes more obvious due to the halation, and information such as the front row of pedestrians is more prominent; the outlines of pedestrians and vehicles in the infrared image are clear, but some important information (such as lane lines, etc.) and other detail information are lacking. The objective evaluation indicators of the fusion images obtained by YUV-Wavelet, the improved IHS-Curvelet transform and the method of the present invention (see Figure 3 , Figure 4 , Figure 11 ) are shown in Table 1.

[0145] Table 1 Objective evaluation indicators of fusion images in night halation scenes

[0146]

[0147] From Figure 3 it can be seen that although the fusion image of YUV-Wavelet weakens the halation, the halation is still obvious, the overall image clarity is poor, and the pedestrians, trees around the vehicles, background buildings, etc. are relatively blurred. From Figure 4It can be seen that the fused image obtained by improving the IHS-Curvelet transform eliminates veiling glare more thoroughly. However, due to excessive elimination of high-brightness information, some lane line information is erroneously eliminated, leading to new traffic hazards. At the same time, there is severe brightness splitting at the junction of veiling glare and non-veiling glare areas, and the overall visual effect is poor. From the clustering map of visible light images in the night vision veiling glare scenario (see Figure 5 ), it can be seen that the high-brightness region contains a large amount of veiling glare information and a small amount of important useful information (such as lane lines, etc.). Existing night vision anti-veiling glare methods for fusing visible light and infrared images distinguish veiling glare and non-veiling glare information based on brightness, making it difficult to avoid the erroneous elimination of high-brightness useful information.

[0148] From Figure 11 it can be seen that the fused image of the present invention better retains the high-brightness lane lines, has a good effect on eliminating high-brightness veiling glare, and the overall brightness of the image is moderate, which is suitable for human eye visual observation. Compared with Figure 3 , the fused image of the present invention significantly reduces veiling glare interference, the vehicle contour and pedestrians are clearly visible, and the brightness of the non-veiling glare area is relatively high, indicating that the method of the present invention can improve the clarity of the image and the brightness of the dark area while effectively eliminating veiling glare. Compared with Figure 4 , the high-brightness lane lines of the fused image of the present invention are better retained, the connection between the veiling glare and non-veiling glare areas is natural, and the image brightness is moderate, indicating that the present invention can effectively retain high-brightness useful information, reasonably eliminate high-brightness veiling glare, the useful information of the image is complete, the visual effect is good, and it better solves the veiling glare problem during night driving, which is beneficial to night driving safety.

[0149] As can be seen from Table 1, in the veiling glare area, although the D HE of the present invention is slightly lower than that of the improved IHS-Curvelet transform method, this is because the improved IHS-Curvelet transform method excessively eliminates high-brightness information, resulting in a falsely high D HE index, and at the same time causing the non-veiling glare area to become darker, the change rate of image pixel values to become smaller, and the various indexes of the non-veiling glare area are lower than those of the present invention. In the non-veiling glare area, for the objective indexes of the fused image of the present invention, except that the mean value μ is lower than that of YUV-Wavelet, the other indexes are higher than those of the other two methods. This is because the YUV-Wavelet transform method does not eliminate veiling glare thoroughly, resulting in a falsely high mean value reflecting the image brightness, verifying that the method of the present invention can effectively improve the overall brightness of the fused image in the non-veiling glare area, enrich the detail information, and make the edge texture clearer.

[0150] From the above analysis, it can be seen that the proposed method for fusing heterogeneous images based on generating low-frequency sequences for night vision anti-veiling glare of the present invention can reasonably eliminate high-brightness veiling glare information while retaining high-brightness useful information, improve the brightness of the dark area of the image, enhance detail information such as color texture, improve the overall visual effect of the fused image, and is beneficial to improving night driving safety.

[0151] The above embodiments merely illustrate the principles and effects of the present invention. For those of ordinary skill in the art, without departing from the creative concept of the present invention, several modifications and improvements can still be made, and these all fall within the protection scope of the present invention.

Claims

1. A method for fusing heterologous images to resist halo in night vision based on low-frequency sequence generation, characterized in that: including the following steps Step 1: Register the visible light and infrared images of the night halo scene collected simultaneously to obtain a preprocessed image; Step 2: Perform YUV color space transformation on the preprocessed image to obtain three components: luminance Y, chrominance U, and saturation V; Step 3: Perform Curvelet decomposition on the luminance component Y and the infrared image to obtain the low-frequency component L of the luminance component Y Y and the high-frequency component The low-frequency component L of the infrared image IR and the high-frequency component Step 4: Construct a low-frequency sequence generation model, and fuse the luminance low-frequency component L Y with the infrared low-frequency component L IR to form luminance low-frequency sequences with different degrees of veiling glare elimination The specific steps for generating the low-frequency sequence generation model are Step 4.1: Calculate the constraint factor for the nth iteration Step 4.2: Calculate the veiling glare threshold β for the nth time n , and based on β n divide the low-frequency component L Y into veiling glare and non-veiling glare regions; If the low-frequency component L Y at a certain point L Y (x, y) ≥ β n , then the pixel (x, y) is in the veiling glare area in the nth generated low-frequency component; If L Y (x, y) < β n , then the pixel is in the non-halos region; Step 4.3: In the non-halos region, design a pixel mean prior strategy and calculate the weight of the infrared low-frequency coefficient at the nth iteration Step 4.

4. In the veiling glare area, design a non-linear infrared low-frequency weight adjustment strategy and calculate the infrared low-frequency coefficient weight for the nth iteration Step 4.5, generate the nth fused low-frequency component where L IR (x, y) is the pixel value of the infrared low-frequency component at (x, y), and ω IR (n) is the infrared low-frequency weight matrix for the nth iteration, expressed as: Step 4.6, determine the constraint factor whether it is satisfied If not satisfied, repeat Steps 4.1 to 4.5 to generate a new low-frequency component; If the condition is satisfied, stop the iteration and output the low-frequency sequence In step 4.1, calculate the constraint factor for the nth iteration where N represents the number of pixels of the low-frequency luminance component L Y ; N * represents the number of pixels in L Y that are greater than its pixel mean value; In step 4.2, calculate the veiling glare threshold β for the nth time n : wherein, and respectively represent the pixel mean value and the maximum value of L Y , and is the initial constraint factor; In step 4.3, in the non-glare region, calculate the weight value of the infrared low-frequency coefficient for the nth iteration In the formula, respectively represent the pixel mean values of the low-frequency luminance component L Y and the low-frequency infrared component L IR in the non-glare region; is the initial low-frequency coefficient weight value of the non-glare region; In step 4.4, in the vignetting area, calculate the weight value of the infrared low-frequency coefficient for the nth iteration where p is the low-frequency luminance component L Y the pixel value at (x, y), and r is the adjustment factor; Based on the halation critical threshold, map the pixel values of the low-frequency luminance component L Y to the interval [a, b]. The mapped low-frequency luminance component L Y '(x, y) is as follows: Obtain the fused high-frequency component Step 6: Respectively perform Curvelet reconstruction on each component in the low-frequency sequence and the fused high-frequency component to obtain a set of anti-halo luminance component sequences with different degrees of halo elimination; Step 7: Perform illumination estimation through convolution of a multi-scale Gaussian kernel function and the anti-halos luminance component sequence to obtain an illumination component sequence {I (0) , I (1) ,..., I (n)}; Step 8: Design a membership function for maximizing visual information, and assign greater weights to the regions suitable for human eye observation according to the illumination component sequence {I (0) , I (1) ,..., I (n)}, so as to determine the weights of each component in the anti-glare luminance component sequence participating in the fusion, and then perform weighted summation on the sequence to obtain a new luminance component Y'; The specific steps for designing the visual information maximization membership function are as follows: Step 8.1, calculate the anti-halos sequence The weight matrix W of the nth component in n (x, y) is as follows: Step 8.2: The new luminance component Y' is: where M represents the number of components of the anti-halos sequence Y AH ; Step 9: Perform YUV inverse transformation on the new luminance component Y' and the original hue U and saturation V components, and output the fused image.

2. The method for generating an infrared image fusion night vision anti-glare based on a low-frequency sequence according to claim 1, wherein The said Take 0.

3.

3. The method for generating a heterologous image fusion night vision anti-glare based on a low-frequency sequence according to claim 1, wherein The r takes 75; the mapping interval is [0, 5].

Images