Extremely low illumination visible light and near infrared image fusion method based on information complementarity

Through the extremely low illuminance visible light and near-infrared image fusion method based on information complementarity, the weighted guide filtering and extended Gaussian differential model are used to solve the color distortion and edge artifacts of image fusion under low illumination conditions in the prior art, achieving high Quality image fusion.

CN118799203BActive Publication Date: 2024-11-19BEIHANG UNIV +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411273076.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-12
Publication Date
2024-11-19
Estimated Expiration
2044-09-12

Smart Images

  • Figure CN118799203B_ABST
    Figure CN118799203B_ABST
Patent Text Reader

Abstract

The invention discloses an extremely low illumination visible light and near infrared image fusion method based on information complementarity, belonging to the field of image fusion technology. In the method, weighted guided filtering and guided filtering decomposition strategy are used to extract texture layer, edge layer and basic layer respectively; the difference map of visible light and near infrared texture layer and edge layer is calculated by using extended Gaussian difference model, and gradient information is extracted as the initial value of information complementarity weight; and a guided map is constructed to adjust the complementary weight range; then a mapping function is designed to adaptively adjust the fusion weight according to the guided map; then the near infrared texture and edge layer are processed by adaptive wavelet filtering, and the near infrared texture layer and edge layer are reconstructed; finally, the visible light and near infrared texture layer and edge layer are weighted fused respectively, and the fused layers are added to generate the final fused image. The method effectively utilizes spectral characteristics and information complementarity, and avoids color distortion and edge artifacts while maintaining natural colors.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of image fusion, and more particularly to an extremely low illumination visible light and near infrared image fusion method based on information complementarity. Background Art

[0002] With the advancement of modern imaging technology, imaging systems that use silicon-based sensors to simultaneously acquire visible light and near-infrared images to provide high-quality images have been widely used in security monitoring and medical imaging. Especially under low-light imaging conditions at night, by adding a near-infrared fill light device, a near-infrared grayscale image with a high signal-to-noise ratio and clear texture can be obtained. At the same time, by fusing it with a low-signal-to-noise ratio color visible light image, a clear image suitable for human eye viewing and machine processing can be obtained, thereby greatly improving the working efficiency of the night imaging system.

[0003] However, according to spectral reflectance theory, there is a big difference between the physical information represented by visible light and near-infrared images. The reflection of the visible light spectrum is based on the chromogenic molecular structure, so the visible light image is colorful and suitable for human visual perception. However, the near-infrared spectrum reflects the molecular structure information of most types of organic compounds, that is, the near-infrared radiation is dependent on the material and has no color information. For example, in vegetation-covered areas such as soil and wet ground, the near-infrared spectrum has high reflectance; in areas such as clouds, water, and snow, the near-infrared spectrum has low reflectance. At the same time, under harsh imaging conditions such as lack of light or uneven light, the differences in different spectral reflectance characteristics lead to large differences in the resulting image data, which brings huge challenges to fusion.

[0004] At present, traditional methods are mainly based on multi-scale transformation of grayscale images, such as pyramid transformation, wavelet transformation, contourlet transformation and non-subsampled contourlet transformation (NSCT). These methods transfer visible light images to a certain color space, divide them into color information and brightness information, and use multi-scale image decomposition methods, and design fusion rules for each scale for fusion, and finally perform inverse transformation to obtain the fused brightness map, and combine the original color information to obtain the final fused image. This kind of fusion method lacks the analysis of the imaging spectral characteristics of the original image, uses simple and unadaptable fusion rules, and has no constraints on the processing of color information and brightness information. Therefore, the final fused image may have a large color deviation, and the scene texture edge is also more prone to halo artifacts.

[0005] Subsequently, inspired by the theory of deep learning, some data-driven methods were developed to fuse visible light and infrared images using a deep learning framework including encoders, fusion layers, and decoder models. Although this fusion method can achieve good results on the test data set, its disadvantage is that it requires a large number of training samples and equipment computing power. The quality of image fusion depends on many aspects such as the quality of the training data set, the size of the data set, the capacity of the model, the selection of loss functions, the selection of hyperparameters, etc., and the fusion algorithm lacks scene applicability. Therefore, in order to obtain clear visible light and near-infrared fusion images suitable for human visual observation, it is necessary to take into account the physical properties of the two spectra and the complementarity of information. Summary of the invention

[0006] In view of this, the present invention provides an extremely low illumination visible light and near-infrared image fusion method based on information complementarity that solves at least some of the above-mentioned technical problems, effectively utilizes spectral characteristics and information complementarity, avoids color distortion and edge artifacts while maintaining natural colors, effectively improves the quality of the fused image, and can obtain high-quality fused images with clear edges, natural colors, and rich details under low light conditions.

[0007] To achieve the above object, the technical solution adopted by the present invention is:

[0008] An embodiment of the present invention provides an extremely low illumination visible light and near infrared image fusion method based on information complementarity, the method comprising the following steps:

[0009] A. Standardize the visible light three-band image and the near-infrared band image, and obtain the texture basic layer of the four bands of the visible light three-band and the near-infrared band through weighted guided filtering; subtract the original image from the texture basic layer to obtain the texture layer; and use guided filtering to process the texture basic layer to obtain the edge basic layer, and then subtract the visible light texture basic layer from the visible light edge basic layer to obtain the visible light edge layer;

[0010] B. Calculate the grayscale histogram of the near-infrared image, determine the threshold using the Otsu automatic threshold segmentation algorithm, and perform threshold segmentation on the near-infrared image and visible light to obtain a binary image; perform edge detection and extraction on the binary image to obtain an edge map, and perform a morphological dilation operation on the edge map to obtain an extended edge map; at the same time, on the texture basic layer and edge basic layer obtained in step A, calculate the difference map of the texture basic layer and edge basic layer of the three visible light bands and the near-infrared band, and extract the gradient information of the difference map through the extended Gaussian difference model as the initial value of the information complementarity weight;

[0011] C. Calculate the extended Gaussian difference gradient map of the near-infrared texture basic layer and the edge basic layer, and add them to the corresponding texture basic layer and edge basic layer respectively to obtain a guide map for calculating the fusion weight based on information complementarity; and establish a mapping function based on the information complementarity theory, use the guide map to guide the gradient map of the texture layer and the edge layer, and generate the information complementarity weight between the bands;

[0012] D. On the extended edge map obtained in step B, calculate the distance from each pixel to the visible light edge layer, and adaptively filter the extended edge map according to the distance to obtain a near-infrared edge layer; adaptively decompose the near-infrared texture layer and the edge layer by wavelet filtering to obtain low-frequency components and high-frequency components of the near-infrared texture layer and the edge layer; directly add the low-frequency components, and use the binary image obtained in step B to constrain the high-frequency components for wavelet reconstruction to obtain a reconstructed near-infrared texture layer and edge layer;

[0013] E. For the visible light and reconstructed near-infrared texture layer and edge layer, the three visible light bands and the near-infrared band are fused respectively according to the inter-band information complementarity weights of step C to obtain the fused texture layer and edge layer of the three visible light bands; the sum of the fused visible light three-band edge layer and the visible light three-band edge basic layer is calculated to obtain the fused texture basic layer of the three visible light bands, and the basic layer is added to the fused texture layer of the three visible light bands to obtain the final fused image of the three visible light bands.

[0014] Furthermore, in step A, weighted guided filtering and guided filtering methods are used to decompose layer by layer to obtain texture layers and edge layers of four bands of three visible light bands and near infrared bands, and the calculation formula is:

[0015] ;

[0016] ;

[0017] In the formula, Represents images of three visible light bands and near infrared bands, Represents three bands of visible light and near-infrared band, represents the texture base layer, represents the edge basic layer, WGuidF represents the weighted guided filter, GuidF represents the guided filter, and denote the parameters of weighted guided filtering and guided filtering respectively, Represents a four-band texture layer, Represents the edge layer of the four bands.

[0018] Furthermore, in step B, the difference map of the texture basic layer and the edge basic layer of the three visible light bands and the near infrared band is calculated, and the gradient information of the difference map is extracted by the extended Gaussian difference model as the initial value of the information complementarity weight. The calculation formula is:

[0019] ;

[0020] ;

[0021] In the formula, The difference map of the texture basic layer and edge basic layer of the three visible light bands and the near infrared band. Represents the texture basic layer and edge basic layer of the three bands of visible light. Represents the texture base layer and edge base layer of the near infrared band, Used to distinguish between texture layer and edge layer, Represents the three bands of visible light, represents the near-infrared band, represents the extended Gaussian difference model, Represents the initial value of information complementarity weight.

[0022] Furthermore, in step C, the calculation formula of the guide map is:

[0023] ;

[0024] In the formula, Represents the guidance map used to calculate the weights based on information complementarity fusion, Extended Difference of Gaussian gradient map representing near-infrared texture base layer and edge base layer;

[0025] The mapping function established based on information complementarity theory is:

[0026] ;

[0027] In the formula, represents the mapping function based on information complementarity theory, Represents pixels;

[0028] The guide map is used to guide the gradient map of the texture layer and the edge layer to generate the inter-band information complementarity weight, and the calculation formula is:

[0029] ;

[0030] In the formula, Represents the weight of information complementarity between bands.

[0031] Furthermore, in step D, the extended edge map is adaptively filtered, and the calculation formula is:

[0032] ;

[0033] In the formula, Represents pixels in an image The strength value of Represents the pixel in the image after adaptive filtering The strength value of Represents a binary image, where its value is 1 in the high response area and 0 otherwise; Represents pixels Distance to the edge of visible light; and It is a tuning parameter used to control the strength of adaptive filtering; is the parameter of the filter range.

[0034] Furthermore, in step D, the low-frequency components are directly added, and the high-frequency components are constrained by the binary image obtained in step B to perform wavelet reconstruction to obtain a reconstructed near-infrared texture edge layer. The calculation formula includes:

[0035] ;

[0036] In the formula, represents the fused low-frequency component, Represents the texture layers of the near infrared image and edge layer The low-frequency components are obtained by wavelet decomposition. represents the fused high-frequency component, Represents the texture layers of the near infrared image and edge layer Perform wavelet decomposition to obtain high-frequency components. represents a binary image, represents the visible light edge with additional constraint on the near-infrared edge layer, To adjust the parameters, It represents the fused near-infrared texture layer and edge layer obtained by wavelet reconstruction, and WaveletReconstruction represents wavelet reconstruction.

[0037] Furthermore, in the step B, a difference map of the texture basic layer and the edge basic layer of the three visible light bands and the near infrared band is calculated, which contains all the information from the visible light and the near infrared.

[0038] Compared with the prior art, the present invention has at least the following beneficial effects:

[0039] The present invention provides an extremely low-illumination visible light and near-infrared image fusion method based on information complementarity. The method effectively utilizes spectral characteristics and information complementarity, avoids color distortion and edge artifacts while maintaining natural colors, and effectively improves the quality of visible light and near-infrared image fusion. Compared with the current mainstream methods, the present invention can obtain high-quality fused images with clear edges, natural colors and rich details under low-light conditions.

[0040] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description and the accompanying drawings.

[0041] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention.

[0044] Figure 1 A schematic flow chart of a method for extremely low illumination visible light and near infrared image fusion based on information complementarity provided in an embodiment of the present invention.

[0045] Figure 2 A schematic diagram of a texture layer in the visible light red band provided by an embodiment of the present invention.

[0046] Figure 3 A schematic diagram of the edge layer of the visible light red band provided by an embodiment of the present invention.

[0047] Figure 4 A schematic diagram of the comparison of visible light and near-infrared colors and materials provided in an embodiment of the present invention.

[0048] Figure 5 The embodiment of the present invention provides Function diagram.

[0049] Figure 6 A schematic diagram of subjective results of visible light and near-infrared image fusion in an outdoor scene provided by an embodiment of the present invention.

[0050] Figure 7 A schematic diagram of subjective results of visible light and near-infrared image fusion in an indoor scene provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0051] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. In addition, various serial numbers are only used for descriptive purposes and cannot be understood as indicating or implying relative importance.

[0052] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] Reference Figure 1 As shown, the present invention provides an extremely low illumination visible light and near infrared image fusion method based on information complementarity, the method mainly includes:

[0054] A. Standardize the visible light three-band image and the near-infrared band image, and obtain the texture basic layer of the four bands of the visible light three-band and the near-infrared band through weighted guided filtering; subtract the original image from the texture basic layer to obtain the texture layer; and use guided filtering to process the texture basic layer to obtain the edge basic layer, and then subtract the visible light texture basic layer from the visible light edge basic layer to obtain the visible light edge layer;

[0055] B. Calculate the grayscale histogram of the near-infrared image, determine the threshold using the Otsu automatic threshold segmentation algorithm, and perform threshold segmentation on the near-infrared image and visible light to obtain a binary image; perform edge detection and extraction on the binary image to obtain an edge map, and perform a morphological dilation operation on the edge map to obtain an extended edge map; at the same time, on the texture basic layer and edge basic layer obtained in step A, calculate the difference map of the texture basic layer and edge basic layer of the three visible light bands and the near-infrared band, and extract the gradient information of the difference map through the extended Gaussian difference model as the initial value of the information complementarity weight;

[0056] C. Calculate the extended Gaussian difference gradient map of the near-infrared texture basic layer and the edge basic layer, and add them to the corresponding texture basic layer and edge basic layer respectively to obtain a guide map for calculating the fusion weight based on information complementarity; and establish a mapping function based on the information complementarity theory, use the guide map to guide the gradient map of the texture layer and the edge layer, and generate the information complementarity weight between the bands;

[0057] D. On the extended edge map obtained in step B, calculate the distance from each pixel to the visible light edge layer, and adaptively filter the extended edge map according to the distance to obtain a near-infrared edge layer; adaptively decompose the near-infrared texture layer and the edge layer by wavelet filtering to obtain low-frequency components and high-frequency components of the near-infrared texture layer and the edge layer; directly add the low-frequency components, and use the binary image obtained in step B to constrain the high-frequency components for wavelet reconstruction to obtain a reconstructed near-infrared texture layer and edge layer;

[0058] E. For the visible light and reconstructed near-infrared texture layer and edge layer, the three visible light bands and the near-infrared band are fused respectively according to the inter-band information complementarity weights of step C to obtain the fused texture layer and edge layer of the three visible light bands; the sum of the fused visible light three-band edge layer and the visible light three-band edge basic layer is calculated to obtain the fused texture basic layer of the three visible light bands, and the basic layer is added to the fused texture layer of the three visible light bands to obtain the final fused image of the three visible light bands.

[0059] The principles and specific implementation methods involved in the method of the present invention are described in detail below. The method mainly includes the following five parts:

[0060] 1. Layer-by-layer decomposition based on weighted guided filtering and guided filtering:

[0061] In this embodiment, in order to better distinguish the noise and useful texture edge information in the visible light and near-infrared images under low illumination imaging conditions, the present invention proposes weighted guided filtering and guided filtering layer-by-layer decomposition to obtain the texture layer and edge layer of the three visible light bands and the near-infrared band respectively. The guided filtering has the advantage of maintaining the large-scale image edge, and the weighted guided filtering can further avoid the gradient inversion problem of the large-scale edge. The weighted guided filtering is first used to extract the texture layer, and then the guided filtering is used to extract the large-scale edge, which can avoid the halo artifact phenomenon of the fusion result.

[0062] like Figure 2 and Figure 3 As shown, Figure 2 and Figure 3 The texture layer and edge layer of the red band of the extracted visible light are shown in Figure 1. It can be seen that the texture layer has a lot of noise, while the edge layer has almost no noise and is relatively clear. The texture layer and edge layer of the four bands of visible light and near infrared are calculated using the following formula:

[0063] ;

[0064] In the formula, Represents images of three visible light bands and near infrared bands, Represents three bands of visible light and near-infrared band, represents the texture base layer, represents the edge basic layer, WGuidF represents the weighted guided filter, GuidF represents the guided filter, and Respectively represent the parameters of weighted guided filtering and guided filtering;

[0065] The corresponding texture layer and edge layer come from the difference between the original image and the texture base layer and the difference between the texture base layer and the edge base layer:

[0066] ;

[0067] In the formula, Represents a four-band texture layer, Represents the edge layer of the four bands.

[0068] 2. Calculation of the initial value of information complementarity weight:

[0069] Considering that the reflection of visible light and near-infrared light depends on different material molecular groups, the physical information they represent is also different, and thus the apparent characteristics of image data will be quite different, such as Figure 4 As shown, the edge of the color disk can be seen in visible light due to color changes, but in near-infrared light, because the color disk material is the same, there is no texture edge. At the same time, since there may be materials with strong reflection in the near-infrared spectral region in the image, these objects will appear extremely bright in the near-infrared image, which exceeds the dynamic range of the camera and causes overexposure, causing the object to have edge diffusion in the near-infrared image. During fusion, the edge diffusion effect will cause unnatural bright edges to appear in the fusion result, reducing the realism and detail clarity of the image. In order to simultaneously include information of images in the visible and near-infrared bands and remove the edge diffusion phenomenon, in this embodiment, the grayscale histogram of the near-infrared image is first calculated, the threshold is determined using the Otsu threshold segmentation algorithm, and the near-infrared image and visible light are threshold segmented to obtain a binary image. Further, an edge detection algorithm is applied to the binary image to obtain an edge map, and a morphological dilation operation is performed on the edge map to obtain an extended edge map. Secondly, the difference map of the three visible light bands and the near infrared band image is calculated respectively. The difference map will contain all the similarities and differences of the three visible light bands and the near infrared band image data. The smaller the value of the difference map, the smaller the difference between the two bands; conversely, the greater the difference between the bands. The calculation formulas for threshold segmentation and difference map are as follows:

[0070] ;

[0071] ;

[0072] In the formula, Represents pixels It is a high response area. Represents pixels in an image The intensity value of the image is , and T represents the threshold determined by Ostu adaptive segmentation. The texture base layer and edge base layer representing the three bands of visible light, Represents the texture base layer and edge base layer of the near-infrared band. Represents the difference map of the texture and edge basic layer of the three visible light bands and the near infrared band, where Used to distinguish between texture layer and edge layer, Represents the three bands of visible light, Represents the near-infrared band.

[0073] At the same time, in this embodiment, the extended Gaussian difference model is used to extract the gradient information of the difference map as the initial value of the information complementary weight of the layered fusion, and the formula is:

[0074] ;

[0075] In the formula, To extend the Difference of Gaussian model, is the initial value of the information complementarity weight.

[0076] 3. Calculation of information complementarity weight:

[0077] After obtaining the initial value of the information complementation weight, the high signal-to-noise ratio advantage of the near-infrared image after the near-infrared additional fill light under low illumination conditions at night is used to extract the sum of the gradient value of the near-infrared texture and edge layer and the corresponding basic layer pixel value using the extended Gaussian difference model of the image as the guidance map of the information complementation weight, which can reflect whether the information in the area of ​​the near-infrared image is a texture with a high signal-to-noise ratio or noise with a low signal-to-noise ratio. The calculation of the guidance map is as follows:

[0078] ;

[0079] ;

[0080] In the formula, Represents the guidance map used to calculate the weights based on information complementarity fusion, Extended Difference of Gaussian gradient maps representing the near-infrared texture basis layer and edge basis layer.

[0081] Furthermore, in this embodiment, a mapping function is designed Through the information complementation guidance graph, the initial value of the information complementation weight is adaptively adjusted. This function can achieve that when the near-infrared signal-to-noise ratio is high, it needs to change within a lower range according to the initial value; when the near-infrared signal-to-noise ratio is low, it needs to change within a higher range according to the initial value. Figure 5As shown, when the near-infrared signal-to-noise ratio is high, it will be adjusted within a lower range according to the initial value of the information complementary weight; when the near-infrared signal-to-noise ratio is low, it will be adjusted within a higher range according to the initial value of the information complementary weight. The function is as follows:

[0082] ;

[0083] The corresponding information complementarity weight is:

[0084] ;

[0085] In the formula, Represents the weight of information complementarity between bands.

[0086] 4. Calculation of edge texture detail constraint enhancement:

[0087] Earlier, we obtained the extended edge map through threshold segmentation, edge detection and morphological operations, and decomposed the original image into a texture layer and an edge layer through weighted guided filtering and guided filtering layer by layer. For the edge layer, due to the difference in edge processing between visible light and near infrared, it is necessary to use the transmission characteristics of the near infrared image to enhance the details of the visible light edge layer. First, for each pixel in the extended edge map, , calculate its distance to the visible light edge layer The closer the distance is, the closer the pixel is to the edge area. Then, according to The size of the extended edge map is adaptively filtered to obtain a near-infrared adaptive filtered image. The idea of ​​adaptive filtering is to determine whether the pixel is in the edge area. If so, the pixel value of the input image is retained, and the distance weight is introduced to smooth the pixel value far from the edge. Otherwise, the pixel value of the input image is smoothed, and the distance weight is introduced to retain the pixel value close to the edge. The adaptive filtering formula is as follows:

[0088] ;

[0089] In the formula, Represents pixels in an image The strength value of Represents the pixel in the image after adaptive filtering The strength value of Represents a binary image, where its value is 1 in the high response area and 0 otherwise; Represents pixels Distance to the edge of visible light; and It is a tuning parameter used to control the strength of adaptive filtering; is the parameter of the filter range.

[0090] In a specific embodiment, in order to balance the details, edges and clarity of the image, obtain an ideal fusion effect, and supplement the edge texture details lost during multi-layer filtering decomposition, it is necessary to use wavelet decomposition and use detail constraints on the texture layer and edge layer to restore more complete information. and edge layer Perform wavelet decomposition separately to obtain low-frequency components and high frequency components . Decomposition of the texture layer:

[0091] ;

[0092] Decomposition of the edge layer:

[0093] ;

[0094] In the formula, is the low frequency component, is the high frequency component. Then the image is constrained by visible light details, and the constraint calculation and wavelet reconstruction are processed as follows:

[0095] ;

[0096] ;

[0097] ;

[0098] in: It is the fused low-frequency component, directly adding the adaptive filtering result; It is the high-frequency component after fusion, using the binary image Constrained adaptive filtering results. Additional introduction of visible light edges constrained near-infrared edge layer; It is an adjustable parameter that controls the visible light edge constraint strength; It is the fused texture layer and edge layer obtained by wavelet reconstruction; WaveletReconstruction represents wavelet reconstruction.

[0099] 5. Generation of fusion results:

[0100] According to the above analysis, the present invention uses the difference between the all-1 matrix and the final fusion matrix of the three visible light bands as the fusion weight of the near-infrared band for the final fusion weight of the texture layer and edge layer of the near-infrared band. The three visible light bands and the near-infrared band are fused separately according to the information complementary fusion weight (information complementarity weight between bands) to obtain the texture layer and edge layer of the fused three visible light bands. The sum of the fused edge layer of the three visible light bands and the edge basic layer of the three visible light bands is calculated and added to the texture layer of the fused three visible light bands to obtain the final fused image of the three visible light bands. The calculation formula is:

[0101] ;

[0102] ;

[0103] In the formula, represents the fused texture base layer, Represents the fused image.

[0104] Furthermore, in order to verify the effectiveness of the method of the present invention, in this embodiment, the method of the present invention is compared with the most advanced fusion methods, including joint scale map recovery fusion, bilateral filtering fusion, color preservation fusion, Laplacian pyramid fusion and spectral edge fusion. The fusion algorithms are compared on data sets of two scenes, outdoor and indoor, and the lighting conditions and noise levels contained in the data sets are quite different. Figure 6 and Figure 7 The results of image fusion effects comparison using different methods for visible light and near-infrared images in imaging scenes with different illumination levels are shown. The results verify that the image fused by the method of the present invention has clearer texture and more natural color. In order to verify the preservation of spectral characteristics and the naturalness of color, the reference-free spectral distortion index, color distance metric, peak signal-to-noise ratio, visual information fidelity, etc. are used as objective evaluation indicators to compare and verify the superiority of the method of the present invention. The verification results are shown in Table 1, which shows the objective comparison results of the above fusion methods on the above evaluation indicators.

[0105]

[0106] From the comparison results in Table 1, it can be seen that the method of the present invention has obvious advantages over other fusion methods in the fusion of near-infrared and visible light. The method of the present invention can make good use of spectral characteristics and information complementarity, and obtain a fused image with clear texture and natural color.

[0107] From the description of the above embodiments, those skilled in the art can know that: in order to solve the problem that the existing methods lack analysis of spectral characteristics of different bands, and fail to effectively utilize complementary information of different bands under low illumination conditions, resulting in color distortion and low clarity of the fusion results. The present invention proposes an extremely low illumination visible light and near-infrared image fusion method based on information complementarity. This method aims at the shortcomings of traditional visible light and near-infrared image fusion methods, deeply analyzes the spectral imaging process and the physical information represented by the image data, and designs a fusion weight model based on information complementarity, especially for extremely low illumination imaging conditions, to adaptively adjust the fusion weight according to the quality of the data itself.

[0108] First, in order to better distinguish the noise and useful texture edge information in visible light and near-infrared images under low-light imaging conditions, the present invention proposes a strategy of weighted guided filtering and layer-by-layer decomposition of guided filtering. Guided filtering has the advantage of maintaining large-scale image edges, while weighted guided filtering can further avoid the gradient inversion problem of large-scale edges. First, weighted guided filtering is used to extract the texture layer, and then guided filtering is used to extract large-scale edges, which can avoid the halo artifact phenomenon of the fusion result. And the extended edge map of the near-infrared is obtained through threshold segmentation, edge detection and morphological expansion.

[0109] Secondly, considering that the reflection of visible light and near-infrared light depends on different material molecular groups, the physical information they represent is also different, and thus the apparent characteristics of the image data will be quite different. By subtracting the three bands of visible light from the near-infrared band image on the extracted texture and edge basic layers, the corresponding difference map is obtained. The difference map will contain all the similarities and differences of the three bands of visible light and the near-infrared band image data, and the extended Gaussian difference model is used to extract the gradient information of the difference map as the initial value of the information complementarity weight of the layered fusion. The near-infrared texture edge layer is decomposed by wavelet, and the visible light edge information is used for constraint, which effectively solves the edge halo artifact phenomenon caused by the large difference in texture structure after fusion.

[0110] Then, due to the low illumination conditions at night, there will be near-infrared fill light to improve the signal-to-noise ratio. The extended Gaussian difference model is used to extract the gradient values ​​of the near-infrared texture and edge basic layer, and the sum of the corresponding basic layer pixel values ​​is used as a guidance map to determine whether the area is texture edge information with a high signal-to-noise ratio or noise with a low signal-to-noise ratio.

[0111] Finally, a mapping function is designed Through the information complementarity guidance map, the initial value of the information complementarity weight is adaptively adjusted, the information complementarity of the nighttime visible light and near-infrared images is fitted, and the final fusion weight based on information complementarity is obtained. The three-band visible light and near-infrared band images are fused layer by layer according to the final fusion weight, and finally the basic layer is merged to obtain a fused image with low noise and natural color.

[0112] In summary, the method of the present invention effectively utilizes spectral characteristics and information complementarity, avoids color distortion and edge artifacts while maintaining natural colors, effectively improves the quality of the fused image, and can obtain a high-quality fused image with clear edges, natural colors, and rich details under low light conditions.

[0113] Furthermore, an embodiment of the present invention also provides an electronic device for realizing extremely low illumination visible light and near infrared image fusion based on information complementarity. The electronic device can be implemented by software and / or hardware, and the device includes a memory and a processor; wherein the memory and the processor can be connected by a bus or other means, and the processor can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components and other chips, or a combination of the above types of chips. The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs, non-transient computer executable programs and modules, such as programs or instructions corresponding to the visible light and near infrared image fusion method. The processor executes various functional applications and data processing of the processor by running the non-transient software programs or instructions stored in the memory, that is, realizing the extremely low illumination visible light and near infrared image fusion method based on information complementarity in the above method embodiment.

[0114] In addition, a storage medium is provided in an embodiment of the present invention, on which one or more programs readable by a computing device are stored, and the one or more programs include instructions. When the instructions are executed by the computing device, the computing device executes the extremely low illumination visible light and near-infrared image fusion method based on information complementarity in the above method embodiment.

[0115] In this embodiment, the storage medium may be, for example, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples of storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, and any suitable combination thereof.

[0116] It should be understood by those skilled in the art that embodiments of the present invention may be provided as methods, electronic devices, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0117] It should be noted that the word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several distinct components, and by means of a suitably programmed computer.

[0118] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0119] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for extremely low illumination visible light and near infrared image fusion based on information complementarity, characterized in that: The method comprises the following steps: A. Standardize the visible light three-band image and the near-infrared band image, and obtain the texture basic layer of the four bands of the visible light three-band and the near-infrared band by weighted guided filtering; subtract the original image from the texture basic layer to obtain the texture layer; and use guided filtering to process the texture basic layer to obtain the edge basic layer, and then subtract the visible light three-band texture basic layer from the visible light three-band edge basic layer to obtain the visible light three-band edge layer; B. Calculate the grayscale histogram of the near-infrared image, use the Otsu automatic threshold segmentation algorithm to determine the threshold, and perform threshold segmentation on the near-infrared band image and the visible light three-band image to obtain a binary image; perform edge detection and extraction on the binary image to obtain an edge map, and perform a morphological dilation operation on the edge map to obtain an extended edge map; at the same time, on the texture basic layer and edge basic layer obtained in step A, calculate the difference map of the texture basic layer and edge basic layer of the visible light three-band and the near-infrared band, and extract the gradient information of the difference map through the extended Gaussian difference model as the initial value of the information complementarity weight; C. Calculate the extended Gaussian difference gradient map of the near-infrared texture basic layer and the edge basic layer, and add them to the corresponding texture basic layer and edge basic layer respectively to obtain a guide map for calculating the fusion weight based on information complementarity; and establish a mapping function based on the information complementarity theory, use the guide map to guide the gradient map of the texture layer and the edge layer, and generate the information complementarity weight between the bands; D. On the extended edge map obtained in step B, calculate the distance from each pixel to the visible light edge layer, and adaptively filter the extended edge map according to the distance to obtain a near-infrared edge layer; adaptively decompose the near-infrared texture layer and the edge layer by wavelet filtering to obtain low-frequency components and high-frequency components of the near-infrared texture layer and the edge layer; directly add the low-frequency components, and use the binary image obtained in step B to constrain the high-frequency components for wavelet reconstruction to obtain a reconstructed near-infrared texture layer and edge layer; The extended edge map is adaptively filtered, and the calculation formula is: Where I(x,y) represents the intensity value of the pixel (x,y) in the image; I'(x,y) represents the intensity value of the pixel (x,y) in the image after adaptive filtering; B(x,y) represents a binary image, which is 1 in the high response area and 0 otherwise; d(x,y) represents the distance from the pixel (x,y) to the edge of the visible light; β and α are adjustment parameters used to control the intensity of the adaptive filtering; σ r is the parameter of the filtering range; The low-frequency components are directly added, and the high-frequency components are constrained by the binary image obtained in step B to perform wavelet reconstruction to obtain the reconstructed near-infrared texture layer and edge layer. The calculation formula includes: D' n =WaveletReconstruction(L combined ,H constrained ) Where, L combined represents the fused low-frequency component, Represents the texture layers of the near infrared image and edge layer The low-frequency components obtained by wavelet decomposition are H constrained represents the fused high frequency component, Represents the texture layers of the near infrared image and edge layer The high-frequency components are obtained by wavelet decomposition, B represents the binary image, represents the visible light edge of the additional near-infrared edge layer, β is the adjustment parameter, and D' n represents the fused near-infrared texture layer and edge layer obtained by wavelet reconstruction, and WaveletReconstruction represents wavelet reconstruction; E. For the visible light and reconstructed near-infrared texture layer and edge layer, the three visible light bands and the near-infrared band are fused respectively according to the inter-band information complementarity weights of step C to obtain the fused texture layer and edge layer of the three visible light bands; the sum of the fused visible light three-band edge layer and the visible light three-band edge basic layer is calculated to obtain the fused texture basic layer of the three visible light bands, and the basic layer is added to the fused texture layer of the three visible light bands to obtain the final fused image of the three visible light bands.

2. The method for extremely low illumination visible light and near infrared image fusion based on information complementarity according to claim 1, characterized in that: In step A, weighted guided filtering and guided filtering methods are used to decompose layer by layer to obtain texture layers and edge layers of four bands of three visible light bands and four near infrared bands. The calculation formula is: In the formula, R p represents the three-band visible light and near-infrared band images, p represents the three-band visible light and near-infrared band, represents the texture base layer, represents the edge basic layer, WGuidF represents the weighted guided filter, GuidF represents the guided filter, r,ε and w,σ represent the parameters of the weighted guided filter and the guided filter respectively, Represents a four-band texture layer, Represents the edge layer of the four bands.

3. The method for extremely low illumination visible light and near infrared image fusion based on information complementarity according to claim 2, characterized in that: In step B, the difference map of the texture basic layer and the edge basic layer of the three visible light bands and the near infrared band is calculated, and the gradient information of the difference map is extracted by the extended Gaussian difference model as the initial value of the information complementarity weight. The calculation formula is: In the formula, The difference map of the texture basic layer and edge basic layer of the three visible light bands and the near infrared band. Represents the texture basic layer and edge basic layer of the three bands of visible light. represents the texture basic layer and edge basic layer of the near-infrared band, i is used to distinguish the texture layer and the edge layer, c represents the three bands of visible light, n represents the near-infrared band, XDoG represents the extended Gaussian difference model, doogV c i Represents the initial value of information complementarity weight.

4. The method for extremely low illumination visible light and near infrared image fusion based on information complementarity according to claim 3, characterized in that: In step C, the calculation formula of the guide map is: In the formula, dogN i represents the guidance map used to calculate the weights based on information complementarity fusion, doogN i Extended Difference of Gaussian gradient map representing near-infrared texture base layer and edge base layer; The mapping function established based on information complementarity theory is: Where arctan_I represents the mapping function based on information complementarity theory, (x, y) represents pixels; The guide map is used to guide the gradient map of the texture layer and the edge layer to generate the inter-band information complementarity weight, and the calculation formula is: In the formula, Represents the weight of information complementarity between bands.

5. The method for extremely low illumination visible light and near infrared image fusion based on information complementarity according to claim 1, characterized in that: In the step B, a difference map of the texture basic layer and the edge basic layer of the three visible light bands and the near infrared band is calculated, which contains all the information from the visible light and the near infrared.