An adaptive image characteristic visible and infrared fusion method

Abstract

The application discloses a visible light and infrared fusion method with adaptive image characteristics, and belongs to the field of image fusion, which comprises the following steps: firstly, calculating the similarity of two source images, and selecting a corresponding fusion algorithm based on the similarity and a preset matching threshold; secondly, calculating the regional energy and global neighborhood variance of the two source images, and calculating an adaptive threshold factor according to the regional energy and the global neighborhood variance; finally, bringing the adaptive threshold factor into the selected fusion algorithm to perform image fusion processing; the application guarantees the rationality and stability of the process of processing different characteristic input sources, and solves the problem of poor robustness of the result caused by the too fixed threshold selection and the too single fusion strategy of the algorithm.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image fusion, in particular to a visible light and infrared fusion method with adaptive image characteristics. BACKGROUND

[0002] The statements in this section merely provide background information related to the present disclosure and can not constitute the prior art.

[0003] Due to the complex form of the battlefield, the cross and complementary use of multiple sensors has become a normalized action; the infrared sensor perceives the surrounding objective environment by using infrared radiation, and the image has the advantages of good environmental adaptability, good concealment, and high recognition degree of disguised targets; however, due to the difficulty of manufacturing process and the purity of materials of the infrared detector, the infrared image generally has the problems of low resolution, large noise, low image contrast, and narrow gray scale range; the image of the visible light sensor has the advantages of rich spectral information, large dynamic range, much detail information, and good visuality; however, the anti-interference ability of the visible light image is poor, and the effect of the visible light image is poor in the case of low light, foggy weather, and disguised targets; therefore, the images obtained by different sensors in the same state or the same sensor in different states can be comprehensively used through the image fusion technology, the quality and information amount of the image are improved, and the target can be more comprehensively and accurately analyzed and understood.

[0004] Traditional image fusion technologies mainly include linear weighting method, high-pass filter method, IHS transformation method, etc., these methods are simple to operate and easy to implement, but these methods are insufficient in detail processing of source images, and the fused image will have "ringing effect" and block phenomenon, resulting in a decrease in image quality; the image fusion method based on multi-scale geometric transformation decomposes and reconstructs the low-frequency component and high-frequency component of the source image in the time domain, and the fused image can effectively retain the detail and edge information of the image, so this algorithm has rapidly become a research hotspot.

[0005] The image fusion method based on multi-scale geometric transformation decomposes and reconstructs the low-frequency component and high-frequency component by selecting a certain threshold to obtain the fused image; at present, most of the technologies select a fixed threshold, which cannot be changed for input sources with different characteristics, and cannot stably ensure the quality of the fused image; at the same time, the image fusion methods of different rules of multi-scale geometric transformation mainly solve different problems, and the current technology mainly uses a single means to process the image, which often appears the phenomenon of "trade-off", and the fused image will have the situation that one characteristic is improved while other image characteristics are ignored. SUMMARY

[0006] The present application aims at the problem that the threshold selection is too fixed and the fusion strategy is too single in the current multi-scale geometric transformation algorithm, causing poor robustness of the fused image quality, and provides a visible light and infrared fusion method with adaptive image characteristics, which selects a suitable multi-scale geometric transformation image fusion algorithm by comparing the similarity of two source images, obtains an adaptive threshold factor after calculating the regional energy and global neighborhood variance of the source images, processes the images through a suitable fusion rule, ensures the rationality and stability of the process of processing different characteristic input sources, makes up for the deficiency of poor robustness of image quality caused by too fixed threshold selection and too single fusion strategy, and thus solves the above problems.

[0007] The technical scheme of the present application is as follows:

[0008] A visible light and infrared fusion method with adaptive image characteristics, comprising:

[0009] Step S1: calculate the similarity of two source images, and select a corresponding fusion algorithm based on the similarity and a preset matching threshold;

[0010] Step S2: calculate the regional energy and global neighborhood variance of the two source images, and calculate an adaptive threshold factor according to the regional energy and global neighborhood variance;

[0011] Step S3: bring the adaptive threshold factor into the selected fusion algorithm for image fusion processing.

[0012] Further, the step S1 comprises:

[0013] Step S11: calculate the neighborhood directional contrast of the source images VIS_Img and IR_Img respectively;

[0014] Step S12: calculate the similarity of the two source images according to the neighborhood directional contrast of the source images VIS_Img and IR_Img;

[0015] Step S13: if the similarity is greater than the matching threshold, select a wavelet transformation image fusion algorithm to process the source images VIS_Img and IR_Img; if the similarity is less than the matching threshold, select an NSCT image fusion algorithm to process the source images VIS_Img and IR_Img.

[0016] Further, the step S11 comprises:

[0017]

[0018] In the formula:

[0019] is the neighborhood directional contrast of the source image VIS_Img;

[0020] is the local directional contrast of the source image IR_Img in the neighborhood direction;

[0021] is the local directional contrast of the source image IR_Img in the direction i at resolution 2 i is the local directional contrast of the source image IR_Img in the direction i at resolution 2

[0022] is the local directional contrast of the source image IR_Img in the direction i at resolution 2 i is the local directional contrast of the source image IR_Img in the direction i at resolution 2

[0023] Further, the step S12 comprises:

[0024]

[0025] wherein:

[0026] t a is the similarity of the two source images;

[0027] S, T are the size of the local region.

[0028] Further, the local energy of the two source images is calculated by the following formula:

[0029]

[0030]

[0031] wherein:

[0032] M x N represents the size of the image region;

[0033] and are the gray mean values of the source images VIS_Img and IR_Img in the region M x N, respectively;

[0034] E v and E i are the local energies of the source images VIS_Img and IR_Img, respectively.

[0035] Further, the global neighborhood variance of the two source images is calculated by the following formula:

[0036]

[0037]

[0038] wherein:

[0039] K x L represents the region size;

[0040] P VIS (m,n) represents the pixel value of the source image VIS_Img at point (m,n);

[0041] P IR (m,n) represents the pixel value of the source image IR_Img at point (m,n);

[0042] μ VIS represents the mean value of the source image VIS_Img;

[0043] μ IR represents the mean value of the source image IR_Img;

[0044] S v and S i are the global neighborhood variance of the source images VIS_Img and IR_Img, respectively.

[0045] Further, the adaptive threshold factor is calculated by the following formula:

[0046]

[0047] In the formula:

[0048] A t is the adaptive threshold factor.

[0049] Further, when the wavelet transform image fusion algorithm is selected to perform image fusion processing on the source images VIS_Img and IR_Img, the step S3 comprises:

[0050] Step S31: creating a convolution kernel of KxL size, performing wavelet decomposition on the source images VIS_Img and IR_Img to obtain the wavelet decomposition coefficients f v and f i of the source images;

[0051] Step S32: calculating the average change rate C t of the two source images VIS_Img and IR_Img;

[0052] Step S33: comparing the average change rate C t and the adaptive threshold factor A t , if C t >A t , adopting the weighted average fusion rule to obtain the wavelet coefficient f1 of the fused image; if C t <A t , calculating the wavelet coefficient f0 of the fused image according to the modulus maximum fusion rule;

[0053] Step S34: inverse wavelet transform is performed on the wavelet coefficients f1 or f0 of the fused image to obtain an image fused based on wavelet transform, denoted as Img_wave.

[0054] Further, when the NSCT image fusion algorithm is selected to perform image fusion processing on the source images VIS_Img and IR_Img, the step S3 comprises:

[0055] Step S3A: multi-layer NSCT transform is performed on the two source images VIS_Img and IR_Img to obtain the coefficients corresponding to the two source images, i.e. the corresponding obtained coefficients are n v ={low frequency coefficients nl1, high frequency coefficients nh1}, n i ={low frequency coefficients nl2, high frequency coefficients nh2};

[0056] Step S3B: the average change rate C t of the two source images VIS_Img and IR_Img is calculated.

[0057] Step S3C: the average change rate C t and the adaptive threshold factor A t are compared, if C t >A t , the weighted average fusion rule is adopted to obtain the NSCT coefficients n1 of the fused image, if C t <A t , the modulus maximum fusion rule is adopted to calculate the NSCT coefficients n0 of the fused image.

[0058] Step S3D: NSCT inverse transform is performed on the NSCT coefficients n1 or n0 of the fused image to obtain an image fused based on NSCT transform, denoted as Img_nsct.

[0059] Further, the average change rate C t is calculated by the following formula:

[0060]

[0061] Compared with the prior art, the present application has the following beneficial effects:

[0062] 1. An adaptive image characteristic visible light and infrared fusion method, which selects a suitable fusion algorithm for source images with different characteristics, avoids the problem of ignoring image uncertainty in the prior fusion algorithm, and is beneficial to enhancing the self characteristics of the source images and improving the stability of the fusion result.

[0063] 2. The adaptive image characteristic visible light and infrared fusion method selects a suitable fusion strategy through an adaptive threshold factor, avoids the problem of source image information loss caused by the over-absolute fusion strategy in the existing fusion algorithm, and is beneficial to retaining low-frequency and high-frequency component information in the source image. BRIEF DESCRIPTION OF DRAWINGS

[0064] Figure 1 A flowchart of the adaptive image characteristic visible light and infrared fusion method. DETAILED DESCRIPTION

[0065] It should be noted that the relational terms, such as first and second, and the like, are used solely to distinguish one from another entity or action, without necessarily requiring or implying any actual relationship or order between such entities or actions. Moreover, the terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element preceded by "comprises... a" does not, without more constraints, foreclose the existence of additional identical elements in the process, method, article, or apparatus that comprises the recited element.

[0066] The features and performance of the present application will be further described in detail below with reference to the embodiments.

[0067] Embodiment One

[0068] Please refer to Figure 1 An adaptive image characteristic visible light and infrared fusion method specifically comprises the following steps:

[0069] Step S1: Calculate the similarity of two source images, and select a corresponding fusion algorithm based on the similarity and a preset matching threshold; preferably, the two source images are a source image VIS_Img and a source image IR_Img; the matching threshold T is set as an empirical value 0.6; *

[0070] Step S2: Calculate the regional energy and global neighborhood variance of the two source images, and calculate an adaptive threshold factor according to the regional energy and the global neighborhood variance; that is, calculate the regional energy E v and E i , the global neighborhood variance S v and S i of the source image VIS_Img and the source image IR_Img; and calculate the adaptive threshold factor based on E v , E i , S v and S i ​Computing an adaptive threshold factor A t The size of the adaptive threshold factor A in a multi-scale geometric transform image fusion algorithm t ;

[0071] Step S3: bring the adaptive threshold factor into the selected fusion algorithm, and perform image fusion processing; preferably, the fusion algorithm includes a wavelet transform image fusion algorithm and an NSCT image fusion algorithm.

[0072] In this embodiment, specifically, the step S1 includes:

[0073] Step S11: respectively calculate the neighborhood directional contrast of the source images VIS_Img and IR_Img.

[0074] Step S12: according to the neighborhood directional contrast of the source images VIS_Img and IR_Img, calculate the similarity of the two source images.

[0075] Step S13: if the similarity is greater than the matching threshold, select the wavelet transform image fusion algorithm to perform image fusion processing on the source images VIS_Img and IR_Img, and obtain the fused image Img_wave; if the similarity is less than the matching threshold, select the NSCT image fusion algorithm to perform image fusion processing on the source images VIS_Img and IR_Img, and obtain the fused image Img_nsct.

[0076] In this embodiment, specifically, the step S11 includes:

[0077]

[0078] In the formula:

[0079] is the neighborhood directional contrast of the source image VIS_Img;

[0080] is the neighborhood directional contrast of the source image IR_Img;

[0081] is the homogeneity of the source image in the i direction on the 2 i resolution, and the mean value is centered on the point (m, n);

[0082] is the brightness of the source image in the i direction on the 2 i resolution, and the mean value is centered on the point (m, n).

[0083] In this embodiment, specifically, the step S12 includes:

[0084]

[0085] wherein:

[0086] t a is the similarity of the two source images;

[0087] S, T are the size of the local region; that is, SxT is the size of the local region, which is set to 3x3, and the matching degree threshold T * is 0.6, and the fusion algorithm conforming to the image characteristics is selected by comparing the similarity t α and the size of the matching degree threshold T * .

[0088] In the embodiment, specifically, the region energy of the two source images is calculated by the following formula:

[0089]

[0090]

[0091] wherein:

[0092] MxN represents the size of the image region; preferably, it is set to 3x3;

[0093] and are the average gray values of the source images VIS_Img and IR_Img in the region MxN, respectively;

[0094] E v and E i are the region energies of the source images VIS_Img and IR_Img, respectively.

[0095] In the embodiment, specifically, the global neighborhood variance of the two source images is calculated by the following formula:

[0096]

[0097]

[0098] wherein:

[0099] KxL represents the region size; preferably, it is set to 3x3;

[0100] P VIS (m,n) represents the pixel value of the source image VIS_Img at the point (m, n);

[0101] P IR (m,n) represents the pixel value of the source image IR_Img at the point (m, n);

[0102] μ VISa mean value of the source image VIS_Img;

[0103] μ IR a mean value of the source image IR_Img;

[0104] S v and S i are global neighborhood variances of the source images VIS_Img and IR_Img respectively.

[0105] In the embodiment, specifically, the adaptive threshold factor is calculated by the following formula:

[0106]

[0107] In the formula:

[0108] A t is the adaptive threshold factor.

[0109] In the embodiment, specifically, when the wavelet transform image fusion algorithm is selected to perform image fusion processing on the source images VIS_Img and IR_Img, the step S3 comprises:

[0110] Step S31: creating a convolution kernel of KxL size to perform wavelet decomposition on the source images VIS_Img and IR_Img to obtain wavelet decomposition coefficients f v and f i of the source images; preferably, a convolution kernel of 3x3 size is created;

[0111] Step S32: calculating the average change rate C t of the two source images VIS_Img and IR_Img;

[0112] Step S33: comparing the average change rate C t and the adaptive threshold factor A t , if C t >A t , adopting the weighted average fusion rule to obtain the wavelet coefficient f1 of the fusion image; if C t <A t , calculating the wavelet coefficient f0 of the fusion image according to the modulus maximum fusion rule;

[0113] Step S34: performing inverse wavelet transform on the wavelet coefficient f1 or f0 of the fusion image to obtain an image based on wavelet transform image fusion, denoted as Img_wave.

[0114] In the embodiment, specifically, when the NSCT image fusion algorithm is selected to perform image fusion processing on the source images VIS_Img and IR_Img, the step S3 comprises:

[0115] Step S3A: multi-layer NSCT transform is performed on the two source images VIS_Img and IR_Img to obtain the coefficients corresponding to the two source images, i.e. the obtained coefficients are n v ={low frequency coefficients nl1, high frequency coefficients nh1}, n i ={low frequency coefficients nl2, high frequency coefficients nh2};

[0116] Step S3B: the average change rate C t of the two source images VIS_Img and IR_Img is calculated.

[0117] Step S3C: the average change rate C t and the adaptive threshold factor A t are compared, if C t >A t , the weighted average fusion rule is adopted to obtain the NSCT coefficient n1 of the fused image, if C t <A t , the modulus maximum fusion rule is adopted to calculate the NSCT coefficient n0 of the fused image.

[0118] Step S3D: NSCT inverse transform is performed on the NSCT coefficient n1 or n0 of the fused image to obtain the image fused based on the NSCT transform image, denoted as Img_nsct.

[0119] In the embodiment, the average change rate C t is calculated by the following formula:

[0120]

[0121] In the formula:

[0122] C t is the average change rate.

[0123] The above embodiments only express the specific implementation of the present application, and the description is more specific and detailed, but it cannot be understood as the limitation of the protection scope of the present application. It should be pointed out that for ordinary skilled in the art, without departing from the technical scheme concept of the present application, a number of modifications and improvements can be made, which are all within the protection scope of the present application.

[0124] This Background section is provided for the purpose of generally presenting the context of the application The work of the present inventors, to the extent it is described in this Background section, and the work of others in this field as described in this section, as well as the work described in this section as it relates to the present application, is not necessarily prior art and should not be considered as such in interpreting the scope of the present application.

Claims

1. A visible light and infrared fusion method for adaptive image characteristics, characterized in that, include: Step S1: Calculate the similarity between the two source images, and select the corresponding fusion algorithm based on the similarity and a preset matching threshold; Step S2: Calculate the region energy and global neighborhood variance of the two source images, and calculate the adaptive threshold factor based on the region energy and global neighborhood variance; Step S3: Input the adaptive threshold factor into the selected fusion algorithm to perform image fusion processing; Step S1 includes: Step S11: Calculate the neighborhood directional contrast of the source images VIS_Img and IR_Img respectively; Step S12: Calculate the similarity between the two source images based on the neighborhood orientation contrast of the source images VIS_Img and IR_Img; Step S13: If the similarity is greater than the matching threshold, select the wavelet transform image fusion algorithm to perform image fusion processing on the source images VIS_Img and IR_Img; if the similarity is less than the matching threshold, select the NSCT image fusion algorithm to perform image fusion processing on the source images VIS_Img and IR_Img. When the wavelet transform image fusion algorithm is selected to perform image fusion processing on the source images VIS_Img and IR_Img, step S3 includes: Step S31: Create K × L A convolution kernel of a certain size is used to perform wavelet decomposition on the source images VIS_Img and IR_Img to obtain the wavelet decomposition coefficients of the source images. f v and f i ; Step S32: Calculate the average rate of change of the two source images VIS_Img and IR_Img. ; Step S33: Compare the average rate of change and adaptive threshold factor The size, if > The wavelet coefficients of the fused image are obtained by applying a weighted average fusion rule. f 1; if < The wavelet coefficients of the fused image are calculated according to the modulus maxima fusion rule. f 0; Step S34: Calculate the wavelet coefficients of the fused image. f 1 or f Perform inverse wavelet transform on 0 to obtain an image based on wavelet transform image fusion, denoted as Img_wave; When the NSCT image fusion algorithm is selected to perform image fusion processing on the source images VIS_Img and IR_Img, step S3 includes: Step S3A: Perform multi-level NSCT transformation on the two source images VIS_Img and IR_Img to obtain the corresponding coefficients of the two source images, i.e., the corresponding coefficients are n. v ={low-frequency coefficient nl1, high-frequency coefficient nh1}, n i ={low-frequency coefficient nl2, high-frequency coefficient nh2}; Step S3B: Calculate the average rate of change of the two source images VIS_Img and IR_Img. ; Step S3C: Compare the average rate of change and adaptive threshold factor The size, if > A weighted average fusion rule is applied to obtain the NSCT coefficients n1 of the fused image. < The NSCT coefficients n0 of the fused image are calculated according to the modulus maximum fusion rule; Step S3D: Perform inverse NSCT transform on the NSCT coefficients n1 or n0 of the fused image to obtain an image based on NSCT transform image fusion, denoted as Img_nsct.

2. The visible light and infrared fusion method for adaptive image characteristics according to claim 1, characterized in that, Step S11 includes: In the formula: The neighborhood directional contrast of the source image VIS_Img; The neighborhood directional contrast of the source image IR_Img; For the source image in 2 i In terms of resolution i Uniformity in direction at points ( m , n The mean centered at ) For the source image in 2 i In terms of resolution i Brightness in direction at point ( m , n The mean centered at ).

3. The visible light and infrared fusion method for adaptive image characteristics according to claim 2, characterized in that, Step S12 includes: In the formula: The similarity between the two source images; S , T This represents the size of the local region.

4. The visible light and infrared fusion method for adaptive image characteristics according to claim 3, characterized in that, The region energy of the two source images is calculated using the following formula: In the formula: M × N Indicates the size of the image region; and These are the source images VIS_Img and IR_Img in the region. M × N The average gray level; and These are the region energies of the source images VIS_Img and IR_Img, respectively.

5. The visible light and infrared fusion method for adaptive image characteristics according to claim 4, characterized in that, The global neighborhood variance of the two source images is calculated using the following formula: In the formula: K × L Indicates the size of the region; P VIS ( m , n () represents the pixel value of the source image VIS_Img at point (m, n); P IR ( m , n () represents the pixel value of the source image IR_Img at point (m, n); This represents the mean value of the source image VIS_Img; This represents the mean value of the source image IR_Img; S v and S i These are the global neighborhood variances of the source images VIS_Img and IR_Img, respectively.

6. The visible light and infrared fusion method for adaptive image characteristics according to claim 5, characterized in that, The adaptive threshold factor is calculated using the following formula: In the formula: This is an adaptive threshold factor.

7. The visible light and infrared fusion method for adaptive image characteristics according to claim 6, characterized in that, The average rate of change Calculated using the following formula: 。

Images