A method for fusion of infrared image and visible light image based on non-global pre-enhancement

Abstract

The present invention relates to a method for fusing infrared and visible light images based on non-global pre-enhancement. Under weak infrared background energy conditions, the method considers the morphology of the local image space and employs a fuzzy processing method more suitable for infrared image structure. The image is decomposed into a target region, a transition region, and a background region. The background region is then histogram-enhanced using the FPDE algorithm, resulting in an enhanced image through reconstruction. A hybrid fusion strategy based on the expected value maximization algorithm and principal component analysis is designed to address the differences between the features of infrared and visible light images. Compared to existing technologies, the present invention achieves superior fusion performance.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a method for fusing infrared images and visible light images based on non-global pre-enhancement. Background Art

[0002] Visible light cameras typically sense wavelengths between 380nm and 780nm. Consequently, visible light images possess high contrast, spatial resolution, and rich detail, more consistent with the perception of the human eye. However, visible light imaging is highly dependent on a favorable imaging environment. Poor lighting, smoke, and other obstacles can significantly affect visible light imaging.

[0003] Over the years, researchers in academia have proposed various fusion algorithms based on diverse mathematical theories. Image fusion involves algorithmically processing source images of the same target, collected by multiple sensors, to maximize the extraction of information of interest and synthesize a highly usable enhanced image. The goal of image fusion is to combine two source images, one from an infrared sensor and one from a visible light sensor, by assigning different weights to them through a fusion algorithm. This results in a single image of the actual scene. The final image contains the information of greatest interest to humans from the source images: the visible light image provides information perceptible to the human eye, while the infrared image provides complementary information with high thermal radiation. The fusion of infrared and visible light images is a widely used branch of image processing, primarily in military surveillance, medical imaging systems, and unmanned driving. The key issue is how to maximize the fusion of information from different sensors so that the resulting single image produces the most realistic scene. Therefore, the key to image fusion technology is to ensure that the fused image reflects the complementary information of the source images to the greatest extent possible.

[0004] To achieve this goal, scholars both domestically and internationally have proposed numerous image fusion methods. Based on their theoretical foundations, these algorithms can be broadly categorized into non-multiscale and multiscale methods. Methods based on multiscale transformations primarily consider the hierarchical distribution of images. For example, Durga et al. proposed an image fusion algorithm based on a fourth-order partial differential equation (FPDE). The FPDE algorithm separates an image into high-frequency and low-frequency components. The high-frequency component contains detailed information such as edges, while the low-frequency component contains background information. They further verified that a fusion strategy based on principal component analysis (PCA) is more suitable for processing high-frequency regions, while using average weights for low-frequency regions can more quickly complete the fusion task. Furthermore, Li et al. proposed an image fusion algorithm based on latent low-rank decomposition (MDlatLRR). Wavelet transforms, guided filtering, and non-subsampled shearlet transforms have also been applied in the field of image fusion. The common thread among these multiscale transformation-based methods is that they extract interesting features from the source image in the transform domain, decompose the image into sub-images of different scales, and apply appropriate fusion decisions to these sub-images to reconstruct the true scene image. In contrast, there are non-multi-scale image fusion methods, such as the sparse representation-based image fusion method (JSR) proposed by Liu et al. This method uses a saliency detection model to extract redundant regions and complementary information from the source images, and then uses a saliency map calculated from sparse coefficients to guide image fusion. With the rapid development of deep learning, combining deep learning algorithms with image fusion has become a research hotspot. The cascaded convolutional neural network (Densefuse) algorithm proposed by Li et al. uses an encoding network consisting of convolutional layers, fusion layers, and dense blocks to extract image features. The decoder network then reconstructs the fused image, with the output of each layer jumping to the next layer. Similar image fusion algorithms based on deep learning frameworks include ResNet, CNN, VggML, and IFCNN. Furthermore, many researchers have pioneered the application of generative adversarial networks to image fusion tasks, such as FusionGAN, DDcGAN, and MFF-GAN.

[0005] Due to the significant difference in brightness response between infrared and visible light images, areas with lower light intensity typically have extremely low contrast. When their brightness levels are not complementary, details from the infrared image can degrade the original perceived information in the visible light image. Directly fusing the two images results in poor visibility in the fused image in non-complementary areas. Therefore, contrast enhancement of the infrared image information is essential. For example, in the field of autonomous driving, potholes, trees, and certain obstacles on nighttime roads may not be clearly reflected in visible light images, requiring the assistance of infrared light. However, the thermal radiation information from these objects is not very strong, so improving the visibility of the fused image is essential to help assisted driving more accurately avoid obstacles in poor lighting conditions.

[0006] To date, many researchers have proposed various nonlinear enhancement algorithms to address the low-contrast image problem in certain scenarios. However, while the background areas of infrared images have low contrast, the salient areas have high contrast and do not require enhancement. Applying a global, unified enhancement rule to the entire infrared image can result in overexposure in some areas after enhancement, and halos can appear in the infrared texture. Summary of the Invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide a method for fusing infrared images and visible light images based on non-global pre-enhancement.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] A method for fusing infrared images and visible light images based on non-global pre-enhancement, the method comprising the following steps:

[0010] S1: Based on the FRFCM algorithm, perform mask processing on the infrared image to obtain the background area image and detail image of the infrared image, and perform mapping operation on the visible light image to obtain the target area of ​​the visible light. Specifically:

[0011] Based on the FRFCM algorithm, an unsupervised learning method is used to segment the infrared image into three basic areas: background area, transition area and detail part. The mask image of the background area is multiplied with the infrared source image at the pixel level to obtain the extracted image of the background area of ​​the infrared image. The infrared source image and the background area are subtracted to obtain the extracted image of the target area. The target area of ​​the original infrared image is mapped to the visible light image. The mask image of the background area of ​​the infrared image is multiplied with the visible light image at the pixel level to obtain the extracted image of the background area of ​​the visible light image. The extracted image of the background area of ​​the visible light image is subtracted from the visible light source image to obtain the extracted image of the target area.

[0012] S2: Enhance the background area of ​​the infrared image based on the FPDE binary histogram. The specific content is:

[0013] FPDE is used as the guided decomposition algorithm to divide the original infrared image into a base layer image and a detail layer image:

[0014] IR b =FPDE(IR original )

[0015] IR d =IR original -IR b

[0016] Where FPDE is the function of low-pass filtering the source image, IRoriginal is the original infrared image, IR b For the base layer image, IR d is the detail layer image;

[0017] The base layer image IR is calculated as follows: b Histogram based on threshold Figure 2 Value processing:

[0018] G=(S max -S min )*β+S min

[0019] Where G is the binarization histogram threshold, S max and S min are the maximum and minimum values ​​in the histogram of the base layer image, respectively. β is a parameter ranging from 0 to 1, which determines the proportion of invalid pixel values ​​in the image.

[0020] S3: For the visible light source image and the enhanced infrared image, the corresponding FPDE energy functional is established based on the human visual characteristics that are sensitive to local image transformations, and each image is decomposed into high-frequency detail areas and low-frequency background areas.

[0021] S4: Obtain a fusion result of low-frequency components of the low-frequency visible light image and the low-frequency infrared image. Specifically:

[0022] First, establish the image fusion model:

[0023] S N (X,j)=α(X,j)S N (F,j)+β(X,j)+ε(X,j)

[0024] Where, X = A or B represents the infrared sensor mark or visible light sensor mark; j = (x, y) represents the pixel position of the original image X; S N (F, j) is the pixel value of the fused low-frequency image at position j; α(X, j) = ±1 or 0 is the distribution coefficient of the sensor, indicating the degree of participation of each sensor; ε(X, j) represents random noise, which obeys the probability density function of the K-term mixed Gaussian distribution:

[0025]

[0026] Where λ k,X (j) is the weight of the K-term Gaussian distribution, which is used to characterize the degree to which the distribution characteristics tend to a certain Gaussian distribution. is the variance of each Gaussian distribution;

[0027] The complete observation data Y c Defined as:

[0028] Y c ={(S N (X,l),k(X,l)):X=AorB;l=1,…,L}

[0029] Where k(X,l) represents the Gaussian mixture distribution density function, which generates S N The comprehensive F of the estimated parameters is recorded as follows:

[0030]

[0031] Its marginal probability density function is:

[0032]

[0033] Where h c (S N (X, l), k(X, l) | F) is the incomplete data Y under the parameter condition F c The marginal probability density function of N (F,l),α′(X),λ′ k,X ,σ′ k,X Repeat steps S1 to S4, and when the parameter value converges to a certain small range, stop the iteration and calculate the next window area; when all the pixel positions of the low-frequency component are scanned, the fusion result S′ of the low-frequency component is obtained. N (F,l).

[0034] S5: Obtain the fusion result of the high-frequency components of the high-frequency visible light image and the high-frequency infrared image. That is, PCA transformation is performed on the high-frequency visible light image and the high-frequency infrared image, and the principal components are obtained in sequence according to the vector eigenvalue sorting. The high-frequency image is then histogram matched and the fused high-frequency image is obtained by addition.

[0035] Specifically, pixels are extracted from the windows of the high-frequency visible light image and the high-frequency infrared image and concatenated into an array X with a dimension of MN*2. The mean vector, i.e., the mathematical expectation of the array, is calculated, which is defined as:

[0036]

[0037] Where K = M × N, and the covariance definition formula C x =E{(xm x )(xm x ) T}, we can randomly sample the M vector and find its covariance matrix:

[0038]

[0039] Assume e i and λ i (i=1,2,...,N) is C x The eigenvectors and corresponding eigenvalues ​​of are arranged in reverse order so that λ j ≥λ j+1 , j=1,2,...n-1; construct a matrix A, whose first row is the eigenvector corresponding to the maximum eigenvalue of C, and the last row is the eigenvector corresponding to the minimum eigenvalue of C; the expectation of the vector in Y after principal component transformation is 0; then pass A and C x Find the covariance matrix of y:

[0040] C y =A·C x ·A T

[0041] Find C y Maximum eigenvalue λ max =max(λ1,λ2), max The corresponding eigenvector is the largest eigenvector e max , and e is calculated by the following formula max The principal components P1 and P2 are normalized:

[0042]

[0043] The weights determined by principal component analysis are used to obtain the fused image with the largest brightness variance, that is, the high-frequency fused image D fuse :

[0044]

[0045] S6: reconstructing the obtained fusion result of the low-frequency component and the fusion result of the high-frequency component to obtain a final fused image.

[0046] The infrared image and visible light image fusion method based on non-global pre-enhancement provided by the present invention has at least the following beneficial effects compared with the existing technology:

[0047] Under the condition of weak energy of infrared light background, the present invention considers the morphology of the local space of the image and adopts a fuzzy processing method that is more suitable for the structure of infrared images. The image is decomposed into a target area, a transition area and a background area, and the background area is enhanced by histogram enhancement based on the FPDE algorithm. An enhanced image is obtained by reconstruction. The enhancement effect of the background area of ​​the infrared image is significant, and there is no problem of overexposure of some areas, and there is no halo phenomenon in the infrared light texture part. Since the frequency of the high-frequency components close to the low-pass is relatively low, if different areas of the two source images are directly fused, it will lead to loss of edge information or loss of high-frequency fine features, and the overall image after fusion will be unnatural. In addition, the low-frequency component has a poor ability to express image details, and image distortion and other phenomena will occur during fusion. Based on this, a hybrid fusion strategy based on the maximum expected value algorithm and principal component analysis is designed to target the differences between the various features of infrared and visible light images. Compared with other existing fusion methods, it has better fusion performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Figure 1 2 is a flowchart of a method for fusing infrared images and visible light images based on non-global pre-enhancement according to an embodiment of the present invention;

[0049] Figure 2 This is the visualization of the infrared light background enhancement process in the embodiment, where column (a) is the four groups of IR source images, columns (b) and (c) are the fuzzy decomposition process, columns (d) and (e) are the enhancement process, and column (f) is the final enhanced image. DETAILED DESCRIPTION

[0050] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the embodiments described are only a portion of the embodiments of the present invention, not all of them. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without inventive effort should fall within the scope of protection of the present invention.

[0051] Example

[0052] This paper provides a method for fusing infrared and visible light images based on non-global pre-enhancement. The method consists of two main stages: first, regional features in the infrared image are blurred using the FRFCM method, thereby dividing the background and target regions into two parts. Then, low-frequency background information in the infrared light source image is enhanced based on FPDE and histogram, while detail regions retain their original grayscale. This method can better account for the structural differences between the infrared and visible light images, preserving the infrared thermal radiation information while integrating the visible light detail information to the greatest extent possible.

[0053] Since the contrast of visible light image is low, it is impossible to directly use FRFCM to extract the target area corresponding to the infrared image. Therefore, the present invention performs mask processing on the infrared image and then performs mapping operation with the visible light image to obtain the target area of ​​visible light, such as Figure 1 As shown, the method of the present invention specifically comprises the following steps:

[0054] 1. Generation of pre-fused images:

[0055] While traditional FCMs perform well in images without similar information interference, they only consider pixel grayscale information, which can lead to misclassification of areas similar to the target, such as streetlights and trees. To address the over-segmentation problem, the FRFCM (Further FCM based on morphological reconstruction and filtering) model introduces local spatial information segmentation before clustering, optimizes the image's distribution characteristics, and adds non-local spatial constraints to the objective function, making the image's histogram distribution more conducive to clustering and suppressing interference from similar background information. The infrared image segmentation parameter is set to clustering number c = 2, effectively dividing the image into two categories. Furthermore, a binarization method is used to mask the high-energy salient target regions and low-energy background regions in the infrared source image. For different i∈c, the value is set to 0 or 1. i = 1 represents background regions, such as the sky, ground, and trees, with low thermal radiation. The pixel value of these regions is set to 0. i = 2 represents target regions, such as people and streetlights, with high thermal radiation. The pixel value of these regions is set to 1, and the grayscale mean of these regions is higher than that of the background regions.

[0056] Then, the mask image of the background area is multiplied with the infrared light source image at the pixel level to obtain the extracted image of the background area of ​​the infrared image. Use infrared source image and background area Perform subtraction to obtain the extracted image of the target area Since the target thermal radiation of the visible light image is low, if it is directly segmented, most of the details will be lost. Therefore, the target area of ​​the original infrared is used to map the visible light image, and the mask image of the infrared background area is multiplied by the visible light image pixel level to obtain the extracted image of the visible light image background area. And subtract it from the source image to obtain the extracted image of the target area During the mapping process, the salient target area and background area of ​​the visible light image are inverted through a mask to achieve the segmentation of the source image.

[0057] The background feature modeling process based on FCM is as follows:

[0058] The infrared image IR can be expressed as X = {x1, x2, ..., x N}, where xj is the sample element, i.e. the gray value of the jth pixel. N is the total number of pixels in the infrared image X. Under the condition of being divided into i categories of fuzzy membership, the mathematical programming method can be used to solve the clustering result that meets the requirements, and the objective function J is defined as FRFCM for:

[0059]

[0060] The c subsets are represented as V = {v1, v2, ..., v c}, is the membership partition matrix, Represents pixel x in the infrared image i The gray value l corresponds to the fuzzy membership of the i-th category; ||ξ l -v k || 2 for ξ l To the sample center v k The parameter m is the weighted index on each fuzzy membership, which is used to determine the fuzziness of the classification result. It determines the fuzziness of the classification result, m∈[1,∞). The larger its value, the more fuzzy the classification.

[0061] In the classification task of FRFCM, m=2 is generally taken. l is the number of grayscale values ​​in the image equal to l, and ξ l is a linearly-weighted sum image factor, which is preformed by the original image and its local neighborhood average image, and is expressed as:

[0062]

[0063] Among them, x l represents the pixel value at pixel position l (l=1, 2, 3..., q), N R Represents x l The set of neighborhood L*L window pixels, α is a key parameter used to weigh the relationship between the image and its corresponding mean filtered image. When α = 0, the parameter ξ l Invalid, equivalent to the traditional FCM. When α→∞, the clustering obtained is the result of FCM after median filtering. For the above mathematical programming problem, the Lagrange multiplier method is used to obtain the minimum value. By introducing the parameter λ j (j=1,2,3...,r), combine the objective function and the constraints into a new function:

[0064]

[0065] in, Represents the membership matrix. Iterate and update the cluster center repeatedly, and update the membership matrix at the same time. The algorithm stops iterating until the updated membership matrix and the previous matrix meet ε=0.0001. The maximum number of iterations is 100. In this process, the objective function J is constantly changing. and v i The algorithm is considered to have converged when J gradually approaches a stable value, at which point the cluster centers and membership matrix are obtained. The present invention segments the infrared image into three basic regions: R1, R2, and R3. The segmentation results are reflected in the segmented image, where R1 represents the black region (background region), R2 represents the gray region (transition region), and R3 represents the white region (details). By multiplying with the source image, an image of the background region and a detail image can be obtained.

[0066] 2. Detailed area decomposition:

[0067] From a visual perspective, the low-frequency components of an image contain background information. This part of the image primarily provides features similar to the source image and is referred to as an approximate image. The texture information of the high-frequency components primarily provides gradient components of details or edges and is referred to as a detail image. Since the high-frequency components close to the low-pass filter have a lower frequency, directly fusing different regions of the two source images will result in loss of edge information or high-frequency fine features, and the fused image will appear unnatural overall. Furthermore, the low-frequency components have a poor ability to express image details, resulting in image distortion during fusion. Therefore, it is necessary to establish a corresponding FPDE energy functional for the enhanced IR image (infrared image) and VIS source image (visible light image), based on the human visual characteristic of being sensitive to local image transformations, to decompose the image into high-frequency detail regions and low-frequency background regions.

[0068] Unlike general frequency-domain digital image processing tools, methods based on partial differential equations (PDEs) treat images as continuous objects. By iterating infinitesimal operations on the image, the image decomposition can be made more accurate. The basic idea of ​​the FPDE method is to smooth the image while maintaining the original information of the source image, achieving a certain degree of optimization. Based on the human visual characteristic of being sensitive to local image transformations, a corresponding energy functional is established, and the image decomposition process is transformed into a problem of minimizing the energy functional. First, to find the minimum value of the energy functional, an energy functional is established in the continuous image space under the Ω set. Then, the source image is decomposed into an approximate image through an edge-preserving decomposition process (SAVIS, SAIR).

[0069] This paper proposes to use binarized histogram processing based on fourth-order partial differential equations (FPDE) to adaptively enhance the background area of ​​infrared images. FPDE has good smoothing and edge preservation properties and good real-time performance. Using FPDE as a guided decomposition algorithm, the original infrared image is divided into a base layer image and a detail layer image:

[0070] IR b =FPDE(IR original )

[0071] IR d =IR original -IR b

[0072] In the above two formulas, FPDE is the function of low-pass filtering the source image, IR original Represents the original image, IR b For the base layer image, IR d For the detail layer image. Base layer IR b The image contains rich scene information of the original image. b The binarization histogram threshold G is as follows:

[0073] G=(S max -S min )*β+S min

[0074] Where S max and S min are the maximum and minimum values ​​in the base layer image histogram, respectively. β is a parameter ranging from 0 to 1 that determines the proportion of invalid pixel values ​​in the image. Grayscale values ​​within a 95% confidence interval are generally considered valid, while the remaining 5% are considered invalid. Therefore, the β value in the above formula is set to 0.05.

[0075] Figure 2 The process of visual infrared light background enhancement of the present invention is shown. (a) is 4 groups of IR source images, (b) and (c) are the fuzzy decomposition process, (d) and (e) are the enhancement process, and (f) is the final enhanced image. Figure 2 It can be seen that the enhancement effect of the background area of ​​the infrared image is significant, and there is no problem of overexposure in some areas, and there is no halo phenomenon in the infrared light texture part.

[0076] 3. Integration strategy:

[0077] Due to the different image structures of the background and target regions, direct fusion will result in some information loss. Therefore, a fast fusion strategy tailored to the image structure is developed for the FPDE decomposed image. For the target region, consideration is given to maximizing the preservation of the infrared image's high-brightness information while integrating the visible image's texture details. Image fusion methods based on statistical models can reduce the impact of noise on the fusion result and enhance the signal-to-noise ratio of the fused image. Therefore, this algorithm can reduce noise interference in the visible image without causing noticeable artifacts, making this statistical model well-suited for target region fusion. However, the statistical characteristics of the image approximately follow a Gaussian mixture model. Simulating the passband signal as noise can easily lead to loss of high-frequency information in the fusion result. Therefore, a non-Gaussian distribution (Gaussian mixture distribution) is assumed for the image. Furthermore, given the varying offsets of different sensors relative to the scene, the imaging sensor offset parameters are incorporated into the image formation model. A 5×5 neighborhood window centered on pixel j, i.e., the entire image, is processed using a sliding window approach. The resulting new parameters are then substituted back into the pre-fusion image generation step and the detail region decomposition step, and the process continues. During the above iterative process, when the parameters converge and stabilize, the target area image obtained by fusion of the low-frequency components is determined. For the fusion of high-frequency components in the background areas of the infrared and visible light images, an algorithm based on principal component analysis (PCA) is selected. The dimensionality reduction processing method of principal component analysis is often used in image processing to easily obtain texture and detail information at large scales. Therefore, applying PCA to the fusion of background areas can effectively capture details, lines, and edges in the background area, thereby preserving the key details of the image.

[0078] Specifically, first, the present invention establishes an image fusion model as follows:

[0079] S N (X, j) = α(X, j)S N (F, j) + β(X, j) + ε(X, j)

[0080] Where X = A or B represents the infrared or visible light sensor marker; j = (x, y) represents the pixel position of the original image X. N (F, j) is the pixel value at position j in the fused low-frequency image and is a parameter to be estimated. α(X, j) = ±1 or 0 refers to the distribution coefficient of the sensor, indicating the degree of participation of each sensor. ε(X, j) represents random noise and follows the probability density function of a K-term mixed Gaussian distribution:

[0081]

[0082] Among them, λ k,X(j) is the weight of the K-term Gaussian distribution, which is used to characterize the degree to which the distribution characteristics tend to a certain Gaussian distribution. is the variance of each Gaussian distribution. k,X (j) is the standard deviation of each Gaussian distribution.

[0083] The complete observation data Y c The definition is as follows:

[0084] Y c ={(S N (X,l),k(X,l)):X=AorB;l=1,…,L}

[0085] Among them, k(X,l) represents the Gaussian mixture distribution density function, which generates S N The comprehensive F of the parameters to be estimated is recorded as:

[0086]

[0087] Its marginal probability density function is:

[0088]

[0089] Among them, h c (S N (X, l), k(X, l)|F) is the incomplete data Y under the parameter condition F c The marginal probability density function of . Using the updated parameters S′ N (F, l), α′(X), λ′ k，X ,σ′ k，X Repeat the steps of generating the pre-fused image and decomposing the detail area with β′(X). When the parameter value converges to a certain small range, stop the iteration and proceed to the calculation of the next window area. When all the pixel positions of the low-frequency component are scanned, the fusion result S′ of the low-frequency component is obtained. N (F,l).

[0090] By calculating the principal components that are relatively few in number relative to the original sample, replacing all dimensions of the original sample, the original data can be represented as much as possible, and they are unrelated to each other, thereby achieving data dimensionality reduction and converting the high-frequency visible light image S VIS·D (i, j) and high-frequency infrared light image S IR·D (i, j) is transformed by PCA, and the principal components P1 and P2 are obtained in sequence according to the order of the vector eigenvalues. Then the high-frequency image is matched with P1 and P2 by histogram, and the fused high-frequency image is obtained by adding them. Specifically:

[0091] First, extract pixels from the windows of the two high-frequency images and concatenate them into an array X of dimension MN*2. Then calculate the mean vector, which is the mathematical expectation of the array, which is defined as:

[0092]

[0093] Where K = M × N. The covariance definition formula C x =E{(xm x )(xm x ) T}, we can randomly sample the M vector and find its covariance matrix:

[0094]

[0095] Assume e i and λ i (i=1,2,...,N) is C x The eigenvectors and corresponding eigenvalues ​​of are arranged in reverse order so that λ j ≥λ j+1 ,j=1,2,...n-1. Construct a matrix A so that its first row is the eigenvector corresponding to the maximum eigenvalue of C, and its last row is the eigenvector corresponding to the minimum eigenvalue of C. The expectation of the vector in Y after principal component transformation is 0. Next, through A and C x Find the covariance matrix of y:

[0096] C y =A·C x ·A T

[0097] Find C y Maximum eigenvalue λ max =max(λ1,λ2), and max The corresponding eigenvector is regarded as the largest eigenvector e max , and e is obtained by the following formula max The principal components P1 and P2 are normalized:

[0098]

[0099] The weights determined by principal component analysis can be used to obtain a fusion image with the largest brightness variance, that is, the high-frequency fusion image D fuse :

[0100]

[0101] The low-frequency fusion image and the high-frequency fusion image obtained according to the above method are reconstructed to obtain the final enhanced fusion image.

[0102] Under the condition of weak energy of infrared light background, the present invention considers the morphology of the local space of the image and adopts a fuzzy processing method that is more suitable for the structure of infrared images. The image is decomposed into a target area, a transition area and a background area, and the background area is enhanced by histogram enhancement based on the FPDE algorithm. An enhanced image is obtained by reconstruction. The enhancement effect of the background area of ​​the infrared image is significant, and there is no problem of overexposure of some areas, and there is no halo phenomenon in the infrared light texture part. Since the frequency of the high-frequency components close to the low-pass is relatively low, if different areas of the two source images are directly fused, it will lead to loss of edge information or loss of high-frequency fine features, and the overall image after fusion will be unnatural. In addition, the low-frequency component has a poor ability to express image details, and image distortion and other phenomena will occur during fusion. Based on this, a hybrid fusion strategy based on the maximum expected value algorithm and principal component analysis is designed to target the differences between the various features of infrared and visible light images. Compared with other existing fusion methods, it has better fusion performance.

[0103] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and such modifications or substitutions are intended to be within the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.

Claims

1. A method for fusion of infrared image and visible light image based on non-global pre-enhancement, characterized in that: The following steps are involved: 1) Based on the FRFCM algorithm, the infrared image is masked to obtain the background area image and detail image of the infrared image, and the visible light image is mapped to obtain the target area of ​​the visible light; 2) Enhance the background area of ​​the infrared image based on the binary histogram of FPDE; 3) For visible light source images and enhanced infrared images, we establish corresponding FPDE energy functionals based on the human visual characteristics of being sensitive to local image transformations, and decompose each image into high-frequency detail regions and low-frequency background regions; 4) Obtaining the fusion result of low-frequency components of low-frequency visible light images and low-frequency infrared images; 5) Obtain the fusion result of high-frequency components of high-frequency visible light images and high-frequency infrared images; 6) Reconstruct the obtained fusion results of the low-frequency components and the high-frequency components to obtain the final fused image.

2. The infrared image and visible light image fusion method based on non-global pre-enhancement according to claim 1, characterized in that: The specific contents of step 1) are: Based on the FRFCM algorithm, an unsupervised learning method is used to segment the infrared image into three basic areas: background area, transition area and detail part. The mask image of the background area is multiplied with the infrared source image at the pixel level to obtain the extracted image of the background area of ​​the infrared image. The infrared source image and the background area are subtracted to obtain the extracted image of the target area. The target area of ​​the original infrared image is mapped to the visible light image. The mask image of the background area of ​​the infrared image is multiplied with the visible light image at the pixel level to obtain the extracted image of the background area of ​​the visible light image. The extracted image of the background area of ​​the visible light image is subtracted from the visible light source image to obtain the extracted image of the target area.

3. The infrared image and visible light image fusion method based on non-global pre-enhancement according to claim 1, characterized in that: The specific content of enhancing the background area of ​​the infrared image based on the FPDE binary histogram is as follows: FPDE is used as the guided decomposition algorithm to divide the original infrared image into a base layer image and a detail layer image: Where, is a function for low-pass filtering the source image. is the original infrared image, is the base layer image, is the detail layer image; The base layer image is calculated as follows: Perform histogram binarization according to the threshold: Where, G is the binarization histogram threshold, and are the maximum and minimum values ​​in the histogram of the base layer image, It is a parameter ranging from 0 to 1 that determines the proportion of invalid pixel values ​​in the image.

4. The infrared image and visible light image fusion method based on non-global pre-enhancement according to claim 1, characterized in that: The specific contents of step 4) are: First, establish the image fusion model: Where, or Indicates an infrared sensor mark or a visible light sensor mark; Represents the original image The pixel position of The fused low-frequency image is The pixel value at ; Or 0 is the distribution coefficient of the sensor, which indicates the degree of participation of each sensor. β(X,j) is the offset parameter of the imaging sensor, specifically the offset pixel value of the image X at 𝑗; represents random noise, obeying Probability density function of the Gaussian mixture distribution : Where, for The weight of the Gaussian distribution is used to characterize the degree to which the distribution characteristics tend to a certain Gaussian distribution. ; is the variance of each Gaussian distribution; The data will be fully observed Defined as: Where, Indicates that in the Gaussian mixture distribution density function, The additional random noise term, the comprehensive Denoted as: Its marginal probability density function is: Where, For the parameter conditions For incomplete data The marginal probability density function of 、 、 、 and Repeat steps 1) to 4). When the parameter value converges to a certain small range, stop the iteration and calculate the next window area. When all the pixel positions of the low-frequency component are scanned, the fusion result of the low-frequency component is obtained. .

5. The infrared image and visible light image fusion method based on non-global pre-enhancement according to claim 1, characterized in that: In step 5), the high-frequency visible light image and the high-frequency infrared light image are subjected to PCA transformation, and the principal components are obtained in sequence according to the sorting of vector eigenvalues. The high-frequency image is then histogram matched and the fused high-frequency image is obtained by addition.

6. The infrared image and visible light image fusion method based on non-global pre-enhancement according to claim 5, characterized in that: The specific contents of step 5) are: Extract pixels from the high-frequency visible light image and high-frequency infrared image windows and splice a dimension of Array X , calculate the mean vector, that is, the mathematical expectation of the array m x , which is defined as: in, , defined by the covariance formula Available, for Vector random sampling, x k Extract pixels from the high-frequency visible light image and the high-frequency infrared image window and splice them into the k-th pixel sample in the array X of dimension MN∗2, and calculate its covariance matrix: set up and for The eigenvectors and corresponding eigenvalues ​​of are arranged in reverse order so that ; Construct a matrix , so that its first line The eigenvector corresponding to the maximum eigenvalue of The eigenvector corresponding to the minimum eigenvalue of The expectation of the vector is 0; then by and Obtain The covariance matrix of : Find out Maximum eigenvalue ,Will The corresponding eigenvector is taken as the largest eigenvector , through the following formula The principal component of and Perform normalization: The weights determined by principal component analysis are used to obtain the fused image with the largest brightness variance, that is, the high-frequency fused image. : in: S IR⋅D is the high-frequency detail area after decomposition of the infrared image, S VIS⋅D It is the high-frequency detail area image after the visible light image is decomposed.

Images