An infrared and visible image fusion method based on multi-scale decomposition and partial differential equation

Abstract

The application relates to an infrared and visible light image fusion method based on multi-scale decomposition and partial differential equations, and belongs to the technical field of image fusion. The method solves the problems of blurred target edges and details in a current fusion image and large resource consumption. The method comprises the following steps: obtaining an infrared image and a visible light image including a target object through synchronous acquisition, and carrying out pretreatment and adaptive multi-scale decomposition; the number of layers of a Gaussian pyramid is obtained based on the complexity of the visible light image and the infrared image; the pretreated visible light image and the infrared image are subjected to multi-scale decomposition by using the Gaussian pyramid corresponding to the number of layers; each decomposition image is subjected to smoothing filtering and detail enhancement to obtain an enhanced image of each scale; the enhanced images of the infrared and visible light of the same scale are subjected to dynamic fusion, and all the scale images obtained through the fusion are reconstructed to obtain a final fusion image. The method realizes the reservation of target edges and details in the fusion image by using small resources.

Description

Technical Field

[0001] This invention relates to the field of image fusion technology, and in particular to an infrared-visible image fusion method based on multi-scale decomposition and partial differential equations. Background Technology

[0002] In recent years, the fusion of infrared and visible light images has attracted considerable research interest. Visible light images have a wide dynamic range, capable of simultaneously capturing details in both bright and dark areas of an object, but they are significantly affected by lighting conditions, failing to produce clear images in poor lighting environments. Infrared images, on the other hand, are unaffected by lighting conditions and maintain good imaging quality even in low-light or adverse environments such as nighttime or hazy weather. However, infrared images are typically grayscale, lacking depth and color information, resulting in relatively blurry visuals and less detail and texture. Fusion of visible light and infrared images combines the image information from these two different imaging modalities, yielding a new image that incorporates the advantages of both. This fused image provides more comprehensive and accurate information and has broad application value.

[0003] Existing image fusion techniques, such as those based on Laplacian pyramids, wavelet transforms, and adversarial networks, have laid the foundation for image fusion technology. Despite the progress made in these techniques, problems remain, including blurred target edges and details in the resulting fused image, and high resource consumption. Summary of the Invention

[0004] Based on the above analysis, the present invention aims to provide an infrared and visible light image fusion method based on multi-scale decomposition and partial differential equations, in order to solve the problems of blurred target edges and detailed features and high resource consumption in the fused images obtained by existing fusion techniques.

[0005] This invention provides an infrared-visible image fusion method based on multi-scale decomposition and partial differential equations, comprising:

[0006] Simultaneously acquire infrared and visible light images of the target object, and preprocess the acquired visible light and infrared images respectively;

[0007] Adaptive multi-scale decomposition is performed on the preprocessed visible light image and infrared image respectively. The number of Gaussian pyramid layers is obtained based on the complexity of the visible light image and infrared image. The preprocessed visible light image and infrared image are then decomposed into infrared and visible light images at each scale using the Gaussian pyramid with the corresponding number of layers.

[0008] The infrared and visible light decomposed images were subjected to smoothing filtering based on second-order partial differential equations and detail enhancement based on fourth-order partial differential equations, respectively, to obtain images enhanced at each scale.

[0009] The enhanced images of infrared and visible light at the same scale are dynamically fused, and the images at all scales obtained by fusion are reconstructed to obtain the final fused image.

[0010] Further improvements to the above method, the steps for obtaining the complexity of visible light images and infrared images include:

[0011] The edge density of an image is obtained based on the number of edges in the image;

[0012] The entropy of an image is obtained from its grayscale matrix.

[0013] The complexity of the image is obtained based on its edge density and entropy.

[0014] Based on a further improvement of the above method, the edge density of the image is obtained using the following formula:

[0015]

[0016] Where (x, y) represents the coordinates of a pixel in the image, E(x, y) represents the edge image corresponding to the image, and N e The value represents the number of pixels containing the edge in the edge image, W and H represent the length and width of the image, respectively, and ρ represents the edge density of the image.

[0017] Based on a further improvement of the above method, the complexity of the image is obtained using the following formula:

[0018]

[0019] Where O represents the complexity of the image, S represents the entropy of the image, and Q represents the side length of the sampling window for calculating the texture.

[0020] Based on further improvements to the above method, the corresponding number of layers for the visible light image and the infrared image are calculated using the following formula, and the larger value is selected as the number of layers in the Gaussian pyramid:

[0021]

[0022] Where n represents the corresponding layer number, This indicates downsampling.

[0023] Based on the above method, further improvements are made to dynamically fuse enhanced images of the same scale in infrared and visible light using the following formula:

[0024]

[0025] I f (x,y)=w I (x,y)·I I (x,y)+w V (x,y)·I V (x,y)

[0026] Among them, G I (x,y) and G V (x, y) represent the local standard deviations of the enhanced infrared and visible light images at the same scale, respectively. I (x,y) and w V (x,y) represent the fusion weights for infrared and visible light, respectively, I I (x,y) and I V (x, y) represent the enhanced infrared and visible light images at the same scale, respectively. f (x,y) represents the dynamically fused image at the corresponding scale.

[0027] Based on a further improvement to the above method, the preprocessing of the obtained visible light image and infrared image respectively includes:

[0028] The obtained visible light image is converted to grayscale to obtain the converted grayscale image;

[0029] The converted grayscale and infrared images are dynamically rescaled to obtain preprocessed grayscale and infrared images.

[0030] Based on a further improvement of the above method, the converted grayscale image is obtained using the following formula:

[0031] G(i,j)=0.2989P(i,j,1)+0.5870P(i,j,2)+0.1140P(i,j,3)

[0032] Where (i, j) represents the coordinates of each pixel in the visible light image, 1, 2, and 3 represent the red, green, and blue color channels, respectively, P represents the acquired visible light image, and G represents the converted grayscale image.

[0033] Based on the above method, a further improvement is made, using the following formula for dynamic rescaling:

[0034]

[0035] Where G represents the converted grayscale image, P v Indicates the acquired infrared image, I greyscaled I represents the preprocessed visible light image obtained after dynamic rescaling. rescaledThis represents the preprocessed infrared image obtained after dynamic rescaling.

[0036] Based on a further improvement of the above method, the synchronous acquisition of infrared and visible light images of the target object includes:

[0037] Images of the target object are simultaneously acquired using a fixed-position infrared camera and a visible light camera, respectively, to obtain infrared and visible light images of the target object.

[0038] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0039] 1. An adaptive multi-scale decomposition algorithm is adopted to dynamically adjust the number of Gaussian pyramid layers according to the image content, thereby achieving the preservation of target edges and detailed features in the fused image with less resources;

[0040] 2. Employing partial differential equation-based smoothing and detail enhancement, the quality of the fused image is improved by selectively enhancing important details and suppressing noise points and artifacts, achieving effective smoothing, edge preservation, and detail enhancement in the fused image;

[0041] 3. The local dynamic fusion method combines complementary information from the infrared and visible light channels, improving the overall quality of the fused image.

[0042] In this invention, the above-described technical solutions can be combined with each other to achieve more preferred combinations. Other features and advantages of this invention will be set forth in the following description, and some advantages may become apparent from the description or be learned by practicing the invention. The objects and other advantages of this invention can be realized and obtained from what is particularly pointed out in the description and drawings. Attached Figure Description

[0043] The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Throughout the drawings, the same reference numerals denote the same parts.

[0044] Figure 1 This is a flowchart illustrating an infrared-visible image fusion method based on multi-scale decomposition and partial differential equations, as shown in an embodiment of the present invention. Detailed Implementation

[0045] Preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, which form part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not intended to limit the scope of the present invention.

[0046] A specific embodiment of the present invention discloses an infrared-visible image fusion method based on multi-scale decomposition and partial differential equations, such as... Figure 1As shown.

[0047] Specifically, including:

[0048] S1: Simultaneously acquire infrared and visible light images of the target object, and preprocess the acquired visible light and infrared images respectively;

[0049] S2: Perform adaptive multi-scale decomposition on the preprocessed visible light image and infrared image respectively; wherein, based on the complexity of the visible light image and infrared image, the number of Gaussian pyramid layers is obtained; the preprocessed visible light image and infrared image are decomposed into multiple scales using the Gaussian pyramid with the corresponding number of layers to obtain the infrared and visible light decomposed images corresponding to each scale.

[0050] S3: Perform smoothing filtering based on second-order partial differential equations and detail enhancement based on fourth-order partial differential equations on each infrared and visible light decomposed image to obtain images enhanced at each scale.

[0051] S4: Dynamically fuse the enhanced images of infrared and visible light at the same scale, and reconstruct the images at all scales obtained by fusion to obtain the final fused image.

[0052] Specifically, the simultaneous acquisition of infrared and visible light images of the target object includes: simultaneously acquiring images of the target object using a fixed-position infrared camera and a visible light camera, respectively, to obtain infrared and visible light images of the target object, without changing the position of the cameras during acquisition.

[0053] Further, the obtained visible light image and infrared image are preprocessed separately, including steps S11 and S12:

[0054] S11: Perform grayscale conversion on the obtained visible light image to obtain the converted grayscale image;

[0055] S12: Dynamically rescale the converted grayscale and infrared images to obtain preprocessed grayscale and infrared images;

[0056] Specifically, the converted grayscale image is obtained using the following formula:

[0057] G(i,j)=0.2989P(i,j,1)+0.5870P(i,j,2)+0.1140P(i,j,3)

[0058] Where (i, j) represents the coordinates of each pixel in the visible light image, 1, 2, and 3 represent the brightness values ​​of the red, green, and blue color channels, respectively, P represents the acquired visible light image, and G represents the converted grayscale image; where P has a size of M×N×3, where M and N represent the height and width of the visible light image, respectively, and 3 represents the three color channels. It is worth noting that since infrared images are inherently grayscale images, to simplify subsequent processing steps and unify the processing of infrared and visible light images, the visible light image also needs to be converted to grayscale.

[0059] Furthermore, dynamic rescaling is performed using the following formula:

[0060]

[0061] Where G represents the converted grayscale image, P v Indicates the acquired infrared image, I greyscaled I represents the preprocessed visible light grayscale image obtained after dynamic rescaling. rescaled This represents the preprocessed infrared image obtained after dynamic rescaling. It's worth noting that the grayscale image and the infrared image obtained after dynamic rescaling have similar dynamic ranges, making them more suitable for subsequent image processing steps such as adaptive multi-scale decomposition, partial differential equation-based smoothing, and detail enhancement, thus improving the accuracy of edge and detail feature processing.

[0062] Further, adaptive multi-scale decomposition is performed on the preprocessed visible light image and infrared image respectively, including steps S21 and S22:

[0063] S21: Based on the complexity of the visible light image and the infrared image, obtain the number of layers of the Gaussian pyramid; specifically, the steps to obtain the complexity of the visible light image and the infrared image include S211-S214:

[0064] S211: Based on the number of edges in the image, the edge density of the image is obtained using the following formula:

[0065]

[0066] Where (x, y) represents the coordinates of a pixel in the image, E(x, y) represents the edge image corresponding to the preprocessed visible light image and infrared image, and N e The edge density represents the number of pixels containing the edges in the edge image, W and H represent the length and width of the image, respectively, and ρ represents the edge density of the image. An edge image is an image formed by extracting edge information from preprocessed visible light and infrared images using edge detection algorithms; edge density can assess the richness of detail in an image, with higher edge density indicating more detail.

[0067] S212: Obtain the entropy S of the image based on the grayscale matrix of the image, and calculate the sampling window size of the texture as Q*Q. For example, Q is 7. Entropy is a measure of the complexity and randomness of image texture, reflecting the degree of disorder of the image. High entropy indicates that the texture of the image is complex.

[0068] S213: Based on the edge density and entropy of the image, the image complexity is obtained using the following formula:

[0069]

[0070] Where O represents the image complexity, S represents the image entropy, and Q represents the side length of the sampling window for calculating the texture; the above formula normalizes the entropy S, making ρ and The values ​​are all between 0 and 1, which facilitates subsequent calculations.

[0071] S214: Calculate the corresponding number of layers for the visible light image and the infrared image using the following formula, and select the larger value as the number of layers in the Gaussian pyramid:

[0072]

[0073] Where n represents the corresponding layer number, This indicates downsampling. It's worth noting that since the value of O is between 0 and 2, the value of n can range from 2 to 5, which aligns with the actual requirements for the number of layers in a Gaussian pyramid.

[0074] It should be noted that the number of Gaussian pyramid layers is dynamically determined by the complexity of the image. In areas with rich image features or many details, the number of Gaussian pyramid layers is increased to capture more details; in images with fewer details, the number of Gaussian pyramid layers is reduced to save computing resources, thus achieving a balance between resource consumption and detail capture.

[0075] S22: The preprocessed visible light image and infrared image are decomposed into multiple scales using Gaussian pyramids of corresponding levels to obtain infrared and visible light decomposed images at each scale. For example, when the corresponding level n1 of the visible light image is 3 and the corresponding level n2 of the infrared image is 4, n2 is selected as the level of the Gaussian pyramid. Thus, after decomposition by Gaussian pyramid, one original image and three sub-images of different scales are obtained.

[0076] It should be noted that Gaussian pyramid decomposition recursively applies low-pass filtering and downsampling operations to the input image, generating images of different levels, with the scale (resolution) of each level decreasing progressively. The decomposed images at each scale serve as input for subsequent processing steps. Multi-scale decomposition based on Gaussian pyramids effectively extracts specific features and structures at each scale from the input image, while considering information from various resolutions within the input image. By integrating a comprehensive representation of the input image, the entire fusion process is enhanced, thereby improving the quality and fidelity of the fused image.

[0077] Furthermore, smoothing filtering based on second-order partial differential equations and detail enhancement based on fourth-order partial differential equations are performed on each infrared and visible light decomposed image to obtain images enhanced at each scale.

[0078] Specifically, the smoothing filter based on the second-order partial differential equation is implemented using the following formula:

[0079]

[0080] M(x,y,0)=M0(x,y)

[0081] Where t represents the evolution time, M0(x,y) represents each input infrared and visible light decomposed image, (x,y) represents the pixel coordinates in the image, and M represents the image intensity at time t. The gradient represents the image intensity, including magnitude and direction information. div represents the divergence operator, c(x,y,t) represents the diffusion coefficient, which is an edge function or edge stopping function, T(x,y) represents the gradient of the image pixels, including only magnitude information, κ(t) represents the time-varying function, κ(t)=κ(0)+λt, κ(0) represents the initial value of the time-varying function, and λ represents the rate of change; for example, κ(0) is 0 and λ is 1, and the value of t is stopped when the visually optimal image is obtained.

[0082] By solving the above equations, smoothed images of the infrared and visible light decomposed images at different times t are obtained. The visually optimal image is selected as the final smoothed image. This smoothed image can effectively suppress noise points and small-scale features in the image while maintaining the integrity of the image edges, which is crucial for improving the quality of the fused image.

[0083] Furthermore, the following formula is used to achieve detail enhancement based on fourth-order partial differential equations:

[0084]

[0085] Where I(x,y,t) represents the position of the pixel at coordinates (x,y) and time. t The intensity of the image at that time, γ is a scalar parameter that controls the degree of enhancement, for example, γ is 2;

[0086] The equation is solved to obtain detail-enhanced images at different times t, and the visually optimal image is selected as the final detail-enhanced image.

[0087] Detail enhancement highlights fine details and textures while maintaining edge sharpness, ensuring that the merged image exhibits enhanced visual clarity and fidelity, preparing it for subsequent fusion steps.

[0088] Furthermore, the enhanced images of infrared and visible light at the same scale are dynamically fused, and the images at all scales obtained by fusion are reconstructed, including steps S41-S43.

[0089] S41: Dynamically fuse enhanced images of infrared and visible light at the same scale;

[0090] Specifically, the enhanced images of infrared and visible light at the same scale are dynamically fused using the following formula:

[0091]

[0092]

[0093] I f (x,y)=w I (x,y)·I I (x,y)+w V (x,y)·I V (x,y)

[0094] Among them, G I (x,y) and G V (x, y) represent the local standard deviations of the enhanced infrared and visible light images at the same scale, respectively. I (x,y) and w V (x,y) represent the fusion weights for infrared and visible light, respectively, I I (x,y) and I V (x, y) represent the enhanced infrared and visible light images at the same scale, respectively. f (x,y) represents the dynamically fused image at the corresponding scale; a weighted average fusion method is used to combine information from two modalities to generate a unified, high-fidelity fused image.

[0095] It should be noted that the dynamic fusion algorithm dynamically adjusts the weights based on local contrast and edge intensity, and uses gradient magnitude for measurement to gain a deeper understanding of the variability and sharpness of features in the image, preserving important image features. At the same time, it integrates the intensity of infrared and visible light imaging modes to ensure visual results.

[0096] S42: Perform post-processing on the dynamically fused images at each scale;

[0097] Specifically, post-processing can improve the visual quality of fused images at various scales, including sharpening, adaptive contrast adjustment, and adaptive brightness adjustment.

[0098] For example, the post-processing method used in this embodiment is gamma correction. Gamma correction is used to adjust the brightness and contrast of an image, and is performed using the following formula:

[0099]

[0100] Where μ represents a parameter controlling the overall brightness; μ greater than 1 reduces image brightness, while μ less than 1 enhances image brightness. F represents the dynamically fused image at one scale, (x, y) represents the pixel coordinates of the image, and F′ represents the post-processed image at one scale. For example, in this embodiment, μ is 0.75. Gamma correction helps ensure that the visual appearance of the fused image is consistent with human perception and ensures that the post-processed image has high clarity, contrast, and fidelity, thereby maximizing the utility of the fused image in various fields.

[0101] S43: Reconstruct images at all scales obtained from post-processing;

[0102] This embodiment uses the Laplacian pyramid to reconstruct the post-processed fused images at various scales. The Laplacian pyramid can seamlessly merge rich information from different scales, capturing essential features and subtle differences to generate the final high-resolution fused image.

[0103] Compared with existing technologies, the method provided in this embodiment employs an adaptive multi-scale decomposition algorithm to dynamically adjust the number of Gaussian pyramid layers according to the image content, thereby achieving the preservation of target edges and detailed features in the fused image with less resources. Furthermore, it adopts smoothing and detail enhancement based on partial differential equations to improve the quality of the fused image by selectively enhancing important details and suppressing noise points and artifacts, achieving effective smoothing, edge preservation, and detail enhancement in the fused image. In addition, the local dynamic fusion method combines complementary information from the infrared and visible light channels, improving the overall quality of the fused image.

[0104] Those skilled in the art will understand that all or part of the processes of the methods described in the above embodiments can be implemented by a computer program instructing related hardware, and the program can be stored in a computer-readable storage medium. The computer-readable storage medium may be a disk, optical disk, read-only memory, or random access memory, etc.

[0105] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. An infrared-visible image fusion method based on multi-scale decomposition and partial differential equations, characterized in that, include: Simultaneously acquire infrared and visible light images of the target object, and preprocess the acquired visible light and infrared images respectively; Adaptive multi-scale decomposition is performed on the preprocessed visible light image and infrared image respectively. The number of Gaussian pyramid layers is obtained based on the complexity of the visible light image and infrared image. The preprocessed visible light image and infrared image are then decomposed into infrared and visible light images at each scale using the Gaussian pyramid with the corresponding number of layers. The infrared and visible light decomposed images were subjected to smoothing filtering based on second-order partial differential equations and detail enhancement based on fourth-order partial differential equations, respectively, to obtain images enhanced at each scale. The enhanced images of infrared and visible light at the same scale are dynamically fused, and the images at all scales obtained by fusion are reconstructed to obtain the final fused image.

2. The infrared-visible image fusion method based on multi-scale decomposition and partial differential equations according to claim 1, characterized in that, The steps to obtain complex visible light and infrared images include: The edge density of an image is obtained based on the number of edges in the image; The entropy of an image is obtained from its grayscale matrix. The complexity of the image is obtained based on its edge density and entropy.

3. The infrared-visible image fusion method based on multi-scale decomposition and partial differential equations according to claim 2, characterized in that, The edge density of the image is obtained using the following formula: Where (x, y) represents the coordinates of a pixel in the image, E(x, y) represents the edge image corresponding to the image, and N e The value represents the number of pixels containing the edge in the edge image, W and H represent the length and width of the image, respectively, and ρ represents the edge density of the image.

4. The infrared-visible image fusion method based on multi-scale decomposition and partial differential equations according to claim 3, characterized in that, The complexity of the image can be obtained using the following formula: Where O represents the complexity of the image, S represents the entropy of the image, and Q represents the side length of the sampling window for calculating the texture.

5. The infrared-visible image fusion method based on multi-scale decomposition and partial differential equations according to claim 4, characterized in that, The corresponding number of layers in the visible light image and the infrared image are calculated using the following formulas, and the larger value is selected as the number of layers in the Gaussian pyramid: Where n represents the corresponding layer number, This indicates downsampling.

6. The infrared-visible image fusion method based on multi-scale decomposition and partial differential equations according to claim 1, characterized in that, The following formula is used to dynamically fuse enhanced images of the same scale in infrared and visible light: I f (x,y)=w I (x,y)·I I (x,y)+w V (x,y)·I V (x,y) Among them, G I (x,y) and G V (x, y) represent the local standard deviations of the enhanced infrared and visible light images at the same scale, respectively. I (x,y) and w V (x,y) represent the fusion weights for infrared and visible light, respectively, I I (x,y) and I V (x, y) represent the enhanced infrared and visible light images at the same scale, respectively. f (x,y) represents the dynamically fused image at the corresponding scale.

7. The infrared-visible image fusion method based on multi-scale decomposition and partial differential equations according to claim 1, characterized in that, The preprocessing of the obtained visible light image and infrared image includes: The obtained visible light image is converted to grayscale to obtain the converted grayscale image; The converted grayscale and infrared images are dynamically rescaled to obtain preprocessed grayscale and infrared images.

8. The infrared-visible image fusion method based on multi-scale decomposition and partial differential equations according to claim 7, characterized in that, The converted grayscale image is obtained using the following formula: G(i,j)=0.2989P(i,j,1)+0.5870P(i,j,2)+0.1140P(i,j,3) Where (i, j) represents the coordinates of each pixel in the visible light image, 1, 2, and 3 represent the red, green, and blue color channels, respectively, P represents the acquired visible light image, and G represents the converted grayscale image.

9. The infrared-visible image fusion method based on multi-scale decomposition and partial differential equations according to claim 8, characterized in that, Dynamic rescaling is performed using the following formula: Where G represents the converted grayscale image, P v Indicates the acquired infrared image, I greyscaled I represents the preprocessed visible light image obtained after dynamic rescaling. rescaled This represents the preprocessed infrared image obtained after dynamic rescaling.

10. The infrared-visible image fusion method based on multi-scale decomposition and partial differential equations according to claim 1, characterized in that, The synchronous acquisition includes infrared and visible light images of the target object, including: Images of the target object are simultaneously acquired using a fixed-position infrared camera and a visible light camera, respectively, to obtain infrared and visible light images of the target object.

Images