An image fusion method based on adaptive edge-preserving smoothing pyramid

Abstract

The present invention discloses an image fusion method based on an adaptive edge-preserving smoothing pyramid, which relates to the field of image processing technology. The method comprises the following steps: obtaining an image sequence; obtaining an initial weight map of the image sequence based on improved exposure weights, local contrast weights, and saturation weights; performing weighted guided filtering on the initial weight map using the Gaussian pyramid of the brightness component of the image sequence as a guide pyramid to obtain a weight map pyramid; constructing an adaptive edge-preserving smoothing pyramid of the weight map using the detail layer of the Laplacian pyramid of the image sequence and the coefficients of the weighted guided filtering; and reconstructing the adaptive edge-preserving smoothing pyramid and the Laplacian pyramid to obtain a fused image. The method has the following advantages: considering spatial domain information, achieving a more natural transition between light and dark, adapting to the overall dark or bright situation of the input image sequence, better preserving the details of the darkest and brightest areas while effectively suppressing the generation of halo artifacts.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and more particularly to an image fusion method based on adaptive edge-preserving smoothing pyramid. Background Art

[0002] The dynamic range of brightness of most natural scenes is usually much larger than the range that can be captured in a single shot by an ordinary digital camera. For example, from faint starlight to bright sunlight, the dynamic range of natural scenes can reach 8 orders of magnitude, however, the image sensors used in ordinary digital cameras can only capture a dynamic range of two orders of magnitude. The contradiction between the high dynamic range (HDR) of the real world and the low dynamic range (LDR) of digital cameras makes it difficult to retain all the details in a single image taken by a camera, and it is easy to have problems such as low image contrast, serious loss of detail information, and overexposure or underexposure in local areas. This challenge can be solved by HDR imaging, which can recover information from multiple LDR images with different exposures in the same HDR scene. Currently, HDR technology is used in medical imaging, video surveillance, satellite remote sensing and other fields.

[0003] One way to obtain HDR images is to use dedicated HDR equipment or develop special sensors that can capture a wide dynamic range. However, these dedicated devices are not common and are too expensive for ordinary consumers. In addition, current display devices do not support the display of HDR images. Therefore, this method has not been widely used. Unlike hardware-based solutions, software solutions are cheaper, more efficient, and easier to implement. In recent years, HDR imaging technology has been able to generate HDR images using ordinary cameras by taking multi-exposure images. Using HDR technology, a sequence of LDR images containing details in different brightness ranges is combined into an image that can more accurately retain most of the information of the real scene and has better visual effects. Current HDR imaging technology mainly includes two methods: tone mapping-based and multi-exposure image fusion (MEF)-based.

[0004] Tone mapping-based methods involve two main steps: HDR image construction and tone mapping. First, an HDR image is constructed from the LDR image. Then, tone mapping is used to display the HDR image on a standard LDR display device. This method requires specialized equipment and the calculation of multiple exposure parameters, making it time-consuming and limited.

[0005] Unlike tone mapping-based methods, the MEF method avoids the construction of intermediate HDR images. Instead, it directly extracts comprehensive image information from multiple low dynamic range images without being restricted by lighting and camera parameters, and synthesizes it into a fused image with good contrast, bright colors, more information content and perceptual appeal. In addition, the obtained high-quality fused image can be directly displayed on the LDR device without any additional processing. This method is simple and effective, but it often has a slow operation speed and suffers from problems such as blurred details, local color distortion, and halo artifacts. The existing MEF method is not sufficiently adaptable to situations where the overall brightness of the input image sequence is dark or bright, and the details of the brightest or darkest areas cannot be well preserved in the fused image.

[0006] Therefore, how to solve the problems of blurred details and halo artifacts in existing solutions and improve the adaptability to input images are problems that those skilled in the art urgently need to solve. Summary of the Invention

[0007] In view of this, the present invention provides an image fusion method based on an adaptive edge-preserving smoothing pyramid, which combines an improved exposure weight algorithm with an adaptive edge-preserving smoothing pyramid fusion algorithm to effectively suppress the generation of halo artifacts while better preserving the details of the darkest or brightest areas.

[0008] To achieve the above object, the present invention adopts the following technical solution: an image fusion method based on adaptive edge-preserving smoothing pyramid, comprising:

[0009] Get image sequence;

[0010] Improved exposure weights are calculated based on the overall brightness of the image sequence and the relative brightness between adjacent images;

[0011] A dense scale-invariant feature descriptor is used to extract local contrast information from the grayscale image of the source image, and the local contrast weight is calculated according to the maximum value of the same pixel position in all images;

[0012] The saturation weight is calculated using the standard deviation of each pixel in the R, G, and B channels;

[0013] Obtaining an initial weight map of the image sequence according to the improved exposure weight, the local contrast weight, and the saturation weight;

[0014] Using the Gaussian pyramid of the brightness component of the image sequence as a guide pyramid, performing weighted guided filtering on the initial weight map to obtain a weight map pyramid;

[0015] An adaptive edge-preserving smoothing pyramid of weight map is constructed using the detail layer of the Laplacian pyramid of the image sequence and the coefficients of the weighted guided filter;

[0016] The adaptive edge-preserving smoothing pyramid and the Laplacian pyramid are reconstructed to obtain a fused image.

[0017] Preferably, calculating the improved exposure weight specifically includes:

[0018] Performing grayscale processing on the image sequence to obtain a grayscale image;

[0019] performing normalization processing on the grayscale image;

[0020] Obtaining normalized pixel values ​​of the grayscale image;

[0021] Calculating the normalized average brightness of the image sequence according to the pixel values;

[0022] The exposure weight of the dark part or the bright part is adaptively adjusted according to the normalized average brightness of the image sequence.

[0023] In order to better adapt to the situation where the input image sequence is darker or brighter as a whole, the present invention adaptively adjusts the weight of the dark or bright part according to the overall average brightness of the input image sequence.

[0024] Preferably, adaptively adjusting the exposure weight of the dark portion or the bright portion according to the normalized average brightness of the image sequence specifically includes:

[0025] When the normalized average brightness is less than the well-exposed pixel value m, the exposure weight is:

[0026]

[0027] When the normalized average brightness is greater than the well-exposed pixel value m, the exposure weight is:

[0028]

[0029] When the normalized average brightness is equal to the well-exposed pixel value m, the exposure weight is:

[0030]

[0031] Among them, m represents the good exposure pixel value. When the normalized pixel value of the image sequence is m, it is given the highest weight, m∈[0,1], I n (i, j) represents the pixel value at position (i, j) after normalization of the nth image. Represents the pixel value at position (i, j) after normalization of the grayscale image of the nth image. The value range of n is 1, 2, ..., N, where N represents the number of input image sequences, α represents the global brightness adaptation factor, and λ represents the relative brightness adaptation factor.

[0032] Based on the well-exposed pixel value m, when the average brightness of all image sequences is less than m, this means that the input image is generally dark. In this case, the weight of the dark areas should be appropriately increased, while the weight of the bright areas should be appropriately decreased to better preserve the dark information. When the average brightness of all image sequences is greater than m, this means that the input image is generally bright. In this case, the weight of the bright areas should be appropriately increased, while the weight of the dark areas should be appropriately decreased to better preserve the bright information.

[0033] Preferably, the global brightness adaptation factor α reflects the deviation between the overall exposure of the image sequence and the well-exposed pixel value m, and is defined as follows:

[0034]

[0035] Where h and w represent the length and width of the image respectively;

[0036] In addition, when the brightness difference between an image in the input image sequence and its adjacent exposed image is large, it generally contains more well-exposed pixels. Therefore, the present invention proposes a relative brightness adaptive factor λ.

[0037] The relative brightness adaptation factor λ is used to adjust the normalized value to be close to the range of the well-exposed m pixel value with a weight of 1, and is defined as follows:

[0038]

[0039] Among them, mean(I n ) represents the average brightness of the nth input image, μ represents a fixed parameter, and the value of μ is 0.25.

[0040] This method comprehensively considers the overall brightness of the image sequence and the relative brightness between adjacent images. Using adaptive factors, it can adaptively adjust the weights of dark or bright areas in the image sequence based on the input image. This improves the ability to adapt to overall dark or bright input images, resulting in a better appearance and more image details.

[0041] Preferably, the Laplacian pyramid decomposes the image sequence into a base layer and a detail layer, wherein the base layer is used to capture low-frequency information and the detail layer is used to capture high-frequency information; the Laplacian pyramid has L layers, where L is obtained by the following formula:

[0042]

[0043] Where h and w represent the length and width of the image respectively. Indicates that an integer less than or equal to log2min(h,w) is returned.

[0044] Preferably, the Gaussian pyramid of the brightness component of the image sequence is used as a guide pyramid, and the weighted guided filtering process is performed on the initial weight map to obtain a weight map pyramid, which specifically includes:

[0045] The Gaussian pyramid of the brightness component of the image sequence is used as the guide pyramid, and G{Y n} (l) express;

[0046] The initial weight map is decomposed into a Gaussian pyramid of L layers, using G{W n} (l) express;

[0047] Based on weighted guided filtering, G{Y n} (l) The structure is transferred to G{W n} (l) , perform preliminary smoothing on the initial weight map, the specific algorithm is as follows:

[0048]

[0049] Wherein, 0≤l≤L, L represents the number of layers of the Gaussian pyramid, and They represent the coefficients of weighted guided filtering, Represents a weighted graph pyramid that has been initially smoothed by weighted guided filtering.

[0050] In order to reduce the computational cost, the present invention only calculates and as well as and Two-layer coefficient, in other layers, if the value of l is less than 4, and is through and The upsampling interpolation is generated layer by layer. If the value of l is greater than 4, the remaining layers are generated by and This is achieved by fixing the radius ζ and the regularization parameter λ to 2 and 1 / 1024 respectively.

[0051] Preferably, constructing an adaptive edge-preserving smoothing pyramid of a weight map using the detail layer of the Laplacian pyramid of the image sequence and the coefficients of the weighted guided filter specifically includes:

[0052] The coefficients of the weighted guided filter are introduced into the detail layer of the Laplacian pyramid to refine the weight map pyramid, and an adaptive edge-preserving smoothing pyramid of the weight map is constructed. The specific algorithm is as follows:

[0053]

[0054] in, Indicates that a Gaussian smoothing filter is used to Smoothing is performed to ensure consistency and reduce halo; |L1{L{I n} (l)}| represents the high-frequency information of the Laplacian pyramid, that is, the edge between the abnormal exposure area and the normal exposure area, which can be used to correct the inappropriate weighting caused by Gaussian smoothing, ensuring that the weighting of the appropriate exposure area is not affected by smoothing; Represents the coefficient of weighted guided filtering. In the above formula, Also |L1{L{I n} (l) The coefficient of |, which controls the magnitude of the high-frequency signal, Determines the gradient retention ability at the edge The larger the value, the better the gradient preservation effect; in the flat area The smaller the value, the better the smoothing effect.

[0055] Preferably, the specific algorithm for reconstructing the refined weight map pyramid and the Laplacian pyramid to obtain a fused image is as follows:

[0056]

[0057] Where L{F(i,j)} (l) Represents the Laplacian pyramid image of the lth layer, L{I n (i,j)} (l) It represents the pixel value at the position (i, j) of the first layer of the Laplacian pyramid of the nth input image, AES{W n (i,j)} (l) The weight map is represented by an adaptive edge-preserving smoothing pyramid, and then the fused result is obtained by inverse Laplace transform.

[0058] Preferably, local contrast measurement of an image sequence is used to preserve important details of the source image, such as texture and edges. This texture and edge information is contained in gradient changes. The present invention uses dense scale-invariant feature transform (SIFT) descriptors to extract local contrast information from the grayscale image of the source image. The specific algorithm is as follows:

[0059]

[0060] Among them, C n (i,j) represents a simple local contrast measurement index, represents the l1 norm, Represents the operator for computing the non-normalized dense SIFT map of the input image. To better utilize memory, in each cell, an 8-bin orientation histogram and a 2×2 cell array are used to generate the descriptor. As mentioned before, for the activity level measurement of the associated pixel, a considerable number of elements in the non-normalized descriptor are used. Since no element in the SIFT descriptor is negative, at each pixel, the grayscale image The l1 norm of the mapping is C n (i, j). Then a winner-takes-all weight distribution strategy is adopted, that is, the maximum value of the same pixel position in all images is used to calculate the local contrast weight:

[0061]

[0062] in Represents the local contrast weight value of the n-th image (i, j) position.

[0063] Preferably, calculating the saturation weight specifically includes: as the exposure time of the photo increases, the color produced will become unsaturated and the visual effect will be poor. Saturated colors are ideal and make the image look vivid. The standard deviation of the R, G and B channels of each pixel is used as the measurement value S. The present invention calculates the standard deviation of each pixel in the R, G and B channels to obtain the color saturation weight

[0064] Preferably, an initial weight map of the image sequence is obtained according to the improved exposure weight, the local contrast weight and the saturation weight, and the initial weight map is normalized:

[0065]

[0066]

[0067] in, Indicates improved exposure weight, represents the local contrast weight, Indicates color saturation weight; represents the initial weight obtained by improving exposure weight, local contrast weight and color saturation weight; W n (i, j) represents the initial weight map after normalization.

[0068] Through the above technical solution, it can be seen that compared with the prior art, the present invention provides an image fusion method based on an adaptive edge-preserving smoothing pyramid, including: obtaining an image sequence; calculating an improved exposure weight based on the overall brightness of the image sequence and the relative brightness between adjacent images; using a dense scale-invariant feature descriptor to extract local contrast information from the grayscale image of the source image, and calculating the local contrast weight based on the maximum value of the same pixel position in all images; calculating the saturation weight based on the standard deviation of each pixel in the R, G and B channels; obtaining an initial weight map of the image sequence based on the improved exposure weight, the local contrast weight and the saturation weight; using the Gaussian pyramid of the brightness component of the image sequence as a guide pyramid, performing weighted guided filtering on the initial weight map to obtain a weight map pyramid; constructing an adaptive edge-preserving smoothing pyramid of the weight map using the detail layer of the Laplacian pyramid of the image sequence and the coefficients of the weighted guided filtering; and reconstructing the adaptive edge-preserving smoothing pyramid and the Laplacian pyramid to obtain a fused image.

[0069] Compared with existing solutions, the proposed solution has the following advantages: it considers spatial domain information, provides more natural light-dark transitions, and adapts to the overall dark or bright conditions of the input image sequence. By improving the exposure assessment weight function and combining local contrast and color saturation weights to calculate the fusion weight for each pixel, the proposed solution proposes an adaptive edge-preserving smoothing pyramid to refine the weight map, ultimately producing a high-quality fused image that better preserves the details of the darkest and brightest areas while effectively suppressing the generation of halo artifacts, resulting in a better visual effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are merely embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0071] Figure 1 A flow chart of an image fusion method based on adaptive edge-preserving smoothing pyramid provided by the present invention;

[0072] Figure 2 An improved exposure weight function diagram provided by an embodiment of the present invention;

[0073] Figure 3 An input multi-exposure image sequence provided by an embodiment of the present invention;

[0074] Figure 4 The final fused image result and local magnified image provided by the embodiment of the present invention;

[0075] Figure 5 These are the results and local magnification images obtained using four image fusion algorithms in the existing technology. DETAILED DESCRIPTION

[0076] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0077] The embodiment of the present invention discloses an image fusion method based on improved exposure weight and smoothing pyramid to obtain an image sequence; Figure 1 As shown, the image sequence is an LDR image sequence;

[0078] Improved exposure weights are calculated based on the overall brightness of the image sequence and the relative brightness between adjacent images;

[0079] A dense scale-invariant feature descriptor is used to extract local contrast information from the grayscale image of the source image, and the local contrast weight is calculated according to the maximum value of the same pixel position in all images;

[0080] The saturation weight is calculated using the standard deviation of each pixel in the R, G, and B channels;

[0081] Obtaining an initial weight map of the image sequence according to the improved exposure weight, the local contrast weight, and the saturation weight;

[0082] Using the Gaussian pyramid of the brightness component of the image sequence as a guide pyramid, performing weighted guided filtering on the initial weight map to obtain a weight map pyramid;

[0083] An adaptive edge-preserving smoothing pyramid of weight map is constructed using the detail layer of the Laplacian pyramid of the image sequence and the coefficients of the weighted guided filter;

[0084] The adaptive edge-preserving smoothing pyramid and the Laplacian pyramid are reconstructed to obtain a fused image.

[0085] An adaptive edge-preserving smoothing pyramid is used to further refine the weight map pyramid. The coefficients of the weighted guided filter are used to introduce the detail layer of the Laplacian pyramid of the image sequence to refine the weight map pyramid and construct an adaptive edge-preserving smoothing pyramid of the weight map.

[0086] Specifically, an embodiment of the present invention proposes an improved exposure weight algorithm that comprehensively considers the overall brightness of the image sequence and the relative brightness between adjacent images; utilizes an adaptive factor to adaptively adjust the weight of the dark or bright part of the image sequence according to the input image, thereby improving the algorithm's adaptability to the overall dark or bright input image, so as to obtain a better appearance and more image details.

[0087] In this embodiment of the present invention, m=0.5, that is, pixels with a normalized value around 0.5 are given the highest weight, and pixels far from 0.5 are given a lower weight. The exposure weight is:

[0088]

[0089] in Represents the pixel value at position (i, j) after normalization of the grayscale image of the nth image, and the value range of n is 1, 2, ..., N, where N is the number of input image sequences; the above formula is the exposure weight when the overall brightness and relative brightness are not considered. In order to better adapt to the situation where the input image is darker or brighter as a whole, this embodiment first adaptively adjusts the weight of the dark or bright part according to the overall average brightness of the input image sequence, and calculates the average brightness of all input image sequences. If the average brightness is less than 0.5, this means that the input image is dark as a whole. At this time, the weight of the dark part should be appropriately increased, and the weight of the bright part should be appropriately reduced to better retain the information of the dark part. If the average brightness is greater than 0.5, this means that the input image is brighter as a whole. At this time, the weight of the bright part should be appropriately increased, and the weight of the dark part should be appropriately reduced. This embodiment uses a global brightness adaptive factor α, which can reflect the offset between the overall exposure of the input image and 0.5. The adaptive factor α is defined as follows:

[0090]

[0091] where h and w represent the length and width of the image, respectively. Furthermore, when the brightness difference between an image in the input image sequence and its adjacent exposed image is large, it generally contains more well-exposed pixels. Therefore, a relative brightness adaptive factor λ is proposed to adjust the normalized value to a range close to 0.5, where the pixel weight is 1.

[0092]

[0093] Among them, mean(I n ) represents the average brightness of the nth input image, and μ represents a fixed parameter.

[0094] In summary, when the normalized average brightness is less than 0.5, the final exposure weight is:

[0095]

[0096] When the normalized average brightness is greater than 0.5, the final exposure weight is

[0097]

[0098] like Figure 2 As shown in Figure 2, the dotted line indicates that the average brightness of all input image sequences is less than 0.5, and the solid line indicates that the average brightness of all input images is greater than 0.5. When the normalized average brightness is equal to 0.5, the exposure weight adopts the original form.

[0099] Specifically, the present invention proposes a local contrast weighting algorithm that measures the local contrast of an input image sequence to preserve important detail information in the source image, such as texture and edges. This texture and edge information is contained in gradient variations. This embodiment uses a dense scale-invariant feature transform (SIFT) descriptor to extract local contrast information from the grayscale image of the source image. The specific algorithm is as follows:

[0100]

[0101] Among them C n (i,j) is a simple local contrast measure, represents the l1 norm, Represents the operator for computing the non-normalized dense SIFT map of the input image. To better utilize memory, in each cell, an 8-bin orientation histogram and a 2×2 cell array are used to generate the descriptor. As mentioned before, for the activity level measurement of the associated pixel, a considerable number of elements in the non-normalized descriptor are used. Since no element in the SIFT descriptor is negative, at each pixel, the grayscale image The l1 norm of the mapping is C n (i, j). Then a winner-takes-all weight distribution strategy is adopted, that is, the maximum value of the same pixel position in all images is used to calculate the local contrast weight:

[0102]

[0103] in Represents the local contrast weight value of the n-th image (i, j) position.

[0104] Specifically, the embodiment of the present invention proposes a method for calculating saturation weights. As the exposure time of a photo increases, the resulting colors become desaturated, resulting in poor visual effects, while saturated colors are ideal and make the image appear more vivid. The standard deviation of the R, G, and B channels of each pixel in the image sequence is used as the measurement value S, and the standard deviation of each pixel in the R, G, and B channels is calculated to obtain the color saturation weight. The specific algorithm is as follows:

[0105]

[0106] in, Represents the pixel value at position (i, j) after normalization of the R, G, and B channels of the nth image, γ n (i,j) R,G,B Represents the average value of the three channels R, G, and B of the nth input image at (i, j), which is defined as follows:

[0107]

[0108] in, Represents the pixel of the image at (i, j) which is a function of the three channels R, G, and B; k represents the three color channels R, G, and B.

[0109] Specifically, the three metrics (improved exposure weight, local contrast weight, and color saturation weight) above represent the contribution of each input pixel to the final result. The initial weight map is constructed from the three calculated metrics and normalized:

[0110]

[0111]

[0112] in, Indicates improved exposure weight, represents the local contrast weight, Indicates color saturation weight; represents the initial weight obtained by improving exposure weight, local contrast weight and color saturation weight; W n (i, j) represents the initial weight map after normalization.

[0113] The initial weight map obtained by the above formula is noisy and discontinuous. If it is used directly for fusion operations, unsatisfactory problems such as seams and obvious halo artifacts are likely to occur. Therefore, it is crucial to smooth and denoise these weight maps before using them for fusion processing. In order to solve this problem, a multi-resolution method can be used, but this method does not retain the details of the brightest or darkest areas well and there is a risk of generating halo artifacts. Edge-preserving filters have been widely used to enhance details and multi-exposure image fusion. The embodiment of the present invention uses edge-preserving smoothing technology to improve the Gaussian pyramid of the weight map. Among them, the weight map and the LDR image sequence are decomposed into L-layer Gaussian pyramids and L-layer Laplacian pyramids, respectively, where the Laplacian pyramid decomposition decomposes the input image into a base layer and a detail layer. The highest layer is the base layer, which captures low-frequency information (global color information of the image), and the remaining layers are detail layers, which capture high-frequency information (image edges and detail information). L is given by the following formula:

[0114]

[0115] in Indicates returning the nearest integer less than or equal to log2min(h,w), and min(h,w) represents the minimum function. As can be seen from the above formula, the number of pyramid layers in the present invention is two fewer than that of the traditional pyramid. This is because appropriately reducing the number of layers can reduce the details lost during the pyramid decomposition and reconstruction process, but this will also bring about the problem of halo artifacts. A careful observation of the weight map pyramid reveals that improper smoothing around the edges is the main cause of halo artifacts. For example, overexposed areas along the boundaries of normally exposed areas often receive higher weights after improper smoothing, which leads to halo artifacts around the edges.

[0116] Different from using Gaussian pyramid G{W n} (l) To fuse images with different exposures, the present invention proposes a novel adaptive edge-preserving smoothing pyramid to fuse them. The present invention is based on weighted guided filtering (WGIF). First, the weight map is decomposed into a Gaussian pyramid, G{W n} (l) is the weight map pyramid to be smoothed, with the pyramid of the luminance component of the input LDR image sequence as the guide pyramid (G{Y n} (l) is the Gaussian pyramid of the luminance component of the input image). The proposed pyramid is based on an observation that WGIF can convert G{Y n} (l) The structure is transferred to G(W n} (l) . Let the coefficients of WGIF be expressed as and In order to reduce the computation cost, the embodiment of the present invention only calculates and as well as and If the value of l is less than 4, the other layers and is through and The upsampling interpolation is generated layer by layer. If the value of l is greater than 4, the remaining layers are generated by and This is achieved by fixing the radius ζ and the regularization parameter λ to 2 and 1 / 1024 respectively. Then the WGIF-based pyramid is given as follows:

[0117]

[0118] Smoothed by WGIF Due to inappropriate smoothing around high-level edges, details cannot be well preserved and halo artifacts exist. The method to eliminate halo artifacts is to smooth inappropriate weights to a wider area to make them invisible. A Gaussian smoothing filter is used for each layer to attenuate halo artifacts. However, the smoothing process reduces the weights of the properly exposed areas, resulting in loss of details in bright areas. To solve this problem, the embodiment of the present invention introduces a Laplacian pyramid detail layer of the input image to refine the weight map, and uses the coefficients of the weighted guided filter to reduce the weight map. To construct an adaptive edge-preserving smoothing pyramid of the weight graph. The specific algorithm is as follows:

[0119]

[0120] Indicates that a Gaussian smoothing filter is used to Smoothing is performed to ensure consistency and reduce halo; |L1{L{I n} (l)}| represents the high-frequency information of the Laplacian pyramid, that is, the edge between the abnormal exposure area and the normal exposure area, which can be used to correct the inappropriate weighting caused by Gaussian smoothing, ensuring that the weighting of the appropriate exposure area is not affected by smoothing; Yes |L1{L{I n} (l)}|, which controls the magnitude of the high-frequency signal; Determines the gradient retention ability at the edge The larger the value, the better the gradient preservation effect; in the flat area The smaller the value, the better the smoothing effect.

[0121] Finally, the present invention uses pyramid fusion to obtain the final fused image. The specific algorithm is as follows:

[0122]

[0123] where L{F(i,j)} (l) Represents the Laplacian pyramid image of the lth layer, L{I n (i,j)} (l) Represents the pixel value at the lth layer position (i, j) of the nth input image Laplacian pyramid, AES{W n (i,j)} (l) Adaptive edge-preserving pyramids for representing weight maps.

[0124] Figure 3 A multi-exposure image sequence input for an embodiment of the present invention, Figure 4 The final fused image result and local magnified image provided by the embodiment of the present invention; Figure 5 The results and local magnification images obtained by using four image fusion algorithms in the existing technology; Figure 4 and Figure 5 From the local enlarged image, we can see that Figure 4 The brightest cloud details are more distinct, and haloing around the edges of the hot air balloon is effectively suppressed. The proposed method effectively suppresses halo artifacts while better preserving details in the brightest or darkest areas, resulting in a better visual effect. By adapting the overall brightness and shading of the input image sequence and introducing a detail layer to refine the weight map, the proposed method constructs an adaptive edge-preserving smoothing pyramid of the weight map, effectively preserving details in the darkest and brightest areas while effectively suppressing halo artifacts.

[0125] The proposed solution offers the following advantages over existing solutions: it considers spatial domain information, provides more natural light-dark transitions, and adapts to the overall dark or bright conditions of the input image sequence. It innovatively proposes an improved exposure weighting algorithm, which combines local contrast weighting and color saturation weighting to calculate the fusion weight for each pixel. It also innovatively proposes an adaptive edge-preserving smoothing pyramid to refine the weight map, ultimately producing a high-quality fused image that better preserves details in the darkest and brightest areas while effectively suppressing halo artifacts.

[0126] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Reference can be made to the common and similar parts between the various embodiments. For the devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method description.

[0127] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An image fusion method based on adaptive edge-preserving smoothing pyramid, characterized in that: include: Get image sequence; Improved exposure weights are calculated based on the overall brightness of the image sequence and the relative brightness between adjacent images; A dense scale-invariant feature descriptor is used to extract local contrast information from the grayscale image of the source image, and the local contrast weight is calculated according to the maximum value of the same pixel position in all images; The saturation weight is calculated using the standard deviation of each pixel in the R, G, and B channels; Obtaining an initial weight map of the image sequence according to the improved exposure weight, the local contrast weight, and the saturation weight; Using the Gaussian pyramid of the brightness component of the image sequence as a guide pyramid, performing weighted guided filtering on the initial weight map to obtain a weight map pyramid; An adaptive edge-preserving smoothing pyramid of weight map is constructed using the detail layer of the Laplacian pyramid of the image sequence and the coefficients of the weighted guided filter; The adaptive edge-preserving smoothing pyramid and the Laplacian pyramid are reconstructed to obtain a fused image.

2. The image fusion method based on adaptive edge-preserving smoothing pyramid according to claim 1, characterized in that: Calculating the improved exposure weight specifically includes: Performing grayscale processing on the image sequence to obtain a grayscale image; performing normalization processing on the grayscale image; Obtaining normalized pixel values ​​of the grayscale image; Calculating the normalized average brightness of the image sequence according to the pixel values; The exposure weight of the dark part or the bright part is adaptively adjusted according to the normalized average brightness of the image sequence.

3. The image fusion method based on adaptive edge-preserving smoothing pyramid according to claim 2, characterized in that: Adaptively adjusting the exposure weight of the dark portion or the bright portion according to the normalized average brightness of the image sequence, specifically comprising: When the normalized average brightness is less than the well-exposed pixel value m When , the exposure weight is: ； When the normalized average brightness is greater than the well-exposed pixel value m When , the exposure weight is: ； When the normalized average brightness is equal to the well-exposed pixel value m When , the exposure weight is: ； in, m Indicates a well-exposed pixel value. When the normalized pixel value of the image sequence is close to m The highest weight is given to , Indicates the n After normalization, the image The pixel value of the position, Indicates the n After the grayscale image of the image is normalized The pixel value of the position, n The value range is 1, 2, ..., N, where N represents the number of input image sequences. represents the global brightness adaptation factor, Represents the relative brightness adaptation factor.

4. The image fusion method based on adaptive edge-preserving smoothing pyramid according to claim 3, characterized in that: Global brightness adaptation factor Reflects the deviation between the overall exposure of the image sequence and the well-exposed pixel value m, which is defined as follows: ； in, h and w Respectively represent the length and width of the image; relative brightness adaptation factor It is used to adjust the normalized value close to the range of good exposure m pixel value weight 1, which is defined as follows: ;in, Indicates the n The average brightness of the input images, Indicates fixed parameters.

5. The image fusion method based on adaptive edge-preserving smoothing pyramid according to claim 1, characterized in that: The Laplacian pyramid decomposes the image sequence into a base layer and a detail layer, wherein the base layer is used to capture low-frequency information and the detail layer is used to capture high-frequency information. The Laplacian pyramid has L layers, where L is obtained by the following formula: ;in, h and w Represent the length and width of the image respectively, Returns a value that is less than or equal to An integer.

6. The image fusion method based on adaptive edge-preserving smoothing pyramid according to claim 5, characterized in that: The Gaussian pyramid of the brightness component of the image sequence is used as a guide pyramid, and the initial weight map is subjected to weighted guided filtering to obtain a weight map pyramid, specifically including: The Gaussian pyramid of the brightness component of the image sequence is used as the guide pyramid, and express; Decompose the initial weight map into L-layer Gaussian pyramids, using express; Based on weighted guided filtering The structure is transferred to , perform preliminary smoothing on the initial weight map, the specific algorithm is as follows: ;in, , L represents the number of layers of the Gaussian pyramid, and They represent the coefficients of weighted guided filtering, Represents a weighted graph pyramid that has been initially smoothed by weighted guided filtering.

7. The image fusion method based on improved exposure weight and smoothing pyramid according to claim 6, characterized in that: The adaptive edge-preserving smoothing pyramid of the weight map is constructed using the detail layer of the Laplacian pyramid of the image sequence and the coefficients of the weighted guided filter, specifically including: The coefficients of the weighted guided filter are introduced into the detail layer of the Laplacian pyramid to refine the weight map pyramid, and an adaptive edge-preserving smoothing pyramid of the weight map is constructed. The specific algorithm is as follows: ;in, Indicates that a Gaussian smoothing filter is used to Smoothing Represents the high-frequency information of the Laplace pyramid, Represents the coefficients of the weighted guided filter.

8. The image fusion method based on improved exposure weight and smoothing pyramid according to claim 1, characterized in that: The specific algorithm for reconstructing the adaptive edge-preserving smoothing pyramid and the Laplacian pyramid to obtain a fused image is as follows: ;in, represents the Laplacian pyramid image of the lth layer, It means the n The Laplacian pyramid of the input image l Layer Position The pixel value at An adaptive edge-preserving pyramid is used to represent the weight map, and then the fused result is obtained by inverse Laplace transform.

9. The image fusion method based on improved exposure weight and smoothing pyramid according to claim 3, characterized in that: Before performing weighted guided filtering on the initial weight map, the initial weight map is normalized: ； ;in, Indicates improved exposure weight, represents the local contrast weight, Indicates color saturation weight; represents the initial weight obtained by improving the exposure weight, local contrast weight and color saturation weight; Represents the initial weight map after normalization.

Images