A Multi-Frame Image Fusion Enhancement Method and System Based on Gradient Domain and Fusion Weights
By employing gradient domain and fusion weights, the challenges of noise suppression and detail preservation in multi-frame image fusion enhancement are addressed. This approach enables high-quality image fusion in complex motion and occluded regions, reduces computational complexity, and is suitable for real-time processing on mobile devices.
Patent Information
- Application Number
- CN202611113097.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-25
- Publication Date
- 2026-11-17
AI Technical Summary
Existing multi-frame image fusion enhancement methods are prone to fusion artifacts when dealing with complex motion and occluded areas. They are difficult to balance noise suppression and detail preservation, and have high computational complexity, making it difficult to meet real-time processing requirements.
A gradient domain-based and fusion weight-based approach is adopted. By selecting reference frames, normalizing brightness, analyzing local frame differences, and removing moving regions, a gradient domain fusion framework is constructed. Adaptive fusion is then performed by combining local noise estimation, gradient consistency constraints, and spatiotemporal nonlocal similarity. Finally, Poisson reconstruction and discrete cosine transform are used for image reconstruction.
It improves the robustness of fusion in complex motion scenes and occluded areas, enhances image quality, and reduces computational overhead, adaptability, and real-time processing capabilities, meeting the multi-frame image fusion enhancement requirements of mobile devices.
Smart Images

Figure CN123118375A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image enhancement technology, and specifically to a method and system for multi-frame image fusion enhancement based on gradient domain and fusion weights. Background Technology
[0002] Digital images are inevitably contaminated by various noise sources during acquisition, including sensor thermal noise, readout noise, photon shot noise, and quantization noise. Multi-frame image fusion enhancement methods utilize the temporal redundancy information between consecutively acquired frames, integrating complementary information from multiple frames into a single high-quality image through appropriate fusion strategies. The theoretical basis of this approach is that for static scenes, noise components in multiple frames are independent and identically distributed in the temporal dimension, while signal components are highly correlated. Therefore, by performing appropriate weighted averaging or more complex fusion operations on multiple frames, noise power can be reduced by the inverse of the frame number while preserving signal energy. Theoretically, the improvement in signal-to-noise ratio (SNR) is proportional to the square root of the number of frames involved in the fusion. Multi-frame image fusion enhancement methods have broad application prospects in mobile photography, video surveillance, astronomical observation, and medical imaging. Especially in handheld device photography, due to physical space limitations, small sensor size, and limited photosensitive area per pixel, the SNR is naturally low, making multi-frame fusion technology a key means to improve image quality.
[0003] Existing multi-frame image fusion enhancement methods typically first perform motion compensation or image registration on continuously acquired multi-frame images, then fuse the spatial and temporal redundancy information in the multi-frame images to obtain a fused image with lower noise and higher quality, and finally output the fused image.
[0004] While existing technologies can improve image quality using multiple frames, they still suffer from several limitations, including difficulty in balancing detail preservation and noise suppression, high dependence of the fusion process on motion estimation accuracy, susceptibility to fusion artifacts in complex motion and occluded areas, difficulty in adapting to noise associated with real image signals using a unified noise model, insufficient utilization of temporal information, high computational complexity of high-quality fusion methods, and difficulty in balancing fusion quality and real-time processing performance. These issues restrict the enhancement effect and practical application capabilities of multi-frame image fusion. Summary of the Invention
[0005] In view of the above-mentioned shortcomings of the existing technology, the present invention provides a multi-frame image fusion enhancement method and system based on gradient domain and fusion weight.
[0006] To achieve the above objectives, the present invention provides the following technical solution:
[0007] A multi-frame image fusion enhancement method based on gradient domain and fusion weights is characterized by the following steps:
[0008] S1, reference frame selection, brightness normalization, local frame difference analysis, motion region elimination and signal related noise estimation are performed on the multi-frame images to obtain a preprocessed image and a local noise variance;
[0009] S2, converting the preprocessed image from a pixel domain to a gradient domain, specifically comprising the following steps:
[0010] S201, defining a gradient field through the preprocessed image;
[0011] S202, obtaining horizontal and vertical gradients of the preprocessed image by difference calculation:
[0012] S203, taking the horizontal and vertical gradients as gradient fusion basis data, weighting and fusing the horizontal and vertical gradients respectively according to the calculated fusion weight to obtain fused horizontal and vertical gradients, and constructing a gradient field;
[0013] S3, based on the preprocessed image, the local noise variance and the gradient field, respectively calculating noise level confidence, gradient consistency confidence, spatio-temporal non-local similarity and motion occlusion confidence, and fusing to obtain the final fusion weight corresponding to each pixel position of each frame image;
[0014] S4, obtaining the final fusion weight, weighting and fusing the horizontal and vertical gradients of each pixel position of each frame image to generate a fused gradient, calculating a gradient consistency index, and performing nonlinear enhancement on the fused gradient according to the gradient consistency index to obtain a fused gradient field;
[0015] S5, based on the fused gradient field, constructing a corresponding Poisson equation, calculating the divergence of the target gradient field, and solving the Poisson equation in the frequency domain using two-dimensional discrete cosine transform, and then restoring the fused image through inverse discrete cosine transform to obtain a reconstructed fused enhanced image;
[0016] S6, post-processing the fused enhanced image to output a final fused enhanced image.
[0017] Further, the step S1 specifically comprises the following steps:
[0018] S101, obtaining a plurality of continuously acquired images as input images, and performing brightness normalization to obtain a preprocessed image;
[0019] S102, calculating a frame difference image through the preprocessed image, calculating a local noise variance of a single adjacent frame pair based on the sample variance of the frame difference image in a local window, and fusing the local noise variances of a plurality of adjacent frame pairs to obtain a pre-estimated local noise variance of each frame;
[0020] S103, calculating an adaptive threshold based on the sample variance of the frame difference image in the local window and the minimum local sample variance, obtaining a motion region recognition result, and eliminating the frame difference image data corresponding to the motion region;
[0021] S104, constructing a linear noise model based on the real signal strength and the local noise variance, obtaining the real signal strength through the pixel value of the preprocessed image, and obtaining the local real signal strength estimate value through low-pass filtering, and finally inputting the noise parameter and the local real signal strength estimate value into the linear noise model to obtain the local noise variance.
[0022] Further, in step S103, the sample variance calculation formula is:
[0023]
[0024]
[0025] wherein, is the sample variance of the frame difference image in the local window, is the local mean value in the window, is the length of the local window, is the pixel coordinate in the local window, is the local window, is the frame difference value of the kth frame difference image at the pixel coordinate (u, v) in the local window.
[0026] Further, in step S202, the horizontal gradient and the vertical gradient calculation formula is:
[0027]
[0028]
[0029] wherein, is the horizontal gradient of the kth frame at the pixel is the vertical gradient of the kth frame at the pixel , , , are the pixel values of the adjacent pixel positions, respectively.
[0030] Further, in step S3, the spatio-temporal non-local similarity calculation formula is:
[0031]
[0032] wherein, is the spatio-temporal non-local mean similarity, is a parameter for controlling the sensitivity of the similarity. d is the image block size.
[0033] Further, the final fusion weight in step S3 is calculated by the formula:
[0034]
[0035] wherein, is the final fusion weight of the k-th frame at pixel is the noise level confidence, is the gradient consistency confidence, is the spatio-temporal non-local similarity, is the motion occlusion confidence.
[0036] Further, the gradient consistency index in step S4 is calculated by the formula:
[0037]
[0038] wherein, is the gradient consistency index, is the final fusion weight of the k-th frame at pixel after normalization, is the horizontal gradient of the k-th frame, is the vertical gradient of the k-th frame, is the square of the horizontal gradient, is the square of the vertical gradient, is a small regularization constant;
[0039] The enhanced horizontal and vertical gradients are calculated by the formulas:
[0040]
[0041]
[0042] wherein, is the enhanced horizontal gradient, is the enhanced vertical gradient, is the fused horizontal gradient, is the fused vertical gradient, is the enhancement intensity coefficient (typical value ), is the protection constant to prevent small gradient from being amplified too much (typical value ), is the amplitude of the fused gradient (|Gx|+|Gy|), is the gradient consistency index.
[0043] Further, the step S5 specifically comprises the following steps:
[0044] S501, constructing a Poisson equation based on the fused gradient field, and establishing an image reconstruction equation with the target gradient field as a constraint;
[0045] S502, discretizing the Laplacian operator in the Poisson equation and the divergence operator corresponding to the target gradient field to obtain a divergence field;
[0046] S503, performing two-dimensional discrete cosine transformation on the divergence field to obtain divergence field frequency domain coefficients, and solving in the transform domain combined with the Poisson equation to output fused image coefficients;
[0047] S504, performing two-dimensional inverse discrete cosine transformation on the fused image coefficients to obtain a fused enhanced image.
[0048] Further, in the step S6, the final fused enhanced image calculation formula is:
[0049]
[0050] wherein is a spatial adaptive gamma value:
[0051]
[0052] wherein, is a final fused enhanced image after gamma correction, is a maximum allowed value of a pixel value, is a reconstructed fused enhanced image, is an adaptive gamma value at a pixel, is an average brightness in a local window, is a global average brightness, is a base gamma value. A multi-frame image fusion enhancement system based on a gradient domain and a fusion weight, the system comprising:
[0053] An image preprocessing module: performing reference frame selection, brightness normalization, local frame difference analysis, motion region elimination and signal related noise estimation on the multi-frame images to obtain preprocessed images and local noise variance;
[0054] A gradient field construction module: defining a gradient field through the preprocessed images, simultaneously calculating the gradients of the preprocessed images in the horizontal direction and the vertical direction by using finite difference, taking the gradients in the horizontal direction and the vertical direction as fusion basis data, and performing weighted fusion after calculating the fusion weight to obtain fused horizontal gradients and fused vertical gradients, and constructing a gradient field;
[0055] A fusion weight calculation module: calculating the fusion weight based on the gradient field and the local noise variance, and outputting the fusion weight;
[0056] Weight fusion calculation module: based on the pre-processed image, local noise variance and gradient field, noise level confidence, gradient consistency confidence, spatio-temporal non-local similarity and motion occlusion confidence are calculated respectively, and the fusion weight corresponding to the pixel position of each frame image is obtained by fusion;
[0057] Fusion gradient generation module: obtain the fusion weight, and respectively perform weighted fusion on the horizontal gradient and vertical gradient of the pixel position corresponding to each frame image to generate a fusion gradient, calculate a gradient consistency index, and perform nonlinear enhancement on the fusion gradient according to the gradient consistency to obtain a fusion enhanced image;
[0058] Final image optimization module: post-processing is performed on the fusion enhanced image, and a final fusion enhanced image is output.
[0059] Compared with the prior art, the present application has the following advantages:
[0060] 1. The present application constructs a gradient domain fusion framework, combines local noise estimation, gradient consistency constraint, spatio-temporal non-local mean weight and motion occlusion detection mechanism, and adaptively fuses the effective information in multiple frames of images, realizes the collaborative processing of noise suppression and edge texture detail preservation, improves the fusion robustness under complex motion scene and occlusion area, and can adaptively adjust the fusion weight according to the local noise level, improve the overall quality of the fusion image, and fully utilize the time redundancy information of multiple frames to obtain more accurate fusion results.
[0061] 2. The present application adopts gradient domain fusion combined with Poisson reconstruction to restore the final fusion enhanced image, and reduces the image reconstruction calculation overhead through the fast Poisson solving method based on discrete cosine transform, which guarantees the fusion quality while taking into account the calculation efficiency; at the same time, the image fusion can be completed without relying on large-scale training data, and has good explainability, adaptability and real-time processing capability, which can meet the multi-frame image fusion enhancement demand in mobile devices and resource limited scenes. BRIEF DESCRIPTION OF DRAWINGS
[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.
[0063] Figure 1 The method flow structure diagram of the present application;
[0064] Figure 2 The flow chart of converting the image from the pixel domain to the gradient domain of the present application. DETAILED DESCRIPTION
[0065] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.
[0066] Embodiment one: a multi-frame image fusion enhancement method based on gradient domain and fusion weight, comprising:
[0067] S1, image preprocessing and noise estimation
[0068] The continuous acquisition of multiple frames of images is subjected to reference frame selection, brightness normalization, local frame difference analysis, motion area elimination and signal related noise estimation, to obtain preprocessed images and local noise variance, specifically including the following steps:
[0069] S101, obtaining continuous acquisition of multiple frames of images as input images, and performing brightness normalization to obtain preprocessed images, specifically:
[0070] Let the input be a sequence of N frames of images acquired continuously:
[0071]
[0072] Wherein, each frame of image is defined in the image domain :
[0073]
[0074] The image domain is usually a rectangular region:
[0075]
[0076] In order to ensure that the subsequent multi-frame images can be normalized around the same reference in brightness, gradient comparison, block matching and fusion, the middle frame is selected as the reference frame:
[0077]
[0078]
[0079] Wherein, is the kth input image, k is the frame number (k = 1, 2, …, N), N is the total number of frames, is the image domain, and are the height and width of the image respectively, is the non-negative real number domain, For reference frame number, As a reference frame, This is a floor function;
[0080] Since there may be overall brightness variations between multiple consecutively acquired images (e.g., due to automatic exposure adjustment or changes in ambient lighting), it is first necessary to normalize the brightness of each input image frame to ensure that their brightness levels are consistent with the reference frame. Let the reference frame be... The global mean is , No. The global mean of the frame is Then the brightness normalization factor is The brightness-normalized image of the k-th frame is:
[0081]
[0082] in, For the k-th frame at pixel The preprocessed image after brightness normalization, This is a truncation function. For any pixel location in the image, This is the maximum allowed pixel value (255 for 8-bit images). In subsequent processing, it will be simplified and will still be represented by the symbol '255'. This represents the preprocessed image after brightness normalization.
[0083] S102. Calculate the frame difference image through preprocessing the image, calculate the local noise variance of a single adjacent frame pair based on the sample variance of the frame difference image within the local window, and fuse the local noise variances of multiple adjacent frame pairs to obtain the estimated local noise variance for each frame, specifically:
[0084] For any pixel position Define a local window centered on it. (Window size is) ,generally or Within a window, the image signal can be approximated as a local constant (for slowly changing regions) or contain edge structures. Considering that the signal components between two consecutive frames are approximately equal in stationary regions, while the noise components are independent, the frame difference image mainly reflects noise. Frame and the The formula for calculating the frame difference is:
[0085]
[0086] in, For the first Frame and the Frame in pixel Frame difference at that location, and These are the normalized th Frame and the Frame pixel value, This is a frame difference image;
[0087] In the static region, the frame difference image The local variance is the sum of the noise variances of the two frames. The formula for calculating the sample variance is:
[0088]
[0089] =
[0090] in, The sample variance of the frame difference image within the local window. It is the local mean within the window. This refers to the side length of the local window. These are the pixel coordinates within the local window (usually 7 or 9). For local windows, Let be the frame difference value of the k-th frame difference image at pixel coordinates (u,v) within the local window;
[0091] In the static region, the true signals of two adjacent frames are approximately equal, and the frame difference is mainly composed of independent noise from the two frames. Therefore:
[0092]
[0093]
[0094] in, For frame difference images, and For the first Frame and the Noise components of the frame, For variance calculation;
[0095] If we assume that the noise levels of adjacent frames are similar, then the local noise variance can be approximated as:
[0096]
[0097]
[0098] in, The sample variance of the frame difference image within the local window. To estimate the variance of the obtained local noise, For the first Frame image at pixel position The local noise variance at that location For the first +1 frame image at pixel position The local noise variance at that location pixel position The estimated local noise variance (common noise variance) at the location;
[0099] To obtain a more robust estimate, the predicted local noise variance of the k-th frame is obtained by utilizing information from all adjacent frame pairs:
[0100]
[0101] in, Let Variance be the estimated local noise variance for the k-th frame. It is an indicator function (used to exclude the influence of the motion region). Total number of frames Let be the sample variance of the j-th adjacent frame difference image within the local window at the pixel location;
[0102] S103. Based on the sample variance and minimum local sample variance of the frame difference image within the local window, calculate an adaptive threshold to obtain the motion region recognition result, and remove the frame difference image data corresponding to the motion region, specifically:
[0103] Frame difference images of moving regions have large sample variance within local windows, which may overestimate noise levels. Therefore, they need to be removed. Moving regions are detected using adaptive thresholding.
[0104]
[0105] in, The sample variance of the frame difference image within the local window. The minimum local sample variance for all frame difference images within the local window. It is a preset threshold (usually set to...). arrive between);
[0106] when When the above conditions are met, the pixel location is determined to belong to the moving region, and the corresponding frame difference image is generated. The data at this pixel location is excluded from the local noise estimation process;
[0107] By removing the frame difference image data corresponding to the moving region, the effective noise statistics of the stationary region are retained, providing a reliable data foundation for subsequent local noise variance calculation.
[0108] S104. A linear noise model is constructed based on the true signal strength and the local noise variance. The true signal strength is obtained from the pixel values of the preprocessed image, and the estimated local true signal strength is obtained through low-pass filtering. Finally, the noise parameters and the estimated local true signal strength are input into the linear noise model to obtain the local noise variance, specifically:
[0109] Let the actual signal strength be To establish the correspondence between local noise variance and image signal, local noise variance is used as a basis for further investigation. Compared with the actual signal strength Construct a linear noise model:
[0110]
[0111] in, For local noise variance, This represents the sensor read noise variance (independent of the signal). The sensor gain coefficient. This represents the actual signal strength.
[0112] Due to the actual signal strength Since it cannot be obtained directly, a preprocessed image is used. The pixel values are approximate, but the noise effect needs to be deducted. Considering that the noise is relatively small compared to the signal, it can be used directly. As an estimate of the true signal strength;
[0113] Furthermore, low-pass filtering is performed on the preprocessed image to obtain an estimate of the local true signal strength (low-pass filtering operation). Gaussian filtering can be used).
[0114]
[0115]
[0116] in, This is an estimate of the local true signal strength. This is a low-pass filtering operation (used to reduce the interference of noise in the signal on parameter estimation). For preprocessing images, It is a Gaussian filter kernel. This is an approximation of the true signal strength. The filter radius (usually taken as...) )
[0117] In practice, we pre-calibrate the noise parameters of the sensor. Then, the noise variance is calculated based on the estimated local true signal strength:
[0118]
[0119] in, Let V be the local noise variance of the k-th frame. For sensor read noise variance, The sensor gain coefficient. This is a low-pass filtering operation (used to reduce the interference of noise in the signal on parameter estimation). This is an estimate of the local true signal strength;
[0120] S2. Convert the image from the pixel domain to the gradient domain.
[0121] The gradient field is defined by preprocessing the image, and the gradient components in the horizontal and vertical directions of the preprocessed image are calculated using finite difference. The gradient field of each frame is obtained by combining boundary processing. The specific steps include:
[0122] S201. Define the gradient field based on the preprocessed image:
[0123] The gradient field is defined as follows:
[0124]
[0125] in, For the k-th frame at pixel The gradient vector at that point, The partial derivative in the horizontal direction is... This is the partial derivative in the vertical direction (in actual discrete implementations, the gradient is approximated using finite difference).
[0126] S202. Based on the pixel values of adjacent preprocessed images, calculate the horizontal and vertical gradients using finite difference:
[0127] For the horizontal direction:
[0128]
[0129] For the vertical direction:
[0130]
[0131] in, For the k-th frame at pixel The horizontal gradient at that point For the k-th frame at pixel Vertical gradient at that point , These are the pixel values of adjacent preprocessed images (at the image boundaries, single-sided difference or mirror filling is used).
[0132] S203. By fusing the horizontal and vertical gradients at their respective pixel locations, a gradient field of the fused image at each pixel is generated.
[0133] The horizontal gradient at the corresponding pixel position in each frame image with vertical gradient As the basic data for gradient fusion, horizontal gradient fusion and vertical gradient weighted fusion are performed at corresponding pixel positions to obtain the fused horizontal gradient. With the fused vertical gradient And combine to form a gradient field:
[0134]
[0135] in, To merge images at the pixel level gradient field at, The horizontal gradient after fusion. The vertical gradient after fusion;
[0136] The gradient field should satisfy two constraints simultaneously: First, for regions with high-quality information in the image (low noise, no motion, strong texture), the fused gradient field should be as close as possible to the gradient of the frame with the best quality in that region. Second, the fused gradient field should be "integrable," meaning there exists a scalar image. Make its gradient equal to (This is handled during the Poisson reconstruction stage).
[0137] Gradient domain representation can highlight high-frequency information in an image, such as edges and textures, allowing subsequent fusion operations to process detailed components more effectively.
[0138] S3, Fusion Weight Calculation Stage
[0139] Based on the preprocessed image, local noise variance, and gradient field, the noise level confidence, gradient consistency confidence, spatiotemporal nonlocal similarity, and motion occlusion confidence are calculated respectively, and fusion weights are obtained for the corresponding pixel positions of each frame image. The specific steps include:
[0140] S301, Calculate the confidence level of noise. :
[0141] The confidence level of noise level reflects the first Frame in position Information reliability at a given location, based on local noise variance Calculate the noise confidence weights:
[0142]
[0143] Alternatively, it can be calculated in the following form (the attenuation rate is slower when the noise variance is large):
[0144]
[0145] in, For position Confidence level at the noise level It is an exponential function. For local noise variance, The scaling parameter (controlling the rate of noise attenuation) The typical value is For 8-bit images, the pixel value range is 0-255.
[0146] when hour, near This indicates that the lower the noise at that location, the higher the reliability.
[0147] when hour, Approaching This indicates that the greater the noise at that location, the lower the reliability.
[0148] S302, Calculate gradient consistency confidence. :
[0149] Gradient consistency confidence measure The gradient direction of the first frame is consistent with that of the reference frame. First, calculate the gradient direction of the second frame. Frame in position Gradient direction angle:
[0150]
[0151]
[0152] in, For the first The gradient direction angle of the frame at that pixel. The four-quadrant arctangent function (return value range) ), For the k-th frame at pixel The horizontal gradient at that point For the k-th frame at pixel Vertical gradient at that point The gradient direction angle at this pixel is used as a reference frame. These are the reference horizontal gradient and vertical gradient, respectively;
[0153] The formula for defining gradient direction difference is:
[0154]
[0155] in, For the first The difference in gradient direction between the frame and the reference frame, For the first The gradient direction angle of the frame at that pixel. The gradient direction angle at this pixel is the reference frame.
[0156] Using only the gradient direction may not be sufficient to distinguish all cases, so the consistency of gradient magnitude is also considered. The formula for the gradient magnitude consistency ratio is defined as follows:
[0157]
[0158] in, This is the uniformity ratio of the gradient magnitude. For the k-th frame at pixel gradient magnitude at that point For reference frame in pixels gradient magnitude at that point A small positive number (e.g.) (to prevent division by zero)
[0159] At this point, the gradient magnitude consistency weight is:
[0160]
[0161] in, This is the uniformity ratio of the gradient magnitude. For an exponential parameter (usually taken as...) )
[0162] Considering the consistency of direction and magnitude, the gradient consistency weight is defined as follows:
[0163]
[0164] The final gradient consistency weights are:
[0165]
[0166] in, For pixels Gradient consistency confidence at the location, To control sensitivity to angular differences (typically set to...) (i.e., 30 degrees Celsius) Let be the scaling parameter of the Gaussian decay function. This is the uniformity ratio of the gradient magnitude. It is the square of the gradient direction difference (when the gradient direction difference exceeds 90 degrees, the weight approaches 0). It is an exponential parameter;
[0167] In edge regions, the gradient direction is stable, and the gradient direction between different frames should be basically consistent (unless there is motion). If the gradient direction of a certain frame at a certain position is significantly different from that of other frames, it indicates that there may be motion or registration error at that position, and the contribution of that frame should be reduced.
[0168] S303. Calculate the spatiotemporal nonlocal mean similarity.
[0169] For the position in the reference frame Define the image patch centered on it:
[0170]
[0171] in, For reference frame, by position The image patch centered on For size A real matrix, It is the image patch size (usually taken as...) or );
[0172] For the The corresponding position in the frame (Position after motion compensation), similarly define image patches. ;
[0173] However, to handle non-rigid motion and registration errors, we do not require precise alignment of block centers, but instead search for the most similar block within a local search window:
[0174] Specifically, in the In the frame, with Define the search window by center Window size is (generally or ), search for the reference block in the search window. Most similar blocks:
[0175]
[0176] in, The optimal offset that minimizes block differences. These are the horizontal and vertical offsets of the candidate matching block relative to the current pixel position. It is the image patch size. For reference frame, by position The image patch centered on , This refers to the local coordinate offset within the image patch. This refers to the image patch at the corresponding position in the k-th frame;
[0177] Find the optimal matching offset Then, the normalized cross-correlation (NCC) or mean squared error (MSE) between the reference block and the best matching block is calculated, using a block similarity metric based on Gaussian kernel weighting:
[0178]
[0179] in, This is a weighted difference metric between the reference block and the best matching block in the k-th frame. For reference frame, by position The image patch centered on , This refers to the local coordinate offset within the image patch. This represents the image patch at the corresponding position in the k-th frame. It is a spatial attenuation parameter (which controls the weight of the center pixel to be higher than that of the edge pixels; a typical value is...). (The weighted similarity is more sensitive to differences in the central region).
[0180] The formula for calculating spatiotemporal nonlocal mean similarity is:
[0181]
[0182] in, For spatiotemporal nonlocal mean similarity, It is a parameter that controls similarity sensitivity (typically set to a value of...). For 8-bit images, ), The weighted difference measure between the reference block and the best matching block in the k-th frame is d, where d is the image block size (to keep the weights consistent across different block sizes, the denominator contains...). (Normalization)
[0183] To further improve the utilization of information in the time dimension, we not only calculate the similarity between the current frame and the reference frame, but also the similarity between adjacent frames, thus constructing a spatiotemporal similarity graph:
[0184] set up For the first Frame and the The temporal consistency factor is calculated based on the local block similarity between +1 frames:
[0185]
[0186] in, To represent the temporal consistency factor of the k-th frame in the time series, For average similarity, For the first Frame and the +1 local block similarity between frames It is a parameter sensitive to time consistency;
[0187] This factor measures the degree of anomaly of a frame in a time series: if a frame has low similarity to the frames before and after it at a certain position, it indicates that the position may be a temporary anomaly (such as transient noise or motion occlusion), and its contribution should be reduced.
[0188] Spatiotemporal nonlocal mean similarity considers not only the similarity of individual pixels, but also the overall similarity of the image patch centered on that pixel in the spatiotemporal neighborhood, thus more robustly measuring the information relevance between multiple frames.
[0189] S304. Calculate motion occlusion confidence.
[0190] Motion occlusion is one of the main reasons for multi-frame fusion failure. When an object moves in the scene, some areas may be occluded or exposed in consecutive frames. The information in these areas is inconsistent in the time dimension and should not be included in the fusion.
[0191] To address this, an occlusion detection method based on forward and backward optical flow consistency detection is proposed:
[0192] First calculate from the... Frame to the Forward optical flow field of the frame:
[0193]
[0194] And from the Frame to the Backward optical flow field of the frame:
[0195]
[0196] in, For the first Frame to the Forward optical flow field of the frame, To represent the horizontal component of the forward optical flow, This represents the vertical component of the forward optical flow. For the first Frame to the The backward optical flow field of the frame;
[0197] If a pixel is not occluded, for the th Position in the frame Then from Frame to Frame back The frame should be able to return to its original position, therefore the forward and backward consistency error is defined as:
[0198]
[0199] in, For the forward and backward optical flow consistency error, For the first Frame to the Forward optical flow field of the frame, For the first Frame to the The backward optical flow field of the frame;
[0200] if Greater than the threshold (usually taken) or (pixel), then it is considered that the position is in the th position. The frame is occluded, therefore the first The information at this location in the frame is fused into the first... Frame time unreliable;
[0201] For the Frame relative to reference frame The occlusion confidence score is used to calculate the cumulative occlusion error.
[0202]
[0203] The formula for calculating the confidence score of motion occlusion weight is:
[0204]
[0205] in, To represent the cumulative occlusion error of the k-th frame relative to the reference frame, Let k be the set of adjacent frame paths traversed from the k-th frame to the reference frame. The attenuation constant is typically set to a value of 1. ), For the forward and backward optical flow consistency error, The confidence score for motion occlusion (in the occluded area, this weight is close to 0, thus effectively eliminating false information);
[0206] S305, Calculate the final fusion weights
[0207] The system integrates four factors: noise level confidence, gradient consistency confidence, spatiotemporal nonlocal similarity, and motion occlusion confidence. The final fusion weight is the product of these four components. The final fusion weight is calculated as follows:
[0208]
[0209] in, For the k-th frame at pixel The final fusion weight, For the noise level confidence level, For gradient consistency confidence, For spatiotemporal nonlocal similarity, Confidence level for motion occlusion;
[0210] Therefore, a pixel will only have a large final fusion weight if it simultaneously satisfies the following conditions: low noise level, consistent gradient structure, high spatiotemporal neighborhood similarity, and no motion occlusion. If any of these factors indicates that the pixel has low reliability, the corresponding component will reduce its overall weight, thus reducing the pixel's contribution in the subsequent fusion process.
[0211] S4, Gradient Generation Fusion Stage
[0212] To obtain the final fusion weights, the horizontal and vertical gradients at corresponding pixel positions in each frame are weighted and fused to generate a fusion gradient. A gradient consistency index is calculated, and the fusion gradient is nonlinearly enhanced based on the gradient consistency index to obtain the fusion gradient field. This process includes the following steps:
[0213] S401, Calculate the fusion weights of each frame image at each location. Then, gradient domain fusion is performed to obtain the fused horizontal gradient and the fused vertical gradient.
[0214] For the horizontal gradient component, the fused horizontal gradient is:
[0215]
[0216] For the gradient components in the vertical direction, the fused vertical gradient is:
[0217]
[0218] in, For the fused horizontal gradient, For the k-th frame at pixel The final fusion weight, For the horizontal gradient of the k-th frame, The vertical gradient of the k-th frame is... For a small regularization constant (such as (to prevent the denominator from being zero) The vertical gradient after fusion;
[0219] However, directly using the above weighted average may over-smooth the gradient field. In order to suppress noise while maintaining sharp edges, a gradient domain guided filtering mechanism is introduced. The core idea of this mechanism is: for each position, determine whether the gradient at that position mainly comes from the signal or noise, and adjust the fusion strategy accordingly.
[0220] S402. Define the gradient consistency index using the final fusion weights, horizontal gradient, and vertical gradient:
[0221]
[0222] in, As a gradient consistency index, For the normalized k-th frame at pixel The final fusion weight, For the horizontal gradient of the k-th frame, The vertical gradient of the k-th frame is... It is the square of the horizontal gradient. The square of the vertical gradient. It is a small regularization constant;
[0223] The range of values for this indicator is: .when When the gradient direction is consistent across frames at that location, it indicates the presence of a real edge or texture, and a stronger gradient should be preserved. If the gradient directions of different frames are inconsistent (which may be due to noise or motion), it is more likely to be affected by noise, motion, or occlusion, and the gradient should be suppressed.
[0224] S403. Based on the consistency index, nonlinear enhancement is performed on the fusion gradient to obtain the enhanced horizontal gradient and the enhanced vertical gradient, which together form the enhanced fusion gradient field:
[0225]
[0226]
[0227] in, To enhance the horizontal gradient, To enhance the vertical gradient, For the fused horizontal gradient, For the fused vertical gradient, To enhance the strength coefficient (typical value) ), This is a protection constant (typical value) to prevent small gradients from being over-amplified. ), The magnitude of the fusion gradient ( ), This is a gradient consistency index;
[0228] This nonlinear enhancement mechanism causes the gradient to be decayed in regions of low consistency (noisy regions), while in regions of high consistency (edge regions), the gradient is maintained or even slightly enhanced.
[0229] The enhanced horizontal gradient is combined with the enhanced vertical gradient to form the enhanced fused gradient field:
[0230]
[0231] in, For the enhanced fusion gradient field, To enhance the horizontal gradient, To enhance the vertical gradient;
[0232] The enhanced fusion gradient field is used to uniformly represent the gradient information of the fused image, serving as the target gradient field in the Poisson reconstruction stage, and providing gradient data for subsequent divergence calculation and fused image reconstruction.
[0233] S5, Fusion Image Reconstruction
[0234] Based on the enhanced fused gradient field, the corresponding Poisson equation is constructed, the divergence of the target gradient field is calculated, and the Poisson equation is solved in the frequency domain using two-dimensional discrete cosine transform. Subsequently, the fused image is recovered through inverse discrete cosine transform to obtain the reconstructed fused enhanced image. The specific steps include:
[0235] S501. Construct the Poisson equation based on the fused gradient field, and establish the image reconstruction equation with the target gradient field as a constraint.
[0236] Obtain the image , so that its gradient As close as possible to the target gradient field This can be achieved by solving the following Poisson equation:
[0237]
[0238] in, For the Laplace operator, The pixel values of the fused image to be reconstructed. It is a divergence operator. For the target gradient field;
[0239] S502. Discretize the Laplace operator and the divergence operator corresponding to the target gradient field in the Poisson equation to obtain the divergence field;
[0240] In discrete form, for internal pixels The Laplace operator can be discretized as follows:
[0241]
[0242] The divergence operator can be discretized as follows:
[0243]
[0244] in, For the fused image to be reconstructed in Pixel value at that location, For divergence operators, For the Laplace operator, and These are respectively enhancing the horizontal gradient and enhancing the vertical gradient. For the target gradient field, It is a divergence field;
[0245] S503. Perform a two-dimensional discrete cosine transform on the divergence field to obtain the frequency domain coefficients of the divergence field, and solve the Poisson equation in the transform domain to output the fused image coefficients.
[0246] When dealing with rectangular image regions and applying von Neumann boundary conditions, a fast solution can be obtained using the two-dimensional discrete cosine transform (DCT). The specific steps are as follows:
[0247] First, define the divergence field. :
[0248]
[0249] Then perform a two-dimensional discrete cosine transform:
[0250]
[0251] in, The frequency domain coefficients of the divergence field after a two-dimensional discrete cosine transform are... It is a two-dimensional discrete cosine transform. For frequency domain coordinates, It is a cosine function. Here, H is the divergence field, H is the image height, and W is the image width.
[0252] In the transformation domain, the solution to the Poisson equation is:
[0253]
[0254] in, These are the fused image coefficients in the frequency domain. The frequency domain coefficients of the divergence field after a two-dimensional discrete cosine transform are... For frequency domain coordinates (for The DC component, with a denominator of zero, at this time It can be set arbitrarily, but is usually set to zero).
[0255] S504. Perform a two-dimensional inverse discrete cosine transform on the coefficients of the fused image to obtain the fused enhanced image;
[0256] Finally, an inverse discrete cosine transform is performed to obtain the fused and enhanced image.
[0257]
[0258] in, For the reconstructed fused enhanced image, It is a two-dimensional inverse discrete cosine transform. These are the fused image coefficients in the frequency domain;
[0259] The reconstructed and fused enhanced image is used to preserve the frequency domain information corresponding to the fused gradient field, providing a frequency domain basis for accurate reconstruction of the fused image.
[0260] S6, Image Fusion Optimization
[0261] Post-processing the fused and enhanced image to output the final fused and enhanced image includes the following steps:
[0262] For the luminance channel (if processing a color image, first convert the image to YCbCr space, perform the above fusion process only on the Y channel, and use frame averaging fusion for the Cb and Cr channels), perform adaptive gamma correction:
[0263]
[0264] in It is a spatially adaptive gamma value:
[0265]
[0266] in, This is the final fused and enhanced image after gamma correction. This represents the maximum allowed pixel value. For the reconstructed fused enhanced image, For pixels The adaptive gamma value at that location, It is the average brightness within a local window. It is the global average brightness. It is the base gamma value (usually taken as...). arrive between);
[0267] The final fused and enhanced image can integrate effective information from multiple frames, suppressing noise and motion artifacts while preserving image edge, texture, and color information, thus improving the integrity and visual quality of the fused image.
[0268] Example 2: A multi-frame image fusion enhancement system based on gradient domain and fusion weights, comprising:
[0269] Image preprocessing module: Performs reference frame selection, brightness normalization, local frame difference analysis, motion region removal, and signal-related noise estimation on continuously acquired multi-frame images, and obtains the variance of the preprocessed image and local noise.
[0270] Gradient field construction module: Defines the gradient field through the preprocessed image, calculates the gradient components of the preprocessed image in the horizontal and vertical directions using finite difference, and obtains the gradient field of each frame image by combining boundary processing.
[0271] Weight fusion calculation module: Based on brightness-normalized multi-frame images, local noise variance and gradient field, calculate noise level confidence, gradient consistency confidence, spatiotemporal nonlocal similarity and motion occlusion confidence respectively, and fuse them to obtain the fusion weight of the corresponding pixel position of each frame image;
[0272] The gradient fusion generation module obtains the final fusion weights, performs weighted fusion of the horizontal and vertical gradients at the corresponding pixel positions in each frame image to generate the fusion gradient, calculates the gradient consistency index, and performs nonlinear enhancement on the fusion gradient based on the gradient consistency to obtain the fusion enhanced image.
[0273] Final image optimization module: Post-processes the fused and enhanced image, processes the brightness channel according to the image type, and outputs the final fused and enhanced image.
[0274] The above embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions will not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A multi-frame image fusion enhancement method based on gradient domain and fusion weights, characterized in that, Includes the following steps: S1. Perform reference frame selection, brightness normalization, local frame difference analysis, motion region removal, and signal-related noise estimation on multiple frames of images to obtain the variance of the preprocessed image and local noise. S2. Transform the preprocessed image from the pixel domain to the gradient domain, specifically including the following steps: S201. Define the gradient field by preprocessing the image; S202. Obtain the horizontal and vertical gradients of the preprocessed image using differential calculation: S203. Using the horizontal and vertical gradients as the basic data for gradient fusion, and according to the calculated fusion weights, the horizontal and vertical gradients are weighted and fused respectively to obtain the fused horizontal and vertical gradients, and a gradient field is constructed. S3. Based on the preprocessed image, local noise variance, and gradient field, calculate the noise level confidence, gradient consistency confidence, spatiotemporal nonlocal similarity, and motion occlusion confidence, and fuse them to obtain the final fusion weight of the corresponding pixel position in each frame image. S4. Obtain the final fusion weights, perform weighted fusion on the horizontal and vertical gradients of the corresponding pixel positions in each frame image to generate a fusion gradient, calculate the gradient consistency index, and perform nonlinear enhancement on the fusion gradient based on the gradient consistency index to obtain the fusion gradient field. S5. Based on the fused gradient field, construct the corresponding Poisson equation, calculate the divergence of the target gradient field, and solve the Poisson equation in the frequency domain using two-dimensional discrete cosine transform. Then, recover the fused image through inverse discrete cosine transform to obtain the reconstructed fused enhanced image. S6. Perform post-processing on the fused and enhanced image to output the final fused and enhanced image.
2. The multi-frame image fusion enhancement method based on gradient domain and fusion weights according to claim 1, characterized in that, Step S1 specifically includes the following steps: S101. Acquire multiple consecutively acquired images as input images, and perform brightness normalization to obtain a preprocessed image; S102. Calculate the frame difference image through preprocessing the image, calculate the local noise variance of a single adjacent frame pair based on the sample variance of the frame difference image within the local window, and fuse the local noise variances of multiple adjacent frame pairs to obtain the estimated local noise variance of each frame. S103. Based on the sample variance and minimum local sample variance of the frame difference image within the local window, calculate the adaptive threshold to obtain the motion region recognition result, and remove the frame difference image data corresponding to the motion region. S104. Construct a linear noise model based on the true signal strength and the local noise variance. The true signal strength is obtained by the pixel value of the preprocessed image, and the estimated value of the local true signal strength is obtained by low-pass filtering. Finally, the noise parameters and the estimated value of the local true signal strength are input into the linear noise model to obtain the local noise variance.
3. The multi-frame image fusion enhancement method based on gradient domain and fusion weights according to claim 2, characterized in that, In step S103, the formula for calculating the sample variance is: ; = ; in, The sample variance of the frame difference image within the local window. It is the local mean within the window. This refers to the side length of the local window. These are the pixel coordinates within the local window. For local windows, Let be the frame difference value of the k-th frame difference image at pixel coordinates (u,v) within the local window.
4. The multi-frame image fusion enhancement method based on gradient domain and fusion weights according to claim 1, characterized in that, In step S202, the formulas for calculating the horizontal gradient and the vertical gradient are as follows: ; ; in, For the k-th frame at pixel The horizontal gradient at that point For the k-th frame at pixel Vertical gradient at that point , These are the pixel values at adjacent pixel positions.
5. The multi-frame image fusion enhancement method based on gradient domain and fusion weights according to claim 1, characterized in that, In step S3, the formula for calculating spatiotemporal nonlocal similarity is: ; in, For spatiotemporal nonlocal mean similarity, It is a parameter that controls similarity sensitivity. d is the weighted difference measure between the reference block and the best matching block in the k-th frame, where d is the image block size.
6. The multi-frame image fusion enhancement method based on gradient domain and fusion weights according to claim 5, characterized in that, In step S3, the final fusion weight calculation formula is as follows: ; in, For the k-th frame at pixel The final fusion weight, For the confidence level of noise level, For gradient consistency confidence, For spatiotemporal nonlocal similarity, Confidence level for motion occlusion.
7. The multi-frame image fusion enhancement method based on gradient domain and fusion weights according to claim 1, characterized in that, The formula for calculating the gradient consistency index in step S4 is defined as follows: ; in, As a gradient consistency index, For the normalized k-th frame at pixel The final fusion weight, For the horizontal gradient of the k-th frame, The vertical gradient of the k-th frame is... It is the square of the horizontal gradient. The square of the vertical gradient. It is a small regularization constant; The formulas for calculating the enhanced horizontal and vertical gradients are as follows: ; ; in, To enhance the horizontal gradient, To enhance the vertical gradient, For the fused horizontal gradient, For the fused vertical gradient, To enhance the strength coefficient (typical value) ), A protection constant (typical value) to prevent small gradients from being over-amplified. ), The magnitude of the fusion gradient ( ), This is a gradient consistency index.
8. The multi-frame image fusion enhancement method based on gradient domain and fusion weights according to claim 1, characterized in that, Step S5 specifically includes the following steps: S501. Construct the Poisson equation based on the fused gradient field, and establish the image reconstruction equation with the target gradient field as a constraint. S502. Discretize the Laplace operator and the divergence operator corresponding to the target gradient field in the Poisson equation to obtain the divergence field; S503. Perform a two-dimensional discrete cosine transform on the divergence field to obtain the frequency domain coefficients of the divergence field, and solve the Poisson equation in the transform domain to output the fused image coefficients. S504. Perform a two-dimensional inverse discrete cosine transform on the coefficients of the fused image to obtain the fused enhanced image.
9. The multi-frame image fusion enhancement method based on gradient domain and fusion weights according to claim 1, characterized in that, In step S6, the final formula for calculating the fused and enhanced image is: ; in It is a spatially adaptive gamma value: ; in, This is the final fused and enhanced image after gamma correction. This represents the maximum allowed pixel value. For the reconstructed fused enhanced image, For pixels The adaptive gamma value at that location, It is the average brightness within a local window. It is the global average brightness. It is the base gamma value.
10. A multi-frame image fusion enhancement system based on gradient domain and fusion weights, characterized in that, The system is used in the multi-frame image fusion enhancement method based on gradient domain and fusion weights as described in any one of claims 1-9, the system comprising: Image preprocessing module: performs reference frame selection, brightness normalization, local frame difference analysis, motion region removal, and signal-related noise estimation on multiple frames of images, and obtains the variance of the preprocessed image and local noise. Gradient field construction module: Define the gradient field through the preprocessed image, and use finite difference to calculate the gradient of the preprocessed image in the horizontal and vertical directions. Use the gradients in the horizontal and vertical directions as the basic data for fusion, calculate the fusion weights and then perform weighted fusion to obtain the fused horizontal gradient and the fused vertical gradient, and construct the gradient field. Weight fusion calculation module: Based on the preprocessed image, local noise variance and gradient field, calculate the noise level confidence, gradient consistency confidence, spatiotemporal nonlocal similarity and motion occlusion confidence respectively, and fuse them to obtain the fusion weight of the corresponding pixel position of each frame image; Fusion gradient generation module: Obtain fusion weights, perform weighted fusion of the horizontal and vertical gradients at corresponding pixel positions in each frame image to generate fusion gradients, calculate gradient consistency index, and perform nonlinear enhancement on the fusion gradient based on gradient consistency to obtain fusion enhanced image; Final image optimization module: performs post-processing on the fused and enhanced image, and outputs the final fused and enhanced image.
