A Multifocal Image Fusion Method to Eliminate Defocus Diffusion Effect

By generating a multifocal image fusion dataset based on optical imaging principles and combining a two-stage strategy of diffusion model and lightweight image fidelity network, the problem of defocus diffusion effect in multifocal image fusion is solved, and high-quality image fusion effect is achieved.

CN118657668BActive Publication Date: 2026-11-17DALIAN NATIONALITIES UNIVERSITY
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202410686666.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-05-30
Publication Date
2026-11-17
Estimated Expiration
2044-05-30

AI Technical Summary

Technical Problem

Existing multifocal image fusion methods are ineffective in eliminating defocus diffusion (DSE) effects, mainly due to unrealistic datasets, limited backbone network performance, and insufficient fusion strategies, which prevent them from effectively addressing DSE.

Method used

Based on the principle of optical imaging, a multifocal image fusion dataset is generated. Using a diffusion model and a lightweight image fidelity network, a two-stage 'generation and correction' strategy is adopted, combining disk blur and Gaussian blur, to generate high-quality fully focused images.

Benefits of technology

It significantly narrows the gap between the synthetic training dataset and real multifocal images, effectively eliminates the defocus diffusion effect, improves the performance and quality of image fusion, and alleviates color distortion and artifact problems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118657668B_ABST
    Figure CN118657668B_ABST
Patent Text Reader

Abstract

This invention proposes a multifocal image fusion method to eliminate defocus diffusion effects, comprising: generating a multifocal image fusion dataset based on a defocus blur prior according to optical imaging principles; training a diffusion model using the multifocal image fusion dataset; inputting the source image into the trained diffusion model to obtain the output fused image; and inputting the source image and the fused image output by the diffusion model into an image fidelity network to obtain the fused multifocal image. This invention focuses on solving a long-neglected problem in multifocal image fusion (MFIF), namely the unrealistic nature of the dataset, and proposes a novel method to generate multifocal image datasets. This method not only introduces the concept of isometric lines in the mask to guide filter blur, but also combines the advantages of disk blur and Gaussian blur. This method significantly reduces the gap between the synthetic training dataset and the real multifocal image, effectively solving the defocus diffusion effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention belongs to the field of multifocal image fusion technology, and particularly relates to a multifocal image fusion method for eliminating defocus diffusion effects. Background Technology

[0002] Defocus blur (also known as out-of-focus blur) refers to the distortion that occurs when a light source deviates from the focal plane of an image, resulting in objects appearing as blurry spots instead of points when they are not in focus. Due to the limitations of depth of field (DoF) of camera lenses, it is difficult to focus on all objects in complex scenes, causing some objects to appear out of focus. MFIF aims to solve the problem of defocus blur caused by the depth of field of camera lenses. MFIF integrates information from multiple source images with the same scene but different focused objects and generates a "perfectly focused" image.

[0003] The primary challenge facing current multifocal image fusion is defocus diffusion, which refers to the blurring of the boundary region around a foreground object in both source images due to background defocus. Existing fusion methods perform poorly in addressing DSE, mainly due to three factors: dataset, architecture, and fusion strategy. First, there is a significant difference between the synthetic training set and real multifocal images. Due to the lack of training sets containing real multifocal images and ground truth values, the current mainstream approach is to synthesize training data from sharp images using Gaussian or mean filters. However, real defocus blur is more complex than the blur generated by these filters because it is affected by factors such as device, lighting, and object distance. Therefore, the two types of blur are not entirely consistent, and in particular, these filters cannot simulate real DSE, thus severely limiting network performance—a problem that has long been overlooked in the MFIF field. Second, current mainstream MFIF algorithms often use convolutional neural networks (CNNs) or Transformers as the backbone network and generative adversarial networks (GANs) as the generative strategy. However, the drawback of these mainstream architectures is that they increase the complexity of DSE removal. The local receptive field of CNNs limits the model's ability to perceive the global focus of an object, while focal separation (DSE) typically occurs near the contours and edges of the focused object, making it difficult for CNNs to accurately identify and resolve DSE. Problems such as gradient vanishing, mode collapse, and convergence failure in GANs make it difficult to train models to achieve the desired results. Transformers, based on patch-level global interactions and long-range dependencies, cannot solve the DSE problem at the pixel level. Finally, current state-of-the-art algorithms mainly employ two fusion strategies: concatenation and generation. Concatenation strategies typically merge focus regions from source images based on binary decision maps. However, since DSE exists in multiple source images simultaneously, concatenation cannot eliminate DSE regardless of the approach. Generation strategies involve using a network to extract and fuse effective features from multiple source images and reconstruct the fused image. Clearly, generation strategies have the potential to resolve DSE, but they inevitably lead to severe color distortion and artifacts. Summary of the Invention

[0004] The purpose of this invention is to propose a multifocal image fusion method that eliminates the defocus diffusion effect, which can effectively solve the defocus diffusion effect.

[0005] To achieve the above objectives, the present invention provides a multifocal image fusion method for eliminating defocus diffusion effects, comprising: Based on the defocus blur prior of optical imaging principle, a multifocal image fusion dataset is generated. A diffusion model is trained using the multifocal image fusion dataset. The source image is input into the trained diffusion model to obtain the output fused image. The source image and the fused image output by the diffusion model are input into the image fidelity network to obtain the fused multifocal image.

[0006] Optionally, based on the defocus blur prior of optical imaging principles, the generated multifocal image fusion dataset includes: A randomly generated mask is used to guide the synthesis of multifocal images to obtain the mask image; For the region of interest in the mask image, dense equidistant lines are set to obtain a mask image Md with several equidistant lines; The mask Md image is used to guide the generation of multifocal source images, simulating far-focus and near-focus source images; Based on the image degradation simulation method, the far-focus source image and the near-focus source image are processed to obtain the multifocal image fusion dataset.

[0007] Optionally, the mask image is in, M This represents the generated binary mask. This represents the mask generation function. I 0 The matrix is ​​a set of all zeros of the same size as I, where I represents the input source image. ρ 1. ρ 2 and ρ 3 represents the area, shape, and location of the region of interest, respectively.

[0008] Optionally, setting dense equidistant lines for the region of interest in the mask image includes: Calculate the Euclidean distance between the non-boundary pixels of the region of interest and each boundary pixel of the region of interest in the mask, and select the shortest distance as D(x,y); where D(x,y) refers to the shortest distance from the non-boundary pixel M(x,y) in the mask image to the boundary pixel. Based on the shortest distance D(x,y) from each pixel to the region of interest, a distance map of the same size as the mask image is formed; in the distance map, all pixels with a value equal to D form an equidistant line; Based on the equidistant lines, a distance set containing several distance thresholds is established, and the dense equidistant lines are set based on the distance set; wherein, each distance threshold in the distance set is generated by a random function, and each distance threshold satisfies a preset condition.

[0009] Optionally, the method for calculating the Euclidean distance is as follows: in,( x si ,ysi ) represents each boundary pixel Si The coordinates, ||·|| represent the Euclidean distance in spatial coordinates, ( x,y () represents the two-dimensional coordinates of an image pixel, used to locate the pixel position. This represents the value of the pixel at coordinates (x, y) in the distance image P, and n represents the total number of boundary pixels in the mask image; The preset conditions that each distance threshold must meet are: in, D For distance set, R 2 is a random function. Max(P) and Min(P) These represent the maximum and minimum values ​​in the distance graph, respectively. k It is a number.

[0010] Optionally, the method for simulating the telephoto source image is as follows: in, I ω Indicates the disk is blurred. ,I ω ( x , y ) represents the image I with ( x , y The number of pixels covered by the window ω centered on the pixel. η Indicates brightness parameter, r Indicates the window radius. Represents source image I A The value of the pixel with spatial coordinates (x, y) in the mid-space. This refers to a disk filter. This represents the value of the pixel with spatial coordinates (x, y) in the masked image M; The method for simulating near-focus source images is as follows: Where γ² represents the Gaussian filter, and σ represents the standard deviation of the Gaussian filter. This represents the value of the pixel with spatial coordinates (x, y) in the source image B that is synthesized.

[0011] Optionally, based on an image degradation simulation method, the far-focus source image and the near-focus source image are processed, including: Use a mean filter to define the boundaries within the mask; Based on the defined boundaries within the mask, the center pixel is classified into non-boundary pixels and boundary pixels; Degradation operations are performed on the far-focus source image and the near-focus source image; Based on the classified non-boundary pixels and boundary pixels, as well as the degraded far-focus source image and near-focus source image, a multifocal source image is synthesized, thus generating the multifocal image fusion dataset.

[0012] Alternatively, the method for classifying the center pixel into non-boundary pixels and boundary pixels is as follows: Where B represents the boundary mapping, Indicates the boundary within the defined mask; The method for performing degradation operations on the far-focus source image and the near-focus source image is as follows: in, k h1 Indicates Gaussian blur, k h2 Indicates a disc-shaped blur, ↓ s and ↑ u This indicates downsampling and upsampling operations, where n represents randomly selected salt-and-pepper noise and Gaussian white noise. This represents the result image after image degradation processing. Represents source image I A Or I B It is uniformly represented as H; The multifocal source image is: in, This represents the initial multifocal source image used in the final synthesis.

[0013] Optionally, the image fidelity network employs a lightweight convolutional network; the lightweight convolutional network is a CNN comprising several convolutional layers; each layer, except for the first and last layers, includes several filters of a preset size for the input image; the first and last layers consist of a single filter of a preset size for image reconstruction, and the loss function is mean squared error. The fused multifocal image is: in, This represents the fused multifocal image. This represents the initial fused image output by the diffusion model. Represents an image fidelity network. Represents source image A, This represents source image B.

[0014] The present invention has the following beneficial effects: This paper addresses a long-neglected issue in multifocal image fusion (MFIF)—the unrealistic nature of the dataset—and proposes a novel method for generating multifocal image datasets. This method not only introduces the concept of isometric lines in the mask to guide filter blur but also combines the advantages of disk blur and Gaussian blur. This approach significantly reduces the gap between the synthetic training dataset and real multifocal images.

[0015] This paper introduces a two-stage fusion strategy of "generation and correction" for the first time. In the generation stage, a diffusion model is used as the network architecture to effectively eliminate the defocus diffusion effect. In the correction stage, a lightweight network is proposed to alleviate the color distortion and artifact problems inherent in the generative model. Experiments demonstrate that this strategy can improve the performance of various generative models.

[0016] Extensive comparative and ablation experiments have demonstrated the performance of the proposed model. Experimental results on seven test sets show that the model effectively addresses the defocusing diffusion effect, achieving optimal performance. Attached Figure Description

[0017] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a schematic flowchart of a multifocal image fusion method for eliminating defocus diffusion effect according to an embodiment of the present invention; Figure 2 This is a schematic diagram of defocusing and defocusing imaging principle in an embodiment of the present invention; Figure 3 This is a visual comparison diagram of different blurring methods according to embodiments of the present invention; Figure 4 This is a schematic diagram of the model principle in an embodiment of the present invention; Figure 5 This is a schematic diagram of the dataset synthesis method according to an embodiment of the present invention. Detailed Implementation

[0018] It should be noted that, unless otherwise specified, the embodiments and features described in the present invention can be combined with each other. The present application will now be described in detail with reference to the accompanying drawings and embodiments.

[0019] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0020] Defocus blur prior: From an optical perspective, camera imaging is the process of light passing through a lens and being captured by a sensor. At the moment the shutter is pressed, the lens's focal length is fixed, and the distance between the lens and the sensor's imaging plane is also fixed. Therefore, there is a focal plane in the camera, and objects located on the focal plane can be focused on the imaging plane and obtain a sharp image. However, during photography, there are often objects in the scene that are far from the defocusing plane. According to geometric optics, the light emitted or reflected by these objects forms a disk on the imaging plane after passing through the lens, called the out-of-focus circle (CoC).

[0021] From a mathematical perspective, defocusing degenerates into: Where (x,y) represents the spatial coordinates of light on the lens, d(x,y) is the defocused image, K(x,y) is the PSF (Power Seer Filter), h(x,y) is the sharp image, and n(x,y) is the noise. Ideally, the PSF can be described as: Where (c,d) represents the center of the lens, and r is the radius of the defocus blur.

[0022] According to geometrical optics, the farther an object is from the focal plane, the larger the CoC (Coarse Optical Contour) becomes, and the closer the object is to the focal plane, the smaller the CoC becomes. However, when the object is very close to the focal plane, factors such as light diffraction and interference, front-end noise, and sensor sampling cause the final image of the light to not appear in the form of CoC.

[0023] Furthermore, in the real world, this embodiment found that bright spots are more noticeable after defocusing. This is because the light emitted or reflected by bright spots passes through the convex lens of the lens to form an image, while dark spots emit or reflect less light and do not participate in the imaging process. Defocus blur characteristics at different distances are as follows... Figure 2 As shown. Based on the foregoing, this embodiment summarizes the following priors: Prior 1. When an object is outside the focal plane and not particularly close to it, the light emitted or reflected by it forms a disk on the image plane after passing through the lens.

[0024] Prior 2. The farther the object is from the focal plane, the larger the CoC; the closer the object is to the focal plane, the smaller the CoC.

[0025] Prior 3. When an object is outside the focal plane and very close to it, factors such as light diffraction and interference, front-end noise, and sensor sampling cause the final image of the light to not appear in the form of CoC, but rather to resemble Gaussian blur.

[0026] Prior 4. Because bright spots are more affected during defocusing, the bright areas are more noticeable after defocusing.

[0027] Based on prior 1, this embodiment applies a disk blur method to the pixels to simulate blur circle diffusion. This embodiment uses Gaussian blur and disk blur respectively to obtain two corresponding images of a high-definition city night scene, as shown below. Figure 3 As shown. Clearly, the image obtained through disk blur is closer to the visual perception of defocus blur in the real world. Especially in luminous areas, Gaussian blur cannot simulate the blur circle, while disk blur can.

[0028] Based on prior 2, when simulating defocus blur, this embodiment needs to design the distance relationship between near and far objects and adjust the degree of pixel diffusion.

[0029] Based on prior 3, when the object is very close to the focal plane, this embodiment uses Gaussian blur to simulate interference from various factors.

[0030] Based on prior 4, this embodiment should make the diffusion of bright areas in the image more obvious to better reflect the real-world scenario, rather than treating all pixels equally.

[0031] Multifocal image fusion effectively improves the visual quality of images with partial focus defects. Defocus diffusion (DSE) is one of the most challenging problems in the field of multifocal image fusion. To address this issue, this embodiment proposes a comprehensive solution comprising three key aspects: dataset generation, backbone network selection, and fusion strategy. Regarding dataset generation, this embodiment addresses the significant gap between synthetic datasets and real multifocal images by summarizing Defocus Blur Prior (DBP) based on optical imaging principles. Supported by DBP, this embodiment designs an innovative dataset generation technique. This technique defines the concept of mask equidistant lines and effectively combines Gaussian blur and disk blur to simulate distance-sensitive near-focus and far-focus blur. Regarding backbone network selection, this embodiment finds that diffusion models are more suitable for MFIF than CNNs and Transformers. By using synthetic multi-source images as additional conditions for DDPM training, DSE in the fusion process is effectively eliminated. Regarding the fusion strategy, this embodiment proposes a two-stage "generation and correction" strategy, introducing a lightweight image fidelity network that mitigates the inherent image distortion problem of the generative model. The benchmark results show that: (i) the generative model in this embodiment has the best performance, producing fully focused images without DSE; (ii) the dataset generation method and image fidelity network proposed in this embodiment are model-independent, effectively enhancing various MFIF algorithms.

[0032] This embodiment employs a comprehensive approach comprising three parts: a dataset generation strategy, a diffusion model, and an image fidelity network, such as... Figure 4As shown in the diagram. During training, this embodiment proposes a novel data generation method to obtain a more realistic multifocal image fusion (MFIF) dataset, which is then used to train the diffusion model in the first stage. In the second stage, the source image and its corresponding image generated by the diffusion model are input into an image fidelity network to obtain a corrected image. During the fusion (test) process, the model receives the real multifocal source image as input and outputs a fully focused fused image.

[0033] The multifocal image fusion method for eliminating defocus diffusion effect proposed in this embodiment includes: Based on the defocus blur prior of optical imaging principle, a multifocal image fusion dataset is generated. A diffusion model is trained using a multifocal image fusion dataset. The source image is input into the diffusion model to obtain the corresponding image. The source image and its corresponding image are input into an image fidelity network to obtain a fused multifocal image.

[0034] Furthermore, based on the defocus blur prior of optical imaging principles, the multifocal image fusion dataset is generated as follows: A randomly generated mask is used to guide the synthesis of multifocal images to obtain the mask image; For the region of interest in the mask image, dense equidistant lines are set to obtain a mask image Md with several equidistant lines; The mask Md image is used to guide the generation of multifocal source images, simulating far-focus and near-focus source images; Based on the image degradation simulation method, the far-focus source image and the near-focus source image are processed to obtain the multifocal image fusion dataset.

[0035] Furthermore, setting dense equidistant lines for the region of interest in the mask image includes: Calculate the Euclidean distance between the non-boundary pixels of the region of interest and each boundary pixel of the region of interest in the mask, and select the shortest distance as D(x,y); Based on the shortest distance D(x,y) from each pixel to the region of interest, a distance map of the same size as the mask image is formed; For each defined distance d, all pixels in the distance map with a value equal to d form an equidistant line; Based on the equidistant lines, a distance set containing several distance thresholds is established, and the dense equidistant lines are set based on the distance set; wherein, each distance threshold in the distance set is generated by a random function, and each distance threshold satisfies a preset condition.

[0036] Furthermore, based on the image degradation simulation method, the distant-focus source image and the near-focus source image are processed, including: Use a mean filter to define the boundaries within the mask; Based on the defined boundaries within the mask, the center pixel is classified into non-boundary pixels and boundary pixels; Degradation operations are performed on the far-focus source image and the near-focus source image; Based on the classified non-boundary pixels and boundary pixels, as well as the degraded far-focus source image and near-focus source image, a multifocal source image is synthesized, thus generating the multifocal image fusion dataset.

[0037] Specifically, the dataset generation method in this embodiment is as follows: Due to the lack of real multifocal datasets, this embodiment introduces a novel dataset generation method, which is a core contribution of the model in this embodiment. Experimental observations show that even with the diffusion model, other data generation methods still cannot eliminate the defocus spread effect (DSE). The data generation method in this embodiment includes four steps: random mask generation, dense equidistant line setting, multifocal image fusion, and simulated DSE image degradation. These steps will be discussed in detail below. For clarity, a high-resolution natural image is defined as... Where H and W represent the dimensions of I. The method flow is as follows: Figure 5 As shown.

[0038] 1) Random Mask Generation: Consistent with existing methods in the field, this embodiment utilizes randomly generated masks to guide the synthesis of multifocal images. These masks contain the random location, area, and shape of the region of interest (ROI), as described in Equation (3).

[0039] (3) Where M represents the generated binary mask (black pixels represent the background, and white pixels represent the region of interest); φ represents the mask generation function; I0 is a zero matrix of the same size as I; ρ1, ρ2 and ρ3 are random parameters generated according to formula (4), representing the area, shape and position of the region of interest, respectively.

[0040] (4) Here, R1 represents the function used to generate random parameters.

[0041] The advantage of randomly generated masks lies in their ability to generate a large number of multifocal source images on a limited dataset, circumventing the limitations of manually created masks. Although these random masks lack consideration for image content and semantic information, this does not prevent the network from generating high-quality fused images, since image sharpness is the key feature of the task.

[0042] 2) Dense isoline setting: According to Prior 1 to Prior 3, it can be clearly seen that in actual imaging scenarios, defocus blur is highly sensitive to distance. Therefore, this embodiment proposes the concept of mask isolines. That is, the boundary of the region of interest is regarded as the depth-of-field boundary of the lens, pixels equidistant from the boundary form isolines, and different isolines are set at different distances.

[0043] Specifically, assume that the set of boundary pixels of the region of interest is expressed as ; this embodiment calculates the Euclidean distance between the non-boundary pixel (x,y) and each boundary pixel Si in the mask, and selects the shortest distance as D(x,y), as shown in Equation (5).

[0044] (5) where (xsi, ysi) represents the coordinate of pixel Si, and ||·|| represents the Euclidean distance in spatial coordinates. Therefore, each pixel has its shortest distance to the ROI, forming a distance map P with the same size as M. For each determined distance D, all pixels in P with a value equal to D form an isoline. To set dense isolines, this embodiment establishes a distance set containing k distance thresholds . Each threshold in D is generated by a random function R2, satisfying D1<D2<...<Dk, as shown in Equation (6): (6) where Max(P) and Min(P) respectively represent the maximum value and the minimum value in P.

[0045] Through these steps, this embodiment obtains a mask Md with k isolines. As shown in Figure 5 , each isoline perfectly reflects the shape of the ROI region, similar to the inward or outward diffusion of the ROI shape. In actual shooting scenarios, the boundary of the DoF is often determined by the shape of the focused object, and it is obvious that this phenomenon of ROI shape diffusion is consistent with reality.

[0046] 3) Multi-focus image generation: The Md with isolines is used to guide the generation of multi-focus source images. Prior 2 to Prior 3 clearly point out the difference between near focus and far focus: near focus is more consistent with disc-shaped blur, while far focus is similar to Gaussian blur. Therefore, this embodiment adopts the following steps to simulate the far-focus source image IA and the near-focus source image IB.

[0047] For IA, this embodiment uses disc-shaped blur to simulate far defocus, as shown in Equation (7).

[0048] (7) where Iω denotes disc blur; Iω(x,y) denotes the pixels covered by the window ω centered at (x,y) in image I; η denotes the luminance parameter; r denotes the window radius, which is calculated by Equation (8).

[0049] (8) where r1, r2, ..., rk are integers randomly selected from the interval (0,50] and satisfy r1<r2<...<rk. The random generation from r1 to rk effectively simulates defocus blur at different distances. It is clearly apparent from Equation (8) that the farther the distance is from the DoF boundary, the larger r becomes, the higher the blur degree, and the more pronounced the diffusion effect.

[0050] For IB, the present embodiment uses Gaussian blur to simulate near defocus, while introducing lens and front-end noises, as shown in Equation (9).

[0051] (9) where γ2 denotes a Gaussian filter; r is the window radius, which is also calculated by Equation (8); σ denotes the standard deviation of the Gaussian filter, which is determined by Equation (10).

[0052] (10) where σ1, ..., σk are numbers randomly selected from the interval (0,100] and satisfy σ1<σ2<...<σk. In the case of near-focus blur, the farther the target distance from the DoF boundary, the closer the target is to the lens, resulting in increased blur.

[0053] Simulating DSE image degradation: Considering that DSE is a mixture of various noises and blurs, the present embodiment designs the following image degradation process for simulation. First, a mean filter is used to define the boundaries within the mask, as shown in Equation (11) (11) where 2r+1 denotes the window size. If the mean value of the window is equal to 0 or 1, the center pixel is classified as a non-boundary pixel; otherwise, it is regarded as a boundary pixel, as shown in Equation (12) (12) where B denotes the boundary map. Inspired by a blind image super-resolution model, the present embodiment performs a degradation operation on the synthesized multi-focus source image IA or IB (collectively referred to as H), as shown in Equation (13) (13) where k h1 denotes Gaussian blur, k h2 denotes disc blur; ↓ s and ↑ uThe downsampling and upsampling operations are used to add additional loss to sharpness; n represents randomly selected salt-and-pepper noise and Gaussian white noise. Finally, the final multifocal source image is synthesized based on the boundary map B. (14) Among them, H f This represents the source image used in the final synthesis.

[0054] Specifically, in this embodiment, the method for training the diffusion model is as follows: Because generative models perform poorly in the MFIF domain, a diffusion model is introduced. The dataset was obtained during the dataset generation process. . x i and y i A composite source image representing partial defocus. z i This represents a "fully focused" image with high resolution. Additionally, D also represents a sample drawn from an unknown conditional distribution p(z|(x,y)).

[0055] A purely noisy image can be viewed as the result of gradually adding Gaussian noise to a signal through a fixed Markov chain. Assuming the algorithm generates a noisy image in T steps, starting with a normal image z0, the algorithm iteratively optimizes the image (z0,...,zT-1,zT) according to the transition distribution q(zt|zt-1), such that zT∼N(0,I), where t∈{0,1,...,T}, N represents the Gaussian noise distribution, and I is the same identity matrix as the initial state z0. The task of MFIF is to learn an inverse Markov chain conditioned on (x,y), which can recover the noisy image to z0. Specifically, the model needs to iteratively converge from zT∼N(0,I) to z0∼p(z|(x,y)) based on its learned pθ(zt-1|zt,(x,y)).

[0056] The method to obtain pθ(zt-1|zt,(x,y)) is to train a noise estimation network, as shown in Algorithm 1. The noise estimation network uses U-net and has the same number of layers as DDPM. The process of recovering from the noisy image is shown in Algorithm 2. The derivation of the specific formula is similar to that of DDPM.

[0057] Specifically, the image fidelity network proposed in this embodiment is: The goal of image fidelity networks is to reduce color distortion and artifacts caused by inherent defects in generative models. Inspired by Jiwon Kim, this embodiment employs a lightweight convolutional network (CNN). The CNN consists of 20 convolutional layers. Except for the first and last layers, each layer uses 64 filters of size 3×3×64 for the input image. The first and last layers consist of a single 3×3×64 filter for image reconstruction. The loss function is mean squared error. This embodiment uses a set of data as an example to illustrate its working principle. G represents the trained diffusion generative model. F, IA, and IB represent the image fidelity network and two images to be fused, respectively.

[0058] The input to F is three images, including a pair of source images and their output image generated by a post-trained diffusion model. Therefore, this embodiment uses G, IA, and IB to obtain the initial fused image Iinitial, i.e., (15) During training F, this embodiment first generates the dataset for this stage, and then inputs the source images into the diffusion model to obtain the final dataset for training.

[0059] Based on the superiority of residual learning proven by Jiwon Kim, F does not directly output the final fused image Ifinal, but instead outputs the residual between Ifinal and Iinitial. The final fused image is shown in Equation (16): (16) The following examples from related work illustrate the advantages and disadvantages of the method in this embodiment: 1. Deep learning-based multifocal image fusion; Mainstream deep learning-based multifocal image fusion (MFIF) algorithms typically employ cascading and generative strategies to generate fused images. The primary goal of cascading-based fusion algorithms is to create high-quality binary decision maps, which play a crucial role in reflecting focus region detection and directly determine the quality of the fused image. Liu et al. proposed the first CNN-based decision map algorithm, which uses a CNN to learn the mapping from the original image to the decision map to obtain a high-quality decision map, providing a prototype for many subsequent algorithms. Tang et al. proposed a pixel-level convolutional neural network (p-CNN), a method for learning focus measurements to identify focus and non-focus pixels in the source image. Li et al. proposed DRPL, which directly converts the entire image into a binary mask, thus eliminating the need for patching operations. This method effectively solves the problem of estimating blur levels between focus and non-focus regions. To mitigate the negative impact of scarce datasets on model performance, this embodiment proposes a self-supervised model based on a novel residual gradient prior, building upon previous work. This model achieves good results without MFIF training data. Furthermore, this embodiment proposes VSPFuse based on a visual saliency prior, using salient object detection as a pre-training task. Recently, this embodiment proposes an edge-aware layer based on image manifold theory and domain transformation, which can improve the performance of CNN architectures. However, since decision maps cannot recover details of common defocused regions, none of the above methods can effectively solve DSE (Discrete Edge Sensing). Since the method in this embodiment is essentially an end-to-end generative method, this drawback can be avoided to a limited extent.

[0060] For models employing generative strategies, their goal is to generate fused images end-to-end from the source input. These models can bypass the need for decision maps and are consistent with other image fusion tasks. Zhang et al. introduced a CNN-based general image fusion framework, IFCNN, for multi-image fusion tasks. Zhao et al. proposed a CNN-based multi-layer fusion architecture and used image super-resolution to improve image details. Xu et al. proposed U2Fusion, which employs an unsupervised learning strategy and designs a well-designed loss function and an efficient backbone network, achieving significant performance improvements in unified image fusion and laying the foundation for this field. Huang et al. proposed a GAN-based end-to-end fusion method that recovers image details through generative adversarial learning. Ma et al. proposed Swin-Fusion using Swin-Transformer as the backbone network. Leveraging the powerful feature representation capabilities of Transformer, Swin-Fusion achieved state-of-the-art performance in multiple image fusion tasks. These models can only slightly alleviate DSE, but inevitably lead to color distortion and obvious artifacts. In contrast, this embodiment employs a two-stage training strategy, where a diffusion model effectively suppresses DSE, and a lightweight image fidelity network alleviates the color distortion problem.

[0061] 2. Diffusion Model Recent research has shown that diffusion models have the ability to generate high-quality images, videos, molecules, and other types of data. The basic idea of ​​diffusion models is to systematically perturb the distribution of data during forward diffusion and then recover the data distribution by learning the reverse diffusion process.

[0062] Diffusion models offer significant advantages over CNNs, Transformers, and GANs. First, they employ a unique generation strategy that refines the probability distribution of data iteratively rather than directly generating samples, making them more flexible in handling diverse data types. Second, diffusion models excel in image generation, particularly in tasks involving high-quality, high-resolution images. In contrast, CNNs and Transformers typically require larger model sizes to achieve similar results, increasing computational costs. Finally, diffusion models perform well in handling sparse data during training because their generation process does not rely on dense point-to-point information propagation, whereas CNNs and Transformers generally have higher data requirements.

[0063] In the context of image fusion, diffusion models offer a valuable application. They provide a refreshing alternative for image fusion through a probabilistic framework. By iteratively diffusing information across the input image, they effectively integrate content from multiple sources. Furthermore, their ability to handle complex multimodal data distributions makes them particularly suitable for fusing images with different characteristics or features. For example, Zhao et al. treated the fusion task as a conditional generation problem and proposed a multimodal image fusion method (DDFM) based on a denoising diffusion model. This method leverages robust generative priors, effectively mitigating the training instability and lack of interpretability common in GAN-based generative techniques. Li et al. proposed FusionDiff based on DDPM for the MFIF task. FusionDiff is robust to noise and can produce excellent fusion results from noisy source images. However, this method still falls short in eliminating DSE and addressing the problem of scarce datasets.

[0064] The model in this embodiment is similar to FusionDiff, also utilizing the classic DDPM to generate fused images. However, it is worth noting that the method in this embodiment effectively mitigates DSE and is able to generate more realistic multifocal image datasets.

[0065] 3. MFIF Dataset Generation Due to the lack of real MFIF training sets and benchmark data, several methods for generating training datasets have been proposed. Guo et al. synthesized a large-scale multifocal image dataset and used a normalized point spread function for the first time to simulate defocus; however, it ignored the problem of varying diffusion levels among different pixels within the same image due to brightness differences. Li et al. used 12 hand-drawn masks combined with Gaussian blur to generate synthetic data for training, but this made the network prone to overfitting on these masks. Ma et al. first used an α-mask boundary defocus model and Gaussian blur to generate training data with DSE effects, but the Gaussian blur made the generated data lack realism. Nian et al. generated training data based on light field data, utilizing the ability of a light field camera to record the cumulative intensity and depth information of light to refocus. However, due to the limitations of the light field camera itself, the actual generated effect had a uniform and slight degree of blur.

[0066] Unlike the methods described above, the dataset generation method proposed in this embodiment fully considers imaging priors. This method introduces the concept of equidistant lines for the first time to guide mask-based image filtering. Furthermore, it combines the advantages of Gaussian blur and disk blur to realistically simulate defocus blur and DSE.

[0067] To effectively address the defocus diffusion effect, this embodiment proposes solutions for the three aspects mentioned above. Regarding the dataset, to simulate more realistic defocus blur, this embodiment summarizes prior knowledge based on imaging principles, including concepts such as point spread function (PSF), defocus circle, focal plane, far-field defocus, and near-field defocus. Supported by these prior knowledge points, this embodiment introduces a novel method to generate a multifocal image dataset. In this method, the concept of equidistant lines in a mask is introduced, consisting of pixels with equal shortest distances to the region of interest (ROI). This embodiment densely arranges equidistant lines within the mask, ensuring that each pixel in the mask is assigned a corresponding distance range. For pixels closer to the ROI, this embodiment assumes that the objects associated with them in the imaging scene are closer to the depth of focus; therefore, Gaussian blur is used to simulate noise introduced by factors such as lens and circuitry. For pixels farther from the ROI, this embodiment employs a disk blur more similar to real defocus blur, effectively simulating DSE caused by light spots. This method significantly reduces the gap between the synthetic training dataset and real multifocal images.

[0068] In terms of architecture, this embodiment employs the diffusion model, which has recently demonstrated excellent performance in the field of image generation. There are two main reasons for using a diffusion model for the MFIF task: its strong adaptability to synthesizing information from zoom regions and its tolerance to noise. On the one hand, diffusion models excel at adaptively synthesizing information from different image regions. Therefore, they can also merge regions at different focal levels, ensuring accurate integration of details obtained from different focal planes. On the other hand, diffusion models inherently possess denoising capabilities, improving the quality of the fused image by reducing noise artifacts; this noise reduction greatly benefits the elimination of DSE (Discrete Sequence of Effects). To adapt to the MFIF task, this embodiment uses two synthetic source images and one noisy image as conditions for the model, with the corresponding high-resolution natural image as the ground truth (GT). Experiments demonstrate that even when the original diffusion model DDPM is combined with the synthetic dataset, it achieves optimal performance.

[0069] Regarding the fusion strategy, this embodiment introduces a two-stage "generation and correction" strategy for the first time, which includes a diffusion model and a lightweight image fidelity network. For images generated by the diffusion model, the image fidelity network effectively eliminates color distortion and artifacts, ensuring high fidelity between the fused image and the source image. Experiments demonstrate that the image fidelity network is not only effective for images generated by the diffusion model, but also improves the quality of fused images generated by other models such as Transformer and CNN.

[0070] The contributions of this invention are summarized as follows: This paper addresses a long-neglected issue in multifocal image fusion (MFIF)—the unrealistic nature of the dataset—and proposes a novel method for generating multifocal image datasets. This method not only introduces the concept of isometric lines in the mask to guide filter blur but also combines the advantages of disk blur and Gaussian blur. This approach significantly reduces the gap between the synthetic training dataset and real multifocal images.

[0071] Based on imaging principles, the priors for defocus blur in MFIF are summarized. These priors include concepts such as defocus circle, point spread function (PSF), object distance, near-field defocus, and far-field defocus, which strongly support the rationality of the proposed dataset generation method.

[0072] This paper introduces a two-stage fusion strategy of "generation and correction" for the first time. In the generation stage, a diffusion model is used as the network architecture to effectively eliminate the defocus diffusion effect. In the correction stage, a lightweight network is proposed to alleviate the color distortion and artifact problems inherent in the generative model. Experiments demonstrate that this strategy can improve the performance of various generative models.

[0073] Extensive comparative and ablation experiments demonstrated the performance of the proposed model. Experimental results on seven test sets show that the model in this embodiment can effectively solve the defocusing diffusion effect and achieves optimal performance.

[0074] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A multifocal image fusion method for eliminating defocus diffusion effects, characterized in that, include: Based on the defocus blur prior of optical imaging principle, a multifocal image fusion dataset is generated. A diffusion model is trained using the multifocal image fusion dataset. The source image is input into the trained diffusion model to obtain the output fused image. The source image and the fused image output by the diffusion model are input into the image fidelity network to obtain the fused multifocal image; Based on the prior knowledge of defocus blur in optical imaging principles, a multifocal image fusion dataset is generated, including: A randomly generated mask is used to guide the synthesis of multifocal images to obtain the mask image; For the region of interest in the mask image, dense equidistant lines are set to obtain a mask image Md with several equidistant lines; The mask Md image is used to guide the generation of multifocal source images, simulating far-focus and near-focus source images; Based on the image degradation simulation method, the far-focus source image and the near-focus source image are processed to obtain the multifocal image fusion dataset; The mask image is: in, M This represents the generated binary mask. This represents the mask generation function. I 0 The matrix is ​​a set of all zeros of the same size as I, where I represents the input source image. ρ 1. ρ 2 and ρ 3 represents the area, shape, and location of the region of interest, respectively; Setting dense equidistant lines for the region of interest in the masked image includes: Calculate the Euclidean distance between the non-boundary pixels of the region of interest and each boundary pixel of the region of interest in the mask, and select the shortest distance as D(x,y); where D(x,y) refers to the shortest distance from the non-boundary pixel M(x, y) in the mask image to the boundary pixel. Based on the shortest distance D(x,y) from each pixel to the region of interest, a distance map of the same size as the mask image is formed; in the distance map, all pixels with a value equal to D form an equidistant line; Based on the equidistant lines, a distance set containing several distance thresholds is established, and the dense equidistant lines are set based on the distance set; wherein, each distance threshold in the distance set is generated by a random function, and each distance threshold satisfies a preset condition.

2. The multifocal image fusion method for eliminating defocus diffusion effect according to claim 1, characterized in that, The method for calculating the Euclidean distance is as follows: in,( x si ,y si ) represents each boundary pixel Si The coordinates, ||·|| represent the Euclidean distance in spatial coordinates, ( x,y () represents the two-dimensional coordinates of an image pixel. This represents the value of the pixel at coordinates (x, y) in the distance image P, and n represents the total number of boundary pixels in the mask image; The preset conditions that each distance threshold must meet are: in, D For distance set, R 2 is a random function. Max(P) and Min(P) These represent the maximum and minimum values ​​in the distance graph, respectively. k It is a number.

3. The multifocal image fusion method for eliminating defocus diffusion effect according to claim 1, characterized in that, The method for simulating the telephoto source image is as follows: in, I ω Indicates the disk is blurred. ,I ω ( x , y ) represents the image I with ( x , y The number of pixels covered by the window ω centered on the pixel. η Indicates brightness parameter, r Indicates the window radius. Represents source image I A The value of the pixel with spatial coordinates (x, y) in the mid-space. This refers to a disk filter. This represents the value of the pixel with spatial coordinates (x, y) in the masked image M; The method for simulating near-focus source images is as follows: Where γ² represents the Gaussian filter, and σ represents the standard deviation of the Gaussian filter. This represents the value of the pixel with spatial coordinates (x, y) in the source image B that is synthesized.

4. The multifocal image fusion method for eliminating defocus diffusion effect according to claim 1, characterized in that, Based on the image degradation simulation method, the far-focus source image and the near-focus source image are processed, including: Use a mean filter to define the boundaries within the mask; Based on the defined boundaries within the mask, the center pixel is classified into non-boundary pixels and boundary pixels; Degradation operations are performed on the far-focus source image and the near-focus source image; Based on the classified non-boundary pixels and boundary pixels, as well as the degraded far-focus source image and near-focus source image, a multifocal source image is synthesized, thus generating the multifocal image fusion dataset.

5. The multifocal image fusion method for eliminating defocus diffusion effect according to claim 4, characterized in that, The method for classifying the center pixel into non-boundary pixels and boundary pixels is as follows: Where B represents the boundary mapping, Indicates the boundary within the defined mask; The method for performing degradation operations on the far-focus source image and the near-focus source image is as follows: in, k h1 Indicates Gaussian blur, k h2 Indicates a disc-shaped blur, ↓ s and ↑ u This indicates downsampling and upsampling operations, where n represents randomly selected salt-and-pepper noise and Gaussian white noise. This represents the result image after image degradation processing. Represents source image I A Or I B It is uniformly represented as H; The multifocal source image is: in, This represents the initial multifocal source image used in the final synthesis.

6. The multifocal image fusion method for eliminating defocus diffusion effect according to claim 1, characterized in that, The image fidelity network employs a lightweight convolutional network; the lightweight convolutional network is a CNN comprising several convolutional layers; except for the first and last layers, each layer includes several filters of a preset size for the input image; the first and last layers consist of a single filter of a preset size for image reconstruction, and the loss function is mean squared error. The fused multifocal image is: in, This represents the fused multifocal image. This represents the initial fused image output by the diffusion model. Represents an image fidelity network. Represents source image A, This represents source image B.

Citation Information

Patent Citations

  • Image fusion processing system and method of display

    CN115690149A

  • Multi-focus image fusion method based on attention mechanism

    CN117475283A