Image fusion method based on diffusion posterior sampling

Abstract

The application discloses an image fusion method based on diffusion posterior sampling, and relates to the field of image processing. The method first initializes variables of a diffusion posterior sampling process, including setting an initial diffusion sampling time step and randomly taking an initial diffusion noise image; on the basis of the initialized variables, an estimation of a final fusion image at a current time step is calculated; a current time step noise image is preliminarily updated based on the estimation of the final fusion image at the current time step, and a preliminary update result of a next time step noise image is obtained; the preliminary update result of the next time step noise image is input with source images to obtain a corrected noise image; a time step is updated, and the step of calculating the estimation of the final fusion image at the current time step is iteratively calculated until the current time step is zero, and the corrected noise image is output as a final fusion image. The method can effectively avoid loss of effective information caused by generative adversarial training, and enhance information completeness of the fusion image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to an image fusion method based on diffusion posterior sampling. BACKGROUND

[0002] The purpose of image fusion is to integrate image data of multiple modes to capture target features more comprehensively. The image fusion method currently used in the field of image processing is an image fusion method based on a generative adversarial network. This kind of method models the image fusion problem as an adversarial game problem between a generator and a discriminator, and forces the discriminator to generate a fusion image with a source image distribution. The disadvantage of the generative adversarial network algorithm is that it is not easy to maintain the balance between the generator and the discriminator during training, and it is easy to generate a blurred fusion image and lose gradient information. SUMMARY

[0003] In view of the problems pointed out in the background, the purpose of the present application is to provide an image fusion method based on diffusion posterior sampling to avoid effective information loss caused by generative adversarial training and enhance the information completeness of the fusion image.

[0004] To achieve the above-mentioned purpose, the present application provides the following solutions.

[0005] In a first aspect, the present application provides an image fusion method based on diffusion posterior sampling, comprising:

[0006] Initializing variables of the diffusion posterior sampling process, including setting an initial diffusion sampling time step and randomly taking an initial diffusion noise image;

[0007] On the basis of initializing the variables, calculating an estimation of the final fusion image at the current time step;

[0008] Based on the estimation of the final fusion image at the current time step, preliminarily updating the noise image at the current time step to obtain a preliminary update result of the noise image at the next time step;

[0009] Inputting a source image to correct the preliminary update result of the noise image at the next time step to obtain a corrected noise image; the source image includes a visible light image and a SAR image;

[0010] Updating the time step, iteratively calculating the step of calculating the estimation of the final fusion image at the current time step until the current time step is zero, and outputting the corrected noise image as the final fusion image.

[0011] Optionally, the initializing variables of the diffusion posterior sampling process specifically includes:

[0012] Setting an initial diffusion sampling time step t=T, and randomly taking an initial diffusion noise image x0 that obeys a normal distribution an initial diffusion noise image f T ; where I n is an identity matrix of the same size as the noise image;

[0013] a pre-trained noise approximation network s θ (f t , t) is denoted as; where t is the current time step of diffusion sampling; f t represents the noise image at the t-th step, also called the current time step noise image.

[0014] Optionally, the pre-trained noise approximation network s θ (f t , t) is pre-trained by a U-Net network, and the input is f t and t, and the output is f t the noise added compared to the original clean image f0.

[0015] Optionally, the calculating the current time step estimate of the final fused image specifically comprises:

[0016] the current time step estimate of the final fused image is calculated by the formula where α t = 1-β t , β t is the single-step noise variance of the t-th step in the forward noise adding process of the diffusion model.

[0017] Optionally, the preliminary updating the current time step noise image based on the current time step estimate of the final fused image to obtain the preliminary update result of the next time step noise image specifically comprises:

[0018] the current time step noise image f t is preliminarily updated based on the current time step estimate of the final fused image by the formula to obtain the preliminary update result f′ t-1 of the next time step noise image; where α t-1 = 1-β t-1 , β t-1 is the single-step noise variance of the t-1-th step in the forward noise adding process of the diffusion model; σ t is the noise variance added to the current time step noise image compared to the original clean image in the forward noise adding process of the diffusion model; and z is a Gaussian noise image. Optionally, the Gaussian noise image z is randomly taken from a noise image obeying a Gaussian distribution.

[0019] ​​​

[0020] Optionally, the input source image corrects a preliminary update result of the next time step noise image to obtain a corrected noise image, and the correction specifically includes:

[0021] The input visible light image o and the SAR image s as source images, and the formula The preliminary update result f' of the next time step noise image is corrected t-1 to obtain the corrected noise image f t-1 ; wherein p, g, h1 and h2 are weight coefficients; sgn represents a sign function; ▽ is a gradient operator; and div is a divergence operator.

[0022] Optionally, the update time step returns the step of calculating the estimation of the current time step on the final fusion image for iterative calculation until the current time step is zero, and outputs the corrected noise image as the final fusion image, and the iteration specifically includes:

[0023] The update time step t is returned to t-1, and the step of calculating the estimation of the current time step on the final fusion image is iteratively calculated, starting from the initialization t=T and iterating until t=0, and the corrected noise image f0 is output as the final fusion image.

[0024] In a second aspect, the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor executes the computer program to realize the image fusion method based on diffusion posterior sampling.

[0025] In a third aspect, the present application provides a computer readable storage medium having a computer program stored thereon, and the computer program is executed by a processor to realize the image fusion method based on diffusion posterior sampling.

[0026] In a fourth aspect, the present application provides a computer program product comprising a computer program, and the computer program is executed by a processor to realize the image fusion method based on diffusion posterior sampling.

[0027] According to the specific embodiments provided by the present application, the following technical effects are disclosed.

[0028] The image fusion method based on diffusion posterior sampling provided by the present application comprehensively combines the complementary information of the visible light image and the SAR image under the diffusion posterior sampling framework to obtain a fusion image containing more complete information, which can effectively avoid the loss of effective information caused by the generation of adversarial training, not only enhances the information completeness of the fusion image, but also improves the feature saliency of the fusion image. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed in the embodiments will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only represent some embodiments of the present application, and can help those skilled in the art to obtain other drawings without any inventive effort.

[0030] Figure 1 A flowchart of an image fusion method based on diffusion posterior sampling according to an embodiment of the present application is shown in the figure.

[0031] Figure 2 A visible light image according to an embodiment of the present application is shown in the figure.

[0032] Figure 3 A SAR image according to an embodiment of the present application is shown in the figure.

[0033] Figure 4 An image fusion result obtained by the method according to an embodiment of the present application is shown in the figure.

[0034] Figure 5 A comparison of image fusion results of land scene is shown in the figure.

[0035] Figure 6 A comparison of image fusion results of ocean scene is shown in the figure. DETAILED DESCRIPTION

[0036] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any inventive effort fall within the scope of the present application.

[0037] The purpose of the present application is to provide an image fusion method based on diffusion posterior sampling to avoid effective information loss caused by generative adversarial training and enhance the information completeness of the fused image.

[0038] In order to make the above-mentioned purposes, features and advantages of the present application more apparent and easy to understand, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0039] In an exemplary embodiment, the present application provides an image fusion method based on diffusion posterior sampling, as shown in the figure, comprising the following steps 1 to 5. Figure 1

[0040] Step 1: Initialize the variables of the diffusion posterior sampling process, including setting the initial diffusion sampling time step and randomly taking the initial diffusion noise image. ​

[0041] This application proposes an image fusion method based on diffusion a posteriori sampling, which uses visible light images and SAR (Synthetic Aperture Radar) images to guide the diffusion a posteriori sampling process to generate a fused image. In this application, the visible light image is defined as... SAR image is fused image Where H and W are the width and height of the image, respectively. It represents the real number field space of dimension H×W.

[0042] The diffusion posterior sampling process first requires initializing variables, specifically including: setting the initial diffusion sampling time step t = T, and randomly selecting the initial diffusion noise image f. T Follows a normal distribution Among them I n It is an identity matrix of the same size as the noisy image. The pre-trained noise approximation network s... θ (f t ,t) use This is represented as follows: where t is the current time step (or simply the current step) of the diffusion sampling, which is a variable, and T is the initial value of t; that is, the initialization of f. t =f T f t The noise image representing the diffused object at step t is also called the noise image at the current time step (current step).

[0043] Among them, the pre-trained noise approximation network s θ (f t (t) is obtained by pre-training the U-Net network, and its input is the noisy image f at the current time step. t Given the current time step t, the output is the predicted noise image f at the current time step. t The noise added to f0 is compared to the original clear image.

[0044] Step 2: Based on the initialized variables, calculate the estimate of the final fused image at the current time step.

[0045] The estimate of the final fused image at the current time step is calculated using formula (1).

[0046]

[0047] Where f t The current step's noisy image; For the pre-trained noise approximation network; intermediate variables in the computation process α t =1-β t ,β tThe single-step noise variance of the tth step in the forward noise adding process of the diffusion model.

[0048] Step 3: Based on the estimation of the final fused image at the current time step, the current time step noise image is preliminarily updated to obtain a preliminary update result of the next time step noise image.

[0049] Taking a Gaussian noise image z, combining the estimation of the final fused image at the current time step The current step noise image f is updated together t The update process expression is:

[0050]

[0051] Wherein f' t-1 represents the preliminary update result of the next time step (t-1th step) noise image. The intermediate variable of the calculation process is α t-1 = 1-β t-1 , β t-1 is the single-step noise variance of the t-1th step in the forward noise adding process of the diffusion model. σ t is the noise variance added to the current time step noise image f t compared with the original clear image f0. The taken Gaussian noise image z refers to a noise image obeying Gaussian distribution as z.

[0052] Step 4: Input the source image to correct the preliminary update result of the next time step noise image to obtain a corrected noise image; the source image includes a visible light image and a SAR image.

[0053] The visible light image and the SAR image are taken as the source image, and the preliminary update result f' t-1 of the next time step noise image is corrected by the following formula:

[0054]

[0055] Wherein f t-1 is the corrected noise image. ρ, γ, η1 and η2 are all weight coefficients; wherein the weight coefficients ρ and η1 are respectively used to balance the features from the SAR image and the texture features from the SAR image in the fused image; the weight coefficients γ and η2 are respectively used to balance the features from the visible light image and the texture features from the visible light image in the fused image. sgn represents the sign function; div is the divergence operator. The gradient operator is ▽. The total variation of a certain image ω is defined as Wherein D 1 , D 2respectively, are 1-dimensional difference matrices of the image ω in the x and y directions; I represents the identity matrix.

[0056] Step 5: update the time step, return to the step of calculating the estimate of the current time step on the final fused image, and iterate the calculation until the current time step is zero, output the corrected noise image as the final fused image.

[0057] Update the time step t <- t - 1, return to step 2 and iterate the calculation using formulas (1) to (3), starting from the initialized t = T and iterating until t = 0, output the corrected noise image f0 as the final fused image, that is, the original clear image f0 recovered as described above.

[0058] In recent years, denoising diffusion probabilistic model (DDPM) and its continuous form have rapidly developed in the field of image processing. They have rapidly achieved remarkable results in multiple fields with their multifunctional pre-training framework, powerful deep generative prior, interpretable generation process and stable training process. These features effectively address the challenges faced by traditional deep learning image fusion methods and provide new ideas for image fusion. Denoising diffusion probabilistic model has shown good performance in a series of inverse problems such as image denoising, image restoration and image super-resolution. However, unlike traditional inverse problems, the forward process model of the image fusion task is not clear, which makes it difficult for diffusion models such as DDPM to be applied to image fusion. To this end, the present application uses the traditional variational model of image fusion to construct the posterior distribution of the fused image, thereby improving the diffusion posterior sampling scheme for image fusion. Compared with the disadvantage of the generative adversarial network algorithm that easily loses gradient information, leading to the generation of blurred fused images, the diffusion model takes variational lower bound maximization as the training target, embeds time step encoding, has a stable training process, and can avoid the loss of effective information caused by generative adversarial training. Its interpretable generation process is superior to the black box characteristics of traditional deep information methods, and the deep generative prior provided by the diffusion model adapts to diversified tasks.

[0059] The diffusion modeling problem based on image fusion is the key to solving the existing image fusion method problem with a diffusion model. The present application uses the sparse prior of the fused image (reflected in These two items) and the bounded variation prior term (for example ) An optimization model combining a sparse approximation term and a total variation regularization term is designed, and then the optimization problem is converted into a maximum likelihood estimation problem of a fused image Laplace distribution; then a guide model for diffusion sampling of the fused image is established under the diffusion posterior sampling framework, as shown in formula (3). Accordingly, an image fusion algorithm based on diffusion posterior sampling is designed. Compared with existing image fusion methods, the present application not only enhances the information completeness of the fused image, but also improves the feature saliency of the fused image.

[0060] To verify the effect of the present application, a specific embodiment is provided below for illustration. In this embodiment, the fused visible light image is as shown in Figure 2 , and the SAR image is as shown in Figure 3 , both of which contain land scene (Land Scene) images and sea scene (Sea Scene) images, are single-channel images of 256x256 and the brightness is adjusted to be consistent. Among them, the visible light image of the land scene is as shown in Figure 2 (a) part, and the SAR image of the land scene is as shown in Figure 3 (a) part; the visible light image of the sea scene is as shown in Figure 2 (b) part, and the SAR image of the sea scene is as shown in Figure 3 (b) part.

[0061] The image fusion method based on diffusion posterior sampling proposed in the present application focuses on establishing an image fusion diffusion posterior sampling diffusion optimization model at each time step in the inverse process of the diffusion model, and the specific steps are as described in the previous steps 1 to 5. In this embodiment, the weight parameter in formula (3) is set as: The visible light image as shown in Figure 2 and the SAR image as shown in Figure 3 are fused by using the above steps, and the fusion result is as shown in Figure 4 , wherein the fused image of the land scene is as shown in Figure 4 (a) part, and the fused image of the sea scene is as shown in Figure 4 (b) part.

[0062] To verify the image fusion effect, the fusion images of the present application method are compared with the fusion images of the CF-Net and DDcGAN methods. Among them, the land scene image fusion result comparison is as shown in Figure 5 , and the sea scene image fusion result comparison is as shown in Figure 6 . Figure 5 The (a), (b) and (c) parts of Figure 6 respectively show the land scene fused images obtained by using the CF-Net, DDcGAN and the present application method.The (a), (b) and (c) parts in FIG. 1 respectively show the ocean scene fusion images obtained by using CF-Net, DDcGAN and the method of the present application. Figures 4 to 6 The boxed part in FIG. 1 indicates the complementary information worthy of attention.

[0063] In order to quantitatively evaluate the performance of the above method, the present application selects five indicators, i.e. peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), cross entropy (CE), mutual information (MI) and VIFF (visual fidelity), and the greater the values of these indicators, the higher the image quality. The comparison results are shown in Table 1.

[0064] Table 1: Quantitative evaluation results of image fusion indicators

[0065]

[0066]

[0067] The results shown in Table 1 show that the method proposed in the present application is superior to the comparison methods in terms of image quality improvement and image fidelity.

[0068] The image fusion method based on diffusion posterior sampling provided by the present application mainly aims to integrate the complementary information of two source images to obtain an image containing more complete information. For this purpose, the present application designs an optimization model combining sparse approximation and total variation regularization and converts it into a maximum likelihood estimation problem of Laplace distribution of the fusion image; and then establishes an image fusion model guided by the fusion image of the visible light image and the SAR image under the diffusion posterior sampling framework, which not only can enhance the information completeness of the fusion image and improve the feature saliency of the fusion image, but also has better visual effect, peak signal-to-noise ratio, structural similarity and other quantitative indicator values of the fusion image than the existing methods, and has a more outstanding image fusion effect.

[0069] In an exemplary embodiment, the present application also provides a computer device which can be a server or a terminal. The computer device comprises a processor, a memory, an input / output interface and a communication interface. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for running the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is configured to exchange information between the processor and external devices. The communication interface of the computer device is configured to communicate with external terminals through a network connection. The computer program, when executed by the processor, implements the image fusion method based on diffusion posterior sampling.

[0070] In an exemplary embodiment, the present application also provides a computer readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image fusion method based on diffusion posterior sampling.

[0071] In an exemplary embodiment, the present application also provides a computer program product comprising a computer program, which, when executed by a processor, implements the image fusion method based on diffusion posterior sampling.

[0072] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by computer program instructions related to hardware, and the computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiment methods. Among them, any reference to memory or other medium in each embodiment provided by the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (Read Only Memory, ROM), magnetic tape, floppy disk, flash memory, optical storage, high-density embedded non-volatile memory, resistive memory (ReRAM), magnetoresistive random access memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric memory (Ferroelectric Random Access Memory, FRAM), phase change memory (Phase Change Memory, PCM), graphene memory, etc. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, etc. As an illustration but not limitation, RAM can be in various forms, such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc.

[0073] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant regulations.

[0074] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist contradictory, they should be considered as the scope of the present application.

[0075] The principles and implementation modes of the present application are described by specific examples in this paper, and the above embodiment description is only used to help understand the method and its core idea of the present application; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. An image fusion method based on diffusion posterior sampling, characterized in that, include: Initialize the variables for the diffusion posterior sampling process, including setting the initial diffusion sampling time step and randomly selecting the initial diffusion noise image; The variables for initializing the diffusion post-amplitude sampling process specifically include: Set the initial diffusion sampling time step to t = T, and randomly select samples that follow a normal distribution. The initial diffuse noise image f T ; where I n It is an identity matrix of the same size as the noisy image; The pre-trained noise approximation network s θ (f t ,t) use Indicates; where t is the current time step of the diffusion sampling; f t The noise image at step t is also called the noise image at the current time step. Based on the initialized variables, calculate the estimate of the final fused image at the current time step; Based on the estimation of the final fused image at the current time step, the noise image at the current time step is initially updated to obtain the initial update result of the noise image at the next time step. The initial update result of the noise image at the next time step is corrected by using the input source image to obtain the corrected noise image; the source image includes a visible light image and a SAR image; The input source image is used to correct the preliminary update result of the noise image at the next time step, resulting in a corrected noise image, specifically including: Input visible light image o and SAR image s as source images, and use the formula Correct the initial update result f′ of the noisy image at the next time step. t-1 The corrected noisy image f is obtained. t-1 Where ρ, γ, η1, and η2 are all weighting coefficients; sgn represents the sign function; It is the gradient operator; div is the divergence operator; This is an estimate of the final fused image at the current time step; Update the time step, return to the step of calculating the estimate of the final fused image at the current time step and iterate until the current time step is zero, and output the corrected noise image as the final fused image.

2. The image fusion method based on diffusion posterior sampling according to claim 1, characterized in that, The pre-trained noise approximation network s θ (f t ,t) is obtained from the pre-training of the U-Net network, with input f t And t, the output is f t The noise added to f0 is compared to the original clear image.

3. The image fusion method based on diffusion posterior sampling according to claim 1, characterized in that, The calculation of the estimate of the final fused image at the current time step specifically includes: Using formula Calculate the estimate of the final fused image at the current time step. in α t =1-β t ,β t Let t be the single-step noise variance at step t during the forward noise addition process of the diffusion model.

4. The image fusion method based on diffusion posterior sampling according to claim 3, characterized in that, The preliminary update of the noise image at the current time step based on the estimation of the final fused image at the current time step, to obtain the preliminary update result of the noise image at the next time step, specifically includes: Estimation of the final fused image based on the current time step Using formula For the noisy image f at the current time step t A preliminary update is performed to obtain the preliminary update result f′ of the noisy image at the next time step. t-1 ;in α t-1 =1-β t-1 ,β t-1 σ represents the single-step noise variance at step t-1 during the forward noise addition process of the diffusion model. t denoted by z, represents the variance of the noise added to the current time step noise image compared to the original clear image during the forward noise addition process of the diffusion model; z represents the Gaussian noise image.

5. The image fusion method based on diffusion posterior sampling according to claim 4, characterized in that, The Gaussian noise image z is randomly selected from noise images that follow a Gaussian distribution.

6. The image fusion method based on diffusion posterior sampling according to claim 5, characterized in that, The update time step iteratively calculates the step of calculating the estimate of the final fused image at the current time step until the current time step is zero, and outputs the corrected noise image as the final fused image, specifically including: Update time step t←t-1, return to the step of calculating the estimate of the final fused image at the current time step and perform iterative calculation, starting from the initial t=T and iterating until t=0, outputting the corrected noise image f0 as the final fused image.

Images