Image fusion method based on diffusion posterior sampling

Abstract

The invention discloses an image fusion method based on diffusion posterior sampling, and relates to the field of image processing. The method comprises the following steps: firstly, initializing variables of a diffusion posterior sampling process, including setting an initial diffusion sampling time step and randomly taking an initial diffusion noise image; on the basis of the initialization variable, calculating the estimation of the current time step on the final fusion image; performing preliminary updating on the noise image of the current time step based on the estimation of the final fusion image of the current time step to obtain a preliminary updating result of the noise image of the next time step; inputting the source image to correct a preliminary updating result of the noise image in the next time step to obtain a corrected noise image; and updating the time step, returning to the step of calculating the estimation of the current time step on the final fusion image to carry out iterative calculation until the current time step is zero, and outputting the corrected noise image as the final fusion image. According to the method, effective information loss caused by generative adversarial training can be effectively avoided, and the information completeness of the fused image is enhanced.

Description

Technical Field

[0001] The present application relates to the technical field of image processing, and in particular to an image fusion method based on diffusion posterior sampling. Background Art

[0002] The purpose of image fusion is to integrate image data of multiple modes and capture the target features more comprehensively. The image fusion method currently used in the field of image processing is the image fusion method based on generative adversarial networks. This type of method models the image fusion problem as an adversarial game problem between the generator and the discriminator, allowing the discriminator to force the generator to generate a fused image with the distribution of the source image. The disadvantage of the generative adversarial network algorithm is that it is not easy to maintain the balance between the generator and the discriminator during training, and it is easy to generate blurred fused images and lose gradient information. Summary of the invention

[0003] In response to the problems pointed out in the background technology, the purpose of this application is to provide an image fusion method based on diffuse posterior sampling to avoid the loss of effective information caused by generative adversarial training and enhance the information completeness of the fused image.

[0004] To achieve the above objectives, this application provides the following solutions.

[0005] In a first aspect, the present application provides an image fusion method based on diffusion posterior sampling, comprising:

[0006] Initialize the variables of the diffusion posterior sampling process, including setting the initial diffusion sampling time step and randomly taking the initial diffusion noise image;

[0007] Based on the initialized variables, the estimation of the final fused image at the current time step is calculated;

[0008] Based on the estimation of the final fused image at the current time step, the noise image at the current time step is preliminarily updated to obtain the preliminary update result of the noise image at the next time step;

[0009] The input source image corrects the preliminary update result of the noise image in the next time step to obtain a corrected noise image; the source image includes a visible light image and a SAR image;

[0010] The time step is updated, and the step of calculating the estimation of the final fused image at the current time step is returned to perform iterative calculation until the current time step is zero, and the corrected noise image is output as the final fused image.

[0011] Optionally, the variables of the initialization diffusion posterior sampling process specifically include:

[0012] Set the initial diffusion sampling time step t = T, and randomly select a sample that follows a normal distribution. The initial diffuse noise image f T ; where I n is the identity matrix of the same size as the noise image;

[0013] Approximate the pre-trained noise to the network s θ (f t ,t) Represents; where t is the current time step of diffusion sampling; f t Represents the noise image at step t, also called the current time step noise image.

[0014] Optionally, the pre-trained noise approximation network s θ (f t ,t) is obtained by U-Net network pre-training, and the input is f t and t, the output is f t Compared to the original clear image f0, the noise is added.

[0015] Optionally, calculating an estimate of the final fused image at the current time step specifically includes:

[0016] Using formula Calculate the estimate of the final fused image at the current time step in α t =1-β t , β t is the single-step noise variance of the tth step in the forward noise addition process of the diffusion model.

[0017] Optionally, the preliminarily updating the noise image at the current time step based on the estimation of the final fused image at the current time step to obtain a preliminary updated result of the noise image at the next time step specifically includes:

[0018] Estimate of the final fused image based on the current time step Using formula For the current time step noise image f t Perform a preliminary update to obtain the preliminary update result f′ of the noise image at the next time step t-1 ;in α t-1 =1-β t-1 , β t-1 is the single-step noise variance of the t-1th step in the forward noise addition process of the diffusion model; σ t is the noise variance added to the noise image at the current time step compared to the original clear image during the forward noising process of the diffusion model; z is the Gaussian noise image.

[0019] Optionally, the Gaussian noise image z is a randomly selected noise image that obeys a Gaussian distribution.

[0020] Optionally, the input source image corrects the preliminary update result of the noise image at the next time step to obtain a corrected noise image, which specifically includes:

[0021] Input the visible light image o and SAR image s as source images, and use the formula Correct the initial update result f′ of the noise image at the next time step t-1 , and get the corrected noise image f t-1 ; where ρ, γ, η1 and η2 are weight coefficients; sgn represents the sign function; ▽ is the gradient operator; div is the divergence operator.

[0022] Optionally, the updating time step, returning to the step of calculating the current time step for estimating the final fused image, and performing iterative calculation until the current time step is zero, and outputting the corrected noise image as the final fused image, specifically includes:

[0023] Update the time step t←t-1, return to the step of calculating the estimation of the final fused image at the current time step, perform iterative calculation, start iterating from the initialization t=T until t=0, and output the corrected noise image f0 as the final fused image.

[0024] In a second aspect, the present application provides a computer device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the image fusion method based on diffusion posterior sampling.

[0025] In a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image fusion method based on diffuse posterior sampling.

[0026] In a fourth aspect, the present application provides a computer program product, including a computer program, which implements the image fusion method based on diffusion posterior sampling when executed by a processor.

[0027] According to the specific embodiments provided in this application, this application discloses the following technical effects.

[0028] The present application provides an image fusion method based on diffuse a posteriori sampling. Under the diffuse a posteriori sampling framework, the complementary information of two source images, the visible light image and the SAR image, is integrated to obtain a fused image containing more complete information. The method can effectively avoid the loss of effective information caused by generative adversarial training, and not only enhance the information completeness of the fused image, but also improve the feature significance of the fused image. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0030] Figure 1 A flowchart of an image fusion method based on diffuse posterior sampling in this application;

[0031] Figure 2 A schematic diagram of a visible light image provided in an embodiment of the present application;

[0032] Figure 3 A schematic diagram of a SAR image provided in an embodiment of the present application;

[0033] Figure 4 This is a schematic diagram of the image fusion result obtained by the method of this application;

[0034] Figure 5 This is a schematic diagram for comparing the fusion results of land scene images;

[0035] Figure 6 Schematic diagram of the comparison of ocean scene image fusion results. DETAILED DESCRIPTION

[0036] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0037] The purpose of this application is to propose an image fusion method based on diffuse posterior sampling to avoid the loss of effective information caused by generative adversarial training and enhance the information completeness of the fused image.

[0038] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0039] In an exemplary embodiment, the present application provides an image fusion method based on diffusion posterior sampling, such as Figure 1 As shown, the steps include steps 1 to 5.

[0040] Step 1: Initialize the variables of the diffusion posterior sampling process, including setting the initial diffusion sampling time step and randomly taking the initial diffusion noise image.

[0041] This application proposes an image fusion method based on diffuse posterior sampling, which uses visible light images and SAR (Synthetic Aperture Radar) images to guide the diffuse posterior sampling process to generate a fused image. SAR image is The fused image is Where H and W are the width and height of the image respectively. Represents the real number space of H×W dimensions.

[0042] The diffusion posterior sampling process first requires initializing variables, including: setting the initial diffusion sampling time step t = T, randomly taking the initial diffusion noise image f T Normal distribution Among them I n is the same size as the noise image. θ (f t ,t) Indicates. Where t is the current time step of diffusion sampling (referred to as the current step), is a variable, and T is the initial value of t; that is, initialization f t =f T .f t The noise image representing the diffusion object at step t, also called the noise image at the current time step (current step).

[0043] Among them, the pre-trained noise approximation network s θ (f t ,t) is obtained by pre-training the U-Net network, and its input is the current time step noise image f t and the current time step t, the output is the predicted current time step noise image f t Compared to the original clear image f0, the noise is added.

[0044] Step 2: Based on the initialized variables, calculate the estimate of the final fused image at the current time step.

[0045] Formula (1) is used to calculate the estimate of the final fused image at the current time step:

[0046]

[0047] where f t is the current step noise image; For the pre-trained noise approximation network; intermediate variables in the calculation process α t =1-β t , β tis the single-step noise variance of the tth step in the forward noise addition process of the diffusion model.

[0048] Step 3: Based on the estimation of the final fused image at the current time step, the noise image at the current time step is preliminarily updated to obtain the preliminary updated result of the noise image at the next time step.

[0049] Take the Gaussian noise image z and estimate the final fused image based on the current time step Update the current step noise image f together t , the update process expression is:

[0050]

[0051] Among them, f′ t-1 Represents the preliminary update result of the noise image at the next time step (step t-1). Intermediate variables in the calculation process α t-1 =1-β t-1 , β t-1 is the single-step noise variance of the t-1th step in the forward noise addition process of the diffusion model. t is the noise image f at the current time step in the forward noise addition process of the diffusion model t Compared with the variance of the noise added to the original clear image f0. The Gaussian noise image z taken refers to a noise image that is randomly taken to obey the Gaussian distribution as z.

[0052] Step 4: Input the source image to correct the preliminary update result of the noise image in the next time step to obtain a corrected noise image; the source image includes a visible light image and a SAR image.

[0053] Input visible light image and SAR images As the source image, the preliminary update result f′ of the noise image at the next time step is corrected by the following formula t-1 :

[0054]

[0055] where f t-1 is the corrected noise image. ρ, γ, η1 and η2 are all weight coefficients; the weight coefficients ρ and η1 are used to balance the features from the SAR image and the texture features from the SAR image in the fused image; the weight coefficients γ and η2 are used to balance the features from the visible light image and the texture features from the visible light image in the fused image. sgn represents the sign function; div is the divergence operator. ▽ is the gradient operator. The total variation of an image ω is defined as in D 1 , D 2are the 1-dimensional difference matrices of the image ω in the x-direction and y-direction respectively; I represents the identity matrix.

[0056] Step 5: Update the time step, return to the step of calculating the current time step for estimating the final fused image, perform iterative calculations until the current time step is zero, and output the corrected noise image as the final fused image.

[0057] Update the time step t←t-1, return to step 2 and use formulas (1) to (3) for iterative calculation, starting from the initialization t=T and iterating until t=0, and output the corrected noise image f0 as the final fused image, that is, restoring the original clear image f0 mentioned above.

[0058] In recent years, the denoising diffusion probability model (DDPM) and its continuous form have developed rapidly in the field of image processing. They have rapidly achieved remarkable results in many fields by virtue of their versatile pre-training framework, powerful deep generative prior, interpretable generation process and stable training process. These characteristics effectively address the challenges faced by traditional deep learning image fusion methods and provide new ideas for image fusion. The denoising diffusion probability model has shown good performance in a series of inverse problems such as image denoising, image restoration, and image super-resolution. However, unlike traditional inverse problems, the positive process model of image fusion tasks is unclear, which makes it difficult for diffusion models such as DDPM to be applied to image fusion. In this regard, the present application uses the traditional variational model of image fusion to construct the posterior distribution of the fused image, thereby improving the diffusion posterior sampling scheme for image fusion. Compared with the drawback of the generative adversarial network algorithm that easily loses gradient information and generates blurred fused images, the diffusion model takes the maximization of the variational lower bound as the training goal, embeds time step encoding, has a stable training process, and can avoid the loss of effective information caused by generative adversarial training. Its explainable generation process is superior to the black box characteristics of traditional deep information methods. The deep generation prior provided by the diffusion model is adaptive to diversified tasks.

[0059] The diffusion modeling problem based on image fusion is the key to solving the existing image fusion method problems with diffusion models. This application uses the sparse prior of the fused image (embodied in These two terms) and bounded variation priors (e.g. ) designed an optimization model combining sparse approximation term and total variation regularization term, and then transformed the optimization problem into the maximum likelihood estimation problem of the Laplace distribution of the fused image; then established a guidance model for the diffusion sampling of the visible light image and the SAR image on the fused image under the diffusion posterior sampling framework, as shown in formula (3). Based on this, an image fusion algorithm based on diffusion posterior sampling was designed. Compared with the existing image fusion method, this application can not only enhance the information completeness of the fused image, but also improve the feature significance of the fused image.

[0060] In order to verify the effect of this application, a specific example is provided below for illustration. In this example, the fused visible light image is as follows: Figure 2 As shown, the SAR image is Figure 3 As shown in Figure 1, both contain land scene images and ocean scene images, which are 256×256 single-channel images and have the same brightness. Figure 2 As shown in part (a) of the diagram, the SAR image of the land scene is as follows: Figure 3 The visible light image of the ocean scene is shown in part (a). Figure 2 As shown in part (b) of the diagram, the SAR image of the ocean scene is as follows: Figure 3 As shown in part (b) of .

[0061] The image fusion method based on diffusion posterior sampling proposed in this application focuses on establishing an image fusion diffusion posterior sampling diffusion optimization model at each time step in the inverse process of the diffusion model, and its specific steps are as described in steps 1 to 5 above. In this embodiment, the weight parameter in formula (3) is set as: Use the above steps to Figure 2 The visible light images and Figure 3 The SAR images shown are fused, and the fusion results are shown in Figure 4 As shown in Figure 2, the fused image of the land scene is as follows: Figure 4 As shown in part (a) of the diagram, the fused image of the ocean scene is as follows: Figure 4 As shown in part (b) of .

[0062] In order to verify the image fusion effect, the fusion images of the present application method are compared with those of the CF-Net and DDcGAN methods. Figure 5 As shown in Figure 2, the fusion results of ocean scene images are compared. Figure 6 shown. Figure 5 Parts (a), (b) and (c) in the figure respectively show the fused images of land scenes obtained using CF-Net, DDcGAN and the method of the present application. Figure 6Parts (a), (b) and (c) in the figure respectively show the fused images of the ocean scene obtained using CF-Net, DDcGAN and the method of the present application. Figures 4 to 6 The boxed parts indicate complementary information that deserves special attention.

[0063] In order to quantitatively evaluate the performance of the above method, this application selected five indicators: peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), cross entropy (CE), mutual information (MI) and VIFF (visual fidelity). The larger the value of these indicators, the higher the image quality. The comparison results are shown in Table 1.

[0064] Table 1 Quantitative evaluation results of image fusion indicators

[0065]

[0066]

[0067] The results shown in Table 1 show that the method proposed in this application is superior to the comparison method in terms of image quality improvement and image fidelity.

[0068] The main purpose of the image fusion method based on diffuse posterior sampling provided by the present application is to integrate the complementary information of the two source images to obtain an image with more complete information. For this purpose, the present application designs an optimization model combining sparse approximation terms and total variation regularization and transforms it into the maximum likelihood estimation problem of the Laplace distribution of the fused image; then, an image fusion model guided by visible light images and SAR images for fused images is established under the diffuse posterior sampling framework, which can not only enhance the information completeness of the fused image and improve the feature significance of the fused image, but also the quantitative index values ​​of the visual effect, peak signal-to-noise ratio, structural similarity, etc. of the fused image are better than the existing methods, and has a more outstanding image fusion effect.

[0069] In an exemplary embodiment, the present application also provides a computer device, which may be a server or a terminal. The computer device includes a processor, a memory, an input / output interface, and a communication interface. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the image fusion method based on diffuse posterior sampling is implemented.

[0070] In an exemplary embodiment, the present application also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the image fusion method based on diffusion posterior sampling is implemented.

[0071] In an exemplary embodiment, the present application also provides a computer program product, including a computer program, which implements the image fusion method based on diffusion posterior sampling when executed by a processor.

[0072] It can be understood by a person of ordinary skill in the art that all or part of the processes in the above-mentioned embodiment method can be completed by hardware related to computer program instructions, and the computer program can be stored in a non-volatile computer-readable storage medium, and the computer program can include the process of the embodiment of the above-mentioned method when executed. Among them, any reference to memory or other media in each embodiment provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0073] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0074] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0075] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An image fusion method based on diffusion posterior sampling, characterized in that: include: Initialize the variables of the diffusion posterior sampling process, including setting the initial diffusion sampling time step and randomly taking the initial diffusion noise image; Based on the initialized variables, the estimation of the final fused image at the current time step is calculated; Based on the estimation of the final fused image at the current time step, the noise image at the current time step is preliminarily updated to obtain the preliminary update result of the noise image at the next time step; The input source image corrects the preliminary update result of the noise image in the next time step to obtain a corrected noise image; the source image includes a visible light image and a SAR image; The time step is updated, and the step of calculating the estimation of the final fused image at the current time step is returned to perform iterative calculation until the current time step is zero, and the corrected noise image is output as the final fused image.

2. The image fusion method based on diffuse posterior sampling according to claim 1, characterized in that: The variables of the initialization diffusion posterior sampling process specifically include: Set the initial diffusion sampling time step t = T, and randomly select a sample that follows a normal distribution. The initial diffuse noise image f T ; where I n is the identity matrix of the same size as the noise image; Approximate the pre-trained noise to the network s θ (f t ,t) Represents; where t is the current time step of diffusion sampling; f t Represents the noise image at step t, also called the current time step noise image.

3. The image fusion method based on diffuse posterior sampling according to claim 2, characterized in that: The pre-trained noise approximation network s θ (f t ,t) is obtained by U-Net network pre-training, and the input is f t and t, the output is f t Compared to the original clear image f0, the noise is added.

4. The image fusion method based on diffusion posterior sampling according to claim 2, characterized in that: The calculation of the estimation of the final fused image at the current time step specifically includes: Using formula Calculate the estimate of the final fused image at the current time step in α t =1-β t , β t is the single-step noise variance of the tth step in the forward noise addition process of the diffusion model.

5. The image fusion method based on diffuse posterior sampling according to claim 4 is characterized in that: The method of preliminarily updating the noise image at the current time step based on the estimation of the final fused image at the current time step to obtain a preliminary update result of the noise image at the next time step specifically includes: Estimate of the final fused image based on the current time step Using formula For the current time step noise image f t Perform a preliminary update to obtain the preliminary update result f of the noise image at the next time step t ' -1 ;in α t-1 =1-β t-1 , β t-1 is the single-step noise variance of the t-1th step in the forward noise addition process of the diffusion model; σ t is the noise variance added to the noise image at the current time step compared to the original clear image during the forward noising process of the diffusion model; z is the Gaussian noise image.

6. The image fusion method based on diffuse posterior sampling according to claim 5, characterized in that: The Gaussian noise image z is a noise image that is randomly selected to obey Gaussian distribution.

7. The image fusion method based on diffuse posterior sampling according to claim 5, characterized in that: The input source image corrects the preliminary update result of the noise image at the next time step to obtain a corrected noise image, specifically comprising: Input the visible light image o and SAR image s as source images, and use the formula Correct the initial update result f of the noise image at the next time step t ' -1 , and get the corrected noise image f t-1 ; where ρ, γ, η1 and η2 are weight coefficients; sgn represents the sign function; ▽ is the gradient operator; div is the divergence operator.

8. The image fusion method based on diffuse posterior sampling according to claim 7, characterized in that: The updating time step, returning to the step of calculating the current time step for estimating the final fused image, performs iterative calculation until the current time step is zero, and outputting the corrected noise image as the final fused image, specifically includes: Update the time step t←t-1, return to the step of calculating the estimation of the final fused image at the current time step, perform iterative calculation, start iterating from the initialization t=T until t=0, and output the corrected noise image f0 as the final fused image.

Images