Image fusion method and system based on binary variational diffusion

Through the image fusion method based on binary variational diffusion, the variational optimization model and first-order stochastic gradient optimization algorithm are used to solve the problem of coherent spot noise in remote sensing image fusion, and efficient image fusion and noise suppression are achieved, which improves the Image quality and fidelity.

CN119090734APending Publication Date: 2024-12-06NAT UNIV OF DEFENSE TECH
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411073756.9
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-07
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing remote sensing image fusion method based on diffusion model has room for improvement in computing efficiency and noise suppression, especially when fusion of visible light images and SAR images, due to the influence of coherent spot noise in SAR images, the fusion imaging effect is poor. .

Method used

Using the image fusion method based on binary variational diffusion, starting from the forward process of coherent spot suppression and image fusion, the variational optimization model is constructed using the variational inference method, and the first-order stochastic gradient optimization algorithm is used to replace backpropagation, and the The coherent speckle suppression and image fusion variation optimization model is implemented to realize the binary variation diffusion remote sensing image fusion optimization model to generate a fusion image that suppresses coherent speckle.

Benefits of technology

While achieving rapid fusion of images, effectively suppress coherent spots, improve image fusion quality and visual fidelity, and significantly improve the effective information utilization of source images and the visual effect of fused images.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119090734A_ABST
    Figure CN119090734A_ABST
Patent Text Reader

Abstract

The invention discloses an image fusion method and system based on binary variational diffusion, and relates to the field of remote sensing image processing, and the method comprises the steps: starting from the forward process of speckle suppression and image fusion, constructing a variational optimization model through a variational inference method; a first-order stochastic gradient optimization algorithm is adopted to replace back propagation of the variational optimization model, a variational optimization model in each time step in the diffusion inverse process is constructed, and a final speckle suppression variational optimization model and an image fusion variational optimization model are obtained; based on the two final variational optimization models, constructing a binary variational diffusion remote sensing image fusion optimization model in a single time step; and taking the visible light image and the SAR image as the input of the binary variational diffusion remote sensing image fusion optimization model to carry out iterative solution so as to generate a fusion image for suppressing speckles. According to the invention, speckle suppression can be taken into account while fast image fusion is realized, the image fusion quality can be effectively improved, and the image fidelity effect is ensured.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of remote sensing image processing, and in particular to an image fusion method and system based on binary variational diffusion. Background Art

[0002] The purpose of remote sensing image fusion is to integrate multimodal information to fully obtain the characteristics of the target and facilitate the subsequent interpretation and analysis of the fused image. Recently, the diffusion denoising probabilistic model and its continuous form of score-based model have flourished in the field of image processing. With its pre-training framework that can be used for multiple tasks, powerful deep generative priors, interpretable generation process and stable training process, it has rapidly generalized a large number of new results. These characteristics are sufficient to solve the dilemma of traditional deep learning image fusion methods.

[0003] The application of diffusion model in image fusion can be divided into two categories. One is to use the prior information of the fused image to initialize the forward process of the diffusion denoising model, such as Dif-fusio and Diff-IF; the other is to use the source image to guide the reverse process of the diffusion model to generate a fused image, such as DPS-IF (Diffusion posterior sampling for remote sensing image fusion). However, the former uses the fusion knowledge prior or source image features as the diffusion starting point, and the quality of the source image affects the capture of the image features by the diffusion model, while the latter's posterior sampling requires back propagation of the score function, and this operation involves a large computational complexity when taking derivatives. Existing remote sensing image fusion methods based on diffusion models have a lot of room for improvement in terms of computational efficiency and noise suppression from source images.

[0004] Especially for the task of fusion of visible light image and SAR (Synthetic Aperture Radar), due to the imaging mechanism of SAR, speckle noise is unavoidable, resulting in poor fusion imaging effect. Therefore, how to achieve fast image fusion while taking into account speckle suppression has become a technical problem to be solved in this field. Summary of the invention

[0005] The purpose of this application is to provide an image fusion method and system based on binary variational diffusion, which can achieve rapid image fusion while taking into account coherent speckle suppression, effectively improve image fusion quality, and ensure image fidelity.

[0006] To achieve the above objectives, this application provides the following solutions:

[0007] In a first aspect, the present application provides an image fusion method based on binary variational diffusion, comprising:

[0008] Starting from the forward process of speckle suppression and image fusion, a variational optimization model is constructed using a variational inference method; the variational optimization model includes: a speckle suppression variational optimization model and an image fusion variational optimization model;

[0009] A first-order stochastic gradient optimization algorithm is used to replace the back propagation of the variational optimization model, and a variational optimization model is constructed in each time step in the inverse diffusion process to obtain a final coherent speckle suppression variational optimization model and a final image fusion variational optimization model;

[0010] Based on the final coherent speckle suppression variational optimization model and the final image fusion variational optimization model, a binary variational diffusion remote sensing image fusion optimization model is constructed in a single time step;

[0011] The visible light image and the SAR image are taken as inputs of the binary variational diffusion remote sensing image fusion optimization model and iteratively solved to generate a fused image with suppressed coherent speckles.

[0012] Optionally, starting from the forward process of speckle suppression and image fusion, the process of constructing a variational optimization model for speckle suppression using a variational inference method includes:

[0013] The speckle suppression task is equivalent to the first required distribution, and the first variational distribution is defined by using the variational inference method;

[0014] Minimize the KL divergence estimation distribution between the first variational distribution and the first desired distribution to obtain a first KL divergence estimation distribution result;

[0015] And using the Bayesian formula, based on the first KL divergence estimation distribution result, obtain a first optimization target;

[0016] The first optimization objective is minimized, and the parameter renormalization technique is used to obtain the coherent speckle suppression variational optimization model based on the minimized first optimization objective; the coherent speckle suppression variational optimization model is expressed as:

[0017] In the formula, represents the fidelity term of the variational optimization model for speckle suppression, represents the regularization term of the variational optimization model for speckle suppression, represents the mean of the assumed first variation distribution, and min means minimization.

[0018] Optionally, when the variational optimization model is a coherent speckle suppression variational optimization model, a first-order stochastic gradient optimization algorithm is used to replace the back propagation of the coherent speckle suppression variational optimization model, and a variational optimization model is constructed in each time step in the inverse diffusion process to obtain a final coherent speckle suppression variational optimization model, including:

[0019] The first-order stochastic gradient optimization algorithm is used to simplify the regularization term of the coherent speckle suppression variational optimization model, and the final coherent speckle suppression variational optimization model in each time step in the diffusion inverse process is obtained.

[0020] Optionally, the final coherent speckle suppression variational optimization model is expressed as:

[0021]

[0022] In the formula, lns is denoted as r, s represents the SAR image, and lns represents the logarithm of the SAR image s; represents the optimization loss of the regularization term of the variational optimization model for speckle suppression, sg[·] represents the truncated gradient, represents the variance of the first variational distribution at the tth time step of the inverse diffusion process The estimate, represents the square of the second norm of *, and λ1 represents the weight parameter.

[0023] Optionally, starting from the forward process of speckle suppression and image fusion, the process of constructing an image fusion variational optimization model using a variational inference method includes:

[0024] The image fusion task is equivalent to the second required distribution, and the second variational distribution is defined using the variational inference method;

[0025] Minimize the KL divergence estimation distribution between the second variation distribution and the second desired distribution to obtain a second KL divergence estimation distribution result;

[0026] And using the Bayesian formula, a second optimization objective is obtained based on the second KL divergence estimation distribution result;

[0027] The second optimization objective is minimized, and the parameter renormalization technique is used to obtain the image fusion variational optimization model based on the minimized second optimization objective; the image fusion variational optimization model is expressed as:

[0028] Wherein, μ represents the mean of the second variational distribution, Fide2(μ) represents the fidelity term of the variational optimization model for speckle suppression, Re2(μ) represents the regularization term of the variational optimization model for image fusion, and min represents minimization.

[0029] Optionally, when the variational optimization model is an image fusion variational optimization model, a first-order stochastic gradient optimization algorithm is used to replace the back propagation of the image fusion variational optimization model, and a variational optimization model is constructed in each time step in the diffusion inverse process to obtain the final image fusion variational optimization model, including:

[0030] The regularization term of the image fusion variational optimization model is simplified by using a first-order stochastic gradient optimization algorithm to obtain the final image fusion variational optimization model in each time step during the diffusion inverse process.

[0031] Optionally, the final image fusion variational optimization model is expressed as:

[0032]

[0033] In the formula, o represents visible light image, s represents SAR image, a, b, η and λ2 all represent weight parameters. represents the square of the second norm of *, The gradient term representing the difference between the fused image and the visible light image, represents the optimization loss of the regularization term of the image fusion variational optimization model, represents the estimate of the mean μ of the second variational distribution at time step t of the inverse diffusion process, and sg[·] represents the truncated gradient.

[0034] Optionally, the binary variational diffusion remote sensing image fusion optimization model is expressed as:

[0035]

[0036] In the formula, logs0 is denoted as r0, s0 represents the SAR image in the ideal state after removing the coherent speckle, and logs0 represents the logarithm of the SAR image s0 in the ideal state after removing the coherent speckle; lns is denoted as r, s represents the SAR image, and lns represents the logarithm of the SAR image s; represents the mean of the variational distribution, min represents minimization, represents the optimization loss of the regularization term of the variational optimization model for speckle suppression, sg[·] represents the truncated gradient, The variance of the first variational distribution at the tth time step of the inverse diffusion process The estimate, represents the square of the second norm of *, μ represents the mean of the second variational distribution, o represents the visible light image, represents the SAR image s0 obtained after the previous step of speckle suppression, a, b, η, λ1 and λ2 are all weight parameters. The gradient term representing the difference between the fused image and the visible light image, represents the optimization loss of the regularization term of the image fusion variational optimization model, Represents the estimate of the final fused image at time step t of the inverse diffusion process.

[0037] In a second aspect, the present application provides an image fusion system based on binary variational diffusion, wherein the image fusion system based on binary variational diffusion is used to implement the image fusion method based on binary variational diffusion provided above; the system comprises:

[0038] The first model building module is used to build a variational optimization model based on the forward process of coherent speckle suppression and image fusion by using a variational inference method; the variational optimization model includes: a coherent speckle suppression variational optimization model and an image fusion variational optimization model;

[0039] A second model building module is used to replace the back propagation of the variational optimization model with a first-order stochastic gradient optimization algorithm, build a variational optimization model in each time step in the inverse diffusion process, and obtain a final coherent speckle suppression variational optimization model and a final image fusion variational optimization model;

[0040] The third model building module is used to build a binary variational diffusion remote sensing image fusion optimization model in a single time step based on the final coherent speckle suppression variational optimization model and the final image fusion variational optimization model;

[0041] The image fusion module is used to iteratively solve the visible light image and the SAR image as inputs of the binary variational diffusion remote sensing image fusion optimization model to generate a fused image with suppressed coherent speckles.

[0042] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0043] The present application provides an image fusion method and system based on binary variational diffusion. Starting from the positive process of speckle suppression and image fusion, first-order stochastic gradient optimization is used to establish variational optimization models for speckle suppression and remote sensing image fusion respectively, and the optimization problems of the same time step are integrated together to realize binary variational diffusion. It is possible to suppress speckles from SAR images while fusing remote sensing images, effectively improve the utilization rate of effective information of source images and the visual fidelity of fused images, and significantly improve the image fusion quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0045] Figure 1 This is an application environment diagram of an image fusion method based on binary variational diffusion in an embodiment of the present application;

[0046] Figure 2 A schematic diagram of a flow chart of an image fusion method based on binary variational diffusion provided in one embodiment of the present application;

[0047] Figure 3 An implementation architecture diagram of an image fusion method based on binary variational diffusion provided in one embodiment of the present application;

[0048] Figure 4 A schematic diagram of a visible light image provided by another embodiment of the present application;

[0049] Figure 5 A schematic diagram of a SAR image provided by another embodiment of the present application;

[0050] Figure 6 A schematic diagram of a fused image provided by another embodiment of the present application;

[0051] Figure 7 A schematic diagram for comparing Land Scene image fusion results provided by another embodiment of the present application;

[0052] Figure 8 A schematic diagram showing comparison of image Sea Scene fusion results provided in another embodiment of the present application;

[0053] Fig. 9 A schematic diagram of the structure of a computer device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0054] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0055] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0056] The image fusion method based on binary variational diffusion provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be set up separately, integrated on the server 104, or placed on the cloud or other servers. The terminal 102 can send the visible light image and the SAR image to the server 104. After the server 104 receives the visible light image and the SAR image, the server 104 uses the visible light image and the SAR image as the input of the binary variational diffusion remote sensing image fusion optimization model for iterative solution to generate a fused image with suppressed coherent speckle. Among them, the server 104 starts from the forward process of coherent speckle suppression and image fusion, and uses the variational inference method to construct a variational optimization model. The first-order stochastic gradient optimization algorithm is used to replace the back propagation of the variational optimization model, and the variational optimization model in each time step in the diffusion inverse process is constructed to obtain the final coherent speckle suppression variational optimization model and the final image fusion variational optimization model. Based on the final variational optimization model for speckle suppression and the final variational optimization model for image fusion, a binary variational diffusion remote sensing image fusion optimization model is constructed in a single time step. The server 104 can feed back the obtained fused image with suppressed speckle to the terminal 102. In addition, in some embodiments, the image fusion method based on binary variational diffusion can also be implemented by the server 104 or the terminal 102 alone, such as the terminal 102 can directly perform image fusion processing on the visible light image and the SAR image, or the server 104 can obtain the visible light image and the SAR image from the data storage system and perform image fusion processing on the visible light image and the SAR image.

[0057] The terminal 102 may be, but is not limited to, various desktop computers, laptop computers, smart phones, tablet computers, IoT devices, and portable wearable devices. The IoT devices may be smart speakers, smart TVs, smart air conditioners, smart vehicle-mounted devices, etc. The portable wearable devices may be smart watches, smart bracelets, head-mounted devices, etc. The server 104 may be implemented as an independent server or a server cluster consisting of multiple servers, or may be a cloud server.

[0058] In an exemplary embodiment, Figure 2 As shown, a method for image fusion based on binary variational diffusion is provided. The method is executed by a computer device, and specifically can be executed by a computer device such as a terminal or a server alone, or can be executed by a terminal and a server together. In the embodiment of the present application, the method is applied to Figure 1 The server 104 in the example is used as an example to illustrate, including the following steps 200 to 203. Among them:

[0059] Step 200: Starting from the forward process of speckle suppression and image fusion, a variational optimization model is constructed using a variational inference method. The variational optimization model includes: a variational optimization model for speckle suppression and a variational optimization model for image fusion.

[0060] Step 201: A first-order stochastic gradient optimization algorithm is used to replace the back propagation of the variational optimization model, and a variational optimization model is constructed in each time step in the inverse diffusion process to obtain a final coherent speckle suppression variational optimization model and a final image fusion variational optimization model.

[0061] Step 202: Based on the final coherent speckle suppression variational optimization model and the final image fusion variational optimization model, a binary variational diffusion remote sensing image fusion optimization model is constructed in a single time step.

[0062] Step 203: using the visible light image and the SAR image as inputs of a binary variational diffusion remote sensing image fusion optimization model to perform iterative solution to generate a fused image with suppressed coherent speckles.

[0063] By implementing the above steps 200 to 203, the present application replaces the back propagation of the score function with a first-order stochastic optimization algorithm in the variational optimization model of speckle suppression and image fusion, and establishes a binary variational diffusion optimization model to achieve visible light image and SAR image fusion with speckle suppression performance. It not only has an obvious speckle suppression effect in visual effect, so that the visible light image information at the corresponding position can be fully presented without being interfered by speckle, but also has higher efficiency in solving optimization problems and good effects in improving image quality and fidelity.

[0064] In another exemplary embodiment of the present application, the existing methods are limited by the coherent speckle from SAR images and the huge computational complexity in the application of visible light image and SAR image fusion. Starting from the forward process of coherent speckle suppression and image fusion, the present application uses the variational inference method to design a variational optimization model for two separate tasks, and then replaces the back propagation of the score function with a first-order stochastic gradient optimization algorithm, and provides an optimization model for a single task at a single time step in the inverse process of the diffusion model. Based on this, Figure 3 As shown, in the above-mentioned image fusion method based on binary variational diffusion provided in the present application, the process of establishing the final coherent speckle suppression variational optimization model and the final image fusion variational optimization model includes:

[0065] S1. Establish the final variational optimization model for speckle suppression.

[0066] S1.1. Give the forward process expression of SAR image speckle, for example:

[0067] Due to the imaging mechanism of SAR, the coherent speckle noise of SAR images is multiplicative noise. When is given, there is:

[0068] s=s0+s0n (1)

[0069] At this time, formula (1) is the forward model of the coherent speckle. It is the SAR image in an ideal state after removing the coherent speckle. This means that the speckle noise follows a normal distribution. is the variance of the speckle noise.

[0070] Taking the logarithm of the SAR image s gives:

[0071] lns=lns0+ln(1+n) (2)

[0072] Where n represents the multiplicative noise.

[0073] If we assume |n| < 1, formula (2) can be written as r = r0 + n, where lns is r and logs0 is r0. p(r|r0) is the distribution of r when r0 is known.

[0074] S1.2. Establish a variational optimization model for speckle suppression.

[0075] The speckle suppression task is equivalent to the first distribution to be determined p(r0|r), and the first variational distribution is defined using the variational inference method. The first variational distribution is denoted as

[0076] Minimize the KL divergence estimation distribution between the first variational distribution q1 and the first desired distribution p(r0|r) to obtain the first KL divergence estimation distribution result. The first KL divergence estimation distribution result is recorded as: KL(*) represents the divergence of *, p(r0|r) represents the distribution that r0 obeys when r is known, and q1(r0|r) represents the first variational distribution that r0 obeys when r is known.

[0077] Using the Bayesian formula, the first optimization goal is obtained based on the first KL divergence estimation distribution result. The first optimization goal is recorded as:

[0078]

[0079] In the formula, represents the expectation of * with respect to the distribution q1(r0|r). log* represents the logarithm of *, p(r) represents the distribution of r, and p(r) represents the distribution of r0.

[0080] The third term of formula (3) is irrelevant to the optimization variable and can be omitted in the optimization process. The diffusion model is introduced in the second term, then p(r0) is the final image distribution to be obtained by the diffusion model, and q1 is the inverse distribution in the inverse process of the diffusion model. The KL divergence between the two can be expressed by matching the score functions of all time steps in the inverse process of the diffusion model. Based on this, the first optimization objective is minimized, which is expressed as:

[0081]

[0082] In the formula, is the weight function. represents the noise process r of extracting sample r0 from distribution q1(r0|r) and applying the diffusion model t =α t r0+σ t ε, For the diffusion noise process from r0 to r t The applied noise level, α t =1-β t , β t is the value of β(t) at time t, and β(t) is the value of the diffusion model noise process from r0 to r1. t-1 to r t The variance of the noise applied

[0083] Assume σ r ≈0, yes σ r and are the variance and mean of the assumed variational distribution q1, respectively. By using the parameter renormalization technique, the variational optimization model for speckle suppression is obtained based on the first optimization objective after minimization, which is expressed as:

[0084]

[0085] in, represents the fidelity term of the variational optimization model for speckle suppression, represents the regularization term of the variational optimization model for speckle suppression. It represents the mean of the assumed first variation distribution. Since the variance of the first variation distribution is assumed to be zero in the model, solving the optimization problem about it is equivalent to solving the image after taking the logarithm of the SAR image without coherent speckle, which is equivalent to solving the SAR image without coherent speckle. min means minimization. The time step follows a uniform distribution ω1(t) is a positive function, ε represents the Gaussian noise added during the diffusion model noise addition process, t represents the time step, represents the standard normal distribution, represents uniform distribution and T represents the total diffusion time step.

[0086] S1.3. Give the variational optimization model for each time step of the inverse diffusion process.

[0087] The regularization term Re1 of the coherent speckle suppression variational optimization model is simplified by using a first-order stochastic gradient optimization algorithm (e.g., a first-order stochastic optimizer). Assuming that ω1(0)=0 to meet the diffusion model condition, the gradient of the Re1 term is:

[0088]

[0089] Among them, λ 1t :=(Tα t / σ t )(dw1 / dt),ε θ is the pre-trained score function of the unconditional diffusion model, and := means it is defined as.

[0090] For each time step t~U(0,T), the optimization loss part of the regularization term Re1 is:

[0091]

[0092] in, represents the variance of the first variational distribution at the tth time step of the inverse diffusion process Since the variance of the first variation distribution is assumed to be zero in the model, The estimation of is equivalent to the estimation of the image after taking the logarithm of the final clean SAR image (here the expectation is used to represent this estimation), ε θ It is the pre-trained score function of the unconditional diffusion model. t is an intermediate variable α t =1-β t , β t ∈(0,1) is the diffusion model noise adding process from x t-1 to x t The variance of the noise applied. sg[·] represents the truncated gradient, which means that the variables in the brackets are regarded as constants during the back propagation process and are not differentiated. λ1 is a weight parameter used to balance the first fidelity term and the second regularization term, that is, to balance the function value between the regularization term and the fidelity term. Therefore, the final coherent speckle suppression variational optimization model at each time step in the inverse diffusion process is expressed as:

[0093]

[0094] In the formula, Represents the square of the second norm of *.

[0095] S2. Establish the final image fusion variational optimization model.

[0096] S2.1. Give the expression of the image fusion forward process.

[0097] The traditional variational model for image fusion is:

[0098]

[0099] in, is the fused image, is a visible light image, is a SAR image. a>0 and b>0 are weight parameters to balance the information of the two source images. This model is equivalent to the maximum likelihood estimation of the distribution of the fused image f0 in the probability density function, which is recorded as:

[0100]

[0101] Where K is the normalization constant, and p(o,s|f0) represents the distribution of the probability density function of the fused image f0.

[0102] S2.2. Establish an image fusion variational optimization model.

[0103] The image fusion task is equivalent to solving the distribution of the fused image under the given visible light image and SAR image conditions, that is, the second distribution to be determined p(f0|o,s). Similar to step S1.2, the variational inference method is used to define the second variational distribution, which is recorded as

[0104] Minimize the KL divergence estimation distribution between the second variational distribution q2 and the second distribution to be determined p(f0|o,s) to obtain the second KL divergence estimation distribution result. The second KL divergence estimation distribution result is recorded as:

[0105]

[0106] The same processing method as the minimization of KL divergence in step S1.2 is used to obtain the second optimization objective based on the second KL divergence estimation distribution result using the Bayesian formula. The second optimization objective is expressed as:

[0107]

[0108] In the formula, It represents the expectation of the distribution q2(f0o,s). q2(f0o,s) represents the second variational distribution of f0 given that o and s are known, p(o,s|f0) represents the distribution of o and s given that f0 is known, p(f0) represents the distribution of f0, and p(o,s) represents the distribution of o and s.

[0109] In formula (12), the third term is irrelevant to the optimization variable and can be omitted in the optimization process. The diffusion model is introduced in the second term, then p(f0) is the final image distribution to be obtained by the diffusion model, and q2 is the inverse distribution in the inverse process of the diffusion model. The KL divergence between the two can be expressed by matching the score functions of all time steps in the inverse process of the diffusion model. Based on this, the second optimization objective is minimized, which is expressed as:

[0110]

[0111] in, is the weight function, q2(ft|o,s) represents the noise adding process f of extracting sample f0 from distribution q2(f0|o,s) and applying the diffusion model t =α t f0+σ t ε.

[0112] Assume σ f ≈0, f0≈μ,σ f and μ are the variance and mean of the set variational distribution q2, respectively. By using the parameter renormalization technique, the image fusion variational optimization model is obtained based on the minimized second optimization objective, which is expressed as:

[0113]

[0114] in, Fide2(μ) represents the fidelity term, which ensures that the fused image retains the information of the source images, namely the SAR image and the visible light image. Re2(μ) represents the regularization term of the image fusion variational optimization model, and ω2(t) is a positive function.

[0115] S2.3. Give the variational optimization model of the diffusion inverse process in each time step, and obtain the final image fusion variational optimization model.

[0116] The Re2 term in step S2.2 is simplified using a first-order stochastic gradient optimization algorithm (e.g., a first-order stochastic optimizer). Assuming that ω2(0)=0 to meet the diffusion model condition, the gradient of the Re2 term is:

[0117]

[0118] In the formula, λ 2t :=(Tα t / σ t )(dw2 / dt), represents the expectation of * about t, ε, λ 2t represents the weight parameter of the fidelity term and the regularization term in the balanced image fusion variational optimization model at the tth step of the diffusion inverse process, f t represents the fused image at time step t.

[0119] Therefore, for each time step The optimization loss part of the regularization term Re2 is:

[0120]

[0121] In the formula, λ2 is a weight parameter that balances the function value between the regularization term and the fidelity term. Therefore, in each time step of the inverse process of the diffusion model, the image fusion variational optimization model is:

[0122]

[0123] In order to retain more texture information of the visible light image in the fused image, the gradient l2 norm term with the weight parameter η is added to the optimization model, and the final image fusion variational optimization model is expressed as:

[0124]

[0125] In the formula, a, b, η and λ2 all represent weight parameters. The gradient term that represents the difference between the fused image and the visible light image is added to the optimization term to retain more texture information in the visible light image in the fused image. represents the optimization loss of the regularization term of the image fusion variational optimization model. μ represents the mean of the second variational distribution. Since the variance of the second variational distribution is assumed to be zero in the model, solving the optimization problem about it is equivalent to solving the fused image. Represents the estimation of the mean μ of the second variational distribution at time step t of the inverse diffusion process. Since the variance of the second variational distribution is assumed to be 0, the estimation of μ is equivalent to the estimation of the fused image.

[0126] In another exemplary embodiment of the present application, Figure 3 As shown in Figure 2, the process of establishing a binary variational diffusion remote sensing image fusion optimization model in a single time step to realize an image fusion method with coherent speckle suppression performance can be described as:

[0127] (1): Establish a binary variational diffusion remote sensing image fusion optimization model. Note that the variational optimization model of time step t obtained in the inverse process of the diffusion model in step S1 and step S2 can share the network ε θ and diffusion model parameters. Therefore, in the time step t of the inverse process of the diffusion model, the remote sensing image speckle suppression image fusion can be modeled as a binary variational diffusion remote sensing image fusion optimization model, expressed as:

[0128]

[0129] In the formula, represents the clean SAR image s0 obtained after the previous step of speckle suppression, The gradient term that represents the difference between the fused image and the visible light image is added to the optimization term to retain more texture information in the visible light image in the fused image.

[0130] (2): Given the relevant parameters of the binary variational diffusion remote sensing image fusion optimization model.

[0131] When solving the fusion image with suppressed coherent speckle using the binary variational diffusion remote sensing image fusion optimization model, the quantities that need to be given are: visible light image o, SAR image s, weight parameters a, b, η, λ1 and λ2, and the score function ε pre-trained by the unconditional diffusion model θ , the variance sequence of Gaussian noise in the diffusion model noise process Total diffusion time step T.

[0132] (3): Iterative solution generates a fused image with suppressed coherent speckles.

[0133] First, take the logarithm lns of the input SAR image s and set it to r. I n is an n-row n-column identity matrix, where n = 256, and the rest represents noise. is the standard normal distribution, f T is the fused image at time step T. The iterative calculation process from t = T to t = 0 is:

[0134] 1.

[0135] 2. Calculation

[0136] 3. Use the optimizer to optimize loss1 Among them, loss1 represents a loss function used to suppress coherent speckle noise.

[0137] 4. Update r t-1 , in

[0138]

[0139] 6. Calculation

[0140] 7. Use the optimizer to optimize loss2 to find μ. Here, loss2 represents another loss function.

[0141] 8. Update f t-1 , f t-1=α t μ+σ t ε, where

[0142] After the above calculation process, the final μ is used as the obtained fused image with suppressed coherent speckles.

[0143] In another exemplary embodiment of the present application, it is assumed that the optical image is as follows Figure 4 As shown, the SAR image is Figure 5 As shown. Among them, Figure 4 Part (a) is a visible light image of a land scene. Figure 4 Part (b) is a visible light image of the ocean scene. Figure 5 Part (a) is a SAR image of a land scene. Figure 5 Part (b) is a SAR image of the ocean scene. Figure 4 and Figure 5 The images in are all single-channel images of 256×256 and the brightness is adjusted to be consistent. Based on this, the specific steps of image fusion using the image fusion method based on binary variational diffusion provided in the present application are:

[0144] S1: Establish the final coherent speckle suppression variational optimization model as shown in formula (8).

[0145] S2: Establish the final image fusion variational optimization model as shown in formula (18).

[0146] S3: Establish a binary variational diffusion remote sensing image fusion optimization model as shown in formula (19).

[0147] S4: Given a binary variational diffusion remote sensing image fusion optimization model, find the relevant parameters of the fused image that suppresses coherent speckles:

[0148] o is the visible light image and s is the SAR image, ε θ is the score function pre-trained by the unconditional diffusion model, T = 1000, and the variance sequence of the Gaussian noise in the diffusion model noise addition process is defined as a sequence of 1000 steps of values ​​equally spaced from 0.0001 to 0.02, σ t By formula Given, the weight parameters are set to a=1.2, b=1, η=1, λ1=0.2, λ2=0.2.

[0149] S5: Iteratively solve and generate a fused image with suppressed coherent speckles.

[0150] First, take the logarithm lns of the input SAR image s and set it to r. Iterate from t = T to t = 0:

[0151] 1.

[0152] 2. Calculation

[0153] 3. Use the optimizer to optimize loss1

[0154] 4. Update r t-1 , in

[0155] 5.

[0156] 6. Calculation

[0157] 7. Use the optimizer to optimize loss2 to find μ.

[0158] 8. Update f t-1 , f t-1 =α t μ+σ t ε, where The final μ is the desired fused image with suppressed coherent speckles.

[0159] The above steps are used to analyze the visible light image (such as Figure 4 ) and SAR images (such as Figure 5 ) fusion, the fusion result is as follows Figure 6 As shown in the figure, the comparison methods are DPS-IF and VDIF (Variational Diffusion Method for Remote Sensing Image Fusion). The land scene image fusion results of each method are compared. Figure 7 As shown in Figure 2, the fusion results of ocean scene images are compared. Figure 8 As shown. In order to quantitatively evaluate the performance of the above method, this embodiment selects five indicators, namely peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), cross entropy (CE), mutual information (MI) and VIFF (visual fidelity) for comparison and explanation. The larger the value of these indicators, the higher the image quality. The results of each indicator are shown in Table 1. The results show that the method proposed in this application is superior to the comparison method in terms of image quality improvement and image fidelity. Figure 6 Part (a) is the fused image of the land scene. Figure 6 Part (b) is the fused image of the ocean scene. Figure 7 Part (a) is the VDIF land scene image fusion result. Figure 7Part (b) is the DPS-IF land scene image fusion result. Figure 7 Part (c) is a graph of land scene image fusion results of the binary variational diffusion based image fusion method (abbreviated as BDIF) provided in this application. Figure 8 Part (a) is the VDIF ocean scene image fusion result. Figure 8 Part (b) is the DPS-IF ocean scene image fusion result. Figure 8 Part (c) is a diagram of the ocean scene image fusion result of the binary variational diffusion based image fusion method (abbreviated as BDIF) provided in this application.

[0160] Table 1 Quantitative results of image fusion indicators

[0161] method PSNR(dB) SSIM CE MI VIFF VDIF 16.78 0.44 0.71 1.29 0.43 DPS-IF 16.70 0.41 0.63 0.69 0.32 BDIF 17.10 0.49 0.98 2.08 0.56

[0162] In summary, the present application can be used to integrate the complementary information of optical images and SAR images, improve the availability of optical images and SAR images, and at the same time have the function of suppressing coherent speckles from SAR images during the fusion process. The present application designs a new binary variational diffusion optimization model. Starting from the positive process model of coherent speckle suppression and image fusion, the variational diffusion optimization models for coherent speckle suppression and remote sensing image fusion are established by using first-order stochastic gradient optimization, and the optimization problems of the same time step are combined to realize binary variational diffusion, which not only suppresses coherent speckles from SAR images while fusing remote sensing images, but also improves the utilization rate of effective information of the source image and the visual fidelity of the fused image.

[0163] Based on the same inventive concept, the embodiment of the present application also provides an image fusion system based on binary variational diffusion for implementing the above-mentioned image fusion method based on binary variational diffusion. The implementation solution provided by the system to solve the problem is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in one or more embodiments of the image fusion system based on binary variational diffusion provided below can refer to the limitations of the image fusion method based on binary variational diffusion above, and will not be repeated here.

[0164] In an exemplary embodiment, a binary variational diffusion based image fusion system is provided, comprising:

[0165] The first model building module is used to build a variational optimization model based on the forward process of coherent speckle suppression and image fusion by using a variational inference method; the variational optimization model includes: a coherent speckle suppression variational optimization model and an image fusion variational optimization model;

[0166] A second model building module is used to replace the back propagation of the variational optimization model with a first-order stochastic gradient optimization algorithm, build a variational optimization model in each time step in the inverse diffusion process, and obtain a final coherent speckle suppression variational optimization model and a final image fusion variational optimization model;

[0167] The third model building module is used to build a binary variational diffusion remote sensing image fusion optimization model in a single time step based on the final coherent speckle suppression variational optimization model and the final image fusion variational optimization model;

[0168] The image fusion module is used to iteratively solve the visible light image and the SAR image as inputs of the binary variational diffusion remote sensing image fusion optimization model to generate a fused image with suppressed coherent speckles.

[0169] In an exemplary embodiment, a computer device is provided. The computer device may be a server or a terminal. The internal structure diagram thereof may be as follows: Fig. 9 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, referred to as I / O) and a communication interface. Among them, the processor, the memory and the input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store image fusion data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, an image fusion method based on binary variational diffusion is implemented.

[0170] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0171] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0172] In an exemplary embodiment, a computer program product is provided, including a computer program, and when the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0173] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0174] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM may be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0175] The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. The non-relational database may include a distributed database based on blockchain, etc., but is not limited thereto. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but is not limited thereto.

[0176] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0177] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An image fusion method based on binary variational diffusion, characterized in that: The image fusion method based on binary variational diffusion includes: Starting from the forward process of speckle suppression and image fusion, a variational optimization model is constructed using a variational inference method; the variational optimization model includes: a speckle suppression variational optimization model and an image fusion variational optimization model; A first-order stochastic gradient optimization algorithm is used to replace the back propagation of the variational optimization model, and a variational optimization model is constructed in each time step in the inverse diffusion process to obtain a final coherent speckle suppression variational optimization model and a final image fusion variational optimization model; Based on the final coherent speckle suppression variational optimization model and the final image fusion variational optimization model, a binary variational diffusion remote sensing image fusion optimization model is constructed in a single time step; The visible light image and the SAR image are taken as inputs of the binary variational diffusion remote sensing image fusion optimization model and iteratively solved to generate a fused image with suppressed coherent speckles.

2. The image fusion method based on binary variational diffusion according to claim 1, characterized in that: Starting from the forward process of speckle suppression and image fusion, the process of constructing a variational optimization model for speckle suppression using the variational inference method includes: The speckle suppression task is equivalent to the first required distribution, and the first variational distribution is defined by using the variational inference method; Minimize the KL divergence estimation distribution between the first variational distribution and the first desired distribution to obtain a first KL divergence estimation distribution result; And using the Bayesian formula, based on the first KL divergence estimation distribution result, obtain a first optimization target; The first optimization objective is minimized, and the parameter renormalization technique is used to obtain the coherent speckle suppression variational optimization model based on the minimized first optimization objective; the coherent speckle suppression variational optimization model is expressed as: In the formula, represents the fidelity term of the variational optimization model for speckle suppression, represents the regularization term of the variational optimization model for speckle suppression, represents the mean of the assumed first variation distribution, and min means minimization.

3. The image fusion method based on binary variational diffusion according to claim 2, characterized in that: When the variational optimization model is a coherent speckle suppression variational optimization model, a first-order stochastic gradient optimization algorithm is used to replace the back propagation of the coherent speckle suppression variational optimization model, and a variational optimization model is constructed in each time step in the inverse diffusion process to obtain a final coherent speckle suppression variational optimization model, including: The first-order stochastic gradient optimization algorithm is used to simplify the regularization term of the coherent speckle suppression variational optimization model, and the final coherent speckle suppression variational optimization model in each time step in the diffusion inverse process is obtained.

4. The image fusion method based on binary variational diffusion according to claim 3, characterized in that: The final variational optimization model for speckle suppression is expressed as: In the formula, lns is denoted as r, s represents the SAR image, and lns represents the logarithm of the SAR image s; represents the optimization loss of the regularization term of the variational optimization model for speckle suppression, sg[·] represents the truncated gradient, represents the variance of the first variational distribution at the tth time step of the inverse diffusion process The estimate, represents the square of the second norm of *, and λ1 represents the weight parameter.

5. The image fusion method based on binary variational diffusion according to claim 1, characterized in that: Starting from the forward process of speckle suppression and image fusion, the process of constructing an image fusion variational optimization model using variational inference method includes: The image fusion task is equivalent to the second required distribution, and the second variational distribution is defined using the variational inference method; Minimize the KL divergence estimation distribution between the second variation distribution and the second desired distribution to obtain a second KL divergence estimation distribution result; And using the Bayesian formula, a second optimization objective is obtained based on the second KL divergence estimation distribution result; The second optimization objective is minimized, and the parameter renormalization technique is used to obtain the image fusion variational optimization model based on the minimized second optimization objective; the image fusion variational optimization model is expressed as: Wherein, μ represents the mean of the second variational distribution. Since the variance of the second variational distribution is assumed to be zero in the model, solving the optimization problem about it is equivalent to solving the fused image. Fide2(μ) represents the fidelity term of the variational optimization model for speckle suppression, Re2(μ) represents the regularization term of the variational optimization model for image fusion, and min represents minimization.

6. The image fusion method based on binary variational diffusion according to claim 5, characterized in that: When the variational optimization model is an image fusion variational optimization model, a first-order stochastic gradient optimization algorithm is used to replace the back propagation of the image fusion variational optimization model, and a variational optimization model is constructed in each time step in the diffusion inverse process to obtain the final image fusion variational optimization model, including: The regularization term of the image fusion variational optimization model is simplified by using a first-order stochastic gradient optimization algorithm to obtain the final image fusion variational optimization model in each time step during the diffusion inverse process.

7. The image fusion method based on binary variational diffusion according to claim 6, characterized in that: The final image fusion variational optimization model is expressed as: In the formula, o represents visible light image, s represents SAR image, a, b, η and λ2 all represent weight parameters. represents the square of the second norm of *, The gradient term representing the difference between the fused image and the visible light image, represents the optimization loss of the regularization term of the image fusion variational optimization model, represents the estimate of the mean μ of the second variational distribution at time step t of the inverse diffusion process, and sg[·] represents the truncated gradient.

8. The image fusion method based on binary variational diffusion according to claim 1, characterized in that: The binary variational diffusion remote sensing image fusion optimization model is expressed as: In the formula, logs0 is denoted as r0, s0 represents the SAR image in the ideal state after removing the coherent speckle, and logs0 represents the logarithm of the SAR image s0 in the ideal state after removing the coherent speckle; lns is denoted as r, s represents the SAR image, and lns represents the logarithm of the SAR image s; represents the mean of the variational distribution, min represents minimization, represents the optimization loss of the regularization term of the variational optimization model for speckle suppression, sg[·] represents the truncated gradient, represents the variance of the first variational distribution at time step t in the inverse diffusion process The estimate, represents the square of the second norm of *, μ represents the mean of the second variational distribution, o represents the visible light image, represents the SAR image s0 obtained after the previous step of speckle suppression, a, b, η, λ1 and λ2 are all weight parameters. The gradient term representing the difference between the fused image and the visible light image, represents the optimization loss of the regularization term of the image fusion variational optimization model, Represents the estimate of the mean μ of the second variational distribution at time step t in the inverse diffusion process.

9. An image fusion system based on binary variational diffusion, characterized in that: The image fusion system based on binary variational diffusion is used to implement the image fusion method based on binary variational diffusion as claimed in any one of claims 1 to 8; the system comprises: The first model building module is used to build a variational optimization model based on the forward process of coherent speckle suppression and image fusion by using a variational inference method; the variational optimization model includes: a coherent speckle suppression variational optimization model and an image fusion variational optimization model; A second model building module is used to replace the back propagation of the variational optimization model with a first-order stochastic gradient optimization algorithm, build a variational optimization model in each time step in the inverse diffusion process, and obtain a final coherent speckle suppression variational optimization model and a final image fusion variational optimization model; The third model building module is used to build a binary variational diffusion remote sensing image fusion optimization model in a single time step based on the final coherent speckle suppression variational optimization model and the final image fusion variational optimization model; The image fusion module is used to iteratively solve the visible light image and the SAR image as inputs of the binary variational diffusion remote sensing image fusion optimization model to generate a fused image with suppressed coherent speckles.

Citation Information

Cited By

  • Synthetic aperture radar image speckle suppression method based on diffusion sampling

    CN119850459A