Diffusion model-based training-free multi-image fusion method, device and medium

Abstract

This application discloses a training-free multi-image fusion method, device, and medium based on a diffusion model, relating to the field of computer vision technology. The method includes: acquiring multiple source images to be fused; simultaneously inputting the multiple source images into a diffusion model; performing a preset diffusion step on each source image to obtain a noisy image of the same size as the source image; and simultaneously performing a denoising process on the noisy image corresponding to each diffusion step until the noisy image corresponding to the 0th diffusion step is determined as the fused image of the multiple source images. This application aims to solve a series of problems in existing technologies when performing multi-image fusion, such as poor fusion effect, long processing time, and low efficiency, achieving fast and efficient multi-image fusion.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a training-free multi-image fusion method, device and medium based on a diffusion model. Background Technology

[0002] With the development of computer vision, image fusion technology has also evolved. Image fusion aims to integrate complementary information from multiple source images to generate a more comprehensive and higher-quality image, which is beneficial for human observation or subsequent machine analysis.

[0003] However, existing fusion methods have a series of problems:

[0004] For example, task-specific training is required: existing general-purpose fusion methods, such as end-to-end deep learning-based networks, require long-term training or fine-tuning on large-scale, meticulously labeled task-specific datasets. For instance, an infrared-visible image fusion model (IVIF model) needs to be trained on pairs of infrared-visible images, while a multi-focus image fusion model (MFIF model) requires a sequence of multi-focus images for training. This not only significantly increases the model development cycle and computational cost, but also means that the entire process must be restarted when faced with a new, unseen fusion task, severely limiting the model's scalability and practicality.

[0005] For example, there is a lack of task sensitivity and a "neutralization" phenomenon: Existing technologies use general-purpose models (such as diffusion models) pre-trained on ultra-large-scale natural image datasets (such as ImageNet) to achieve training-free image fusion. However, current general-purpose models are trained with the source images as hard constraints (such as through simple loss functions), lacking task sensitivity and often producing "neutralized" fusion results—that is, simply averaging or smoothing the source images, causing important edges, textures, and semantic information to be blurred or lost, resulting in poor image fusion effects. For example, when fusing infrared and visible light images, the model needs to understand the characteristics of infrared images highlighting heat sources and visible light images being rich in texture; when fusing multi-focus images, the model needs to identify and fuse all sharp regions. However, existing fusion methods do not effectively acquire this information, resulting in poor fusion effects.

[0006] For example, existing fusion methods suffer from low efficiency: fusion tasks often involve more than two source images. Existing methods employ a sequential fusion strategy, fusing the first two images first, then fusing the result with the third image, and so on. This strategy has two major drawbacks: First, its inference time is linearly related to the number of input images, resulting in enormous computational overhead for sequences containing dozens or even hundreds of images (such as multi-exposure HDR synthesis); second, the fusion result is highly dependent on the processing order of the input images, and different orders can lead to drastically different outputs, resulting in poor model robustness. Summary of the Invention

[0007] In response to the aforementioned problems and technical requirements, the applicant proposes a training-free multi-image fusion method, device, and medium based on a diffusion model to solve a series of problems in existing technologies, such as poor fusion effect, long time consumption, and low efficiency, thereby achieving fast and efficient multi-image fusion.

[0008] This application provides a training-free multi-image fusion method based on a diffusion model, the method comprising: Multiple source images to be fused are acquired and simultaneously input into the diffusion model. By performing a preset diffusion step on each source image, a noise image with the same size as the source image is obtained. Denoising is performed simultaneously on the noisy image corresponding to each diffusion step until the noisy image corresponding to the 0th diffusion step is determined to be the image fused from multiple source images. The noise reduction process includes: Based on the noise image corresponding to the t-th diffusion step, predict the target image corresponding to the t-th diffusion step; determine the matching degree between the target image corresponding to the t-th diffusion step and the multiple source images corresponding to the t-th diffusion step; correct the noise image corresponding to the t-th diffusion step based on the matching degree corresponding to the t-th diffusion step; predict the corrected noise image corresponding to the t-th diffusion step again using the diffusion model to obtain the noise image corresponding to the (t-1)-th diffusion step, where t is obtained by reversing the order of the diffusion steps that diffuse the source images.

[0009] According to the training-free multi-image fusion method based on a diffusion model provided in this application, the matching degree between the target image corresponding to the t-th diffusion step and the multiple source images corresponding to the t-th diffusion step is determined, including: Input the target image and multiple source images corresponding to the t-th diffusion step into a preset energy function to obtain the gradient change magnitude corresponding to the t-th diffusion step output by the energy function; Calculate the gradient of the energy function with respect to the target image at the t-th diffusion step, where the magnitude of the gradient change and the gradient constitute a gradient vector, which is used to characterize the degree of matching between multiple source images and the target image.

[0010] According to the training-free multi-image fusion method based on a diffusion model provided in the embodiments of this application, the energy function includes: ; in, This represents the energy function, and the corresponding result is the magnitude of the gradient change. Represents n source images, Represents the i-th source image. Indicates the weighting factor. This represents the target image corresponding to the t-th diffusion step.

[0011] According to the training-free multi-image fusion method based on diffusion model provided in the embodiments of this application, the gradient direction of the gradient vector is used to reduce the difference between the target image and the source image.

[0012] According to the training-free multi-image fusion method based on a diffusion model provided in this application, the method predicts the target image corresponding to the t-th diffusion step based on the noisy image corresponding to the t-th diffusion step, including: The noise information in the noise image corresponding to the t-th diffusion step is evaluated using a diffusion model. Input the noise information corresponding to the t-th diffusion step and the noise image corresponding to the t-th diffusion step into the image prediction formula to obtain the target image corresponding to the t-th diffusion step output by the image prediction formula. The image prediction formula includes: ; in, This represents the target image corresponding to the t-th diffusion step. This represents the cumulative variance parameter corresponding to the t-th diffusion step; This represents the noise image corresponding to the t-th diffusion step. This indicates that the t-th diffusion step corresponds to the noise information.

[0013] According to the training-free multi-image fusion method based on diffusion model provided in the embodiments of this application, gradient vectors are used to represent the degree of matching; The noise image corresponding to the t-th diffusion step is corrected based on the matching degree corresponding to the t-th diffusion step, including: Input the gradient vector corresponding to the t-th diffusion step into the preset image correction formula to obtain the corrected noise image corresponding to the t-th diffusion step output by the image correction formula; The image correction formulas include: ; in, This represents the noise image corresponding to the t-th diffusion step after correction. This represents the noise image corresponding to the t-th diffusion step. This represents the scaling factor corresponding to the t-th diffusion step. This represents the gradient corresponding to the t-th diffusion step. This represents the magnitude of the gradient change between multiple source images and the target image corresponding to the t-th diffusion step. Indicates multiple source images. This represents the target image corresponding to the t-th diffusion step. Represents the gradient change vector; in, ; in, Indicates hyperparameters, This represents the cumulative variance parameter corresponding to the t-th diffusion step.

[0014] According to the training-free multi-image fusion method based on a diffusion model provided in the embodiments of this application, the cumulative variance parameter corresponding to the t-th diffusion step is calculated, including: ; in, This represents a preset variable that is monotonically increased with the diffusion step t at a preset initial value.

[0015] According to the training-free multi-image fusion method based on a diffusion model provided in this application, the noise image corresponding to the corrected t-th diffusion step is predicted using a diffusion model to obtain the noise image corresponding to the (t-1)-th diffusion step, including: The new noise information corresponding to the noise image at the t-th diffusion step after correction is evaluated using a diffusion model. Based on the new noise information and the preset unconditional diffusion sampling formula, the noise image corresponding to the (t-1)th diffusion step is obtained.

[0016] This application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps of the diffusion model-based training-free multi-image fusion method as described above.

[0017] This application also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the diffusion model-based training-free multi-image fusion method as described above.

[0018] The training-free multi-image fusion method, device, and medium based on a diffusion model provided in this application acquire multiple source images to be fused, simultaneously input these images into a diffusion model, and perform a preset diffusion step on each source image to obtain a noisy image of the same size as the source image. A denoising process is then performed on the noisy image corresponding to each diffusion step until the noisy image corresponding to the 0th diffusion step is determined as the fused image of the multiple source images. The denoising process includes: predicting the target image corresponding to the tth diffusion step based on the noisy image corresponding to the tth diffusion step; determining the matching degree between the target image corresponding to the tth diffusion step and the multiple source images corresponding to the tth diffusion step; correcting the noisy image corresponding to the tth diffusion step based on the matching degree; and further processing the corrected noisy image corresponding to the tth diffusion step using the diffusion model. The method predicts and obtains the noise image corresponding to the (t-1)th diffusion step, where t is obtained by reversing the diffusion steps that diffuse the source image. In each diffusion step, this application performs denoising processing on multiple noise images simultaneously, which improves the fusion efficiency and effectiveness of the fusion results compared to the serial fusion strategy of the prior art. Furthermore, this application matches the target image and the source image in each diffusion step and corrects the noise image in each diffusion step. By using an alternating sampling method, the optimal balance point between the source image and the noise image is actively determined in each diffusion step, thereby overcoming the "neutralization" problem in the prior art and generating a fused image with rich details, high contrast and clear semantics, thus improving the image fusion effect. Moreover, this application can quickly and accurately obtain the fused image by directly using the existing general model to introduce matching degree calculation and noise image correction in the denoising process. Attached Figure Description

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is one of the flowcharts illustrating the training-free multi-image fusion method based on a diffusion model provided in this application embodiment; Figure 2 This is the second flowchart illustrating the training-free multi-image fusion method based on a diffusion model provided in this application embodiment; Figure 3 This is a schematic diagram illustrating the implementation framework of the ship hull conceptual design method based on typical ship performance provided in this application embodiment; Figure 4This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0021] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0022] This application provides a training-free multi-image fusion method based on a diffusion model. This method can be applied to smart terminals and servers. This application uses the application of this method in a server as an example for illustration, and some other descriptions in the embodiments are illustrative and not intended to limit the scope of protection of this application, and will not be described in detail thereafter. The specific implementation of the method is as follows... Figure 1 As shown: Step 101: Obtain multiple source images to be fused. Input the multiple source images to be fused into the diffusion model at the same time. By performing a preset diffusion step on each source image, a noise image with the same size as the source image is obtained.

[0023] Step 102: Perform a denoising process on the noise image corresponding to each diffusion step simultaneously until the noise image corresponding to the 0th diffusion step is determined to be the image fused from multiple source images.

[0024] The noise reduction process is shown below. Figure 2 : Step 201: Based on the noise image corresponding to the t-th diffusion step, predict the target image corresponding to the t-th diffusion step.

[0025] Step 202: Determine the matching degree between the target image corresponding to the t-th diffusion step and the multiple source images corresponding to the t-th diffusion step.

[0026] Step 203: Correct the noise image corresponding to the t-th diffusion step based on the matching degree corresponding to the t-th diffusion step.

[0027] Step 204: The noise image corresponding to the corrected t-th diffusion step is predicted again using the diffusion model to obtain the noise image corresponding to the (t-1)-th diffusion step.

[0028] Where t is obtained by reversing the diffusion steps of the source image.

[0029] The training-free multi-image fusion method based on a diffusion model provided in this application acquires multiple source images to be fused, simultaneously inputs these source images into a diffusion model, and performs a preset diffusion step on each source image to obtain a noisy image of the same size as the source image. A denoising process is then performed on the noisy image corresponding to each diffusion step until the noisy image corresponding to the 0th diffusion step is determined as the fused image of the multiple source images. The denoising process includes: predicting the target image corresponding to the tth diffusion step based on the noisy image corresponding to the tth diffusion step; determining the matching degree between the target image corresponding to the tth diffusion step and the multiple source images corresponding to the tth diffusion step; correcting the noisy image corresponding to the tth diffusion step based on the matching degree; and predicting the corrected noisy image corresponding to the tth diffusion step again using the diffusion model. The application obtains the noise image corresponding to the (t-1)th diffusion step, where t is obtained by reversing the diffusion steps that diffuse the source image. In each diffusion step, the application performs denoising processing on multiple noise images simultaneously, which improves the fusion efficiency and effectiveness of the fusion results compared to the serial fusion strategy of the prior art. Furthermore, the application matches the target image and the source image in each diffusion step and corrects the noise image in each diffusion step. By using an alternating sampling method, the application actively determines the optimal balance point between the source image and the noise image in each diffusion step, thereby overcoming the "neutralization" problem in the prior art and generating a fused image with rich details, high contrast, and clear semantics, thus improving the image fusion effect. Moreover, the application can quickly and accurately obtain the fused image by directly using the existing general model to introduce matching degree calculation and noise image correction in the denoising process.

[0030] Specifically, the multiple source images to be fused are read to obtain an RGB image set. All source images need to be pre-adjusted to the same size H. W 3.

[0031] Specifically, the diffusion model is used to diffuse to a preset diffusion step, and an initial noise image of the same size as the target image is sampled from a standard Gaussian distribution. ,Right now, This is the noise image corresponding to the preset diffusion step.

[0032] Specifically, for each diffusion step The noise reduction process is executed repeatedly.

[0033] The diffusion step in the diffusion process is: Therefore, t is obtained by reversing the diffusion steps based on the diffusion of the source image.

[0034] In one specific embodiment, the specific implementation of predicting the target image corresponding to the t-th diffusion step based on the noise image corresponding to the t-th diffusion step includes: The noise information in the noisy image corresponding to the t-th diffusion step is evaluated using a diffusion model. The noise information corresponding to the t-th diffusion step and the noisy image corresponding to the t-th diffusion step are input into the image prediction formula to obtain the target image corresponding to the t-th diffusion step, which is output by the image prediction formula.

[0035] The image prediction formula is shown in formula (1): ...(1) in, This represents the target image corresponding to the t-th diffusion step. This represents the cumulative variance parameter corresponding to the t-th diffusion step; This represents the noise image corresponding to the t-th diffusion step. This indicates that the t-th diffusion step corresponds to the noise information.

[0036] The target image is the predicted fused image corresponding to the 0th diffusion step predicted at the tth diffusion step.

[0037] The noise information being evaluated is the noise that needs to be removed from the noise image corresponding to the t-th diffusion step. This is evaluated using a score estimator based on a U-Net structure within the diffusion model, which is pre-trained using sample data.

[0038] In one specific embodiment, the specific implementation of calculating the cumulative variance parameter corresponding to the t-th diffusion step includes: As the denoising process progresses, the calculation of the diffusion steps from the 1st to the tth step is performed. The cumulative product, that is .

[0039] in, This represents a preset variable that is monotonically increased with the diffusion step t at a preset initial value.

[0040] in, It is a variable that is between 0 and 1 and increases monotonically with the diffusion step t.

[0041] For example, setting T=1000 presets the initial value. Preset termination value , exist arrive Linear interpolation is performed between them.

[0042] In one specific embodiment, the specific implementation of determining the matching degree between the target image corresponding to the t-th diffusion step and the multiple source images corresponding to the t-th diffusion step includes: Input the target image and multiple source images corresponding to the t-th diffusion step into a preset energy function to obtain the gradient change magnitude of the energy function output corresponding to the t-th diffusion step. Calculate the gradient of the energy function with respect to the target image corresponding to the t-th diffusion step.

[0043] Specifically, the energy function is calculated with respect to... The gradient.

[0044] The gradient vector is composed of the magnitude of the gradient change and the gradient itself. The gradient vector is used to characterize the degree of matching between multiple source images and the target image.

[0045] The energy function is given by formula (2): ...(2) in, This represents the energy function, and the corresponding result is the magnitude of the gradient change. Represents n source images, Represents the i-th source image. Indicates the weighting factor. This represents the target image corresponding to the t-th diffusion step.

[0046] Specifically, the gradient direction of the gradient vector is used to reduce the difference between the target image and the source image.

[0047] Specifically, an energy function independent of the specific fusion task (such as infrared and visible light image fusion task, or multi-focus image fusion task) is created in advance. The L2 norm of the average deviation between the target image and all source images is calculated through this energy function to quantify the compatibility between the two.

[0048] In the diffusion step iterative calculation process, the energy function simultaneously receives all source images for processing, which is a symmetrical aggregation of all source images. Its gradient calculation naturally supports parallel processing. Therefore, the processing of a diffusion step is a single complete diffusion trajectory, without the need for serial fusion processing or consideration of the image input order, thus achieving fast and efficient multi-input fusion.

[0049] In a specific embodiment, the specific implementation of correcting the noisy image corresponding to the t-th diffusion step based on the matching degree corresponding to the t-th diffusion step includes: Input the gradient vector corresponding to the t-th diffusion step into the preset image correction formula to obtain the corrected noise image corresponding to the t-th diffusion step output by the image correction formula.

[0050] The image correction formula is shown in formula (3): ... (3) in, This represents the noise image corresponding to the t-th diffusion step after correction. This represents the noise image corresponding to the t-th diffusion step. This represents the scaling factor corresponding to the t-th diffusion step. This represents the gradient corresponding to the t-th diffusion step. This represents the magnitude of the gradient change between multiple source images and the target image corresponding to the t-th diffusion step. Indicates multiple source images. This represents the target image corresponding to the t-th diffusion step. This represents the gradient change vector.

[0051] in, .

[0052] in, Indicates hyperparameters, This represents the cumulative variance parameter corresponding to the t-th diffusion step.

[0053] Different diffusion steps correspond to different scaling factors, and different diffusion steps correspond to different cumulative variance parameters.

[0054] In one specific embodiment, the modified Right now The data is then fed back into the fraction estimator, which re-evaluates the noise information and calculates the noise image for the next diffusion step based on standard unconditional diffusion sampling formulas (such as the update rules for DDPM or DDIM). .

[0055] The standard diffusion sampling formula is shown in formula (4): ... (4) in, .

[0056] in, This represents the noise image corresponding to the next diffusion step (t+1 diffusion step). This represents a pre-defined variable. This represents standard Gaussian noise.

[0057] Below, through Figure 3 The overall framework of this application is explained as follows: This application takes n source images as input, and the diffusion model includes two core parts: energy-guided sampling and unconditional diffusion sampling.

[0058] In this system, a task-independent energy function serves as a conditional guide, and the CooED mechanism (corresponding to the denoising process) achieves collaborative optimization of the source domain and natural priors through alternating sampling, resulting in the output... This is the final fused image.

[0059] To better illustrate this invention, the following detailed explanation uses publicly available IVIF, MFIF, and MEIF datasets as examples.

[0060] Example 1: Model Configuration and Experiment Setup This application was fully validated on the following six public datasets: MSRS and M3FD for infrared-visible image fusion (IVIF); Lytro and MFI-WHU for multi-focus image fusion (MFIF); and MEFB and SICE for multi-exposure image fusion (MEIF). All input images were resized to 256×256 pixels. This application uses the DDPM model pre-trained on ImageNet as the base diffusion model, with U-Net as its backbone network. The total number of diffusion steps T was set to 1000, but DDIM was used for acceleration during inference, resulting in an actual sampling number of 50 steps. The global hyperparameter λ was uniformly set to 1.0. Table 1 summarizes the detailed experimental parameters.

[0061]

[0062] Table 1 Experimental Parameter Settings To more intuitively demonstrate the advantages of this application in terms of detail preservation, edge sharpness, and overall visual consistency, we will take the Lytro dataset as an example to analyze in depth the performance of the proposed method in image fusion tasks.

[0063] Example 2: Detailed Implementation of the Cooperative Optimization Mechanism (CooED) At each diffusion step t, the system first utilizes the current noise image. Predicting a preliminary clean image Next, the energy function immediately intervenes, calculating a gradient vector that points in the direction that reduces the average difference between the predicted image and all source images. This gradient is used to correct the original noise. This yields an intermediate state that is closer to the source image. However, at this time This may not conform to the statistical regularities of natural images. Therefore, in the second step, the system will... The data is then fed back into the pre-trained U-Net. Leveraging its knowledge learned from massive amounts of natural images, U-Net... The process involves "purifying" and "enhancing" the image to output a new noise that retains key information from the original image while possessing high-quality, natural image characteristics. This process is repeated over 1000 time steps (or 50 after acceleration), eventually "growing" a perfectly fused image from pure noise.

[0064] Example 3: Advantages Analysis of Energy-Guided Parallel Fusion Strategy (Eg-PFS) The Eg-PFS strategy in this application will include all source images Simultaneously, the data is fed into the energy function. The summation operation inside the energy function naturally aggregates information from all inputs, and its output gradient is the result of the combined effect of all source images. This means that regardless of whether n is 2 or 100, the diffusion model runs only once during the entire sampling process, with a constant time complexity. Meanwhile, since the summation operation satisfies the commutative law, changing the order of the input images will not change the final gradient direction, thus ensuring the consistency of the results.

[0065] Example 4: Ablation Experiment and Performance Comparison To verify the effectiveness of each component of this invention, an ablation study was conducted on the proposed Cooperative Energy Guided Denoising (CooED) module on the Lytro dataset. The results are shown in Table 2. Two sets of ablation experiments were specifically designed: "w / o refined score": Preserves the energy-guided mechanism but does not introduce a refined score estimate. That is, the original fractional function of DDPM is still used; “w / o co-optimization”: Uses the modified score, but disables the co-optimization process between the image and the score, degenerating into a one-way energy-guided strategy.

[0066] As shown in Table 2, any variant leads to a significant decrease in fusion performance, indicating that both the modified score estimation and the collaborative optimization mechanism make crucial contributions to the overall performance. Our application (CDFM) achieves optimal results across all evaluation metrics, fully validating the effectiveness and necessity of the proposed CooED module.

[0067]

[0068] Table 2 Ablation Experiment Results Furthermore, this application compares its performance with state-of-the-art general image fusion methods on the Lytro dataset, including training-required methods (such as U2Fusion) and training-free methods (such as DeFusion and DDFM). As shown in Table 3, this invention (CDFM) achieves best or highly competitive performance in the vast majority of cases, especially leading across the board among training-free methods, fully demonstrating its superiority.

[0069]

[0070] Table 3. Performance Comparison of Existing Techniques and the Present Application This application provides a flexible and training-free guiding signal for general image fusion by designing a concise energy function that directly affects pixel intensity. Secondly, the COOED mechanism decomposes the diffusion sampling process into two alternating sub-steps, enabling the diffusion model to actively extract high-quality natural image knowledge from a pre-defined model while adhering to source image conditions. This achieves one-time, parallel processing of all input images, fundamentally solving the efficiency and order sensitivity issues.

[0071] This application completely eliminates the dependence on any dedicated image fusion dataset. The entire framework requires only a diffusion model pre-trained on general datasets such as ImageNet, which can be directly applied to various novel fusion tasks, greatly reducing the deployment threshold and maintenance costs of the model.

[0072] This application employs an alternating sampling method, enabling the model to proactively seek the optimal balance between "faithfulness to the source image" and "naturalness" at each step. This mechanism effectively activates the natural image priors in the pre-trained model and transforms them into the ability to solve specific fusion tasks, thereby fundamentally overcoming the "neutralization" phenomenon and generating images with rich details, high contrast, and clear semantics.

[0073] This application makes the computational complexity of fusing n images almost the same as that of fusing 2 images, and the inference speed is extremely fast. More importantly, because the energy function treats all inputs equally, the final fusion result remains highly consistent regardless of the order in which the user provides the source images, ensuring the stability and reliability of the system in practical applications.

[0074] This application demonstrates excellent performance in terms of accuracy and visual quality across a variety of image fusion tasks.

[0075] Figure 4 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 4 As shown, the electronic device may include a processor 401, a communications interface 402, a memory 403, and a communication bus 404. The processor 401, communications interface 402, and memory 403 communicate with each other via the communication bus 404. The processor 401 can call logical instructions from the memory 403 to execute a training-free multi-image fusion method based on a diffusion model.

[0076] Furthermore, the logical instructions in the aforementioned memory 403 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0077] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program stored on a non-transitory computer-readable storage medium, the computer program including program instructions, and when the program instructions are executed by a computer, the computer is able to execute the training-free multi-image fusion method based on the diffusion model provided by the above methods.

[0078] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to perform the training-free multi-image fusion method based on the diffusion model provided in the above embodiments.

[0079] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0080] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0081] Finally, it should be noted that the above descriptions are merely preferred embodiments of this application, and this application is not limited to the above embodiments. It is understood that other improvements and variations directly derived or conceived by those skilled in the art without departing from the spirit and concept of this application should be considered to be included within the protection scope of this application.

Claims

1. A training-free multi-image fusion method based on a diffusion model, characterized in that, The method includes: Multiple source images to be fused are acquired and simultaneously input into the diffusion model. By performing a preset diffusion step on each source image, a noise image with the same size as the source image is obtained. Denoising is performed simultaneously on the noisy image corresponding to each diffusion step until the noisy image corresponding to the 0th diffusion step is determined to be the image fused from multiple source images. The noise reduction process includes: Based on the noise image corresponding to the t-th diffusion step, predict the target image corresponding to the t-th diffusion step; determine the matching degree between the target image corresponding to the t-th diffusion step and the multiple source images corresponding to the t-th diffusion step; correct the noise image corresponding to the t-th diffusion step based on the matching degree corresponding to the t-th diffusion step; predict the corrected noise image corresponding to the t-th diffusion step again using the diffusion model to obtain the noise image corresponding to the (t-1)-th diffusion step, where t is obtained by reversing the order of the diffusion steps that diffuse the source images.

2. The training-free multi-image fusion method based on a diffusion model according to claim 1, characterized in that, Determine the matching degree between the target image at the t-th diffusion step and the multiple source images at the t-th diffusion step, including: Input the target image and multiple source images corresponding to the t-th diffusion step into a preset energy function to obtain the gradient change magnitude corresponding to the t-th diffusion step output by the energy function; Calculate the gradient of the energy function with respect to the target image at the t-th diffusion step, where the magnitude of the gradient change and the gradient constitute a gradient vector, which is used to characterize the degree of matching between multiple source images and the target image.

3. The training-free multi-image fusion method based on a diffusion model according to claim 2, characterized in that, The energy function includes: ； in, This represents the energy function, and the corresponding result is the magnitude of the gradient change. Represents n source images, Represents the i-th source image. Indicates the weighting factor. This represents the target image corresponding to the t-th diffusion step.

4. The training-free multi-image fusion method based on a diffusion model according to claim 2, characterized in that, The gradient direction of the gradient vector is used to reduce the difference between the target image and the source image.

5. The training-free multi-image fusion method based on a diffusion model according to any one of claims 1-4, characterized in that, Based on the noisy image corresponding to the t-th diffusion step, predict the target image corresponding to the t-th diffusion step, including: The noise information in the noise image corresponding to the t-th diffusion step is evaluated using a diffusion model. Input the noise information corresponding to the t-th diffusion step and the noise image corresponding to the t-th diffusion step into the image prediction formula to obtain the target image corresponding to the t-th diffusion step output by the image prediction formula. The image prediction formula includes: ； in, This represents the target image corresponding to the t-th diffusion step. This represents the cumulative variance parameter corresponding to the t-th diffusion step; This represents the noise image corresponding to the t-th diffusion step. This indicates that the t-th diffusion step corresponds to the noise information.

6. The training-free multi-image fusion method based on a diffusion model according to any one of claims 1-4, characterized in that, The gradient vector is used to represent the degree of matching. The noise image corresponding to the t-th diffusion step is corrected based on the matching degree corresponding to the t-th diffusion step, including: Input the gradient vector corresponding to the t-th diffusion step into the preset image correction formula to obtain the corrected noise image corresponding to the t-th diffusion step output by the image correction formula; The image correction formulas include: ； in, This represents the noise image corresponding to the t-th diffusion step after correction. This represents the noise image corresponding to the t-th diffusion step. This represents the scaling factor corresponding to the t-th diffusion step. This represents the gradient corresponding to the t-th diffusion step. This represents the magnitude of the gradient change between multiple source images and the target image corresponding to the t-th diffusion step. Indicates multiple source images. This represents the target image corresponding to the t-th diffusion step. Represents the gradient change vector; in, ; in, Indicates hyperparameters, This represents the cumulative variance parameter corresponding to the t-th diffusion step.

7. The training-free multi-image fusion method based on a diffusion model according to claim 6, characterized in that, Calculate the cumulative variance parameter corresponding to the t-th diffusion step, including: ； in, This represents a preset variable that is monotonically increased with the diffusion step t at a preset initial value.

8. The training-free multi-image fusion method based on a diffusion model according to any one of claims 1-4, characterized in that, The noise image corresponding to the corrected t-th diffusion step is predicted using a diffusion model to obtain the noise image corresponding to the (t-1)-th diffusion step, including: The new noise information corresponding to the noise image at the t-th diffusion step after correction is evaluated using a diffusion model. Based on the new noise information and the preset unconditional diffusion sampling formula, the noise image corresponding to the (t-1)th diffusion step is obtained.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the training-free multi-image fusion method based on the diffusion model as described in any one of claims 1 to 8.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the computer program implements the steps of the training-free multi-image fusion method based on a diffusion model as described in any one of claims 1 to 8.

Images