Multimodal Image Fusion and Super-Resolution Method Based on Conditional Diffusion Probability Model

Through the multimodal image fusion and super-resolution method based on the conditional diffusion probability model, the problem of generating high-quality fusion images on low-resolution source images is solved, and the recovery of source image details and texture information is realized, and the method is improved practicality.

CN118314022BActive Publication Date: 2024-11-19FOSHAN UNIVERSITY
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410747138.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-06-11
Publication Date
2024-11-19
Estimated Expiration
2044-06-11

AI Technical Summary

Technical Problem

The existing multimodal image fusion method is difficult to generate high-quality fusion images on low-resolution source images in real scenes, and it is impossible to effectively restore the degraded details and texture information in the source images.

Method used

Using the multimodal image fusion and super-resolution method based on the conditional diffusion probability model, a conditional diffusion probability model including the multimodal information fusion module and iterative denoising module is constructed by fuzzing and fusion processing on multiple groups of source images. The model is trained to recover details and texture information in low-resolution source images.

Benefits of technology

It realizes the generation of higher resolution fusion images on low-resolution source images, restores details and texture information in the source images, and improves the practicality of the method.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118314022B_ABST
    Figure CN118314022B_ABST
Patent Text Reader

Abstract

The present application belongs to the field of image processing technology, and discloses a multimodal image fusion and super-resolution method based on a conditional diffusion probability model, which performs blurring and fusion processing on a multimodal source image to obtain a blurred image and a true value fused image, which are used to train a conditional diffusion probability model including a multimodal information fusion module and an iterative denoising module, so that the conditional diffusion probability model can fuse the multimodal images and complete iterative denoising at the same time, obtain a fused image with a higher resolution than the source image, and restore the degraded details and texture information in the source image, and has high practicality.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multimodal image fusion and super-resolution method based on a conditional diffusion probability model. Background Art

[0002] Multimodal image fusion integrates feature information from multiple sensors with different modal source images to generate a fused image with cross-modal information, which is widely used in semantic segmentation, target detection and other fields. Among them, the purpose of fusion of infrared images and visible light images is to retain the high-spatial-resolution texture information in visible light images and the thermal target information in infrared images that is insensitive to light, which is widely used in application fields such as drone reconnaissance and autonomous driving.

[0003] However, most of the existing multimodal image fusion methods work on high-resolution source images. In real scenes, due to adverse environmental conditions and imaging system performance, high-resolution source images are sometimes not available, which limits the practicality of existing multimodal image fusion methods in real scenes. Therefore, it is necessary to find a method that can generate high-quality fused images on multimodal low-resolution source images to restore the degraded details and texture information in the source images. Summary of the invention

[0004] The purpose of this application is to provide a multimodal image fusion and super-resolution method based on a conditional diffusion probability model, which can generate a relatively high-resolution fused image based on a relatively low-resolution source image to restore the degraded details and texture information in the source image.

[0005] In a first aspect, the present application provides a multimodal image fusion and super-resolution method based on a conditional diffusion probability model, comprising the steps of:

[0006] A1. Acquire multiple sets of source images for training; each set of source images includes a registered first modality image and a second modality image;

[0007] A2. performing blur processing and fusion processing on each group of source images to obtain corresponding blurred first modality images, blurred second modality images and true value fusion images;

[0008] A3. constructing a conditional diffusion probability model, and training the conditional diffusion probability model using the blurred first modality image, the blurred second modality image and the true value fusion image corresponding to each group of the source images to obtain a trained conditional diffusion probability model; the conditional diffusion probability model includes a multimodal information fusion module and an iterative denoising module;

[0009] A4. Using the trained conditional diffusion probability model, fuse the registered first modality image to be fused and the second modality image to be fused.

[0010] This method blurs and fuses the multimodal source images in the training set (i.e., the image set composed of source images used for training) to obtain blurred images and true value fused images, which are used to train the conditional diffusion probability model including the multimodal information fusion module and the iterative denoising module, so that the conditional diffusion probability model can jointly fuse and super-resolution the multimodal images to obtain a high-resolution fused image, and restore the degraded details and texture information in the low-resolution source image, which has high practicality.

[0011] Preferably, the first modality image and the first modality image to be fused are visible light images, and the second modality image and the second modality image to be fused are infrared images.

[0012] Preferably, in step A2, blurring each group of source images includes:

[0013] Downsampling the first modality image and the second modality image to obtain a corresponding first reduced-resolution image and a second reduced-resolution image;

[0014] The first reduced-resolution image and the second reduced-resolution image are upsampled to obtain corresponding blurred first modality images and blurred second modality images; the pixel size of the blurred first modality image is the same as the pixel size of the corresponding first modality image, and the pixel size of the blurred second modality image is the same as the pixel size of the corresponding second modality image.

[0015] After downsampling and upsampling, a blurred image with the same pixel size as the source image is obtained, which can effectively reduce the resolution of the blurred image and ensure that the pixel size of the blurred image and the true value fusion image is the same, so as to facilitate the subsequent training of the conditional diffusion probability model.

[0016] Preferably, in step A2, fusing each group of source images comprises:

[0017] The first modality image and the second modality image in the same group of the source images are fused using a preset image fusion algorithm to obtain the corresponding true value fused image.

[0018] The true value fusion image obtained by directly fusing the source images through the existing fusion method has a higher resolution than the blurred image. The conditional diffusion probability model is trained using the blurred image and the true value fusion image. The conditional diffusion probability model fusion based on the multimodal information fusion module can be used to simultaneously achieve image fusion and super-resolution.

[0019] Preferably, the multimodal information fusion module includes a first two-dimensional convolutional layer, the convolution kernel size of the first two-dimensional convolutional layer is 3×3, and the step size is 1.

[0020] Preferably, the iterative denoising module comprises a downsampling module, an intermediate module, an upsampling module and a first submodule connected in sequence;

[0021] The down-sampling module includes a plurality of first residual modules, a plurality of second residual modules, a plurality of third residual modules, a plurality of fourth residual modules and a plurality of down-sampling convolution blocks; the intermediate module includes a plurality of first residual modules and a plurality of second residual modules; the up-sampling module includes a plurality of third residual modules, a plurality of fourth residual modules and a plurality of up-sampling convolution blocks; the down-sampling module and the up-sampling module form a U-Net structure;

[0022] The first residual module includes a first submodule, a feature dimension refinement layer, and a second submodule connected in sequence, and also includes a first identity activation function layer, in which the input end of the first identity activation function layer is connected to the input end of the first submodule, and the output of the first identity activation function layer is added to the output of the second submodule as the output of the first residual module;

[0023] The second residual module includes a first residual module and a self-attention module connected in sequence;

[0024] The third residual module includes a first submodule, a feature dimension refinement layer, and a second submodule connected in sequence, and also includes a second two-dimensional convolutional layer, in which the input end of the second two-dimensional convolutional layer is connected to the input end of the first submodule, and the output of the second two-dimensional convolutional layer is added to the output of the second submodule as the output of the third residual module;

[0025] The fourth residual module includes a third residual module and a self-attention module connected in sequence;

[0026] The first submodule includes a GroupNorm normalization function layer, a Swish activation function layer, a second Identity activation function layer and a third two-dimensional convolutional layer connected in sequence;

[0027] The second submodule includes a GroupNorm normalization function layer, a Swish activation function layer, a Dropout regularization function layer and a third two-dimensional convolution layer which are connected in sequence.

[0028] Preferably, the downsampling convolution block is a two-dimensional convolution layer with a convolution kernel size of 3×3 and a step size of 2;

[0029] The upsampling convolution block is a two-dimensional convolution layer with a convolution kernel size of 3×3 and a step size of 1.

[0030] Preferably, in step A3, the conditional diffusion probability model is trained using the blurred first modality image, the blurred second modality image and the true value fusion image corresponding to each group of the source images to obtain a trained conditional diffusion probability model, including:

[0031] A301. Using the blurred first modality image, the blurred second modality image and the true value fusion image corresponding to each group of the source images, the conditional diffusion probability model is trained by iteratively adding noise to the forward diffusion process and the back propagation process to obtain the corresponding prediction noise;

[0032] A302. Calculate a loss function based on the iteratively added noise and the predicted noise, and update the model parameters of the conditional diffusion probability model based on the loss function;

[0033] A303. Repeat steps A301 and A302 until the loss function converges.

[0034] Preferably, in step A302, the loss function is calculated according to the following formula:

[0035] ;

[0036] in, is the loss function, is the true value fusion image, x is the blurred first modality image, y is the blurred second modality image, is Gaussian noise, is the noise variance, is the noise predictor function, and They are all expected functions.

[0037] Preferably, step A4 comprises:

[0038] The registered first modality image to be fused and the second modality image to be fused are input into the trained conditional diffusion probability model, and after Gaussian noise is initialized by the trained conditional diffusion probability model, the final fused image is obtained by iteratively denoising based on the initialized Gaussian noise according to the reverse diffusion process.

[0039] Beneficial effect: The multimodal image fusion and super-resolution method based on the conditional diffusion probability model provided in the present application performs blurring and fusion processing on the multimodal source images to obtain blurred images and true value fused images, which are used to train the conditional diffusion probability model including the multimodal information fusion module and the iterative denoising module, so that the conditional diffusion probability model can fuse the multimodal images and complete the iterative denoising at the same time, obtain the fused image with higher resolution than the source image, and restore the degraded details and texture information in the source image, and has high practicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 A flowchart of a multimodal image fusion and super-resolution method based on a conditional diffusion probability model provided in an embodiment of the present application.

[0041] Figure 2 This is a structural diagram of the conditional diffusion probability model.

[0042] Figure 3 Schematic diagram of the structure of the iterative denoising module.

[0043] Figure 4 Schematic diagram of the structure of the first residual module.

[0044] Figure 5 Schematic diagram of the structure of the second residual module.

[0045] Figure 6 Schematic diagram of the structure of the third residual module.

[0046] Figure 7 Schematic diagram of the structure of the fourth residual module.

[0047] Figure 8 It is a schematic diagram of the structure of the first submodule.

[0048] Fig. 9 It is a structural diagram of the second submodule.

[0049] Fig.10 This is the first set of comparison pictures.

[0050] Fig.11 This is the second set of comparison pictures.

[0051] Explanation of numbers: 1. Multimodal information fusion module; 2. Iterative denoising module; 3. Downsampling module; 4. Intermediate module; 5. Upsampling module; 6. First residual module; 7. Second residual module; 8. Third residual module; 9. Fourth residual module; 10. Downsampling convolution block; 11. Upsampling convolution block. DETAILED DESCRIPTION

[0052] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0053] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0054] Please refer to Figure 1 , Figure 2 , Figure 1 A multimodal image fusion and super-resolution method based on a conditional diffusion probability model in some embodiments of the present application comprises the steps of:

[0055] A1. Acquire multiple sets of source images for training; each set of source images includes a registered first modality image and a second modality image;

[0056] A2. performing fuzzy processing and fusion processing on each group of source images to obtain corresponding fuzzy first modality images, fuzzy second modality images and true value fusion images;

[0057] A3. construct a conditional diffusion probability model, and train the conditional diffusion probability model using the blurred first modality image, blurred second modality image and true value fusion image corresponding to each group of source images to obtain a trained conditional diffusion probability model; the conditional diffusion probability model includes a multimodal information fusion module 1 and an iterative denoising module 2;

[0058] A4. Use the trained conditional diffusion probability model to fuse the registered first modality image to be fused and the second modality image to be fused.

[0059] The method performs blurring and fusion processing on the multimodal source images used for training to obtain blurred images (blurred first modality images and blurred second modality images) and true value fusion images, which are used to train the conditional diffusion probability model including the multimodal information fusion module 1 and the iterative denoising module 2, so that the conditional diffusion probability model can simultaneously fuse and super-resolution the multimodal images and restore the degraded details and texture information in the low-resolution source image (the resolution is a relative probability, and the low-resolution source image here refers to the source image whose resolution is lower than the resolution of the fusion result), and has high practicality.

[0060] In some embodiments, the first modality image and the first modality image to be fused are visible light images, and the second modality image and the second modality image to be fused are infrared images. However, the first modality image and the first modality image to be fused are not limited to visible light images, and the second modality image and the second modality image to be fused are not limited to infrared images, but may also be images of other wavelength bands.

[0061] Specifically, in step A2, blurring each group of source images includes:

[0062] Downsampling the first modality image and the second modality image to obtain corresponding first reduced-resolution image and second reduced-resolution image;

[0063] The first down-resolution image and the second down-resolution image are up-sampled to obtain a corresponding blurred first modality image and a blurred second modality image; the pixel size of the blurred first modality image is the same as the pixel size of the corresponding first modality image, and the pixel size of the blurred second modality image is the same as the pixel size of the corresponding second modality image.

[0064] For example, the first modality image and the second modality image are first downsampled according to the magnification factor N (which is a preset positive integer greater than 1) to obtain the corresponding first reduced-resolution image and second reduced-resolution image, and then the first reduced-resolution image and the second reduced-resolution image are upsampled N times to obtain a blurred first modality image and a blurred second modality image with the same pixel size as the source image.

[0065] After downsampling and upsampling, a blurred image with the same pixel size as the source image is obtained, which can effectively reduce the resolution of the blurred image and ensure that the pixel size of the blurred image and the true value fusion image is the same, so as to facilitate the subsequent training of the conditional diffusion probability model.

[0066] Furthermore, in step A2, the fusion processing of each group of source images includes:

[0067] The first modality image and the second modality image in the same group of source images are fused using a preset image fusion algorithm to obtain a corresponding true value fused image.

[0068] The true value fusion image obtained directly by fusing the source images has a higher resolution than the blurred image. Using the blurred image and the true value fusion image to train the conditional diffusion probability model can ensure that the conditional diffusion probability model jointly realizes image fusion and super-resolution.

[0069] That is, after the above-mentioned blurring and fusion processing, a true fused image with a resolution equivalent to that of the source image and a blurred image with a resolution lower than that of the source image can be obtained from a source image with a certain resolution. The blurred image and the true fused image are used as training set data. After training the conditional diffusion probability model, it can be ensured that the resolution of the fused image output by the conditional diffusion probability model is higher than the multimodal image input therein, thereby achieving image fusion and improving image resolution at the same time.

[0070] The preset image fusion algorithm may be selected as an image fusion algorithm in the prior art as needed, for example, the CDDFuse image fusion algorithm, but is not limited thereto.

[0071] The multimodal information fusion module 1 is used to fuse the input multimodal images, and the iterative denoising module 2 is used to iteratively denoise the fusion feature map obtained by the multimodal information fusion module 1.

[0072] In this embodiment, the multimodal information fusion module 1 includes a first two-dimensional convolution layer, the convolution kernel size of the first two-dimensional convolution layer is 3×3, and the step size is 1. Thus, independent modal information in different modal images can be extracted for fusion.

[0073] Further, see Figure 3 , the iterative denoising module 2 includes a downsampling module 3, an intermediate module 4, an upsampling module 5 and a first submodule connected in sequence;

[0074] The downsampling module 3 includes a plurality of first residual modules 6, a plurality of second residual modules 7, a plurality of third residual modules 8, a plurality of fourth residual modules 9 and a plurality of downsampling convolution blocks 10; the intermediate module 4 includes a plurality of first residual modules 6 and a plurality of second residual modules 7; the upsampling module 5 includes a plurality of third residual modules 8, a plurality of fourth residual modules 9 and a plurality of upsampling convolution blocks 11; the downsampling module 3 and the upsampling module 5 form a U-Net structure;

[0075] The first residual module 6 includes a first submodule, a feature dimension refinement layer, and a second submodule connected in sequence, and also includes a first identity activation function layer. In the first residual module 6, the input end of the first identity activation function layer is connected to the input end of the first submodule, and the output of the first identity activation function layer is added to the output of the second submodule as the output of the first residual module (such as Figure 4 shown);

[0076] The second residual module 7 includes the first residual module 6 and the self-attention module (such as Figure 5 shown);

[0077] The third residual module 8 includes a first submodule, a feature dimension refinement layer, and a second submodule connected in sequence, and also includes a second two-dimensional convolutional layer. In the third residual module 8, the input end of the second two-dimensional convolutional layer is connected to the input end of the first submodule, and the output of the second two-dimensional convolutional layer is added to the output of the second submodule as the output of the third residual module 8 (such as Figure 6 shown);

[0078] The fourth residual module 9 includes the third residual module 8 and the self-attention module (such as Figure 7 shown);

[0079] The first submodule includes a GroupNorm normalization function layer, a Swish activation function layer, a second Identity activation function layer, and a third two-dimensional convolutional layer (such as Figure 8 shown);

[0080] The second submodule includes a GroupNorm normalization function layer, a Swish activation function layer, a Dropout regularization function layer, and a third two-dimensional convolutional layer (such as Fig. 9 as shown).

[0081] Among them, the number of the first residual module 6, the second residual module 7, the third residual module 8, the fourth residual module 9 and the downsampling convolution block 10 in the downsampling module 3 can be set according to actual needs, and the modules in the downsampling module 3 are connected in series in sequence, wherein each downsampling convolution block 10 is connected after at least one group of other modules of the downsampling module 3, and each group of other modules of the downsampling module 3 includes at least one other module in the downsampling module 3 except the downsampling convolution block 10; the serial connection order of the modules in the downsampling module 3 can be set according to actual needs.

[0082] Among them, the number of first residual modules 6 and second residual modules 7 in the intermediate module 4 can be set according to actual needs, and the modules in the intermediate module 4 are connected in series in sequence, wherein the series connection order of the modules in the intermediate module 4 can be set according to actual needs.

[0083] Among them, the number of the third residual module 8, the fourth residual module 9 and the upsampling convolution block 11 in the upsampling module 5 can be set according to actual needs, and the modules in the upsampling module 5 are connected in series in sequence, wherein each upsampling convolution block 11 is connected to at least one group of other modules in the upsampling module 5, and each group of other modules in the upsampling module 5 includes other modules other than at least one upsampling convolution block 11 in the upsampling module 5; the serial connection order of the modules in the upsampling module 5 can be set according to actual needs.

[0084] For example Figure 3 In the iterative denoising module 2, the downsampling module 3 includes six first residual modules 6, one second residual module 7, two third residual modules 8, one fourth residual module 9 and four downsampling convolution blocks 10, which are connected in the following order from front to back: the first residual module 6, the first residual module 6, the downsampling convolution block 10, the third residual module 8, the first residual module 6, the downsampling convolution block 10, the third residual module 8, the first residual module 6, the downsampling convolution block 10, the fourth residual module 9, the second residual module 7, the downsampling convolution block 10, the first residual module 6, and the first residual module 6;

[0085] The intermediate module 4 includes a second residual module 7 and a first residual module 6, and these modules are connected in sequence from front to back in the following order: the second residual module 7, the first residual module 6;

[0086] The upsampling module 5 includes eleven third residual modules 8, three fourth residual modules 9 and four upsampling convolution blocks 11, which are connected in the following order from front to back: third residual module 8, third residual module 8, third residual module 8, upsampling convolution block 11, fourth residual module 9, fourth residual module 9, fourth residual module 9, upsampling convolution block 11, third residual module 8, third residual module 8, third residual module 8, upsampling convolution block 11, third residual module 8, third residual module 8, third residual module 8, upsampling convolution block 11, third residual module 8, third residual module 8, third residual module 8, upsampling convolution block 11, third residual module 8, third residual module 8, third residual module 8, upsampling convolution block 11, third residual module 8, third residual module 8;

[0087] Among them, the modules in the downsampling module 3 are sorted from front to back (there are a total of 14 modules in the downsampling module 3), and the modules in the upsampling module 5 except the upsampling convolution block 11 are sorted from front to back (there are a total of 14 modules in the upsampling module 5 except the upsampling convolution block 11), the i-th module in the downsampling module 3 is jump-connected with the 15-i-th module in the upsampling module 5 except the upsampling convolution block 11, i=1,2,…,14.

[0088] Preferably, the downsampling convolution block 10 is a two-dimensional convolution layer with a convolution kernel size of 3×3 and a step size of 2; the upsampling convolution block 11 is a two-dimensional convolution layer with a convolution kernel size of 3×3 and a step size of 1.

[0089] By adopting the iterative denoising module 2, it is possible to ensure that the image is effectively denoised, thereby effectively restoring the degraded details, texture and other information in the source image.

[0090] Specifically, in step A3, the conditional diffusion probability model is trained using the blurred first modality image, the blurred second modality image and the true value fusion image corresponding to each group of source images to obtain a trained conditional diffusion probability model, including:

[0091] A301. Using the blurred first modality image, blurred second modality image and true value fusion image corresponding to each group of source images, the conditional diffusion probability model is trained by iteratively adding noise in the forward diffusion process and the back propagation process to obtain the corresponding prediction noise;

[0092] A302. Calculate a loss function based on the iteratively added noise and the predicted noise, and update the model parameters of the conditional diffusion probability model based on the loss function;

[0093] A303. Repeat steps A301 and A302 until the loss function converges.

[0094] After the blurred first modality image, the blurred second modality image and the true value fusion image are input into the conditional diffusion probability model, the multimodal information fusion module 1 integrates the features of the blurred first modality image and the blurred second modality image, and the integrated features (i.e., the fusion feature map) and the true value fusion image are input into the iterative denoising module 2. The iterative denoising module 2 gradually adds noise (generally Gaussian noise) to the true value fusion image, and the iterative denoising module 2 predicts the noise added each time, and calculates the loss function based on the real noise and the predicted noise added to the true value fusion image, and then updates the model parameters of the iterative denoising module 2 according to the loss function. The model parameters of the iterative denoising module 2 can be updated according to the loss function by using the existing technology, which will not be described in detail here.

[0095] Generally, before training begins, the model parameters of the conditional diffusion probability model need to be initialized, that is, before step A301, the step A300 is also included: Initialize the model parameters of the conditional diffusion probability model. Among them, the Kaiming initialization method can be used to initialize the model parameters of the conditional diffusion probability model (the Kaiming initialization method is a prior art and is not described in detail here), but is not limited to this.

[0096] In step A302, the loss function calculation formula in the prior art may be selected to calculate the loss function according to actual needs, or the loss function may be calculated using the following formula:

[0097] ;

[0098] in, is the loss function, is the true value fusion image, x is the blurred first modality image, y is the blurred second modality image, is Gaussian noise, is the noise variance, is the noise predictor function (thus, is the output of the noise predictor, i.e. the predicted noise), and They are all expected functions.

[0099] Specifically, step A4 includes:

[0100] The registered first modality image to be fused and the second modality image to be fused are input into the trained conditional diffusion probability model. After Gaussian noise is initialized by the trained conditional diffusion probability model, the final fused image is obtained by iterative denoising based on the initialized Gaussian noise according to the reverse diffusion process.

[0101] Among them, after the first modality image to be fused and the second modality image to be fused are input into the trained conditional diffusion probability model, the multimodal information fusion module 1 first fuses the first modality image to be fused and the second modality image to be fused into an initial fused image, and then the iterative denoising module 2 performs Gaussian noise initialization. Based on the initialized Gaussian noise, the initial fused image is iteratively denoised according to the reverse diffusion process to obtain the final fused image (Gaussian noise initialization and iterative denoising according to the reverse diffusion process are both existing technologies and will not be described in detail here).

[0102] Fig.10 and Fig.11 A comparison diagram of the fusion results of the multimodal image fusion and super-resolution method based on the conditional diffusion probability model of the present application and several existing fusion methods (hereinafter referred to as comparison methods) is shown, wherein the comparison methods include a fusion method of dual-branch driving plus conditional diffusion probability model, a fusion method of cyclic correction network plus conditional diffusion probability model, a fusion method of target perception dual adversarial learning plus conditional diffusion probability model, a fusion method of fusion and segmentation multi-interactive feature learning plus conditional diffusion probability model, a fusion method based on generative adversarial learning plus conditional diffusion probability model, a fusion method of representation learning guided fusion network plus conditional diffusion probability model and a fusion method based on denoising diffusion probability model plus conditional diffusion probability model.

[0103] Fig.10 The fuzzy source a and Fig.11 The blur sources a in are two blurred first modal images, Fig.10 The fuzzy source b and Fig.11 The blur sources b in are two blurred second modality images, Fig.10 The integration of Fig.11 The fusion and a in the above are two fusion results obtained by the multimodal image fusion and super-resolution method based on the conditional diffusion probability model of this application, respectively. Fig.10 The integration of Fig.11 The fusion and b in the figure are two fusion results obtained by the fusion method of double-branch driving plus conditional diffusion probability model. Fig.10 The fusion and Fig.11 The fusion c in the figure are two fusion results obtained by the fusion method of the cycle correction network plus the conditional diffusion probability model. Fig.10 The fusion and Fig.11 The fusion and d in the figure are two fusion results obtained by the fusion method of target perception dual adversarial learning plus conditional diffusion probability model, Fig.10 The integration of Fig.11 The fusions in the figure are two fusion results obtained by the fusion method of fusion and segmentation multi-interaction feature learning plus conditional diffusion probability model. Fig.10 The fusion and Fig.11 The fusions f in the figure are two fusion results obtained by the fusion method based on generative adversarial learning and conditional diffusion probability model. Fig.10 The integration of Fig.11 The fusion g in the figure are two fusion results obtained by the fusion method of the representation learning guided fusion network plus the conditional diffusion probability model. Fig.10 The melting point and Fig.11 The fusion h in the figure are two fusion results obtained by the fusion method based on the denoising diffusion probability model plus the conditional diffusion probability model; as can be seen from the figure, the fusion results of the multimodal image fusion based on the conditional diffusion probability model and the super-resolution method in the present application are better than the fusion results of the comparative methods, and provide more target and scene information.

[0104] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0105] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0106] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist independently, or two or more modules may be integrated to form an independent part.

[0107] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0108] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multimodal image fusion and super-resolution method based on a conditional diffusion probability model, characterized in that: Includes steps: A1. Acquire multiple sets of source images for training; each set of source images includes a registered first modality image and a second modality image; A2. performing blur processing and fusion processing on each group of source images to obtain corresponding blurred first modality images, blurred second modality images and true value fusion images; A3. constructing a conditional diffusion probability model, and training the conditional diffusion probability model using the blurred first modality image, the blurred second modality image and the true value fusion image corresponding to each group of the source images to obtain a trained conditional diffusion probability model; the conditional diffusion probability model includes a multimodal information fusion module (1) and an iterative denoising module (2); A4. Fusing the registered first modality image to be fused and the second modality image to be fused using the trained conditional diffusion probability model; The first modality image and the first modality image to be fused are visible light images, and the second modality image and the second modality image to be fused are infrared images; The multimodal information fusion module (1) comprises a first two-dimensional convolution layer, wherein the convolution kernel size of the first two-dimensional convolution layer is 3×3 and the step length is 1; The iterative denoising module (2) comprises a down-sampling module (3), an intermediate module (4), an up-sampling module (5) and a first sub-module which are connected in sequence; The down-sampling module (3) comprises a plurality of first residual modules (6), a plurality of second residual modules (7), a plurality of third residual modules (8), a plurality of fourth residual modules (9) and a plurality of down-sampling convolution blocks (10); the intermediate module (4) comprises a plurality of first residual modules (6) and a plurality of second residual modules (7); the up-sampling module (5) comprises a plurality of third residual modules (8), a plurality of fourth residual modules (9) and a plurality of up-sampling convolution blocks (11); the down-sampling module (3) and the up-sampling module (5) form a U-Net structure; The first residual module (6) comprises a first submodule, a feature dimension refinement layer, and a second submodule connected in sequence, and also comprises a first identity activation function layer, wherein in the first residual module (6), an input end of the first identity activation function layer is connected to an input end of the first submodule, and an output of the first identity activation function layer is added to an output of the second submodule as an output of the first residual module (6); The second residual module (7) comprises a first residual module (6) and a self-attention module connected in sequence; The third residual module (8) comprises a first submodule, a feature dimension refinement layer and a second submodule connected in sequence, and also comprises a second two-dimensional convolutional layer, wherein in the third residual module (8), an input end of the second two-dimensional convolutional layer is connected to an input end of the first submodule, and an output of the second two-dimensional convolutional layer is added to an output of the second submodule as an output of the third residual module (8); The fourth residual module (9) comprises a third residual module (8) and a self-attention module connected in sequence; The first submodule includes a GroupNorm normalization function layer, a Swish activation function layer, a second Identity activation function layer and a third two-dimensional convolutional layer connected in sequence; The second submodule includes a GroupNorm normalization function layer, a Swish activation function layer, a Dropout regularization function layer and a third two-dimensional convolutional layer connected in sequence; The downsampling module (3) includes six first residual modules (6), one second residual module (7), two third residual modules (8), one fourth residual module (9) and four downsampling convolution blocks (10), and these modules are connected in sequence from front to back in the following order: first residual module (6), first residual module (6), downsampling convolution block (10), third residual module (8), first residual module (6), downsampling convolution block (10), third residual module (8), first residual module (6), downsampling convolution block (10), fourth residual module (9), second residual module (7), downsampling convolution block (10), first residual module (6), first residual module (6); The intermediate module (4) includes a second residual module (7) and a first residual module (6), and these modules are connected in sequence from front to back in the following order: the second residual module (7), the first residual module (6); The upsampling module (5) comprises eleven third residual modules (8), three fourth residual modules (9) and four upsampling convolution blocks (11), which are connected in sequence from front to back in the following order: third residual module (8), third residual module (8), third residual module (8), upsampling convolution block (11), fourth residual module (9), fourth residual module (9), fourth residual module (9), upsampling convolution block (11), third residual module (8), third residual module (8), third residual module (8), upsampling convolution block (11), third residual module (8), third residual module (8), third residual module (8), upsampling convolution block (11), third residual module (8), third residual module (8), third residual module (8), upsampling convolution block (11), third residual module (8), third residual module (8), third residual module (8), upsampling convolution block (11), third residual module (8), third residual module (8); The modules in the downsampling module (3) are sorted from front to back, and the modules in the upsampling module (5) except the upsampling convolution block (11) are sorted from front to back, and the i-th module in the downsampling module (3) is jump-connected with the 15-i-th modules in the upsampling module (5) except the upsampling convolution block (11), i=1,2,…,14; The downsampling convolution block (10) is a two-dimensional convolution layer with a convolution kernel size of 3×3 and a stride of 2; The upsampling convolution block (11) is a two-dimensional convolution layer with a convolution kernel size of 3×3 and a stride of 1.

2. The multimodal image fusion and super-resolution method based on the conditional diffusion probability model according to claim 1, characterized in that: In step A2, blurring each group of source images includes: Downsampling the first modality image and the second modality image to obtain a corresponding first reduced-resolution image and a second reduced-resolution image; The first reduced-resolution image and the second reduced-resolution image are upsampled to obtain corresponding blurred first modality images and blurred second modality images; the pixel size of the blurred first modality image is the same as the pixel size of the corresponding first modality image, and the pixel size of the blurred second modality image is the same as the pixel size of the corresponding second modality image.

3. The multimodal image fusion and super-resolution method based on conditional diffusion probability model according to claim 1, characterized in that: In step A2, the fusion processing of each group of source images includes: The first modality image and the second modality image in the same group of the source images are fused using a preset image fusion algorithm to obtain the corresponding true value fused image.

4. The multimodal image fusion and super-resolution method based on conditional diffusion probability model according to claim 1, characterized in that: In step A3, the conditional diffusion probability model is trained using the blurred first modality image, the blurred second modality image and the true value fusion image corresponding to each group of the source images to obtain a trained conditional diffusion probability model, including: A301. Using the blurred first modality image, the blurred second modality image and the true value fusion image corresponding to each group of the source images, the conditional diffusion probability model is trained by iteratively adding noise to the forward diffusion process and the back propagation process to obtain the corresponding prediction noise; A302. Calculating a loss function based on the iteratively added noise and the predicted noise, and updating the model parameters of the conditional diffusion probability model based on the loss function; A303. Repeat steps A301 and A302 until the loss function converges.

5. The multimodal image fusion and super-resolution method based on conditional diffusion probability model according to claim 4, characterized in that: In step A302, the loss function is calculated according to the following formula: ; in, is the loss function, is the true value fusion image, x is the blurred first modality image, y is the blurred second modality image, is Gaussian noise, is the noise variance, is the noise predictor function, and They are all expected functions.

6. The multimodal image fusion and super-resolution method based on conditional diffusion probability model according to claim 1, characterized in that: Step A4 includes: The registered first modality image to be fused and the second modality image to be fused are input into the trained conditional diffusion probability model, and after Gaussian noise is initialized by the trained conditional diffusion probability model, the final fused image is obtained by iteratively denoising based on the initialized Gaussian noise according to the reverse diffusion process.