Multi-modal image fusion de-ghosting method based on downstream counting task reverse optimization

Abstract

The application provides a multi-modal image fusion ghost elimination method based on downstream counting task reverse optimization, and the method comprises the following steps: using a diffusion fusion model to synthesize a fusion image for each visible light-infrared image pair to be fused; using a fusion modal generation module to approximate the multi-modal fusion capability of the diffusion fusion model through knowledge distillation; fine-tuning the fusion modal generation module in a multi-modal target counting task; and using the fine-tuned fusion modal generation module to re-fuse the visible light-infrared image pair into a fusion image. The application can fuse visible light and infrared images into a fusion image without ghosting, can work well even in a target dense scene, is suitable for actual application scenarios, the effect of the synthesized fusion image is better than that of a traditional image registration-based ghost elimination method, and the operation time is lower than that of the traditional ghost elimination method.

Description

Technical Field

[0001] This invention belongs to the fields of artificial intelligence and multimodal learning technology, and particularly relates to a multimodal image fusion ghosting removal method based on downstream counting task back optimization. Background Technology

[0002] With the rapid development of artificial intelligence and deep learning technologies, multimodal learning has gradually attracted widespread attention. Multimodal learning acquires, interacts with, and integrates information from different modalities to achieve a comprehensive and accurate understanding of information from multiple sources. Among these, visible light-infrared multimodal learning is an important computer vision task because the color and texture information carried by the visible light modality and the electromagnetic radiation information carried by infrared images have sufficient complementary advantages. For example, under poor lighting conditions, it is difficult to distinguish human figures from visible light images, but thermal infrared images can clearly reflect the contours of each person; heated negative objects look almost identical in thermal infrared images, but they can be easily distinguished from visible light images. Therefore, fully integrating information from these two modalities is of great benefit for visual understanding tasks in complex scenes.

[0003] Visible-infrared image fusion is a widely used visible-infrared multimodal learning method. In recent years, image fusion methods based on generative adversarial networks and diffusion models have emerged. However, the acquisition of multimodal data is mostly accomplished by different probes of the same industrial camera, which are often arranged side-by-side or at an angle. This inevitably leads to angular differences and spatial misalignments between images of different modalities. Since most of the aforementioned methods perform spatial alignment fusion, the positional difference of the same object in the two source modal images in the fused image results in ghosting.

[0004] Currently, most mainstream ghosting removal methods are based on image registration. They first select unique structures in two images to be registered as keypoints, then extract features from each keypoint and attempt to find similar feature point pairs between the two images. Subsequently, a spatial transformation model between the two images is estimated based on the matched feature point pairs, and then the estimated transformation model is used to transform one image to a coordinate system aligned with the other. These methods often work well in natural landscapes, but in dense scenes, they often fail to perform well due to the large number and high similarity of targets, making it difficult for keypoint-based feature pairing algorithms to distinguish different targets, especially in infrared images. This invention adopts a different technical approach, using a downstream task-based reverse optimization method to remove ghosting. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing image registration-based ghost removal methods in distinguishing different targets when feature matching, and to propose a multimodal image fusion ghost removal method based on downstream counting task back optimization.

[0006] This invention is achieved through the following technical solution: It proposes a multimodal image fusion ghosting removal method based on backward optimization of a downstream counting task. The method selects multimodal target counting as the downstream task of visible-infrared image fusion. In the configuration of the downstream task, the fusion modality generation module is optimized through backpropagation, thereby achieving the effect of removing ghosting from the fused modality image. The specific steps of the method are as follows:

[0007] A pre-defined neural network model is trained using a multimodal target counting method based on learnable fusion modalities;

[0008] The trained neural network model is used to synthesize visible light and infrared modal images into a ghost-free fused image.

[0009] Furthermore, in the learnable fusion modality multimodal target counting method, for each visible-infrared image pair, a density map can be predicted. The value of each pixel in the density map represents the number of targets in a corresponding small region in the original image, and the sum of all pixel values ​​in the density map represents the total number of targets in the original image predicted by the method.

[0010] Furthermore, the preset neural network model is trained in the following manner:

[0011] The diffusion fusion model is used to synthesize each visible-infrared image pair in a pre-selected dataset into a single fused modality image, which is then saved.

[0012] By employing knowledge distillation, a fusion modality generation module is used to approximate the multimodal image fusion capability of the diffusion fusion model;

[0013] A learnable fusion modality counting framework based on efficient cloning of diffusion fusion model is used to fine-tune the fusion modality generation module for multimodal target counting tasks.

[0014] Furthermore, the diffusion fusion model synthesizes each visible-infrared image pair in a pre-selected dataset into a single fused modality image through the following process:

[0015] Predict the diffusion noise at the current time step using pre-trained weights of the diffusion model;

[0016] The estimate of the reconstructed fused modality image is updated once using the diffuse noise predicted at the current time step;

[0017] The expectation-maximization algorithm is used to perform a likelihood correction on the reconstructed fusion modality image predicted at the current time step;

[0018] A one-step denoising operation is performed using the corrected prediction to reconstruct the fused modality image.

[0019] Furthermore, the process of generating the fused modal image is as follows:

[0020] The predicted diffusion noise is:

[0021] s = s θ (f t ,t)

[0022] Where s represents the propagation noise predicted by the model, t represents the current time step, and f t The fused modality image representing time step t, s θ This represents a pre-trained diffusion model;

[0023] The estimation of the reconstructed fused modality image is as follows:

[0024]

[0025] Where f0 represents the output image of the diffusion fusion model at the last time step, i.e., the reconstructed fusion modality image. It is the prediction result of the diffusion fusion model for the reconstructed fusion modality image f0 at time step t. These are the hyperparameters preset by the diffusion model;

[0026] The predicted image after one likelihood correction using the expectation-maximization algorithm on the predicted result of the reconstructed fused modality image f0 at the current time step is as follows:

[0027]

[0028] in, Reconstructed fusion modality image representing the prediction at the current time step The predicted image obtained after likelihood correction, where h represents one iteration of the expectation-maximization algorithm, and I and V represent the visible light mode and infrared mode images used to synthesize the fused modal image, respectively.

[0029] The fused modality image obtained at the next time step after performing a one-step denoising operation using the corrected prediction-reconstructed fused modality image is:

[0030]

[0031] Among them, f t-1 The fused modality image representing time step t-1, α t , β tThese are hyperparameters preset in the diffusion model. The bias value is set to zero, and z is the value sampled from a Gaussian distribution. A random variable.

[0032] Furthermore, the multimodal image fusion capability of the fusion modality generation module is approximated by the diffusion fusion model through the following knowledge distillation process:

[0033] The weights of the fusion mode generation module are initialized using the Kaiming method;

[0034] For each visible-infrared modal image pair in the preset training set, a fusion modality generation module is used to synthesize a fused image;

[0035] The mean square error between the fused image synthesized by the fusion modality generation module and the fused modality image synthesized by the saved diffusion fusion model is calculated, and this error is used as the loss function to perform a backpropagation and update the parameters of the fusion modality generation module.

[0036] After training converges, save the weights of the fusion mode generation module;

[0037] The mean square error between the fused image synthesized by the fusion modality generation module and the fused modality image synthesized by the saved diffusion fusion model is calculated as follows:

[0038]

[0039] in, denoted by mean square error, g represents the fusion mode generation module, p represents the diffusion fusion model, R and T represent the visible light and infrared modes in the preset training set, respectively, and K represents the number of samples in the training set.

[0040] Furthermore, the fusion modality generation module includes: a cross-modal contraction module, a cross-modal attention module, and a fusion modality decoding module;

[0041] The cross-modal contraction module and the fusion modality decoding module are implemented by the encoder and decoder of the U-Net neural network, respectively. The cross-modal attention module is implemented by several convolutional layers and a single-layer, four-head cross-attention Transformer encoder. Visible light and infrared images are input to the cross-modal contraction module and the cross-modal attention module. The sum of the output features of these two modules is the input of the fusion modality decoding module, which decodes it into a three-channel fusion modality image.

[0042] The forward propagation process for each part is shown in the following formula:

[0043] Cross-modal contraction module:

[0044] F e=f e ([R;T])

[0045] Where [·; ·] represents the channel merging operation, f e F represents the encoder network of the cross-modal shrinkage module. e This represents the encoding result of the cross-modal contraction module, where R and T represent visible light and infrared images, respectively;

[0046] Cross-modal attention module:

[0047] X r =ξ(R),X t =ξ(T)

[0048] Q = X t W q K = X r W k V = X r W v

[0049] H = Att(Q,K,V) + X t

[0050] F h =FFN(H)+H

[0051] Where ξ represents the convolutional layer in the cross-modal attention module, Q, K, and V represent the embedding vectors of Query, Key, and Value in the attention mechanism, respectively, and W q W k and W v These are the projection weights of Q, K, and V in multi-head attention; FFN represents the feedforward neural network in the Transformer; F h This represents the encoding result of the cross-modal attention module; It is the scaling factor, and Softmax(·) is the normalization exponential function;

[0052] Fusion modal decoding module:

[0053] F = f d (F e +F h )

[0054] Among them, f d F represents the decoder network of the fused modal decoding module. e and F h These are the output features of the cross-modal contraction module and the cross-modal attention module, respectively, and F is the three-channel fused modal image obtained by the fused modal decoding module.

[0055] Furthermore, the learnable fusion modality counting framework based on efficient cloning of diffusion fusion model consists of the following structure: feature extraction module, fusion modality generation module, and regression head;

[0056] The feature extraction module consists of a VGG-19 convolutional neural network and a two-layer, six-head Transformer encoder, while the regression head consists of a three-layer convolutional neural network with a kernel size of 3×3 and two ReLU activation layers in between.

[0057] The forward propagation process of this counting frame is shown in the following formula:

[0058] F = g(R,T)

[0059] D=ρ(ξ(R)+ξ(F)+ξ(T))

[0060] Where F represents the fused modal image, R and T represent the visible light and infrared modal images, D represents the density map generated by the counting frame, and ξ, ρ and g represent the feature extraction module, regression head and fused modal generation module in the counting frame, respectively;

[0061] The loss function used for fine-tuning the counting task is shown in the following formula:

[0062]

[0063] in, This represents the counting loss, where M represents the total number of targets in the sample, and z n These are the coordinates of the nth labeled point in the sample, where <·,·> represents the vector dot product operation. This indicates that the mean is μ and the variance is σ. 2 The distribution is Gaussian, and D represents the density map of the counting frame output.

[0064] Furthermore, the specific steps for synthesizing visible light and infrared modal images into a ghost-free fused image using the trained neural network model are as follows:

[0065] The model was trained using a multimodal target counting method based on learnable fusion modalities, and the trained weights were saved.

[0066] Separate the fusion mode generation module and its weights from the counting framework;

[0067] The visible light and infrared modal images to be fused are input into the fusion modal generation module to obtain the fused modal image and save it.

[0068] Furthermore, the process of obtaining a ghost-free fused image is shown in the following formula:

[0069] F * =g* (R,T)

[0070] Among them, F * G represents a ghost-free merged image. * This represents the fusion modality generation module with weights after training with target counts, where R and T represent the visible light and infrared modal images to be fused, respectively.

[0071] Compared with the prior art, the present invention has the following advantages:

[0072] This invention proposes a visible-infrared modal image fusion ghosting removal method. It uses a downstream multimodal target counting task to inversely optimize the fusion image generation module, thereby removing ghosting from the fused image. Compared to existing deep learning and graphics methods based on image registration or anti-aliasing, this invention achieves superior ghosting removal, especially in scenarios widely encountered in practical applications such as dense crowds. Furthermore, during the inference phase, this invention only requires a lightweight fusion image generation module to synthesize a ghost-free fusion image, resulting in fewer parameters and floating-point calculations compared to existing methods, shorter computation time, and lower requirements for computing power. Attached Figure Description

[0073] Figure 1 This is a flowchart illustrating a method for removing ghosting in visible-infrared image fusion according to an embodiment of the present invention.

[0074] Figure 2 This is a flowchart illustrating the method for training a neural network model using a multimodal target counting method based on learnable fusion modalities, according to an embodiment of the present invention.

[0075] Figure 3 This is a schematic diagram illustrating the method for training a neural network model based on a learnable fusion modality multimodal target counting method according to an embodiment of the present invention.

[0076] Figure 4 This is a flowchart illustrating a method for synthesizing visible-infrared image pairs into a single fused modal image using a diffusion fusion model, as described in an embodiment of the present invention.

[0077] Figure 5 This is a flowchart illustrating a method for approximating the multimodal image fusion capability of a diffusion fusion model using a fusion modality generation module through knowledge distillation, as described in an embodiment of the present invention.

[0078] Figure 6 This is a schematic diagram of the fusion mode generation module structure according to an embodiment of the present invention.

[0079] Figure 7 This is a schematic diagram of the learnable fusion mode counting framework based on efficient cloning of diffusion fusion model according to an embodiment of the present invention. Detailed Implementation

[0080] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0081] It should be noted that the terms "first," "second," etc., used in this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0082] The present invention will now be described in further detail with reference to the accompanying drawings:

[0083] See Figure 1 In one embodiment of the present invention, a multimodal image fusion ghost removal method based on downstream counting task reverse optimization is provided. Specifically, it is a multimodal target counting method based on learnable fusion modalities, and a method for synthesizing a ghost-free fused image using the multimodal target counting method based on learnable fusion modalities. It can synthesize visible light and infrared modal images in the same scene into a ghost-free fused image, and can still work well in scenes with dense targets. It has a fast calculation speed and low requirements for computing power.

[0084] Specifically, the multimodal image fusion ghosting removal method based on downstream counting task back optimization includes the following steps:

[0085] S1: Train a pre-defined neural network model using a multimodal target counting method based on learnable fusion modalities.

[0086] S2: Use a trained neural network model to synthesize visible light and infrared modal images into a ghost-free fused image.

[0087] The dataset used to train the pre-set neural network can be an open-source dataset for a multimodal target counting task, and the visible light and infrared images to be synthesized are determined based on the actual working scenario.

[0088] This invention proposes a visible-infrared modal image fusion ghosting removal method. It uses a downstream multimodal target counting task to inversely optimize the fusion image generation module, thereby removing ghosting from the fused image. This method can synthesize a visible light image and an infrared image from the same scene into a ghost-free fused image. Compared with existing deep learning and graphics methods based on image registration or anti-aliasing, this invention achieves better ghosting removal, especially in scenarios widely encountered in practical applications such as dense crowds. Furthermore, during the inference stage, this invention only requires a lightweight fusion image generation module to synthesize the ghost-free fused image, resulting in fewer parameters and floating-point calculations compared to existing methods, shorter computation time, and lower requirements for computing devices.

[0089] In one possible implementation, the multimodal target counting method based on learnable fusion modalities can predict a density map for each visible-infrared image pair. The value of each pixel in the density map represents the number of targets in a corresponding small region of the original image, and the sum of all pixel values ​​in the density map represents the total number of targets in the original image predicted by this method. See also Figure 2 and Figure 3 The preset neural network model can be trained using the multimodal target counting method based on learnable fusion modalities through the following steps:

[0090] S11: Use a diffusion fusion model to synthesize each visible-infrared image pair in the pre-selected dataset into a fused modality image and save it;

[0091] S12: By using knowledge distillation, the fusion modality generation module is used to approximate the multimodal image fusion capability of the diffusion fusion model;

[0092] S13: A learnable fusion modality counting framework based on efficient cloning of diffusion fusion model is used to fine-tune the fusion modality generation module for multimodal target counting tasks.

[0093] See Figure 4 Optionally, the following method can be used to synthesize visible-infrared image pairs into a single fused modality image using a diffusion fusion model:

[0094] S111: Predict the diffusion noise at the current time step using the pre-trained weights of the diffusion model;

[0095] S112: Update the estimate of the reconstructed fused modality image once using the diffuse noise predicted at the current time step;

[0096] S113: Use the expectation-maximization algorithm to perform a likelihood correction on the reconstructed fusion modality image predicted at the current time step;

[0097] S114: Perform a one-step denoising operation using the corrected prediction to reconstruct the fused modality image.

[0098] The process is shown in the following formula:

[0099] The predicted diffusion noise is:

[0100] s = s θ (f t ,t)

[0101] Where s represents the propagation noise predicted by the model, t represents the current time step, and f t The fused modality image representing time step t, s θ This represents a pre-trained diffusion model;

[0102] The estimation of the reconstructed fused modality image is as follows:

[0103]

[0104] Where f0 represents the output image of the diffusion fusion model at the last time step, i.e., the reconstructed fusion modality image. It is the prediction result of the diffusion fusion model for the reconstructed fusion modality image f0 at time step t. These are the hyperparameters preset by the diffusion model;

[0105] The predicted image after one likelihood correction using the expectation-maximization algorithm on the predicted result of the reconstructed fused modality image f0 at the current time step is as follows:

[0106]

[0107] in, Reconstructed fusion modality image representing the prediction at the current time step The predicted image obtained after likelihood correction, where h represents one iteration of the expectation-maximization algorithm, and I and V represent the visible light mode and infrared mode images used to synthesize the fused modal image, respectively.

[0108] The fused modality image obtained at the next time step after performing a one-step denoising operation using the corrected prediction-reconstructed fused modality image is:

[0109]

[0110] Among them, f t-1 The fused modality image representing time step t-1, α t , β t These are hyperparameters preset in the diffusion model. This is the bias value, which is usually set to zero, and z is the value sampled from a Gaussian distribution. A random variable.

[0111] In the actual implementation, the pre-trained weights of the diffusion sampling denoising model are used without fine-tuning. The maximum number of time steps for the diffusion model is 1000.

[0112] See Figure 5 Optionally, by using knowledge distillation, the multimodal image fusion capability of the diffusion fusion model can be approximated by the fusion modality generation module through the following method:

[0113] S121: Initialize the weights of the fusion mode generation module using the Kaiming method;

[0114] S122: For each visible-infrared modality image pair in the preset training set, use the fusion modality generation module to synthesize a fused image;

[0115] S123: Calculate the mean square error between the fused image synthesized by the fusion modality generation module and the fused modality image synthesized by the saved diffusion fusion model, and use this as the loss function to perform a backpropagation to update the parameters of the fusion modality generation module.

[0116] S124: After training converges, save the weights of the fusion mode generation module.

[0117] The mean square error between the fused image synthesized by the fusion modality generation module and the fused modality image synthesized by the saved diffusion fusion model can be calculated as follows:

[0118]

[0119] in, denoted by mean square error, g represents the fusion mode generation module, p represents the diffusion fusion model, R and T represent the visible light and infrared modes in the preset training set, respectively, and K represents the number of samples in the training set.

[0120] See Figure 6 Optionally, the above-mentioned fusion modality generation module includes: a cross-modal contraction module, a cross-modal attention module, and a fusion modality decoding module.

[0121] The cross-modal contraction module and the fusion modality decoding module are implemented by the encoder and decoder of the U-Net neural network, respectively. The cross-modal attention module is implemented by several convolutional layers and a single-layer, four-head cross-attention Transformer encoder. Visible light and infrared images are both processed by the cross-modal contraction module and the cross-modal attention module. The sum of the output features of these two modules is the input of the fusion modality decoding module, which decodes them into a three-channel fusion modality image.

[0122] Its forward propagation process is shown in the following formula:

[0123] F e =f e ([R;T])

[0124] Where [·; ·] represents the channel merging operation, f e F represents the encoder network of the cross-modal shrinkage module. e This represents the encoding result of the cross-modal contraction module, where R and T represent visible light and infrared images, respectively;

[0125] X r =ξ(R),X t =ξ(T)

[0126] Q = X t W q K = X r W k V = X r W v

[0127] H = Att(Q,K,V) + X t

[0128] F h =FFN(H)+H

[0129] Where ξ represents the convolutional layer in the cross-modal attention module, Q, K, and V represent the embedding vectors of Query, Key, and Value in the attention mechanism, respectively, and W q W k and W v These are the projection weights of Q, K, and V in multi-head attention; FFN represents the feedforward neural network in the Transformer; F h This represents the encoding result of the cross-modal attention module; It is the scaling factor, and Softmax(·) is the normalization exponential function;

[0130] F = f d (F e +F h )

[0131] Among them, f d F represents the decoder network of the fused modal decoding module. e and F h These are the output features of the cross-modal contraction module and the cross-modal attention module, respectively, and F is the three-channel fused modal image obtained by the fused modal decoding module.

[0132] In its implementation, the cross-modal shrinking module upscales the merged 6-channel image to a 256-channel feature map and downsamples the image's length and width to 1 / 8 of their original size. The fusion modality decoding module resamples the 256-channel feature map back to the original image size and generates a 3-channel fused image. The cross-modal attention module has an embedding dimension of 256 and an image block size of 64. Training is performed using an NVIDIA RTX 3090 graphics card with 24GB of VRAM within the PyTorch framework. The Adam optimizer guides gradient descent with an initial learning rate of 0.00001 and a weight decay of 0.0001. To improve the model's generalization ability, images in the training set are randomly cropped to 224×224 pixels and horizontally flipped with a 50% probability as random augmentation. The maximum number of training iterations is 1000, and the batch size is 1.

[0133] See Figure 7 Optionally, the learnable fusion modality counting framework based on efficient cloning of diffusion fusion models consists of the following structure: feature extraction module, fusion modality generation module, and regression head.

[0134] The feature extraction module consists of a VGG-19 convolutional neural network and a two-layer, six-head Transformer encoder, while the regression head consists of a three-layer convolutional neural network with a kernel size of 3×3 and two ReLU activation layers in between.

[0135] The forward propagation process of this counting frame is shown in the following formula:

[0136] F = g(R,T)

[0137] D=ρ(ξ(R)+ξ(F)+ξ(T))

[0138] Where F represents the fused modal image, R and T represent the visible light and infrared modal images, D represents the density map generated by the counting frame, and ξ, ρ and g represent the feature extraction module, regression head and fused modal generation module in the counting frame, respectively;

[0139] The loss function used for fine-tuning the counting task is shown in the following formula:

[0140]

[0141] in, This represents the counting loss, where M represents the total number of targets in the sample, and z n These are the coordinates of the nth labeled point in the sample, where <·,·> represents the vector dot product operation. This indicates that the mean is μ and the variance is σ. 2 The distribution is Gaussian, and D represents the density map of the counting frame output.

[0142] In its implementation, the VGG-19 neural network downsamples the length and width of the input image to 1 / 8 of their original size and boosts the original image's channels from 3 to 512. The Transformer encoder boosts the 512-channel feature map to 768 channels without changing the feature map's length and width. The Transformer encoder has an embedding dimension of 768 and the image is divided into 196 blocks. Training is performed using an NVIDIA RTX 3090 graphics card with 24GB of VRAM within the PyTorch framework. The Adam optimizer guides gradient descent with an initial learning rate of 0.00001 and weight decay of 0.0001. To improve the model's generalization ability, the training set images are randomly cropped to 224×224 pixels and horizontally flipped with a 50% probability as random augmentation. The maximum number of training iterations is 400, and the batch size is 1.

[0143] Optionally, based on the above-described multimodal target counting method based on learnable fusion modalities, the method for synthesizing ghost-free fused images is as follows:

[0144] The model was trained using a multimodal target counting method based on learnable fusion modalities, and the trained weights were saved.

[0145] Separate the fusion mode generation module and its weights from the counting framework;

[0146] The visible light and infrared modal images to be fused are input into the fusion modal generation module to obtain the fused modal image and save it;

[0147] The process of obtaining a ghost-free blended image is shown in the following formula:

[0148] F * =g * (R,T)

[0149] Among them, F * G represents a ghost-free merged image. * This represents the fusion modality generation module with weights after training with target counts, where R and T represent the visible light and infrared modal images to be fused, respectively.

[0150] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0151] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0153] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0154] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multimodal image fusion ghosting removal method based on downstream counting task reverse optimization, characterized in that, The method selects multimodal target counting as a downstream task of visible light-infrared image fusion. In the configuration of the downstream task, backpropagation is used to optimize the fusion modality generation module, thereby achieving the effect of removing ghosting in the fusion modality image. The specific steps of the method are as follows: A pre-defined neural network model is trained using a multimodal target counting method based on learnable fusion modalities; The trained neural network model is used to synthesize visible light and infrared modal images into a ghost-free fused image. The preset neural network model is trained in the following manner: The diffusion fusion model is used to synthesize each visible-infrared image pair in a pre-selected dataset into a single fused modality image, which is then saved. By employing knowledge distillation, a fusion modality generation module is used to approximate the multimodal image fusion capability of the diffusion fusion model; A learnable fusion modality counting framework based on efficient cloning of diffusion fusion model is used to fine-tune the fusion modality generation module for multimodal target counting tasks. The learnable fusion modality counting framework based on efficient cloning of diffusion fusion model consists of the following structure: feature extraction module, fusion modality generation module, and regression head; The feature extraction module consists of a VGG-19 convolutional neural network and a two-layer, six-head Transformer encoder, while the regression head consists of a three-layer convolutional neural network with a kernel size of 3×3 and two ReLU activation layers in between. The forward propagation process of this counting frame is shown in the following formula: F = g(R,T) D=ρ(ξ(R)+ξ(F)+ξ(T)) Where F represents the fused modal image, R and T represent the visible light and infrared modal images, D represents the density map generated by the counting frame, and ξ, ρ and g represent the feature extraction module, regression head and fused modal generation module in the counting frame, respectively; The loss function used for fine-tuning the counting task is shown in the following formula: in, This represents the counting loss, where M represents the total number of targets in the sample, and z n These are the coordinates of the nth labeled point in the sample, where <·,·> represents the vector dot product operation. This indicates that the mean is μ and the variance is μ. 2 The distribution is Gaussian, and D represents the density map of the counting frame output.

2. The method according to claim 1, characterized in that: In the learnable fusion modality multimodal target counting method, for each visible-infrared image pair, a density map can be predicted. The value of each pixel in the density map represents the number of targets in a corresponding small region in the original image, and the sum of all pixel values ​​in the density map represents the total number of targets in the original image predicted by this method.

3. The method according to claim 2, characterized in that: The diffusion fusion model synthesizes each visible-infrared image pair in a pre-selected dataset into a single fused modality image through the following process: Predict the diffusion noise at the current time step using pre-trained weights of the diffusion model; The estimate of the reconstructed fused modality image is updated once using the diffuse noise predicted at the current time step; The expectation-maximization algorithm is used to perform a likelihood correction on the reconstructed fusion modality image predicted at the current time step; A one-step denoising operation is performed using the corrected prediction to reconstruct the fused modality image.

4. The method according to claim 3, characterized in that: The specific process for generating fused modal images is as follows: The predicted diffusion noise is: s=s θ (f t ,t) Where s represents the propagation noise predicted by the model, t represents the current time step, and f t The fused modality image representing time step t, s θ This represents a pre-trained diffusion model; The estimation of the reconstructed fused modality image is as follows: Where f0 represents the output image of the diffusion fusion model at the last time step, i.e., the reconstructed fusion modality image. It is the prediction result of the diffusion fusion model for the reconstructed fusion modality image f0 at time step t. These are the hyperparameters preset by the diffusion model; The predicted image after one likelihood correction using the expectation-maximization algorithm on the predicted result of the reconstructed fused modality image f0 at the current time step is as follows: in, Reconstructed fusion modality image representing the prediction at the current time step The predicted image obtained after likelihood correction, where h represents one iteration of the expectation-maximization algorithm, and I and V represent the visible light mode and infrared mode images used to synthesize the fused modal image, respectively. The fused modality image obtained at the next time step after performing a one-step denoising operation using the corrected prediction-reconstructed fused modality image is: Among them, f t-1 The fused modality image representing time step t-1, α t , β t These are hyperparameters preset in the diffusion model. The bias value is set to zero, and z is the value sampled from a Gaussian distribution. A random variable.

5. The method according to claim 4, characterized in that: The following knowledge distillation process is used to make the fusion modality generation module approximate the multimodal image fusion capability of the diffusion fusion model: The weights of the fusion mode generation module are initialized using the Kaiming method; For each visible-infrared modal image pair in the preset training set, a fusion modality generation module is used to synthesize a fused image; The mean square error between the fused image synthesized by the fusion modality generation module and the fused modality image synthesized by the saved diffusion fusion model is calculated, and this error is used as the loss function to perform a backpropagation and update the parameters of the fusion modality generation module. After training converges, save the weights of the fusion mode generation module; The mean square error between the fused image synthesized by the fusion modality generation module and the fused modality image synthesized by the saved diffusion fusion model is calculated as follows: in, denoted by mean square error, g represents the fusion mode generation module, p represents the diffusion fusion model, R and T represent the visible light and infrared modes in the preset training set, respectively, and K represents the number of samples in the training set.

6. The method according to claim 5, characterized in that: The fusion modality generation module includes: a cross-modal contraction module, a cross-modal attention module, and a fusion modality decoding module; The cross-modal contraction module and the fusion modality decoding module are implemented by the encoder and decoder of the U-Net neural network, respectively. The cross-modal attention module is implemented by several convolutional layers and a single-layer, four-head cross-attention Transformer encoder. Visible light and infrared images are input to the cross-modal contraction module and the cross-modal attention module. The sum of the output features of these two modules is the input of the fusion modality decoding module, which decodes it into a three-channel fusion modality image. The forward propagation process for each part is shown in the following formula: Cross-modal contraction module: F e =f e ([R;T]) Where [·; ·] represents the channel merging operation, f e F represents the encoder network of the cross-modal shrinkage module. e This represents the encoding result of the cross-modal contraction module, where R and T represent visible light and infrared images, respectively; Cross-modal attention module: X r =ξ(R),X t =ξ(T) Q=X t W q ,K=X r W k ,V=X r W v H=Att(Q,K,V)+X t F h =FFN(H)+H Where ξ represents the convolutional layer in the cross-modal attention module, Q, K, and V represent the embedding vectors of Query, Key, and Value in the attention mechanism, respectively, and W q W k and W v These are the projection weights of Q, K, and V in multi-head attention; FFN represents the feedforward neural network in the Transformer; F h This represents the encoding result of the cross-modal attention module; It is the scaling factor, and Softmax(·) is the normalization exponential function; Fusion modal decoding module: F=f d (F e +F h ) Among them, f d F represents the decoder network of the fused modal decoding module. e and F h These are the output features of the cross-modal contraction module and the cross-modal attention module, respectively, and F is the three-channel fused modal image obtained by the fused modal decoding module.

7. The method according to claim 1, characterized in that: The specific steps for using a trained neural network model to synthesize visible light and infrared modal images into a ghost-free fused image are as follows: The model was trained using a multimodal target counting method based on learnable fusion modalities, and the trained weights were saved. Separate the fusion mode generation module and its weights from the counting framework; The visible light and infrared modal images to be fused are input into the fusion modal generation module to obtain the fused modal image and save it.

8. The method according to claim 7, characterized in that: The process of obtaining a ghost-free blended image is shown in the following formula: F * =g * (R,T) Among them, F * G represents a ghost-free merged image. * This represents the fusion modality generation module with weights after training with target counts, where R and T represent the visible light and infrared modal images to be fused, respectively.

Images