Multimodal image fusion ghosting removal method based on downstream counting task reverse optimization

Abstract

The invention provides a multi-modal image fusion ghosting removal method based on downstream counting task reverse optimization. The method comprises the following steps: synthesizing a fusion image for each visible light-infrared image pair to be fused by using a diffusion fusion model; through knowledge distillation, a fusion mode generation module is used for approaching the multi-mode fusion capability of the diffusion fusion model; finely adjusting a fusion modal generation module in the multi-modal target counting task; and fusing the visible light-infrared image pair into a fused image again by using the fine-tuned fusion mode generation module. According to the method, the visible light image and the infrared image can be synthesized into a ghosting-free fusion image, the method can work well even in a target dense scene and fits an actual application scene, the effect of the synthesized fusion image is better than that of a traditional ghosting removal method based on image registration, and the operation time is shorter than that of the traditional ghosting removal method.

Description

Technical Field

[0001] The present invention belongs to the field of artificial intelligence and multimodal learning technology, and in particular relates to a multimodal image fusion and ghost removal method based on reverse optimization of downstream counting tasks. Background Art

[0002] With the vigorous development of artificial intelligence and deep learning technology, multimodal learning has gradually attracted widespread attention. Multimodal learning acquires, interacts, and integrates information from different modalities to obtain a comprehensive and accurate understanding of information from multiple sources. Among them, visible light-infrared multimodal learning is an important computer vision task because the color and texture information carried by the visible light modality and the electromagnetic radiation information carried by infrared images have sufficient complementary advantages. For example, under poor lighting conditions, it is difficult to distinguish portraits from visible light images, but thermal infrared images can clearly reflect the outline of each person; heated negative objects look almost the same in thermal infrared images, but they can be easily distinguished from visible light images. Therefore, fully integrating the information of these two modalities is of great benefit to the visual understanding task of complex scenes.

[0003] Visible light-infrared image fusion is a widely used visible light-infrared multimodal learning method. In recent years, image fusion methods based on generative adversarial networks and diffusion models have emerged. However, the acquisition of multimodal data is mostly completed by different probes of the same industrial camera. Different probes are often arranged in parallel or at a certain angle, which inevitably leads to certain angle differences and spatial misalignment between different modal images. Most of the above methods perform spatial alignment fusion. Therefore, the position difference of the same object in the two source modal images in the fused image leads to ghosting.

[0004] At present, most of the mainstream ghost removal methods are based on image registration. They first select the unique structures in the two images to be registered as key points, then extract the features of each key point, and try to find similar feature point pairs between the two images. After that, the spatial transformation model between the two images is estimated based on the matched feature point pairs, so that one image is transformed into a coordinate system aligned with the other image using the estimated transformation model. In scenes such as natural scenery, they often work well, but in dense scenes, such methods often do not work well. Due to the large number of targets and their strong similarity, it is difficult for the feature pairing algorithm based on key points to distinguish different targets, especially in infrared images. The present invention adopts a different technical route and uses the method of reverse optimization of downstream tasks to remove ghosts. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of existing ghost removal methods based on image registration, which have poor effects in distinguishing different targets during feature pairing, and propose a multimodal image fusion ghost removal method based on reverse optimization of downstream counting tasks.

[0006] The present invention is implemented by the following technical scheme. The present invention proposes a multimodal image fusion ghost removal method based on reverse optimization of downstream counting tasks. The method selects multimodal target counting as the downstream task of visible light-infrared image fusion, and optimizes the fusion modality generation module by back propagation in the configuration of the downstream task, thereby achieving the effect of removing ghosts in the fusion modality image. The specific steps of the method are as follows:

[0007] Train the preset neural network model using a multimodal target counting method based on learnable fusion modalities;

[0008] The visible and infrared modality images are synthesized into a ghost-free fused image using a trained neural network model.

[0009] Furthermore, in the multimodal target counting method that can learn fusion modalities, for each visible light-infrared image pair, a density map can be predicted, and the value of each pixel in the density map represents the number of targets in a corresponding small area in the original image, and the sum of all pixel values ​​in the density map represents the total number of targets in the original image predicted by the method.

[0010] Furthermore, the preset neural network model is trained in the following manner:

[0011] Each visible-infrared image pair in the pre-selected dataset is synthesized into a fused modality image using a diffusion fusion model and saved;

[0012] Through knowledge distillation, a fusion modality generation module is used to approximate the multimodal image fusion capability of the diffusion fusion model;

[0013] A learnable fusion modality counting framework based on efficient cloning of diffusion fusion model is used to fine-tune the fusion modality generation module for multimodal object counting tasks.

[0014] Furthermore, the diffusion fusion model combines each visible light-infrared image pair in the pre-selected data set into a fused modality image through the following process:

[0015] Use the pre-trained weights of the diffusion model to predict the diffusion noise at the current time step;

[0016] Update the estimate of the reconstructed fused modality image once using the diffuse noise predicted at the current time step;

[0017] Use the expectation maximization algorithm to perform a likelihood correction on the reconstructed fusion modality image predicted at the current time step;

[0018] The rectified prediction is used to reconstruct the fused modality image and perform a one-step denoising operation.

[0019] Furthermore, the generation process of the fused modality image is specifically as follows:

[0020] The predicted diffuse noise is:

[0021] s=s θ (f t ,t)

[0022] Where s represents the diffusion noise predicted by the model, t represents the current time step, and f t represents the fused modality image at time step t, s θ represents the pre-trained diffusion model;

[0023] The estimation of the reconstructed fusion modality image is:

[0024]

[0025] Among them, f0 represents the output image of the diffusion fusion model at the last time step, that is, the reconstructed fusion modality image, is the prediction result of the diffusion fusion model on the reconstructed fusion modality image f0 at time step t, is the preset hyperparameter of the diffusion model;

[0026] The prediction result of the reconstructed fusion modality image f0 at the current time step is corrected by the expectation maximization algorithm once, and the predicted image is:

[0027]

[0028] in, Reconstructed fused modality image representing the prediction for the current time step The predicted image obtained after likelihood correction, h represents an iteration process of the expectation maximization algorithm, I and V represent the visible light modality and infrared modality images used to synthesize the fusion modality image, respectively;

[0029] The fused modal image at the next time step obtained after one-step denoising operation using the corrected predicted reconstructed fused modal image is:

[0030]

[0031] Among them, f t-1 represents the fused modality image at time step t-1, α t , β tis the preset hyperparameter of the diffusion model, is the bias value set to zero, and z is sampled from a Gaussian distribution A random variable.

[0032] Furthermore, the fusion modality generation module is made to approach the multimodal image fusion capability of the diffusion fusion model through the following knowledge distillation process:

[0033] Use Kaiming method to initialize the weights of the fusion modality generation module;

[0034] For each visible light-infrared modality image pair in the preset training set, a fused image is synthesized using the fusion modality generation module;

[0035] Calculate the mean square error between the fusion image synthesized by the fusion modality generation module and the fusion modality image synthesized by the saved diffusion fusion model, and use this as the loss function to perform a back propagation to update the parameters of the fusion modality generation module;

[0036] After the training converges, save the weights of the fusion modality generation module;

[0037] Among them, the mean square error between the fused image synthesized by the fusion modality generation module and the fused modality image synthesized by the saved diffusion fusion model is calculated as follows:

[0038]

[0039] in, represents the mean square error, g represents the fusion modality generation module, p represents the diffusion fusion model, R and T represent the images of visible light and infrared modalities in the preset training set respectively, and K represents the number of samples in the training set.

[0040] Furthermore, the fused modality generation module includes: a cross-modality contraction module, a cross-modality attention module and a fused modality decoding module;

[0041] The cross-modal shrinkage module and the fused modality decoding module are implemented by the encoder and decoder of the U-Net neural network respectively, and the cross-modal attention module is implemented by several convolutional layers and a single-layer, four-head cross-attention Transformer encoder. Both visible light and infrared images are input into the cross-modal shrinkage module and the cross-modal attention module. The sum of the output features of these two modules is the input of the fused modality decoding module, which decodes them into a three-channel fused modality image.

[0042] The forward propagation process of each part is shown in the following formula:

[0043] Cross-modal contraction module:

[0044] F e=f e ([R; T])

[0045] where [·;·] represents the channel merging operation, f e represents the encoder network of the cross-modal contraction module, F e represents the encoding result of the cross-modal contraction module, R and T represent visible light and infrared images respectively;

[0046] Cross-modal Attention Module:

[0047] X r =ξ(R),X t =ξ(T)

[0048] Q=X t W q ,K=X r W k ,V=X r W v

[0049] H=Att(Q,K,V)+X t

[0050] F h =FFN(H)+H

[0051] Where ξ represents the convolutional layer in the cross-modal attention module, Q, K, and V represent the embedding vectors of Query, Key, and Value in the attention mechanism, respectively, and W q , W k and W v is the projection weight of Q, K, and V in the multi-head attention. FFN represents the feedforward neural network in Transformer. h Represents the encoding result of the cross-modal attention module; is the scale coefficient, Softmax(·) is the normalized exponential function;

[0052] Fusion modality decoding module:

[0053] F=f d (F e +F h )

[0054] Among them, f d represents the decoder network of the fusion modality decoding module, F e and F h are the output features of the cross-modal contraction module and the cross-modal attention module respectively, and F is the three-channel fused modality image decoded by the fused modality decoding module.

[0055] Furthermore, the learnable fusion modality counting framework based on efficient cloning of diffusion fusion model consists of the following structures: feature extraction module, fusion modality generation module and regression head;

[0056] The feature extraction module consists of a VGG-19 convolutional neural network and a two-layer, six-head Transformer encoder, and the regression head consists of three layers of convolutional neural networks with a convolution kernel size of 3×3 and two ReLU activation layers in between.

[0057] The forward propagation process of the counting framework is shown in the following formula:

[0058] F=g(R,T)

[0059] D=ρ(ξ(R)+ξ(F)+ξ(T))

[0060] Among them, F represents the fused modality image, R and T represent the visible light and infrared modality images, D represents the density map generated by the counting framework, ξ, ρ and g represent the feature extraction module, regression head and fused modality generation module in the counting framework respectively;

[0061] When fine-tuning for counting tasks, the loss function used is shown in the following formula:

[0062]

[0063] in, represents the counting loss, M represents the total number of targets in the sample, and z n is the coordinate of the nth labeled point in the sample, <·,·> represents the vector inner product operation, Indicates that the mean is μ and the variance is σ 2 Gaussian distribution of , D represents the density map output by the counting framework.

[0064] Furthermore, the specific steps of using the trained neural network model to synthesize the visible light and infrared modality images into a ghost-free fused image are as follows:

[0065] Use a multimodal target counting method based on learnable fusion modalities to complete model training and save the trained weights;

[0066] Separate the fused modality generation module and its weights from the counting framework;

[0067] The visible light and infrared modality images to be fused are input into the fusion modality generation module to obtain the fusion modality image and save it.

[0068] Furthermore, the process of obtaining a ghost-free fused image is shown in the following formula:

[0069] F * =g* (R,T)

[0070] Among them, F * represents the ghost-free fused image, g * represents the fusion modality generation module with the weights after target count training, and R and T represent the visible light and infrared modality images to be fused, respectively.

[0071] Compared with the prior art, the present invention has the following advantages:

[0072] The visible light-infrared modality image fusion ghost removal method proposed in the present invention reversely optimizes the fused image generation module based on the downstream multimodal target counting task, thereby removing ghosts in the fused image. Compared with existing deep learning and graphics methods based on image registration or anti-aliasing, the present invention has better ghost removal effects, especially in scenes that are widely seen in practical applications such as dense crowds. At the same time, in the inference stage, the present invention only needs to use a lightweight fused image generation module to complete the synthesis of ghost-free fused images. The number of parameters and floating-point calculations are lower than those of existing methods, the operation time is shorter, and the requirements for computing equipment are lower. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] Figure 1 The present invention is a flowchart of a method for removing ghosting in visible light-infrared image fusion according to an embodiment of the present invention.

[0074] Figure 2 The present invention is a flowchart of a method for training a neural network model based on a multimodal target counting method of a learnable fusion modality according to an embodiment of the present invention.

[0075] Figure 3 A schematic diagram of a method for training a neural network model based on a multimodal target counting method of a learnable fusion modality according to an embodiment of the present invention.

[0076] Figure 4 The present invention is a flowchart of a method for combining a visible light-infrared image pair into a fused modality image using a diffusion fusion model according to an embodiment of the present invention.

[0077] Figure 5 This is a flow chart of a method for using a fusion modality generation module to approximate the multimodal image fusion capability of a diffusion fusion model through knowledge distillation according to an embodiment of the present invention.

[0078] Figure 6 Schematic diagram of the structure of a fusion modality generation module according to an embodiment of the present invention.

[0079] Figure 7 It is a structural schematic diagram of a learnable fusion modality counting framework based on efficient cloning of a diffusion fusion model according to an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0081] It should be noted that the terms "first", "second", etc. in the present invention are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0082] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0083] See also Figure 1 In one embodiment of the present invention, a multimodal image fusion de-ghosting method based on reverse optimization of downstream counting tasks is provided, specifically a multimodal target counting method based on a learnable fusion modality, and a method for synthesizing a ghost-free fused image using the multimodal target counting method based on a learnable fusion modality. The method can synthesize visible light and infrared modality images in the same scene into a ghost-free fused image, and can still work well in scenes with dense targets, with fast calculation speed and low requirements on computing power.

[0084] Specifically, the multimodal image fusion deghosting method based on reverse optimization of downstream counting tasks includes the following steps:

[0085] S1: Train the preset neural network model using a multimodal object counting method based on learnable fusion modalities.

[0086] S2: Use the trained neural network model to synthesize visible and infrared modality images into a ghost-free fused image.

[0087] Among them, the data set used to train the preset neural network can adopt the open source data set of the multimodal target counting task, and the visible light and infrared images to be synthesized are determined according to the actual working scenario.

[0088] The visible light-infrared modality image fusion de-ghosting method proposed in the present invention reversely optimizes the fused image generation module based on the downstream multi-modal target counting task, thereby removing ghosts in the fused image. This method can synthesize a visible light image and an infrared image of the same scene into a ghost-free fused image. Compared with existing deep learning and graphics methods based on image registration or anti-aliasing, the present invention has a better effect of removing ghosts, especially in scenes that are widely used in practical applications such as dense crowds. At the same time, in the reasoning stage, the present invention only needs to use a lightweight fused image generation module to complete the synthesis of ghost-free fused images. The number of parameters and floating-point calculations are lower than those of existing methods, the operation time is shorter, and the requirements for computing equipment are lower.

[0089] In a possible implementation, the multimodal target counting method based on learnable fusion modality can predict a density map for each visible light-infrared image pair, where the value of each pixel in the density map represents the number of targets in a corresponding small area in the original image, and the sum of all pixel values ​​in the density map represents the total number of targets in the original image predicted by the method. Figure 2 and Figure 3 , using the multimodal target counting method based on learnable fusion modality to train a preset neural network model can be obtained by the following steps:

[0090] S11: using a diffusion fusion model to combine each visible light-infrared image pair in the pre-selected dataset into a fused modality image and save it;

[0091] S12: Through knowledge distillation, a fusion modality generation module is used to approximate the multimodal image fusion capability of the diffusion fusion model;

[0092] S13: We use a learnable fusion modality counting framework based on efficient cloning of the diffusion fusion model to fine-tune the fusion modality generation module for multimodal object counting tasks.

[0093] See also Figure 4 Optionally, a method for combining a visible light-infrared image pair into a fused modality image using a diffusion fusion model is as follows:

[0094] S111: Use the pre-trained weights of the diffusion model to predict the diffusion noise of the current time step;

[0095] S112: using the diffuse noise predicted at the current time step to update the estimation of the reconstructed fusion modality image;

[0096] S113: using an expectation maximization algorithm to perform a likelihood correction on the reconstructed fusion modality image predicted at the current time step;

[0097] S114: Use the corrected predicted reconstructed fusion modality image to perform a one-step denoising operation.

[0098] The process is shown in the following formula:

[0099] Among them, the predicted diffusion noise is:

[0100] s=s θ (f t ,t)

[0101] Where s represents the diffusion noise predicted by the model, t represents the current time step, and f t represents the fused modality image at time step t, s θ represents the pre-trained diffusion model;

[0102] The estimation of the reconstructed fusion modality image is:

[0103]

[0104] Among them, f0 represents the output image of the diffusion fusion model at the last time step, that is, the reconstructed fusion modality image, is the prediction result of the diffusion fusion model on the reconstructed fusion modality image f0 at time step t, is the preset hyperparameter of the diffusion model;

[0105] The prediction result of the reconstructed fusion modality image f0 at the current time step is corrected by the expectation maximization algorithm once, and the predicted image is:

[0106]

[0107] in, Reconstructed fused modality image representing the prediction for the current time step The predicted image obtained after likelihood correction, h represents an iteration process of the expectation maximization algorithm, I and V represent the visible light modality and infrared modality images used to synthesize the fusion modality image, respectively;

[0108] The fused modal image at the next time step obtained after one-step denoising operation using the corrected predicted reconstructed fused modal image is:

[0109]

[0110] Among them, f t-1 represents the fused modality image at time step t-1, α t , β t is the preset hyperparameter of the diffusion model, is a bias value that is usually set to zero, and z is sampled from a Gaussian distribution A random variable.

[0111] In the specific implementation, the pre-trained weights of the diffusion sampling denoising model are used without fine-tuning. The maximum time step number of the diffusion model is 1000.

[0112] See also Figure 5 ,Optionally, through knowledge distillation, the fusion modality generation module is used to approximate the multimodal image fusion capability of the diffusion fusion model can be ,achieved by the following methods:

[0113] S121: Initialize the weights of the fusion modality generation module using the Kaiming method;

[0114] S122: for each visible light-infrared modality image pair in the preset training set, synthesize a fused image using a fusion modality generation module;

[0115] S123: calculating the mean square error between the fusion image synthesized by the fusion modality generation module and the fusion modality image synthesized by the stored diffusion fusion model, and performing a back propagation using this as the loss function to update the parameters of the fusion modality generation module;

[0116] S124: After the training converges, the weights of the fusion modality generation module are saved.

[0117] Among them, the mean square error between the fused image synthesized by the fusion modality generation module and the fused modality image synthesized by the saved diffusion fusion model can be calculated as follows:

[0118]

[0119] in, represents the mean square error, g represents the fusion modality generation module, p represents the diffusion fusion model, R and T represent the images of visible light and infrared modalities in the preset training set respectively, and K represents the number of samples in the training set.

[0120] See also Figure 6 Optionally, the above-mentioned fused modality generation module includes: a cross-modal shrinkage module, a cross-modal attention module and a fused modality decoding module.

[0121] Among them, the cross-modal shrinkage module and the fused modality decoding module are implemented by the encoder and decoder of the U-Net neural network respectively, and the cross-modal attention module is implemented by several convolutional layers and a single-layer, four-head cross-attention Transformer encoder; both visible light and infrared images pass through the cross-modal shrinkage module and the cross-modal attention module, and the sum of the output features of the two modules is the input of the fused modality decoding module, which decodes it into a three-channel fused modality image.

[0122] The forward propagation process is shown in the following formula:

[0123] F e =f e ([R; T])

[0124] where [·;·] represents the channel merging operation, f e represents the encoder network of the cross-modal contraction module, F e represents the encoding result of the cross-modal contraction module, R and T represent visible light and infrared images respectively;

[0125] X r =ξ(R),X t =ξ(T)

[0126] Q=X t W q ,K=X r W k ,V=X r W v

[0127] H=Att(Q,K,V)+X t

[0128] F h =FFN(H)+H

[0129] Where ξ represents the convolutional layer in the cross-modal attention module, Q, K, and V represent the embedding vectors of Query, Key, and Value in the attention mechanism, respectively, and W q , W k and W v is the projection weight of Q, K, and V in the multi-head attention. FFN represents the feedforward neural network in Transformer. h Represents the encoding result of the cross-modal attention module; is the scale coefficient, Softmax(·) is the normalized exponential function;

[0130] F=f d (F e +F h )

[0131] Among them, f d represents the decoder network of the fusion modality decoding module, F e and F h are the output features of the cross-modal contraction module and the cross-modal attention module respectively, and F is the three-channel fused modality image decoded by the fused modality decoding module.

[0132] In the specific implementation, the cross-modal shrinkage module upgrades the 6-channel image after channel merging to a 256-channel feature map, and downsamples the length and width of the image to 1 / 8 of the original. The fused modality decoding module resamples the length and width of the 256-channel feature map back to the size of the original image and generates a 3-channel fused image. The embedding dimension of the cross-modal attention module is 256, and the number of image blocks is 64. During training, an NVIDIA RTX 3090 graphics card with 24GB of video memory is used in the Pytorch framework. The Adam optimizer is used to guide gradient descent, with an initial learning rate of 0.00001 and a weight decay of 0.0001. In order to improve the generalization of the model, the images of the training set are randomly cropped to 224×224 size, and horizontal flipping is performed with a probability of 50% as random enhancement. The maximum iteration round of training is 1000, and the batch size is 1.

[0133] See also Figure 7 ,The optional learnable fusion modality counting framework based on ,efficient cloning of diffusion fusion model consists of the ,following structures: feature extraction module, fusion modality generation module and ,regression head.

[0134] The feature extraction module consists of a VGG-19 convolutional neural network and a two-layer, six-head Transformer encoder, and the regression head consists of three layers of convolutional neural networks with a convolution kernel size of 3×3 and two ReLU activation layers in between.

[0135] The forward propagation process of the counting framework is shown in the following formula:

[0136] F=g(R,T)

[0137] D=ρ(ξ(R)+ξ(F)+ξ(T))

[0138] Among them, F represents the fused modality image, R and T represent the visible light and infrared modality images, D represents the density map generated by the counting framework, ξ, ρ and g represent the feature extraction module, regression head and fused modality generation module in the counting framework respectively;

[0139] When fine-tuning for counting tasks, the loss function used is shown in the following formula:

[0140]

[0141] in, represents the counting loss, M represents the total number of targets in the sample, and z n is the coordinate of the nth labeled point in the sample, <·,·> represents the vector inner product operation, Indicates that the mean is μ and the variance is σ 2 Gaussian distribution of , D represents the density map output by the counting framework.

[0142] In the specific implementation, the VGG-19 neural network downsamples the length and width of the input image to 1 / 8 of the original, and increases the number of channels of the original image from 3 to 512. The Transformer encoder increases the feature map of 512 channels to 768 channels without changing the length and width of the feature map. The embedding dimension of the Transformer encoder is 768, and the number of blocks of the image is 196. During training, an NVIDIA RTX 3090 graphics card with 24GB of video memory is used in the Pytorch framework. The Adam optimizer is used to guide gradient descent, with an initial learning rate of 0.00001 and a weight decay of 0.0001. In order to improve the generalization of the model, the images of the training set are randomly cropped to 224×224 size, and horizontal flipping is performed with a probability of 50% as random enhancement. The maximum iteration round of training is 400, and the batch size is 1.

[0143] Optionally, according to the multimodal target counting method based on the learnable fusion modality, a method for synthesizing a ghost-free fusion image is as follows:

[0144] Use a multimodal target counting method based on learnable fusion modalities to complete model training and save the trained weights;

[0145] Separate the fused modality generation module and its weights from the counting framework;

[0146] Input the visible light and infrared modality images to be fused into the fusion modality generation module to obtain the fusion modality image and save it;

[0147] The process of obtaining a ghost-free fused image is shown in the following formula:

[0148] F * =g * (R,T)

[0149] Among them, F * represents the ghost-free fused image, g * represents the fusion modality generation module with the weights after target count training, and R and T represent the visible light and infrared modality images to be fused, respectively.

[0150] The division of modules in the embodiments of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation. In addition, each functional module in each embodiment of the present invention may be integrated into one processor, or may exist physically separately, or two or more modules may be integrated into one module. The above-mentioned integrated modules may be implemented in the form of hardware or in the form of software functional modules.

[0151] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0152] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0153] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0154] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0155] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A multimodal image fusion deghosting method based on reverse optimization of downstream counting tasks, characterized in that: The method selects multimodal target counting as the downstream task of visible light-infrared image fusion, and optimizes the fusion modality generation module through back propagation in the configuration of the downstream task, so as to achieve the effect of removing ghosts in the fusion modality image; the specific steps of the method are as follows: Train the preset neural network model using a multimodal target counting method based on learnable fusion modalities; The visible and infrared modality images are synthesized into a ghost-free fused image using a trained neural network model.

2. The method according to claim 1, characterized in that: In the multimodal target counting method that can learn fusion modalities, for each visible light-infrared image pair, a density map can be predicted. The value of each pixel in the density map represents the number of targets in a corresponding small area in the original image, and the sum of all pixel values ​​in the density map represents the total number of targets in the original image predicted by the method.

3. The method according to claim 2, characterized in that: The preset neural network model is trained in the following manner: Each visible-infrared image pair in the pre-selected dataset is synthesized into a fused modality image using a diffusion fusion model and saved; Through knowledge distillation, a fusion modality generation module is used to approximate the multimodal image fusion capability of the diffusion fusion model; A learnable fusion modality counting framework based on efficient cloning of diffusion fusion model is used to fine-tune the fusion modality generation module for multimodal object counting tasks.

4. The method according to claim 3, characterized in that: The diffusion fusion model combines each visible light-infrared image pair in the pre-selected dataset into a fused modality image through the following process: Use the pre-trained weights of the diffusion model to predict the diffusion noise at the current time step; Update the estimate of the reconstructed fused modality image once using the diffuse noise predicted at the current time step; Use the expectation maximization algorithm to perform a likelihood correction on the reconstructed fusion modality image predicted at the current time step; The rectified prediction is used to reconstruct the fused modality image and perform a one-step denoising operation.

5. The method according to claim 4, characterized in that: The generation process of the fused modality image is as follows: The predicted diffuse noise is: s=s θ (f t ,t) Where s represents the diffusion noise predicted by the model, t represents the current time step, and f t represents the fused modality image at time step t, s θ represents the pre-trained diffusion model; The estimation of the reconstructed fusion modality image is: Among them, f0 represents the output image of the diffusion fusion model at the last time step, that is, the reconstructed fusion modality image, is the prediction result of the diffusion fusion model on the reconstructed fusion modality image f0 at time step t, is the preset hyperparameter of the diffusion model; The prediction result of the reconstructed fusion modality image f0 at the current time step is corrected by the expectation maximization algorithm once, and the predicted image is: in, Reconstructed fused modality image representing the prediction for the current time step The predicted image obtained after likelihood correction, h represents an iteration process of the expectation maximization algorithm, I and V represent the visible light modality and infrared modality images used to synthesize the fusion modality image, respectively; The fused modal image at the next time step obtained after one-step denoising operation using the corrected predicted reconstructed fused modal image is: Among them, f t-1 represents the fused modality image at time step t-1, α t , β t is the preset hyperparameter of the diffusion model, is the bias value set to zero, and z is sampled from a Gaussian distribution A random variable.

6. The method according to claim 5, characterized in that: The fusion modality generation module is made to approach the multimodal image fusion capability of the diffusion fusion model through the following knowledge distillation process: Use Kaiming method to initialize the weights of the fusion modality generation module; For each visible light-infrared modality image pair in the preset training set, a fused image is synthesized using the fusion modality generation module; Calculate the mean square error between the fusion image synthesized by the fusion modality generation module and the fusion modality image synthesized by the saved diffusion fusion model, and use this as the loss function to perform a back propagation to update the parameters of the fusion modality generation module; After the training converges, save the weights of the fusion modality generation module; Among them, the mean square error between the fused image synthesized by the fusion modality generation module and the fused modality image synthesized by the saved diffusion fusion model is calculated as follows: in, represents the mean square error, g represents the fusion modality generation module, p represents the diffusion fusion model, R and T represent the images of visible light and infrared modalities in the preset training set respectively, and K represents the number of samples in the training set.

7. The method according to claim 6, characterized in that: The fused modality generation module includes: a cross-modality contraction module, a cross-modality attention module, and a fused modality decoding module; The cross-modal shrinkage module and the fused modality decoding module are implemented by the encoder and decoder of the U-Net neural network respectively, and the cross-modal attention module is implemented by several convolutional layers and a single-layer, four-head cross-attention Transformer encoder. Both visible light and infrared images are input into the cross-modal shrinkage module and the cross-modal attention module. The sum of the output features of these two modules is the input of the fused modality decoding module, which decodes them into a three-channel fused modality image. The forward propagation process of each part is shown in the following formula: Cross-modal contraction module: F e =f e ([R;T]) where [·;·] represents the channel merging operation, f e represents the encoder network of the cross-modal contraction module, F e represents the encoding result of the cross-modal contraction module, R and T represent visible light and infrared images respectively; Cross-modal attention module: X r =ξ(R),X t =ξ(T) Q=X t W q ,K=X r W k ,V=X r W v H=Att(Q, K, V)+X t F h =FFN(H)+H Where ξ represents the convolutional layer in the cross-modal attention module, Q, K, and V represent the embedding vectors of Query, Key, and Value in the attention mechanism, respectively, and W q , W k and W v is the projection weight of Q, K, and V in the multi-head attention. FFN represents the feedforward neural network in Transformer. h Represents the encoding result of the cross-modal attention module; is the scale coefficient, Softmax(·) is the normalized exponential function; Fusion modality decoding module: F=f d (F e +F h ) Among them, f d represents the decoder network of the fusion modality decoding module, F e and F h are the output features of the cross-modal contraction module and the cross-modal attention module respectively, and F is the three-channel fused modality image decoded by the fused modality decoding module.

8. The method according to claim 7, characterized in that: The learnable fusion modality counting framework based on efficient cloning of diffusion fusion model consists of the following structures: feature extraction module, fusion modality generation module and regression head; The feature extraction module consists of a VGG-19 convolutional neural network and a two-layer, six-head Transformer encoder, and the regression head consists of three layers of convolutional neural networks with a convolution kernel size of 3×3 and two ReLU activation layers in between. The forward propagation process of the counting framework is shown in the following formula: F = g(R, T) D=ρ(ξ(R)+ξ(F)+ξ(T)) Among them, F represents the fused modality image, R and T represent the visible light and infrared modality images, D represents the density map generated by the counting framework, ξ, ρ and g represent the feature extraction module, regression head and fused modality generation module in the counting framework respectively; When fine-tuning for counting tasks, the loss function used is shown in the following formula: in, represents the counting loss, M represents the total number of targets in the sample, and z n is the coordinate of the nth labeled point in the sample, <·,·> represents the vector inner product operation, Indicates that the mean is μ and the variance is σ 2 Gaussian distribution of , D represents the density map output by the counting framework.

9. The method according to claim 1, characterized in that: The specific steps of using the trained neural network model to synthesize the visible light and infrared modality images into a ghost-free fused image are as follows: Use a multimodal target counting method based on learnable fusion modalities to complete model training and save the trained weights; Separate the fused modality generation module and its weights from the counting framework; The visible light and infrared modality images to be fused are input into the fusion modality generation module to obtain the fusion modality image and save it.

10. The method according to claim 9, characterized in that: The process of obtaining a ghost-free fused image is shown in the following formula: F*=g*(R,T) Among them, F * represents the ghost-free fused image, g * represents the fusion modality generation module with the weights after target count training, and R and T represent the visible light and infrared modality images to be fused, respectively.

Images