Unsupervised multi-modal image fusion method

Abstract

The invention relates to an unsupervised multi-modal image fusion method. The method comprises the following steps: constructing a data set based on visible light and infrared multi-source modal videos and image registration data in multiple scenes; building a fusion model, wherein the structure of the fusion model is based on a convolutional neural network containing a residual module; building adiscrimination model; designing a loss function of the generative adversarial model, which is multi-source information loss and is used for improving the multi-source information retention capabilityof the fusion network; using the similarity loss for judging the similarity between the fusion result and the source image; using the adversarial loss for united training direction constraints between the fusion network and the discrimination network; and a step 5, carrying out model joint adversarial training through an iteration step.

Description

technical field [0001] The invention belongs to the fields of deep learning, computer vision and image fusion, and relates to an unsupervised end-to-end, infrared and visible light multimodal image fusion method based on generative confrontation learning and twin network. Background technique [0002] Constrained by the imaging mechanism, all necessary information cannot be obtained from an image of a single source modality. Compared with visible light image (VI, Visible Image), infrared image (IR, Infrared Image) has the following characteristics: it can reduce external influences such as sunlight and smoke, and is sensitive to targets and areas with obvious infrared thermal characteristics. But at the same time, visible light images have higher spatial resolution, richer texture structure details and better human visual feedback [1]. [0003] The task of image fusion (Information Fusion) is to generate a fused image for subsequent visual perception and processing for the ...

Images