A cross-modal image fusion method based on image disentangled representation

Abstract

The application discloses a cross-modal image fusion method based on image disentangled representation, designs a content encoder and a modal encoder based on dense connection, and performs disentangled representation decomposition on content components and modal components of cross-modal images; a reconstruction decoder is further designed, and a multi-scale fusion structure is formed through a multi-scale long connection between the reconstruction decoder and the content encoder; the reconstruction decoder performs convolution reconstruction on the content components; then, a discriminator is designed, reconstruction of cross-modal image content and modal components, modal conversion reconstruction and cyclic decomposition reconstruction are performed through a generative adversarial network, and the content encoder, the modal encoder and the reconstruction decoder are trained. A fusion decoder with double-branch dense connection is further designed, and a fusion image is generated through a multi-scale fusion manner. The fusion image is subjected to disentangled representation decomposition, content reconstruction loss and modal reconstruction loss are constructed through content components and modal components of the fusion image and source images, and the fusion image is ensured to reserve information of the source image to the greatest extent.

Description

Technical Field

[0001] This invention belongs to the technical field of feature separation, saliency extraction, information fusion, and image reconstruction of cross-modal images generated by multi-source sensors, and specifically relates to a cross-modal image fusion method based on image deentanglement representation. Background Technology

[0002] Cross-modal images (CMIs) refer to a set of images captured by multiple sensors, representing the same scene in different forms. Their key characteristic is that different modalities can represent information with different emphases, thus describing scene features from different angles and orientations. However, with the development of sensor technologies, the representation of cross-modal images has become more specialized and diversified. The information presented by a single image often cannot fully describe the scene, making subsequent scene analysis and applications difficult. Therefore, to improve the limitations of single images in describing scene information and to integrate the useful information from each cross-modal image, enabling comprehensive and multi-level scene analysis, the concept of image fusion (IF) is proposed. Image fusion extracts information of interest from cross-modal images separately and fuses them into a composite image according to certain rules. The generated fused image maintains maximum similarity to all input images while having higher information aggregation, greater information content, more significant features, uniform grayscale distribution, clear edge contours, and effectively reduces data dimensionality and storage requirements. It can be widely used in many fields such as medical diagnosis, geological exploration, environmental monitoring, equipment safety monitoring, military reconnaissance, virtual reality, and multimedia entertainment.

[0003] However, due to the significant differences in imaging principles among multi-source sensors, there are substantial differences in the way scene features are represented, sometimes even with completely opposite or mutually exclusive features. Therefore, using general feature extraction methods to extract these features will lead to errors in feature representation, resulting in ghosting or ringing effects in the fused image and affecting the fusion quality. Simultaneously, appropriate rules need to be set to process and fuse these significantly different features, thereby avoiding a decrease in overall image contrast and loss of significant information due to feature averaging and processing errors. The key to cross-modal image fusion lies in the extraction and processing of these mutually exclusive features, which differs significantly from general image feature extraction and fusion methods, and is also the challenge and focus of this invention.

[0004] Currently, when fusing cross-modal images, manually defined, globally uniform feature extraction methods and fusion rules are typically used to tailor specific features or tasks, thereby improving fusion quality. However, this approach often leads to a decrease in local fusion accuracy due to the global nature of the rules, and the fusion effect is unstable when the fusion task is unknown, resulting in poor model generalization. Furthermore, while convolutional neural network-based fusion methods can adaptively extract and fuse features, the lack of real-world fusion images for supervision makes it difficult to guarantee fusion quality. Summary of the Invention

[0005] To address the shortcomings of existing technologies, the present invention aims to provide a cross-modal image fusion method based on image disentangled representation. This method utilizes image disentangled representation (DR) decomposition to obtain the image's content and modal components. Then, through content component reconstruction and modal component injection based on a Generative Adversarial Network (GAN), it avoids interference from mutually exclusive features on content feature reconstruction, achieving adaptive feature extraction and fusion.

[0006] To achieve the above-mentioned objectives, the present invention adopts the following concept:

[0007] Based on the prior information that cross-modal images image the same scene, there should be a shared high-level semantic feature space among them. The feature vectors in this space can constitute the basic representation of the scene information in the image, containing most of the image's energy information. The different ways in which cross-modal images represent the scene can be regarded as different rendering methods of a certain feature vector in this shared feature space by different sensors. These different rendering methods are unique to each cross-modal image, constituting their own independent feature spaces, which contain the remaining small portion of the image's energy information. Considering this characteristic of cross-modal images, we use image deentanglement representation to separate these two spaces, where the shared space is called the content component and the unique space is called the modal component. Since the content component contains most of the image's energy, fusing and reconstructing it can obtain the overall structure and most of the feature information of the fused image. And since the proportion of different image modal components in the fused component can adjust the bias of the image rendering method, the fusion of modal components can be flexibly adjusted according to application requirements. By separating and fusing content and modality separately, the network can focus on feature extraction, avoid the influence of mutually exclusive features on content reconstruction, and is unaffected by the type of cross-modal image. It is a general cross-modal image fusion framework that can generate high-quality fused images.

[0008] Based on the above inventive concept, the present invention adopts the following technical solution:

[0009] A cross-modal image fusion method based on image deentanglement representation is proposed. The method involves decomposing the cross-modal image using a densely connected content encoder and modality encoder with deentanglement representation. The obtained content components are then reconstructed in a densely connected fusion decoder using a multi-scale fusion approach. Finally, the modality components are injected into the content reconstruction using an adaptive normalization method to obtain the fused image. The details are as follows:

[0010] Step 1: Design a densely connected content encoder, reconstruction decoder, and modal encoder. Input the unpaired cross-modal images from training set 1 and encode them to obtain the content and modal components of the images.

[0011] Step 2: Reconstruct, transform, and recursively decompose the content and modal components of the image from Step 1. Design a discriminator by training the content of the generative adversarial network, the modal encoder, and the reconstruction decoder.

[0012] Step 3: Repeat steps 1 and 2 until all images in training set 1 have been trained and the set number of iterations has been reached, to obtain the pre-trained content, modal encoder, and reconstruction decoder.

[0013] Step 4: Load the content and modal encoder from Step 3, input the cross-modal images from Training Set 2, and obtain the content and modal components;

[0014] Step 5: Design a densely connected fusion decoder and input the content and modal components from Step 4 to obtain the fused image. Decompose the fused image into its unentangled representation to obtain the reconstructed content and modal components. Train the fusion decoder using the fused image, the reconstructed content and modal components, and the generative adversarial network.

[0015] Step 6: Repeat steps 4 and 5 until all images in training set 2 have been trained to obtain the trained fusion model.

[0016] Compared with the prior art, the present invention has the following outstanding substantive features and significant progress:

[0017] This invention leverages the shared content feature space and unique modal feature space of cross-modal images. Based on generative adversarial networks (GANs), it constructs separate content encoders and modal encoders to encode and extract content and modal components, and generates a fused image through a constructed fusion decoder. Since the mutually exclusive features between cross-modal images are essentially due to large differences in pixel values ​​caused by different sensor imaging methods, the fusion decoder uses a "content component reconstruction, modal component injection" approach to generate the fused image, which eliminates the interference of mutually exclusive features on the fusion process. Furthermore, this method employs an unsupervised end-to-end generation approach, eliminating the need for real-image supervision during training and directly generating the fused image. This avoids the complex process of manually formulating fusion rules and feature extraction methods, allowing the network to focus on feature extraction and fusion, thus improving the model's generalization and versatility. Attached Figure Description

[0018] Figure 1 This is the overall step of cross-modal image fusion based on image deentanglement representation in the present invention.

[0019] Figure 2 This is the network data flow framework of the present invention.

[0020] Figure 3 The generator structure of this invention includes content, modal encoder and fusion decoder.

[0021] Figure 4 This is the discriminator structure of the present invention.

[0022] Figure 5 This is a training loss diagram for the present invention.

[0023] Figure 6 This is a schematic diagram illustrating the principle of the cross-modal image fusion method based on image deentanglement representation of the present invention. Detailed Implementation

[0024] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0025] like Figure 1 , Figure 2 and Figure 6 As shown, a cross-modal image fusion method based on image deentanglement representation is implemented as follows:

[0026] Step 1: Design a densely connected content encoder, reconstruction decoder, and modal encoder. Input the unpaired cross-modal images from training set 1 and encode them to obtain the content and modal components of the images.

[0027] In step 1, the characteristics of cross-modal images in the high-level semantic space are utilized to separate their shared content feature space from their unique modal feature space. For example... Figure 3 As shown, to perform complete decomposition, a densely connected content encoder and modality encoder were designed and trained. Since the training at this stage does not involve image fusion, the training dataset uses a larger number of unpaired cross-modal images. Taking a two-input network as an example, let the input unpaired cross-modal image be represented as (x1, x2), and the content encoder be... Modal encoder is The image deentanglement representation is then decomposed as follows:

[0028]

[0029] Among them, (c x1 c x2 ) represents the content components of a cross-modal image, (m x1 m x2 ) represents the modal components.

[0030] Step 2: Reconstruct, transform, and reconstruct the content and modal components of the image from Step 1, such as... Figure 4 As shown, the discriminator is designed by generating adversarial network training content, modality encoder, and reconstruction decoder;

[0031] 1) In step 2, firstly, to ensure that the original image can be reconstructed from the obtained content and modal components after image deentanglement, the (c) in step 1 is... x1 c x2 ) and (m x1 m x2 Input to the reconstruction decoder Refactoring is performed in the following ways:

[0032]

[0033] The reconstruction decoder uses convolutional layers to concatenate content components and then reconstructs the image. Adaptive Instance Normalization (AdaIN) is used to inject modal components into the reconstruction process to prevent them from interfering with the expression of content features. Furthermore, the reconstruction decoder and content encoder form a multi-scale fusion structure through long connections, enhancing the expressive hierarchy of the fused image. Finally, the reconstruction loss is calculated between the reconstructed image and the source image.

[0034] L rec =||x1-x 1→1 ||1+||x2-x 2→2 ||1#(3)

[0035] Where ||·||1 represents the L1 norm. The reconstruction loss enables the content encoder and modality encoder to perform accurate feature extraction, and is the most fundamental guarantee for feature reconstruction.

[0036] 2) In step 2, to ensure that the image content and modal components are completely stripped away, (c1, c2) from step 1 is modally converted and reconstructed to generate a modally converted image, i.e.:

[0037]

[0038] Since modality-transformed images cannot be directly L1-normated with the original images, adversarial loss is used to measure their similarity to the corresponding modality source images.

[0039] L cyc_adv =log(Dis1(x1))+log(1-Dis1(x)) 2→1 ))+log(Dis2(x2))+log(1-Dis2(x 1→2 ))#(5)

[0040] Ideally, the discriminator cannot distinguish between a true cross-modal image and a modality transfer image of the corresponding modality. This indicates that the content encoder can strip away the shared content components between cross-modal images and generate images of other modalities through the unique modal components encoded by the modality encoder.

[0041] 3) In step 2, to further improve the performance of the content and modal encoder, the modal transfer image is decomposed and reconstructed again using unentangled representation; this is called cyclic decomposition and reconstruction. First, the modal transfer image is decomposed:

[0042]

[0043] Then perform a loop reconstruction:

[0044]

[0045] Cyclic decomposition loss is used to measure the amount of feature loss during mode transfer:

[0046]

[0047] Cyclic reconstruction loss is used between the recurrently reconstructed image and the source image to enhance the accuracy of content and modality encoder feature extraction and the integrity of separation:

[0048] L cyc_pix =||x 1→2→1 -x1||1+||x 2→1→2 -x2||1#(9)

[0049] Step 3: Repeat steps 1 and 2 until all images in training set 1 have been trained and the set number of iterations has been reached, to obtain the pre-trained content, modal encoder, and reconstruction decoder.

[0050] In step 3, as Figure 5 As shown, the algorithm first checks whether the images in training set 1 have been trained. If so, it is considered to have completed one iteration, and the iteration count is automatically incremented by one. Otherwise, training continues in the current iteration count. Then, it checks whether the set number of iterations has been reached. If so, training stops to obtain the trained content, modal encoder, and reconstruction decoder; otherwise, iterative training continues.

[0051] Step 4: Load the content and modal encoder from Step 3, input the cross-modal images from Training Set 2, and obtain the content and modal components.

[0052] In step 4, to train the fusion decoder, the images in training set two first need to be decomposed into unentangled representations using the pre-trained module from step 3. Since image fusion is involved, the images in training set two need to be strictly registered image pairs, denoted as (y1, y2). Then, the pre-trained module is loaded to decompose the image pairs:

[0053]

[0054] This invention uses a pre-training mode, pre-training the content, modal encoder, and reconstruction decoder first, and then training the fusion decoder. This approach is more stable and results in faster network convergence than training these modules simultaneously.

[0055] Step 5: Design a densely connected fusion decoder and input the content and modal components from Step 4 to obtain a fused image. Decompose the fused image into an unentangled representation to obtain the reconstructed content and modal components. Train the fusion decoder using the fused image, the reconstructed content and modal components, and a generative adversarial network.

[0056] 1) In step 5, the content from step 4 and the modal components are first input into the dual-branch densely connected fusion decoder D. f In this process, a fused image is generated through multi-scale fusion reconstruction:

[0057] f = D f (c y1 c y2 ,AdaIN(m y1 ), AdaIN(m y2 ))#(11)

[0058] AdaIN(·) is an adaptive normalization method. The content components of the two cross-modal images are concatenated and directly fed into the convolutional layer for reconstruction, while the modal components are normalized by AdaIN at each convolutional layer, thus being injected into the reconstruction of the content components and affecting the rendering method of the fused image. After obtaining the fused image, in order to train the fused image to have the highest similarity to the input cross-modal image, this invention proposes a fusion pixel loss:

[0059] L fus_pix =||f-y1||1+||f-y2||1#(12)

[0060] Combining and counteracting losses:

[0061] L fus_adv =log(Dis1(y1))+log(1-Dis1(f))+log(Dis2(γ2))+log(1-Dis2(f))#(13)

[0062] 2) In step 5, to ensure that the fused image contains all the information of the source image to the greatest extent, the fused image is decomposed into unentangled representations using the content and modal encoder from step 3:

[0063]

[0064] Then, a reconstruction loss is calculated between the content and modal components in step 4 and the content and modal components of the fused image to measure the lost content and modal information in the fused image:

[0065] L fus_rec =||c f1 -c y1 ||1+||c f2 -c y2 ||1+||m f1 -m y1 ||1+||m f2 -m y2 ||1#(15)

[0066] Step 6: Repeat steps 4 and 5 until all images in training set 2 have been trained to obtain the trained fusion model.

[0067] In step 6, it is first determined whether the images in training set two have been trained. If so, it is considered that one iteration has been completed, and the iteration count is automatically incremented by one; otherwise, training continues in the current iteration count. Then, it is determined whether the set number of iterations has been reached. If so, training stops to obtain the trained fusion decoder; otherwise, iterative training continues.

[0068] The cross-modal image fusion method based on image deentanglement representation described in the above embodiments of the present invention first designs a content encoder and a modal encoder based on dense connections to decompose the content components and modal components of the cross-modal image into deentangled representations. Then, a reconstruction decoder is designed, and a multi-scale fusion structure is formed between it and the content encoder through multi-scale long connections. The reconstruction decoder performs convolutional reconstruction on the content components and injects the adaptive parameters generated by the modal components through a multilayer perceptron into the content reconstruction process through adaptive normalization, avoiding interference from mutually exclusive modal features. Next, a discriminator is designed, and a generative adversarial network is used to train the content encoder, modal encoder, and reconstruction decoder through the reconstruction of cross-modal image content and modal components, modality transformation reconstruction, and cyclic decomposition reconstruction. Simultaneously, a dual-branch densely connected fusion decoder is designed to generate a fused image through multi-scale fusion. To improve the performance of the fusion decoder, a generative adversarial network is used to optimize the L1 norm loss and adversarial loss of the fused image and the source image. Finally, the fused image is decomposed into an unentangled representation, and the content and modal components of the fused image and the source image are used to construct the content and modal reconstruction loss to ensure that the fused image retains the information of the source image to the greatest extent.

[0069] The embodiments of the present invention have been described above in conjunction with the accompanying drawings. However, the present invention is not limited to the above embodiments. Various changes can be made according to the purpose of the invention. Any changes, modifications, substitutions, combinations or simplifications made based on the spirit and principle of the technical solution of the present invention shall be equivalent substitutions. As long as they meet the purpose of the invention and do not deviate from the technical principle and inventive concept of the present invention, they shall fall within the protection scope of the present invention.

Claims

1. A cross-modal image fusion method based on image deentanglement representation, characterized in that: The cross-modal image is decomposed into unentangled representations using a densely connected content encoder and modality encoder. The obtained content components are then reconstructed in a densely connected fusion decoder using a multi-scale fusion approach. Finally, the modality components are injected into the content reconstruction using an adaptive normalization method to obtain the fused image, as detailed below: Step 1: Design a densely connected content encoder, reconstruction decoder, and modal encoder. Input the unpaired cross-modal images from training set 1 and encode them to obtain the content and modal components of the images. Step 2: Reconstruct, transform, and recursively decompose the content and modal components of the image from Step 1. Design a discriminator by training the content of the generative adversarial network, the modal encoder, and the reconstruction decoder. Step 3: Repeat steps 1 and 2 until all images in training set 1 have been trained and the set number of iterations has been reached, to obtain the pre-trained content, modal encoder, and reconstruction decoder. Step 4: Load the content and modal encoder from Step 3, input the cross-modal images from Training Set 2, and obtain the content and modal components; Step 5: Design a densely connected fusion decoder and input the content and modal components from Step 4 to obtain the fused image; The fused image is decomposed into unentangled representations to obtain the reconstructed content and modal components. The fused decoder is then trained using the fused image, the reconstructed content, the modal components, and the generative adversarial network. Step 6: Repeat steps 4 and 5 until all images in training set 2 have been trained to obtain the trained fusion model.

2. The cross-modal image fusion method based on image deentanglement representation according to claim 1, characterized in that: The image deentanglement representation is decomposed, and a densely connected content encoder, modality encoder, and reconstruction decoder are designed. These are then trained through image reconstruction, modality transformation, and cyclic reconstruction. 1) Following the dense connection model, each convolutional layer in the content encoder uses the feature maps of all outputs from the previous layer as input, thus designing a densely connected content encoder and reconstruction decoder; the modality encoder is designed as a convolutional network with downsampling function, and modal components are generated through a multi-layer perceptron network. 2) Decompose and reconstruct the unpaired cross-modal images using unentangled representations, and construct the image reconstruction loss training content, modal encoder, and reconstruction decoder according to the following formula (1): L rec ||x1-x 1→1 ||1+||x2-x 2→2 ||1 #(1) Where x1 and x2 are unpaired cross-modal images, x 1→1 x 2→2 For the reconstructed image generated by combining content and modal components, ||·||1 is the L1 norm; 3) After swapping the modalities of the image's content components, generate a modality-transformed image, design a discriminator, and construct the image modality transfer adversarial loss training content and modality encoder according to the following formula (2): L cyc_adv =log(Dis1(x1))+log(1-Dis1(x 2→1 ))+log(Dis2(x2))+log(1-Dis2(x 1→2 )) #(2) Where Dis1 and Dis2 are discriminators, x 2→1 x 1→2 This is a modality transformation image; 4) Decompose the modality transformation image into an unentangled representation, and construct the cyclic decomposition loss training content and modality encoder according to the following formula (3): in, For the content and modal components of a modal-transformed image, c x1 c x2 m x1 m x2 Input the content and modal components of the cross-modal image; 5) Input the content and modal components of the modality-transformed image into the reconstruction decoder to generate a cyclic reconstruction image, and construct the cyclic reconstruction loss training content, modality encoder, and reconstruction decoder according to the following formula (4): L cyc_pix =||x 1→2→1 -x1||1+||x 2→1→2 -x2||1 #(4) Where, x 1→2→1 x 2→1→2 To reconstruct the image in a loop.

3. The cross-modal image fusion method based on image deentanglement representation according to claim 1, characterized in that: The content and modal components are fused and reconstructed. A dual-branch densely connected fusion decoder and a multi-scale fusion structure are designed, and pre-trained content and modal encoders are loaded. The fusion decoder is trained by using the fused image and its reconstructed content and modal components. 1) Load the pre-trained content and modal encoder, decompose the pairwise cross-modal images into unentangled representations, input the content and modal components into the fusion decoder to generate the fused image, and construct the fusion pixel loss according to the following formula (5) and the fusion adversarial loss according to the following formula (6) to train the fusion decoder: L fus_pix =||f-y1||1+||f-y2||1 #(5) L fus_adv =log(Dis1(y1))+log(1-Dis1(f))+log(Dis2(y2))+log(1-Dis2(f)) #(6) Where y1 and y2 are paired cross-modal images, and f is the fused image; 2) Decompose the fused image into unentangled representations using the content and modal decoders of the two modalities respectively. Construct the content and modal reconstruction loss between the content and modal components of the obtained fused image and the content and modal components of the input image according to the following formula (7): L fus_rec =||c f1 -c y1 ||1+||c f2 -c y2 ||1+||m f1 -m y1 ||1+||m f2 -m y2 ||1 #(7) Among them, c f1 c f2 m f1 m f2 The content and modal components obtained after decomposing the fused image using two content and modal decoders are c. y1 c y2 m y1 m y2 These represent the content and modal components of the input pair of cross-modal images, respectively.

Images