Multimodal image fusion model and construction method based on asymmetric U-Net structure

Abstract

The present application belongs to the field of image processing technology, and discloses a multimodal image fusion model and construction method of an asymmetric U-Net structure, in which a first coding unit uses visible light feature information gradually extracted by each first coding block to inject into a compression and excitation module to form an attention map of visible light information, which is multiplied with the infrared light feature information extracted by the corresponding second coding block and then assists the next second coding block to extract infrared light feature information, so as to speed up the feature extraction efficiency of the infrared light coding branch for the infrared image, so that the corresponding second coding block can extract more infrared light feature information, and finally fuse the feature images extracted by multiple first coding blocks and multiple second coding blocks, which can reduce the loss of shallow information of the fusion result or the bias towards a single mode, and obtain a fused image with good quality.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a multimodal image fusion model with an asymmetric U-Net structure and a construction method thereof. Background Art

[0002] At present, when visually detecting or perceiving a target scene, different sensors are often used to obtain images of multiple modalities (such as infrared images, visible light images, etc.), and then fuse them. Among them, visible light sensors can provide high-definition and high-resolution image information in a good environment, but their disadvantage is weak anti-interference ability; infrared sensors have the characteristic of identifying objects through thermal radiation, so they can detect thermal target information well in complex environments, but the spatial resolution of the images provided is low, and the imaging scene details are less. Therefore, the use of multimodal image fusion technology can extract effective information from source images taken by different sensors and eliminate redundant information to generate a fused image with richer scene interpretation, thereby increasing the understanding of the real scene.

[0003] However, existing multimodal image fusion methods often perform symmetrical fusion of multimodal images, which can easily lead to the loss of shallow information in some areas of the fusion result or bias towards a single modality, resulting in poor image quality of the obtained fused image. Summary of the invention

[0004] The purpose of this application is to provide a multimodal image fusion model and construction method with an asymmetric U-Net structure, which can reduce the loss of shallow information in the fusion result or the bias towards a single modality, and obtain a fused image with good quality.

[0005] In a first aspect, the present application provides a multimodal image fusion model with an asymmetric U-Net structure, which is used to fuse infrared images and visible light images, including a first encoding unit, a fusion unit, a first decoding unit and an output layer connected in sequence;

[0006] The first encoding unit comprises a visible light encoding branch formed by a first input layer and N first encoding blocks connected in series, an infrared light encoding branch having a second input layer and N second encoding blocks connected in sequence from front to back, and N-2 compression and excitation modules, where N is a positive integer not less than 3; the first input layer is used to input a visible light image, and the second input layer is used to input an infrared image; the i-th first encoding block is connected to the input end of the i-th compression and excitation module, and the output of the i-th compression and excitation module is multiplied by the output of the i+1-th second encoding block and then input into the i+2-th second encoding block, i∈[1,N-2];

[0007] The fusion unit includes N-3 first fusion modules and one second fusion module, the output end of the j-th first coding block and the output end of the j+2-th second coding block are both connected to the input end of the j-th first fusion module, j∈[1,N-3], the output end of the N-2-th first coding block and the output end of the N-th second coding block are both connected to the input end of the second fusion module;

[0008] The first decoding unit includes N first decoding blocks, the second fusion module is connected to the first two first decoding blocks in sequence, the input end of the kth first decoding block is connected to the output end of the Nkth first fusion module and the output end of the k-1th first decoding block, k∈[3, N-1], and the input end of the Nth first decoding block is connected to the output end of the N-1th first decoding block;

[0009] The output layer is connected to the output end of the last of the first decoding blocks.

[0010] In the first coding unit in the above model, the visible light feature information gradually extracted by each first coding block is injected into the compression and excitation module to form an attention map of the visible light information, which is multiplied with the infrared light feature information extracted by the corresponding second coding block to assist the next second coding block in extracting infrared light feature information, so as to speed up the feature extraction efficiency of the infrared light coding branch for the infrared image, so that the corresponding second coding block can extract more infrared light feature information. Finally, the feature images extracted by multiple first coding blocks and multiple second coding blocks are fused, which can reduce the loss of shallow information in the fusion result or the bias towards a single mode, and obtain a fused image with good quality.

[0011] Preferably, N=5.

[0012] Preferably, the first first coding block and the first second coding block both include a Transformer block, and the second to last first coding block and the second to last second coding block both include a first maximum pooling layer and a Transformer block connected in sequence;

[0013] The Transformer block includes a first regularization layer and a multi-head attention operation layer connected front and back, and also includes a second regularization layer and an MLP network connected front and back; the input end of the Transformer block is connected to the input end of the first regularization layer, the input of the Transformer block is residually connected to the output of the multi-head attention operation layer to obtain a first feature, the first feature is input into the second regularization layer, and the first feature is residually connected to the output of the MLP network as the output of the Transformer block.

[0014] Preferably, the compression and excitation module includes a global pooling layer, a first linear layer, a first ReLU activation function layer, a second linear layer and a first Sigmoid activation function layer which are sequentially connected from front to back.

[0015] Preferably, the first fusion module includes a third input layer connected to the output end of the corresponding first coding block, a fourth input layer connected to the output end of the corresponding second coding block, a second maximum pooling layer, a third maximum pooling layer, a fourth maximum pooling layer, a first average pooling layer, a second average pooling layer, a shared MLP network and a second Sigmoid activation function layer;

[0016] The output end of the third input layer is connected to the input end of the second maximum pooling layer, the output of the second maximum pooling layer is added to the output of the fourth input layer to obtain a first preliminary fusion feature, the first preliminary fusion feature is respectively input into the third maximum pooling layer and the first average pooling layer, the output end of the third maximum pooling layer and the output end of the first average pooling layer are both connected to the input end of the shared MLP network, the output end of the shared MLP network is respectively connected to the input end of the fourth maximum pooling layer and the input end of the second average pooling layer, the output of the fourth maximum pooling layer and the output of the second average pooling layer are added and input into the second Sigmoid activation function layer, the output of the second Sigmoid activation function layer is multiplied by the first preliminary fusion feature and serves as the output of the first fusion module.

[0017] The first fusion module actually uses element-by-element addition and channel attention to perform image fusion, which can enhance the generalization ability, robustness and interpretability of the model. The first fusion module realizes the fusion of common features, which is conducive to ensuring the integrity of the basic details of the multimodal feature map.

[0018] Preferably, the second fusion module includes a fifth input layer connected to the output end of the corresponding first coding block, a sixth input layer connected to the output end of the corresponding second coding block, a fifth maximum pooling layer, a compression and excitation module, and a first convolution layer and a third Sigmoid activation function layer connected front and back;

[0019] The output end of the fifth input layer is connected to the input end of the fifth maximum pooling layer, the output of the fifth maximum pooling layer is added to the output of the sixth input layer to obtain a second preliminary fusion feature, the second preliminary fusion feature is respectively input into the compression and excitation module of the second fusion module and the first convolution layer, the output of the compression and excitation module of the second fusion module is multiplied by the second preliminary fusion feature to obtain a third preliminary fusion feature, the output of the third Sigmoid activation function layer is multiplied by the second preliminary fusion feature to obtain a fourth preliminary fusion feature, and the third preliminary fusion feature and the fourth preliminary fusion feature are subjected to element-by-element maximization operation as the output of the second fusion module.

[0020] The second fusion module uses the compression and excitation modules to construct the channel attention branch, which can better ensure the integrity of the basic details of the multimodal feature map. At the same time, the first convolutional layer and the third Sigmoid activation function layer are used to construct the spatial attention branch, thereby improving the saliency of important details from a fine-grained aspect and better retaining the semantic information of the multimodal image.

[0021] In a second aspect, the present application provides a method for constructing a multimodal image fusion model with an asymmetric U-Net structure as described above, comprising the steps of:

[0022] A1. Construct an initial asymmetric U-Net structured multimodal image fusion model;

[0023] A2. Construct an asymmetric U-Net model; the asymmetric U-Net model includes a second encoding unit having the same structure as the first encoding unit, and also includes two second decoding units, wherein one of the second decoding units and the visible light encoding branch in the second encoding unit form a first U-Net network, and the other second decoding unit and the infrared light encoding branch in the second encoding unit form a second U-Net network;

[0024] A3. Train the first U-Net network and the second U-Net network separately to obtain a trained asymmetric U-Net model;

[0025] A4. assigning the model parameters of the second coding unit in the trained asymmetric U-Net model to the first coding unit of the multimodal image fusion model of the initial asymmetric U-Net structure;

[0026] A5. After freezing the model parameters of the first encoding unit of the multimodal image fusion model with the initial asymmetric U-Net structure, the multimodal image fusion model with the initial asymmetric U-Net structure is trained to obtain a trained multimodal image fusion model with the asymmetric U-Net structure.

[0027] Preferably, step A3 comprises:

[0028] A301. After freezing the model parameters of the second U-Net network, training the first U-Net network;

[0029] A302. After freezing the model parameters of the first U-Net network, training the second U-Net network;

[0030] A303. If the number of training times of the first U-Net network and the second U-Net network reaches a preset number threshold, stop training the first U-Net network and the second U-Net network, otherwise, return to execute step A301.

[0031] Preferably, in step A301, the first U-Net network is trained according to the following first mean square error loss function:

[0032] ;

[0033] in, is the first mean square error loss function, H and W are the pixel height and pixel width of the image respectively, is the pixel value of the (i, j) pixel in the input image of the first U-Net network, is the pixel value of the (i, j) pixel in the output image of the first U-Net network;

[0034] In step A302, the second U-Net network is trained according to the following second mean square error loss function:

[0035] ;

[0036] in, is the second mean square error loss function, is the pixel value of the (i, j) pixel in the input image of the second U-Net network, is the pixel value of the (i, j) pixel in the output image of the second U-Net network.

[0037] Preferably, in step A5, the multimodal image fusion model of the initial asymmetric U-Net structure is trained according to the following total loss function:

[0038] ;

[0039] ;

[0040] ;

[0041] ;

[0042] ;

[0043] ;

[0044] in, is the total loss function, is the third mean square error loss function, is the structural loss function, is the detail loss function, , is the weight coefficient, H and W are the pixel height and pixel width of the image respectively, F is the fused image output by the multimodal image fusion model of the initial asymmetric U-Net structure, is the pixel value of the (i, j) pixel in F, VI is the visible light image input into the multimodal image fusion model of the initial asymmetric U-Net structure, is the pixel value of the (i, j) pixel in VI, IR is the infrared image of the multimodal image fusion model of the initial asymmetric U-Net structure input, is the pixel value of the (i, j) pixel in IR, is the structural similarity between F and VI, is the structural similarity between F and IR, is the mean pixel value of VI, is the mean pixel value of F, is the mean pixel value of IR, is the covariance between the pixel values ​​of VI and the pixel values ​​of F, is the covariance between the pixel values ​​of IR and F, is the standard deviation of the pixel values ​​of VI, is the standard deviation of the pixel values ​​of IR, is the standard deviation of the pixel values ​​of F, , is a constant term.

[0045] Beneficial effect: The multimodal image fusion model and construction method of the asymmetric U-Net structure provided in the present application, in which the first coding unit in the model uses the visible light feature information gradually extracted by each first coding block to inject the compression and excitation module to form an attention map of the visible light information, which is multiplied with the infrared light feature information extracted by the corresponding second coding block to assist the next second coding block in extracting the infrared light feature information, so as to speed up the feature extraction efficiency of the infrared light coding branch for the infrared image, so that the corresponding second coding block can extract more infrared light feature information, and finally fuse the feature images extracted by multiple first coding blocks and multiple second coding blocks, which can reduce the loss of shallow information in the fusion result or the bias towards a single modality, and obtain a fused image with good quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 A schematic diagram of the structure of a multimodal image fusion model with an asymmetric U-Net structure provided in an embodiment of the present application.

[0047] Figure 2 It is a schematic diagram of the structure of the first first coding block and the first second coding block.

[0048] Figure 3 It is a schematic diagram of the structure of other first coding blocks and other second coding blocks.

[0049] Figure 4 It is a structural diagram of the compression and excitation module.

[0050] Figure 5 It is a structural diagram of the first fusion module.

[0051] Figure 6 It is a structural diagram of the second fusion module.

[0052] Figure 7 This is a schematic diagram of the structure of the last first decoding block.

[0053] Figure 8 is a schematic diagram of the structure of other first decoding blocks.

[0054] Fig. 9 A flowchart of the construction method provided in the embodiment of the present application.

[0055] Fig.10 Schematic diagram of the structure of the asymmetric U-Net model.

[0056] Fig.11 is a schematic diagram of the structure of the last second decoding block in the second decoding unit.

[0057] Fig.12 is a schematic diagram of the structure of other second decoding blocks in the second decoding unit.

[0058] Fig.13 This is the first set of comparison images.

[0059] Fig.14 This is the second set of comparison images.

[0060] Explanation of the reference numerals: 1. First encoding unit; 101. Visible light encoding branch; 102. Infrared light encoding branch; 103. Compression and excitation module; 1031. Global pooling layer; 1032. First linear layer; 1033. First ReLU activation function layer; 1034. Second linear layer; 1035. First Sigmoid activation function layer; 2. Fusion unit; 201. First fusion module; 202. Second fusion module; 3. First decoding unit; 301. First decoding block; 4. Output layer; 5. First input layer; 6. First encoding block; 7. Second input layer; 8. Second encoding block; 9. Transformer block; 901. First regularization layer; 902. Multi-head attention operation layer; 903. Second regularization layer; 904. MLP network; 10. First most Large pooling layer; 11. Third input layer; 12. Fourth input layer; 13. Second maximum pooling layer; 14. Third maximum pooling layer; 15. Fourth maximum pooling layer; 16. First average pooling layer; 17. Second average pooling layer; 18. Shared MLP network; 19. Second Sigmoid activation function layer; 20. Upsampling block; 2001. Second convolutional layer; 2002. Pixel-Shuffle layer; 21. Third convolutional layer; 22. Second encoding unit; 23. Second decoding unit; 2301. Second decoding block; 24. Up convolutional layer; 25. Fourth convolutional layer; 26. Tanh activation function layer; 27. Fifth input layer; 28. Sixth input layer; 29. ​​Fifth maximum pooling layer; 30. First convolutional layer; 31. Third Sigmoid activation function layer. DETAILED DESCRIPTION

[0061] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0062] It should be noted that similar reference numerals and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and cannot be understood as indicating or implying relative importance.

[0063] Please refer to Figure 1 , Figure 1 It is a multimodal image fusion model of an asymmetric U-Net structure in some embodiments of the present application, which is used for fusing infrared images and visible light images, and includes a first encoding unit 1, a fusion unit 2, a first decoding unit 3 and an output layer 4 connected in sequence;

[0064] The first encoding unit 1 includes a visible light encoding branch 101 formed by a first input layer 5 and N first encoding blocks 6 connected in series, an infrared light encoding branch 102 having a second input layer 7 and N second encoding blocks 8 connected in sequence from front to back, and N-2 compression and excitation modules 103, where N is a positive integer not less than 3; the first input layer 5 is used to input a visible light image, and the second input layer 7 is used to input an infrared image; the i-th first encoding block 6 is connected to the input end of the i-th compression and excitation module 103, and the output of the i-th compression and excitation module 103 is multiplied (element-by-element multiplication) with the output of the i+1-th second encoding block 8 and then input into the i+2-th second encoding block 8, i∈[1, N-2];

[0065] The fusion unit 2 includes N-3 first fusion modules 201 and one second fusion module 202, the output end of the j-th first coding block 6 and the output end of the j+2-th second coding block 8 are both connected to the input end of the j-th first fusion module 201, j∈[1, N-3], the output end of the N-2-th first coding block 6 and the output end of the N-th second coding block 8 are both connected to the input end of the second fusion module 202;

[0066] The first decoding unit 3 includes N first decoding blocks 301, the second fusion module 202 is connected to the first two first decoding blocks 301 in sequence, the input end of the kth first decoding block 301 is connected to the output end of the Nkth first fusion module 201 and the output end of the k-1th first decoding block 301, k∈[3, N-1], and the input end of the Nth first decoding block 301 is connected to the output end of the N-1th first decoding block 301;

[0067] The output layer 4 is connected to the output terminal of the last (ie, the Nth) first decoding block 301 .

[0068] It can be understood that i, j and k are all positive integers, and are serial numbers sorted from front to back according to the direction of information transmission.

[0069] In fact, research has found that when the U-Net network with the same structure is used to extract features from visible light images and infrared images of the same scene, the multi-layer coding unit in the U-Net network can complete the extraction of shallow information faster when extracting features from visible light images, thereby reaching a deeper semantic space faster (faster here does not mean that the time consumed is shorter, but that the number of coding block layers experienced to complete the shallow information extraction is fewer). Therefore, when two multi-layer coding units with the same architecture are used to extract features from visible light images and infrared images, respectively, the feature extraction efficiency of visible light images is higher (higher feature extraction efficiency does not mean that more feature information is extracted per unit time, but that the feature information extracted by each layer of coding blocks in the coding unit is richer, which can more effectively reduce the loss of shallow information), and the feature extraction efficiency of infrared images is lower (so that each layer of coding blocks in the coding unit extracts less feature information and loses more shallow information).

[0070] In the first encoding unit 1 in the above model, the visible light feature information gradually extracted by each first encoding block 6 is injected into the compression and excitation module 103 to form an attention map of visible light information, which is used to multiply with the infrared light feature information extracted by the corresponding second encoding block 8 to assist the next second encoding block 8 in extracting infrared light feature information, so as to speed up the feature extraction efficiency of the infrared light encoding branch 102 for the infrared image, so that the corresponding second encoding block 8 can extract more infrared light feature information, and finally, the feature images extracted by multiple first encoding blocks 6 and multiple second encoding blocks 8 are fused, which can reduce the loss of shallow information in the fusion result or the bias towards a single mode, and obtain a fused image with good quality.

[0071] Among them, N can be set according to actual needs, for example Figure 1 In the example, N=5, which can ensure that the fused image output by the model has good quality. On the other hand, it can avoid the model structure being too complicated and the processing efficiency being too low due to too many layers of coding units.

[0072] refer to Figure 2 and Figure 3 , the first first encoding block 6 and the first second encoding block 8 both include a Transformer block 9, and the second to last first encoding block 6 and the second to last second encoding block 8 both include a first maximum pooling layer 10 and a Transformer block 9 connected in sequence;

[0073] Transformer block 9 includes a first regularization layer 901 and a multi-head attention operation layer 902 connected front and back, and also includes a second regularization layer 903 and an MLP network 904 (i.e., a multi-layer perceptron network) connected front and back; the input end of Transformer block 9 is connected to the input end of the first regularization layer 901, and the input of Transformer block 9 is residually connected to the output of the multi-head attention operation layer 902 to obtain a first feature, which is input into the second regularization layer 903, and the first feature is residually connected to the output of the MLP network 904 as the output of Transformer block 9.

[0074] Compared with general convolutional units, the above-mentioned Transformer block 9 has a stronger inductive bias capability when processing diverse and complex tasks, and therefore has more advantages when performing multimodal image fusion processing.

[0075] Further, see Figure 4 The compression and excitation module 103 includes a global pooling layer 1031, a first linear layer 1032, a first ReLU activation function layer 1033, a second linear layer 1034 and a first Sigmoid activation function layer 1035 which are sequentially connected from front to back.

[0076] The compression and excitation module 103 can effectively obtain the attention map of the visible light information to assist the second encoding block 8 in extracting infrared light features, thereby improving the extraction efficiency of infrared information.

[0077] Specifically, see Figure 5 The first fusion module 201 includes a third input layer 11 connected to the output end of the corresponding first coding block 6, a fourth input layer 12 connected to the output end of the corresponding second coding block 8, a second maximum pooling layer 13, a third maximum pooling layer 14, a fourth maximum pooling layer 15, a first average pooling layer 16, a second average pooling layer 17, a shared MLP network 18 (i.e., a Shared-MLP network), and a second Sigmoid activation function layer 19;

[0078] The output end of the third input layer 11 is connected to the input end of the second maximum pooling layer 13, the output of the second maximum pooling layer 13 is added to the output of the fourth input layer 12 (element-by-element addition) to obtain a first preliminary fusion feature, the first preliminary fusion feature is respectively input into the third maximum pooling layer 14 and the first average pooling layer 16, the output end of the third maximum pooling layer 14 and the output end of the first average pooling layer 16 are both connected to the input end of the shared MLP network 18, the output end of the shared MLP network 18 is respectively connected to the input end of the fourth maximum pooling layer 15 and the input end of the second average pooling layer 17, the output of the fourth maximum pooling layer 15 and the output of the second average pooling layer 17 are added (element-by-element addition) and then input into the second Sigmoid activation function layer 19, the output of the second Sigmoid activation function layer 19 is multiplied by the first preliminary fusion feature (element-by-element multiplication) and then used as the output of the first fusion module 201.

[0079] The first fusion module 201 actually uses element-by-element addition and channel attention to perform image fusion, thereby enhancing the generalization ability, robustness and interpretability of the model. The first fusion module 201 realizes the fusion of common features, which is conducive to ensuring the integrity of the basic details of the multimodal feature map (that is, the output fusion result can retain more basic details of the multimodal feature map).

[0080] Specifically, see Figure 6 The second fusion module 202 includes a fifth input layer 27 connected to the output end of the corresponding first coding block 6, a sixth input layer 28 connected to the output end of the corresponding second coding block 8, a fifth maximum pooling layer 29, a compression and excitation module 103, and a first convolution layer 30 and a third Sigmoid activation function layer 31 connected front and back;

[0081] The output end of the fifth input layer 27 is connected to the input end of the fifth maximum pooling layer 29, the output of the fifth maximum pooling layer 29 is added to the output of the sixth input layer 28 (element-by-element addition) to obtain a second preliminary fusion feature, the second preliminary fusion feature is respectively input into the compression and excitation module 103 and the first convolution layer 30 of the second fusion module 202, the output of the compression and excitation module 103 of the second fusion module 202 is multiplied by the second preliminary fusion feature (element-by-element multiplication) to obtain a third preliminary fusion feature, the output of the third Sigmoid activation function layer 31 is multiplied by the second preliminary fusion feature (element-by-element multiplication) to obtain a fourth preliminary fusion feature, and the third preliminary fusion feature and the fourth preliminary fusion feature are subjected to element-by-element maximization operation as the output of the second fusion module 202.

[0082] The second fusion module 202 uses the compression and excitation module 103 to construct a channel attention branch, so as to better ensure the integrity of the basic details of the multimodal feature map. At the same time, the first convolutional layer 30 and the third Sigmoid activation function layer 31 are used to construct a spatial attention branch, so as to improve the significance of important details from a fine-grained aspect, and better retain the semantic information of the multimodal image (that is, the output fusion result can retain more semantic information of the multimodal feature map).

[0083] Further, see Figure 7 and Figure 8 , the first to N-1th first decoding blocks 301 each include an upsampling block 20 and a Transformer block 9 connected front to back, and the last (i.e., Nth) first decoding block 301 includes an upsampling block 20, a Transformer block 9, and a third convolutional layer 21 connected in sequence from front to back;

[0084] The upsampling block 20 includes a second convolutional layer 2001 and a Pixel-Shuffle layer 2002 (ie, an element reorganization layer) connected front and back.

[0085] The first decoding unit 3 composed of the first decoding block 301 can perform step-by-step upsampling and reconstruction operations on the images output by each fusion module to obtain a fused image with good quality.

[0086] refer to Fig. 9 The present application provides a method for constructing a multimodal image fusion model with an asymmetric U-Net structure as described above, comprising the steps of:

[0087] A1. Construct an initial asymmetric U-Net structure multimodal image fusion model (its structure is as follows Figure 1 shown);

[0088] A2. Construct an asymmetric U-Net model; the asymmetric U-Net model includes a second encoding unit 22 with the same structure as the first encoding unit 1, and also includes two second decoding units 23, wherein one second decoding unit 23 and the visible light encoding branch 101 in the second encoding unit 22 form a first U-Net network, and the other second decoding unit 23 and the infrared light encoding branch 102 in the second encoding unit 22 form a second U-Net network (such as Fig. 9 shown);

[0089] A3. Train the first U-Net network and the second U-Net network separately to obtain a trained asymmetric U-Net model;

[0090] A4. Assign the model parameters of the second encoding unit 22 in the trained asymmetric U-Net model to the first encoding unit 1 of the multimodal image fusion model of the initial asymmetric U-Net structure;

[0091] A5. After freezing the model parameters of the first encoding unit 1 of the multimodal image fusion model with the initial asymmetric U-Net structure, the multimodal image fusion model with the initial asymmetric U-Net structure is trained to obtain a trained multimodal image fusion model with the asymmetric U-Net structure.

[0092] Among them, in step A1, after constructing the model according to the structure of the multimodal image fusion model with an asymmetric U-Net structure in the previous text, the model parameters of the model are initialized (the existing initialization method can be used for initialization, which is not limited here), and the initial multimodal image fusion model with an asymmetric U-Net structure is obtained.

[0093] Among them, reference Fig.10 , the second decoding unit 23 includes N second decoding blocks 2301 connected sequentially from front to back;

[0094] In the first U-Net network, the output end of the Nth first encoding block 6 and the output end of the N-1th first encoding block 6 are connected to the input end of the first second decoding block 2301 after channel cascading, the output end of the n-1th second decoding block 2301 and the output end of the N-1-nth first encoding block 6 are connected to the input end of the nth second decoding block 2301 after channel cascading, n∈[2, N-1], the output end of the N-1th second decoding block 2301 is connected to the input end of the Nth second decoding block 2301, and the output end of the Nth second decoding block 2301 serves as the output end of the first U-Net network;

[0095] In the second U-Net network, the N-2th guided output (the guided output refers to the feature obtained by multiplying the output of the compression and excitation module 103 with the output of the corresponding second encoding block 8; the N-2th guided output is the feature obtained by multiplying the output of the N-2th compression and excitation module 103 with the output of the N-1th second encoding block 8) and the output of the N-2nd encoding block 8 are channel-concatenated and input into the first second decoding block 2301, the output of the o-1th second decoding block 2301 and the N-1-oth guided output are channel-concatenated and input into the oth second decoding block 2301, o∈[2, N-2], the output of the first second encoding block 8 and the output of the N-2th second decoding block 2301 are channel-concatenated and input into the N-1th second decoding block 2301, the output end of the N-1th second decoding block 2301 is connected to the input end of the Nth second decoding block 2301, and the output end of the Nth second decoding block 2301 is used as the output end of the second U-Net network.

[0096] It can be understood that n and o are both positive integers, and are serial numbers sorted from front to back according to the direction of information transmission.

[0097] Further, Fig.11 and Fig.12 In the same second decoding unit 23, the first to the N-1th second decoding blocks 2301 all include a front-to-back connected upper convolution layer 24 and a Transformer block 9, and the last (i.e., the Nth) second decoding block 2301 includes a front-to-back connected fourth convolution layer 25 and a Tanh activation function layer 26.

[0098] Wherein, step A3 comprises:

[0099] A301. After freezing the model parameters of the second U-Net network, training the first U-Net network;

[0100] A302. After freezing the model parameters of the first U-Net network, training the second U-Net network;

[0101] A303. If the training times of the first U-Net network and the second U-Net network reaches a preset threshold, the training of the first U-Net network and the second U-Net network is stopped; otherwise, the process returns to step A301.

[0102] Before step A3, training samples may be collected to form a training sample set (for example, training samples may be extracted from an open source M3FD dataset or MSRS dataset), and each training sample includes a visible light image and an infrared image that are registered with each other. Thus, in step A301, after freezing the model parameters of the second U-Net network, the visible light image and the infrared image in each training sample are sequentially input into the asymmetric U-Net model (wherein the visible light image is input into the first input layer 5 of the asymmetric U-Net model, and the infrared image is input into the second input layer 7 of the asymmetric U-Net model), and the first loss function is calculated using the output image of the first U-Net network and the input image of the first U-Net network (i.e., the visible light image input into the asymmetric U-Net model), and the model parameters of the first U-Net network are updated according to the first loss function (the existing update method may be selected for updating, such as the gradient descent method, but not limited thereto), until the first loss function converges.

[0103] The first loss function may be set according to actual needs. For example, in some preferred implementations, the first loss function may be a mean square error loss function, so that in step A301, the first U-Net network is trained according to the following first mean square error loss function (that is, the first mean square error loss function is the first loss function):

[0104] ;

[0105] in, is the first mean square error loss function, H and W are the pixel height and pixel width of the image respectively, is the pixel value of the (i, j) pixel in the input image of the first U-Net network, is the pixel value of the (i, j) pixel in the output image of the first U-Net network.

[0106] Furthermore, in step A302, after freezing the model parameters of the first U-Net network, the visible light image and the infrared image in each training sample are input into the asymmetric U-Net model in turn (wherein the visible light image is input into the first input layer 5 of the asymmetric U-Net model, and the infrared image is input into the second input layer 7 of the asymmetric U-Net model), and the second loss function is calculated using the output image of the second U-Net network and the input image of the second U-Net network (i.e., the infrared image input into the asymmetric U-Net model), and the model parameters of the second U-Net network are updated according to the second loss function (existing update methods may be used for updating, such as the gradient descent method, but not limited thereto) until the second loss function converges.

[0107] The second loss function can be set according to actual needs. For example, in some preferred implementations, the second loss function can be a mean square error loss function, so that in step A302, the second U-Net network is trained according to the following second mean square error loss function (that is, the second mean square error loss function is the second loss function):

[0108] ;

[0109] in, is the second mean square error loss function, is the pixel value of the (i, j) pixel in the input image of the second U-Net network, is the pixel value of the (i, j) pixel in the output image of the second U-Net network.

[0110] In step A303, the preset number threshold can be set according to actual needs and can be 1 or greater than 1.

[0111] By training the asymmetric U-Net model in the above manner, it is ensured that the asymmetric U-Net model can decompose and reconstruct the input source image (visible light image and infrared image) to obtain an output image that is basically consistent with the source image. At this time, it means that the second encoding unit 22 can fully extract the effective information in the source image.

[0112] Furthermore, in step A4, since the structure of the second encoding unit 22 is the same as that of the first encoding unit 1, the model parameters of the second encoding unit 22 in the trained asymmetric U-Net model can be directly copied to the first encoding unit 1 of the multimodal image fusion model of the initial asymmetric U-Net structure as the values ​​of the corresponding model parameters. Since the second encoding unit 22 in the trained asymmetric U-Net model can fully extract the effective information in the source image, after the model parameters of the second encoding unit 22 are assigned to the first encoding unit 1 of the multimodal image fusion model of the initial asymmetric U-Net structure, the first encoding unit 1 can also fully extract the effective information in the source image.

[0113] Among them, in step A5, after freezing the model parameters of the first encoding unit 1 of the multimodal image fusion model of the initial asymmetric U-Net structure, the visible light image and the infrared image in each training sample are sequentially input into the multimodal image fusion model of the initial asymmetric U-Net structure (wherein, the visible light image is input into the first input layer 5 of the multimodal image fusion model of the initial asymmetric U-Net structure, and the infrared image is input into the second input layer 7 of the multimodal image fusion model of the initial asymmetric U-Net structure), and the total loss function is calculated using the fused image output by the multimodal image fusion model of the initial asymmetric U-Net structure and the visible light image and the infrared image input into the multimodal image fusion model of the initial asymmetric U-Net structure, and the model parameters of the multimodal image fusion model of the initial asymmetric U-Net structure are updated according to the total loss function (specifically, the model parameters of the fusion unit 2 and the first decoding unit 3 are updated, and the existing updating method can be selected for updating, such as the gradient descent method, but not limited thereto), until the total loss function converges.

[0114] The total loss function can be set according to actual needs. For example, in some preferred embodiments, the total loss function includes three components: mean square error loss, structural loss, and detail loss. Specifically, in step A5, the initial asymmetric U-Net structure multimodal image fusion model is trained according to the following total loss function:

[0115] ;

[0116] ;

[0117] ;

[0118] ;

[0119] ;

[0120] ;

[0121] in, is the total loss function, is the third mean square error loss function, is the structural loss function, is the detail loss function, , is the weight coefficient (which can be set according to actual needs), H and W are the pixel height and pixel width of the image (source image and fused image), F is the fused image output by the multimodal image fusion model of the initial asymmetric U-Net structure, is the pixel value of the (i, j) pixel in F, VI is the visible light image of the multimodal image fusion model with the initial asymmetric U-Net structure as input, is the pixel value of the (i, j) pixel in VI, IR is the infrared image of the multimodal image fusion model with the initial asymmetric U-Net structure as input, is the pixel value of the (i, j) pixel in IR, is the structural similarity between F and VI, is the structural similarity between F and IR, is the mean pixel value of VI, is the mean pixel value of F, is the mean pixel value of IR, is the covariance between the pixel values ​​of VI and the pixel values ​​of F, is the covariance between the pixel values ​​of IR and F, is the standard deviation of the pixel values ​​of VI, is the standard deviation of the pixel values ​​of IR, is the standard deviation of the pixel values ​​of F, , is a constant term (used to prevent the denominator in the structural similarity calculation formula from being close to 0, and can be set according to actual needs).

[0122] Using the above total loss function to train the initial asymmetric U-Net structure multimodal image fusion model can ensure that the fused image output by the trained asymmetric U-Net structure multimodal image fusion model has a smaller mean square error loss, structural loss and detail loss with the input source image, thereby ensuring that the fused image has a richer interpretation of the scene and the quality of the fused image is better.

[0123] Fig.13 and Fig.14 The image fusion results of the multimodal image fusion model using the asymmetric U-Net structure of the present application and the image fusion results of several existing fusion algorithms are shown in the comparative image. Fig.13 a and Fig.14 The a in the above are all infrared images in the source image. Fig.13 b and Fig.14 The b in the above are all visible light images in the source image. Fig.13 c and Fig.14 The c in the figure are all fusion images of existing target perception algorithms. Fig.13 d and Fig.14 The d in the figure are all fused images of the existing driver decomposition algorithm. Fig.13 e and Fig.14 The e in each of them are fusion images of the existing multi-scale algorithm. Fig.13 f and Fig.14 The f in the figure are all fused images output by the multimodal image fusion model with an asymmetric U-Net structure of the present application.

[0124] from Fig.13 and Fig.14 It can be seen that the fused images of several existing fusion algorithms have different degrees of detail information loss and contrast reduction; Fig.13 As can be seen from the circled area in FIG, when there is thick smoke occlusion, the fused image output by the multimodal image fusion model of the asymmetric U-Net structure of the present application can well preserve the edge contour and detailed texture information of the mountain captured by the infrared image in the source image; Fig.14 It can be seen from the marked box area in the upper left corner that when the visible light image in the source image is overexposed, the fused image output by the multimodal image fusion model with an asymmetric U-Net structure of the present application can well preserve the house structure in the infrared image in the source image, thereby eliminating overexposure; therefore, the multimodal image fusion model with an asymmetric U-Net structure of the present application can achieve information balance in both visible light images and infrared images through asymmetric fusion of the same feature space, thereby improving the quality of the fused image.

[0125] The multimodal image fusion model with an asymmetric U-Net structure of the present application and the above-mentioned several existing fusion algorithms are used to fuse the source images in the M3FD dataset, and the average values ​​of the five indicators of the fusion results are calculated. The calculation results are shown in Table 1.

[0126] Table 1

[0127] method VIF SSIM PSNR Qabf QUR This application 0.430 0.764 16.335 0.626 0.529 Target perception algorithm 0.462 0.760 14.840 0.246 0.434 Drive decomposition algorithm 0.392 0.725 13.446 0.672 0.511 Multi-scale algorithm 0.424 0.763 15.630 0.582 0.469

[0128] Among them, the index VIF is visual fidelity, the index SSIM is structural similarity, the index PSNR is peak signal-to-noise ratio, the index Qabf is fusion quality, and the index Qcb is human visual perception. As can be seen from Table 1, the fusion results of the multimodal image fusion model with an asymmetric U-Net structure of the present application have the best performance in the two indicators of SSIM and Qcb, indicating that it can retain the structure and contrast of the source image and produce the most consistent fusion results with HVS (human visual system); the fusion results of the multimodal image fusion model with an asymmetric U-Net structure of the present application have good performance in the two indicators of VIF and PSNR, indicating that the fusion results have strong robustness and high information fidelity; in general, the multimodal image fusion model with an asymmetric U-Net structure of the present application has the best performance.

[0129] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0130] In addition, the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, and may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0131] Furthermore, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0132] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any such actual relationship or order between these entities or operations.

[0133] The above description is only an embodiment of the present application and is not intended to limit the protection scope of the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A multimodal image fusion model with an asymmetric U-Net structure, used for fusing infrared images and visible light images, characterized in that: It comprises a first encoding unit (1), a fusion unit (2), a first decoding unit (3) and an output layer (4) which are connected in sequence; The first encoding unit (1) comprises a visible light encoding branch (101) formed by a first input layer (5) and N first encoding blocks (6) connected in series, an infrared light encoding branch (102) having a second input layer (7) and N second encoding blocks (8) connected in sequence from front to back, and N-2 compression and excitation modules (103), where N is a positive integer not less than 3; the first input layer (5) is used to input a visible light image, and the second input layer (7) is used to input an infrared image; the i-th first encoding block (6) is connected to the input end of the i-th compression and excitation module (103), and the output of the i-th compression and excitation module (103) is multiplied by the output of the i+1-th second encoding block (8) and then input into the i+2-th second encoding block (8), i∈[1, N-2]; The fusion unit (2) comprises N-3 first fusion modules (201) and a second fusion module (202), the output end of the j-th first coding block (6) and the output end of the j+2-th second coding block (8) are both connected to the input end of the j-th first fusion module (201), j∈[1, N-3], the output end of the N-2-th first coding block (6) and the output end of the N-th second coding block (8) are both connected to the input end of the second fusion module (202); The first decoding unit (3) comprises N first decoding blocks (301), the second fusion module (202) is connected to the first two first decoding blocks (301) in sequence, the input end of the kth first decoding block (301) is connected to the output end of the Nkth first fusion module (201) and the output end of the k-1th first decoding block (301), k∈[3, N-1], and the input end of the Nth first decoding block (301) is connected to the output end of the N-1th first decoding block (301); The output layer (4) is connected to the output end of the last of the first decoding blocks (301); The first first encoding block (6) and the first second encoding block (8) both include a Transformer block (9), and the second to last first encoding block (6) and the second to last second encoding block (8) both include a first maximum pooling layer (10) and a Transformer block (9) connected in sequence; The Transformer block (9) includes a first regularization layer (901) and a multi-head attention operation layer (902) connected front and back, and also includes a second regularization layer (903) and an MLP network (904) connected front and back; the input end of the Transformer block (9) is connected to the input end of the first regularization layer (901), the input of the Transformer block (9) is residually connected with the output of the multi-head attention operation layer (902) to obtain a first feature, the first feature is input into the second regularization layer (903), and the first feature is residually connected with the output of the MLP network (904) to serve as the output of the Transformer block (9); The compression and excitation module (103) includes a global pooling layer (1031), a first linear layer (1032), a first ReLU activation function layer (1033), a second linear layer (1034) and a first Sigmoid activation function layer (1035) which are connected in sequence from front to back.

2. The multimodal image fusion model with an asymmetric U-Net structure according to claim 1, characterized in that: N=5。 3. The multimodal image fusion model with an asymmetric U-Net structure according to claim 1, characterized in that: The first fusion module (201) comprises a third input layer (11) connected to the output end of the corresponding first coding block (6), a fourth input layer (12) connected to the output end of the corresponding second coding block (8), a second maximum pooling layer (13), a third maximum pooling layer (14), a fourth maximum pooling layer (15), a first average pooling layer (16), a second average pooling layer (17), a shared MLP network (18) and a second Sigmoid activation function layer (19); The output end of the third input layer (11) is connected to the input end of the second maximum pooling layer (13); the output of the second maximum pooling layer (13) is added to the output of the fourth input layer (12) to obtain a first preliminary fusion feature; the first preliminary fusion feature is respectively input to the third maximum pooling layer (14) and the first average pooling layer (16); the output end of the third maximum pooling layer (14) and the output end of the first average pooling layer (16) are both connected to the input end of the shared MLP network (18); the output end of the shared MLP network (18) is respectively connected to the input end of the fourth maximum pooling layer (15) and the input end of the second average pooling layer (17); the output of the fourth maximum pooling layer (15) and the output of the second average pooling layer (17) are added and then input to the second Sigmoid activation function layer (19); the output of the second Sigmoid activation function layer (19) is multiplied by the first preliminary fusion feature and then used as the output of the first fusion module (201).

4. The multimodal image fusion model with an asymmetric U-Net structure according to claim 1, characterized in that: The second fusion module (202) comprises a fifth input layer (27) connected to the output end of the corresponding first encoding block (6), a sixth input layer (28) connected to the output end of the corresponding second encoding block (8), a fifth maximum pooling layer (29), a compression and excitation module (103), and a first convolution layer (30) and a third Sigmoid activation function layer (31) connected to each other. The output end of the fifth input layer (27) is connected to the input end of the fifth maximum pooling layer (29), the output of the fifth maximum pooling layer (29) is added to the output of the sixth input layer (28) to obtain a second preliminary fusion feature, the second preliminary fusion feature is respectively input into the compression and excitation module (103) of the second fusion module (202) and the first convolution layer (30), the output of the compression and excitation module (103) of the second fusion module (202) is multiplied with the second preliminary fusion feature to obtain a third preliminary fusion feature, the output of the third Sigmoid activation function layer (31) is multiplied with the second preliminary fusion feature to obtain a fourth preliminary fusion feature, and the third preliminary fusion feature and the fourth preliminary fusion feature are subjected to an element-by-element maximization operation and serve as the output of the second fusion module (202).

5. A method for constructing a multimodal image fusion model with an asymmetric U-Net structure as described in any one of claims 1 to 4, characterized in that: Includes steps: A1. Construct an initial asymmetric U-Net structured multimodal image fusion model; A2. Constructing an asymmetric U-Net model; the asymmetric U-Net model comprises a second encoding unit (22) having the same structure as the first encoding unit (1), and also comprises two second decoding units (23), wherein one of the second decoding units (23) and the visible light encoding branch (101) in the second encoding unit (22) form a first U-Net network, and the other of the second decoding units (23) and the infrared light encoding branch (102) in the second encoding unit (22) form a second U-Net network; A3. Train the first U-Net network and the second U-Net network separately to obtain a trained asymmetric U-Net model; A4. assigning the model parameters of the second encoding unit (22) in the trained asymmetric U-Net model to the first encoding unit (1) of the multimodal image fusion model of the initial asymmetric U-Net structure; A5. After freezing the model parameters of the first encoding unit (1) of the multimodal image fusion model with the initial asymmetric U-Net structure, the multimodal image fusion model with the initial asymmetric U-Net structure is trained to obtain a trained multimodal image fusion model with the asymmetric U-Net structure.

6. The construction method according to claim 5, characterized in that: Step A3 includes: A301. After freezing the model parameters of the second U-Net network, training the first U-Net network; A302. After freezing the model parameters of the first U-Net network, training the second U-Net network; A303. If the number of training times of the first U-Net network and the second U-Net network reaches a preset number threshold, stop training the first U-Net network and the second U-Net network, otherwise, return to execute step A301.

7. The construction method according to claim 6, characterized in that: In step A301, the first U-Net network is trained according to the following first mean square error loss function: ； in, is the first mean square error loss function, H and W are the pixel height and pixel width of the image respectively, is the pixel value of the (i, j) pixel in the input image of the first U-Net network, is the pixel value of the (i, j) pixel in the output image of the first U-Net network; In step A302, the second U-Net network is trained according to the following second mean square error loss function: ； in, is the second mean square error loss function, is the pixel value of the (i, j) pixel in the input image of the second U-Net network, is the pixel value of the (i, j) pixel in the output image of the second U-Net network.

8. The construction method according to claim 5, characterized in that: In step A5, the multimodal image fusion model of the initial asymmetric U-Net structure is trained according to the following total loss function: ； ； ； ； ； ； in, is the total loss function, is the third mean square error loss function, is the structural loss function, is the detail loss function, , is the weight coefficient, H and W are the pixel height and pixel width of the image respectively, F is the fused image output by the multimodal image fusion model of the initial asymmetric U-Net structure, is the pixel value of the (i, j) pixel in F, VI is the visible light image input into the multimodal image fusion model of the initial asymmetric U-Net structure, is the pixel value of the (i, j) pixel in VI, IR is the infrared image of the multimodal image fusion model of the initial asymmetric U-Net structure input, is the pixel value of the (i, j) pixel in IR, is the structural similarity between F and VI, is the structural similarity between F and IR, is the mean pixel value of VI, is the mean pixel value of F, is the mean pixel value of IR, is the covariance between the pixel values ​​of VI and the pixel values ​​of F, is the covariance between the pixel values ​​of IR and F, is the standard deviation of the pixel values ​​of VI, is the standard deviation of the pixel values ​​of IR, is the standard deviation of the pixel values ​​of F, , is a constant term.

Images