Multi-exposure image fusion method based on depth separable Transform and CNN cascade network

Abstract

The invention discloses a multi-exposure image fusion method based on a depth separable Transform and a CNN cascade network. According to the method, a weight sequence obtained through channel self-attention calculation is used for making up for the potential loss of a synergistic effect between channels. Wherein the learnable weighted summation process for obtaining the weight sequence is beneficial to strengthening the interaction between input feature channels, and provides guidance information for adaptively adjusting the information importance degree of different channels. In the feature extraction module, the model can effectively reuse features and reduce the depth and parameter quantity of the network through the form of a dense connection network and the design of a deep coupled convolution block and a Transform block. According to the method, the calculation amount is greatly reduced, and meanwhile, the weight sequence obtained by channel self-attention calculation is utilized to make up for the potential loss of the synergistic effect between channels, so that the model can effectively reuse the features, and the network depth and the parameter amount are reduced. The method provided by the invention is superior to an existing advanced algorithm in visual performance and fusion speed, and an efficient and effective solution is provided for the field of multi-exposure image fusion.

Description

Technical Field

[0001] The present invention relates to the field of multi-exposure image fusion, and in particular to a multi-exposure image fusion method based on a deep separable Transformer and a CNN cascade network. Background Art

[0002] Due to the limited dynamic range that ordinary imaging sensors can capture, a single image usually suffers from underexposure or overexposure, and a single image is usually not enough to show all the details in the real scene. Currently, the most cost-effective way to solve this problem is to use multi-exposure image fusion technology. Specifically, multi-exposure image fusion technology is the process of fusing multiple images with different exposures to produce a single high dynamic range (HDR) image with good visual performance. It has important research significance and value in the fields of obtaining high-quality photographic images, improving the accuracy of downstream image tasks, and realizing low-cost autonomous driving.

[0003] Multi-exposure image fusion technology can be divided into traditional methods and methods based on deep learning. However, traditional methods have many problems. First, it is difficult to handle complex scenes. This is because the feature fusion rules are manually designed, the algorithm generalization is poor, and it is difficult to maintain detail balance and artifact suppression. Secondly, there are many pre-processing and post-processing steps. For example, histogram equalization of the source image is required before fusion, and edge-preserving filtering of the weight map is performed during fusion. After deep learning technology has achieved amazing results in the field of computer vision, researchers have also proposed a large number of multi-exposure image fusion methods based on deep learning. A large number of studies have also shown that methods based on deep learning are far ahead of traditional MEF methods in terms of fusion performance and algorithm generalization.

[0004] MEF methods based on deep learning can be divided into MEF methods that use convolutional neural networks (CNNs) alone and methods that introduce Transformers according to different feature extraction mechanisms. MEF methods based solely on convolutional neural networks can extract and aggregate the corresponding information in the receptive field corresponding to the convolution kernel. However, for the multi-exposure image fusion task, the final fusion quality of the fused image does not only depend on the information in the receptive field, but also on the global intensity and texture information of the entire image. However, due to the inherent limitations of the small receptive field of CNN, these algorithm frameworks with convolutional layers as basic components are difficult to learn the long-range dependencies between the parts, which leads to the inability to summarize the global information of the source image well. Recently, the self-attention mechanism used by Transformers solves the problems faced by CNNs by capturing the global interactions between contexts. Subsequently, researchers also introduced Transformers into MEF methods and proposed a variety of models. A large number of studies have shown that the MEF method that introduces the self-attention mechanism significantly outperforms the MEF method that uses CNN alone in performance. Therefore, combining CNN with Transformers is the general trend of MEF methods in the future. However, these methods also have some significant problems. One is the limitation on the size of the input image, and the other is the increase in model inference time due to the high computational complexity of the self-attention mechanism. The Transformers introduced in these methods all use self-attention in the spatial dimension to calculate the interaction between all pixels of the input features, which causes the computational complexity of the model to grow quadratically with the size of the input image, which is also the main reason for the significant increase in model inference time. Therefore, the high computational overhead caused by the self-attention mechanism has greatly increased the model inference time, making it difficult to deploy in some application scenarios with high requirements for timeliness. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention discloses a multi-exposure image fusion method based on a deep separable Transformer and a CNN cascade network, which specifically includes the following steps:

[0006] Construct a deep separable Transformer architecture to extract global information based on the synergy of self-attention maps between spatial and channel dimensions;

[0007] Constructing a dense layer with diverse information mining capabilities, wherein the dense layer includes a cascade of a CNN cascade network and a Transformer layer, wherein multiple dense layers constitute a single dense block;

[0008] Densely connect the dense blocks to construct a feature extraction network for feature extraction, and use the feature extraction network to extract feature information flow from the source image;

[0009] Construct a feature reconstruction network, and use the feature reconstruction network to reconstruct the feature information flow to obtain a fused image;

[0010] The feature extraction network and the feature reconstruction network are combined into a deep separable Transformer and CNN cascade network for multi-exposure image fusion;

[0011] A combination of structural similarity, mean square error, and gradient loss functions is used to train a deep separable Transformer and CNN cascade network, which is used to measure the similarity between the fused image and the two source images.

[0012] Furthermore, when constructing a deep separable Transformer architecture, the correlation between pixels is modeled through the synergy of self-attention maps in the spatial dimension and the channel dimension, specifically including: obtaining spatial self-attention features, calculating the cross-channel cross covariance of image features, and then obtaining the cross-channel self-attention map, normalizing the cross-channel self-attention map to generate a channel weight map, and obtaining the composite self-attention feature by guiding the weighted sum process of the spatial attention features through the channel weight map.

[0013] Furthermore, the following method is used to obtain the spatial self-attention feature: the original input feature Separate each channel to obtain C features with a channel number of 1 Where i = [1, 2, ..., C], represents the channel number of the original input feature X, and the feature mapping matrix is ​​obtained after the convolution layer and layer normalization (LN)

[0014] Z i =LN(W1·Y i )

[0015] Where W1 is the convolutional layer, and the feature mapping matrix is ​​split by channel to obtain query (Q i )、key(K i ) and value(V i ), after reshaping, the sizes of the three mapping matrices are

[0016] {Q i ,K i ,V i}={Re(C1(Z i )),Re(C2(Z i )),Re(C3(Z i ))}

[0017] Among them C a Represents the feature Z iThe features of the ath channel cut out in , Re() represents the dimension reshaping operation;

[0018] The paired pixel points in the spatial dimension are explicitly modeled, and the spatial self-attention map is calculated along the height and width directions of the single channel feature Zi:

[0019]

[0020] in Represents the single channel feature Z i The spatial self-attention map calculated in the height direction is represents the calculated spatial self-attention map along the width direction, is a scaling parameter used to control the dot product size of K and Q, d k K i The number of channels;

[0021] Multiply the two obtained spatial self-attention maps with Vi and reconstruct the dimensions to obtain the preliminary self-attention features

[0022] F i1 =Re(M i1 (Q i ,K i )·V i )

[0023]

[0024] where F i1 and Represents the single channel feature Y i Preliminary self-attention features generated along the height and width directions that explicitly encode the global context;

[0025] F i1 With F i2 After splicing, the convolutional layers are used for fusion to obtain the single-channel feature Y i The spatial self-attention feature map F i

[0026] F i =W2·Concat(F i1 ,F i2 )

[0027] in W2 is a convolutional layer. The above operation is repeated on C channels, and the results are concatenated in the channel dimension to obtain the final spatial self-attention feature.

[0028] F s =Concat(F1,F2,...,Fc ).

[0029] Furthermore, the cross-channel self-attention map is obtained by calculating the self-attention map of the input feature matrix along the channel dimension, and then normalizing it row by row to obtain a weight map, and guiding the spatial self-attention feature F based on the weight map s Perform weighted calculations to strengthen the interaction between feature channels: Obtain a self-attention map based on the full channel of the original input feature X. First, pass X through a convolutional layer and layer normalization operation (LN), then split it according to the channel and reshape the dimension to obtain the query (Q) and key (K).

[0030] {Q,K}={Re(C1(LN(W3·X))),Re(C2(LN(W3·X)))}

[0031] in W3 is a convolutional layer with the same configuration as W2. It performs self-attention calculation based on the obtained feature mapping matrices Q and K to obtain a cross-channel self-attention map:

[0032]

[0033] in The channel self-attention map explicitly models the cross-correlation between channels;

[0034] After normalizing the channel self-attention map row by row, we get the channel weight map Channel weight map M W Each row of represents a weight sequence, and each value of the weight sequence is assigned to F S The weight value of the corresponding channel

[0035] C i (M W )=C i (M C (Q,K)) / Sum(C i (M C (Q,K)))

[0036] in The sum of all the values ​​contained in it is 1, and Sum() represents the sum operation.

[0037] Further, according to the channel weight map M W Guided spatial self-attention feature F s Get the final weighted output feature: channel weight map M W Each row of weight value sequence is used to guide F s To carry out information interaction between channels, the calculation process of the composite self-attention feature is:

[0038] Fcomplex =Concat(V i (C i (M W ))×C i (F s ))

[0039] in V i represents the i-th value of the weight sequence,

[0040] The composite self-attention feature is input into LN and the feed-forward neural network FFN to obtain the final output feature vector after the original input feature X is processed by the deep separable Transformer Keeping the dimension of X the same, the process is:

[0041] F final =FFN(LN(F complex ))+F complex

[0042] in V i Represents the i-th value of the weight sequence;

[0043] The composite self-attention feature is input into LN and the feed-forward neural network FFN to obtain the final output feature vector after the original input feature X is processed by the deep separable Transformer Keeping the dimension of X the same, the process is:

[0044] F final =FFN(LN(F complex ))+F complex .

[0045] Furthermore, when constructing a feature extraction network, dense connections are first made between dense blocks: the input and output of a single dense block are spliced ​​in the channel dimension to become the input of the next-level dense block. The feature extraction network includes a front shallow feature extraction layer and three dense blocks, wherein the shallow feature extraction layer includes a convolutional layer and an activation function, and the shallow feature extraction layer is used to fuse the spliced ​​source image sequence and extract mixed shallow features.

[0046] Furthermore, the dense block includes L j dense layers, where j is the number of the densely connected block, ranging from [1-3]. For the first dense layer of each dense block, a 3×3 convolutional layer and an activation function are placed in front, and for the last dense layer, it is placed after it. The remaining dense layers are composed of depthwise separable Transformers.

[0047] The present invention discloses a multi-exposure image fusion method based on a deep separable Transformer and a CNN cascade network. Compared with the traditional CNN, the method of the present invention greatly reduces the amount of calculation by cyclically applying spatial self-attention to a single separation channel instead of all channels. At the same time, the weight sequence calculated by channel self-attention is used to compensate for the loss of potential inter-channel synergy. The learnable weighted summation process of the weight sequence helps to strengthen the interaction between input feature channels and provide guidance information for adaptively adjusting the importance of information of different channels. In the feature extraction module, the form of the feature extraction network and the design of the deep coupled convolution block and the Transformer block enable the model to effectively reuse features and reduce the network depth and parameter amount. For the Transformer block, we adopt an efficient deep separable self-attention mechanism to calculate the spatial self-attention of a single channel in a depth-by-depth separate manner, further reducing the computational complexity. In addition, the feature reconstruction module directly reconstructs features from the information stream after the source image sequence is spliced, without the need for an additional feature fusion module, thereby improving the fusion efficiency. A large number of experiments show that the method proposed by the present invention is superior to the existing advanced algorithms in terms of visual performance and fusion speed, and provides an efficient and effective solution for the field of multi-exposure image fusion. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0049] Figure 1 is a flow chart of the method disclosed in the present invention;

[0050] Figure 2 This is the proposed depth-separable Transformer structure diagram;

[0051] Figure 3 This is a comparative test of some fusion results of the present invention on the MEFB dataset. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0053] like Figure 1 The multi-exposure image fusion method based on a deep separable Transformer and a CNN cascade network shown in FIG. 1 specifically includes the following steps:

[0054] S1: Concatenate the overexposed and underexposed source images in the channel dimension to obtain the input feature matrix;

[0055] S2: Figure 2 As shown in the figure, an efficient deep separable Transformer architecture is constructed to efficiently model the correlation between pixels through the synergy of self-attention maps in the spatial dimension and the channel dimension; an efficient deep separable Transformer architecture is constructed to efficiently model the correlation between pixels through the synergy of self-attention maps in the spatial dimension and the channel dimension; in the calculation of obtaining the self-attention map in the spatial dimension, since the self-attention calculation is performed on a separated single channel, its computational complexity increases quadratically with the length and width of the input feature, which is O(CH 2 +CW 2 ). Instead of growing quadratically with the product of the length and width of the input feature as in the standard self-attention, the computational complexity is O(H 2 W 2 ). In the application of MEF tasks, the resolution of images can usually reach more than 1000×1000. This change greatly reduces the amount of calculation required to obtain the self-attention map. Secondly, self-attention calculation is performed on the channel dimension of the input feature, that is, the cross-channel cross covariance of the feature is calculated. The computational complexity of obtaining the self-attention map in this step is O(C 2 ). The purpose of this step is to enhance the interaction of input features in the channel dimension. Based on the obtained cross-channel self-attention map, it is normalized to generate a channel weight map. Finally, the composite self-attention feature is obtained by the weighted sum of the guided spatial attention features of the normalized channel weight map.

[0056] S2 includes the following steps:

[0057] S21: Spatial dimension: First, the original input features Perform channel-by-channel separation to obtain C features with a channel number of 1 Where i = [1, 2, ..., C], representing the channel number of the original input feature X. Then, after a convolution layer and layer normalization (LN), the feature mapping matrix is ​​obtained Right now:

[0058] Z i =LN(W1·Y i )

[0059] Among them, W1 is the convolution layer, the convolution kernel size is 3×3, the input channel is set to 1, and the output channel is set to 3.

[0060] Then, it is split by channel to get query (Q i ), key(K i ) and value(V i ), after reshaping, the sizes of the three mapping matrices are The process can be summarized as:

[0061] {Q i ,K i ,V i}={Re(C1(Z i )),Re(C2(Z i )),Re(C3(Z i ))}

[0062] Among them C a Represents the feature Z i The features of the ath channel cut out from , Re() represents the dimension reshaping operation.

[0063] Next, in order to better explicitly model the paired pixel points in the spatial dimension, the spatial self-attention map is calculated along the height direction and width direction of the single channel feature Zi. The process can be formulated as:

[0064]

[0065] in Represents the single channel feature Z i The spatial self-attention map calculated in the height direction; is the calculated spatial self-attention map along the width direction. is a scaling parameter used to control the dot product size of K and Q. k K i The number of channels is set to 1 here.

[0066] Then, the two obtained spatial self-attention maps are multiplied with Vi and the dimensions are reconstructed to obtain the preliminary self-attention features. The process can be described as:

[0067] F i1=Re(M i1 (Q i ,K i )·V i )

[0068]

[0069] Among them, F i1 and Represents the single channel feature Y i Preliminary self-attention features generated along the height and width directions that explicitly encode the global context.

[0070] Furthermore, F i1 With F i2 After concatenation, a convolutional layer is used to further refine the concatenated features and obtain a single-channel feature Y. i The spatial self-attention feature map F i . F i The acquisition process can be described as:

[0071] F i =W2·Concat(F i1 ,F i2 )

[0072] in W2 is a convolution layer with a convolution kernel size of 3×3, 2 input channels, and 1 output channel. Concat() represents a concatenation operation based on the channel dimension.

[0073] Repeat the above operation on C channels and concatenate the results in the channel dimension to obtain the final spatial self-attention feature

[0074] F s =Concat(F1,F2,...,F c )

[0075] S22: Channel dimension: Since the above-mentioned spatial self-attention only calculates self-attention separately on the separated channels, compared with the standard self-attention, the information between the channels lacks interaction. If only the spatial self-attention features obtained in the previous step are input into the subsequent feedforward network, the fusion performance of the model may be reduced. To solve this problem, the solution of the present invention is to first calculate the self-attention map of the input feature matrix along the channel dimension, and then normalize it row by row to obtain a weight map to guide the spatial self-attention feature F s Weighted calculations are performed to strengthen the interaction between feature channels.

[0076] First, unlike in the spatial dimension, we no longer perform channel-by-channel separation, but obtain the self-attention map based on the full channel of the original input feature X. First, X passes through a convolutional layer and layer normalization (LN) operation, and then splits it according to the channel and reshapes the dimension to obtain query (Q) and key (K). The process can be described by the following formula:

[0077] {Q,K}={Re(C1(LN(W3·X))),Re(C2(LN(W3·X)))}

[0078] in W3 is a convolutional layer, and its configuration is consistent with W2. Then, self-attention calculation is performed based on the obtained feature mapping matrices Q and K to obtain the channel self-attention map. The calculation process is as follows:

[0079]

[0080] in The channel self-attention map explicitly models the cross-correlations between channels.

[0081] Then, the channel self-attention map is normalized row by row to obtain a channel weight map M W Each row of represents a weight sequence, and each value of the weight sequence is assigned to F S The weight value of the corresponding channel. The calculation process is as follows:

[0082] C i (M W )=C i (M C (Q,K)) / Sum(C i (M C (Q,K)))(4.11)

[0083] in The sum of all the values ​​contained in it is 1. Sum() represents the sum operation.

[0084] S23: Compound self-attention: Finally, the channel weight map M W Guided spatial self-attention feature F s The final weighted output feature is obtained. The learnable weighted summation process helps to strengthen the interaction between input feature channels and provides guidance for adaptively adjusting the importance of information in different channels. W Each row of weight value sequence is used to guide F s To interact information between channels. The calculation process of the composite self-attention feature is:

[0085] F complex =Concat(Vi (C i (M W ))×C i (F s ))

[0086] in V i Represents the i-th value of the weight sequence.

[0087] Finally, the composite self-attention feature is input into LN and the feed-forward neural network FFN to obtain the final output feature vector after the original input feature X is processed by the deep separable Transformer Keep the same dimension as X. The process can be summarized as:

[0088] F final =FFN(LN(F complex ))+F complex

[0089] S3: Construct a dense layer with the ability to mine diverse semantic information. It is composed of a cascaded coupling of a convolutional layer and a Transformer layer. Multiple dense layers form a single dense block. The input and output of a single dense block are concatenated in the channel dimension and then become the input of the next dense block. This is called a dense connection between dense blocks.

[0090] S4: Construct a feature extraction network responsible for feature extraction, which is composed of dense blocks in a densely connected form to achieve feature reuse and reduce the number of model parameters; the feature extraction network consists of a front shallow feature extraction layer and three dense blocks. Among them, the shallow feature extraction layer consists of a 3×3 convolution layer and an activation function layer, which is responsible for simply fusing the spliced ​​source image sequence and extracting mixed shallow features. For the dense blocks, L j The network consists of dense layers, where j is the number of the densely connected block, ranging from [1-3]. The first dense layer of each dense block is preceded by a combination of a 3×3 convolutional layer and an activation function; the last dense layer is followed by a 3×3 convolutional layer. The remaining dense layers are composed of a depthwise separable Transformer.

[0091] As the abstraction level of the extracted features increases, in order to more effectively refine the deep features, the present invention sets a gradually increasing hyperparameter L j, to control the number of dense layers in different dense blocks. It is worth noting that dense connections are used between dense blocks, that is, except for the first dense block, the input of each dense block is composed of the concatenation of the input and output of the previous dense block. This achieves feature reuse and speeds up model training and inference efficiency

[56] , which is exactly what the real-time performance of the MEF task requires. In the feature extraction module, the convolution kernel size of each convolution layer is set to 3×3. To keep the feature scale consistent, the step size of the convolution operation is set to 1, and the padding is also set to 1. The subsequent activation functions are all leaky rectified linear units (Leaky ReLU) functions.

[0092] S5: Construct a feature reconstruction network. The end-to-end network structure does not require additional fusion rules to be designed. It can simply and efficiently reconstruct the single feature information stream extracted after the source image sequence is spliced ​​together to obtain a fused image. The input of the model is the spliced ​​source image sequence, and the network structure is a branchless end-to-end network. This does not require the design of additional feature fusion rules, and the well-exposed fused image is directly fused from the source image sequence. This makes the model structure more concise and compact, reduces the number of model parameters, and speeds up the fusion time. In addition, the robustness of the learnable adaptive fusion rules is better than the manually designed fusion rules, making the model more generalizable.

[0093] In the feature reconstruction network, three cascaded convolutional blocks are used to gradually restore the fused image. The first two convolutional blocks have the same building blocks, which are constructed by a cascade combination of a 3×3 convolutional layer, a batch normalization (BN) layer, and a leaky ReLU activation function with a slope of 0.2. All convolutional layers use a reflection padding operation with padding equal to 1 to reduce boundary artifacts. The step size is set to 1 to keep the feature size unchanged. The activation function of the last convolutional block is replaced with the hyperbolic tangent function (Tanh), which limits the value of the output feature vector to the range of [-1,1], and there is no preceding BN layer. The convolutional layer configuration is consistent with the first two convolutional blocks.

[0094] S6: Construct a loss function combination consisting of structural similarity, mean square error and gradient to train the network to measure the similarity between the fused image and the two source images, so as to retain the information between different source images to the greatest extent and integrate them into one image; due to the lack of real ground truth images in the MEF task, the artificially selected reference images will set an invisible performance ceiling for the model. Therefore, this model uses unsupervised learning to train the model. The loss function consists of three sub-loss function terms.

[0095] The first term is the fused image I fThe structural similarity (SSIM) loss between the source images I1 and I2 is summarized as:

[0096]

[0097] Where SSIM() represents the structural similarity calculation, ω1 and ω2 are weight constants, set to 0.5.

[0098] The second term is the mean square error (MSE) loss. An algorithm with superior performance is expected to generate a fused image with moderate global intensity based on the global intensity information of the source image sequence. This also helps to improve the chromaticity deviation and boundary sharpness anomaly problems caused by the structural similarity sub-loss term. The calculation process can be summarized as:

[0099]

[0100] Where MSE() represents the mean square error calculation.

[0101] The third item is the gradient loss. The gradient information reveals the texture distribution law in the source image, and the gradient maximum selection strategy helps to effectively aggregate the texture detail information in different source image sequences, so that the fused image retains enough and appropriate texture information. The specific calculation process of the gradient loss item is as follows:

[0102]

[0103] where ||·||1 represents the l1 norm, represents the Sobel gradient operator. max() represents the maximum value operation. H and W represent the size parameters of the fused image. Finally, the total loss function of the supervised network model training is the weighted sum of the above three sub-loss items:

[0104]

[0105] Among them, λ1, λ2, and λ3 are sub-item weight factors, which regulate the weight of the sub-loss item in guiding model optimization.

[0106] Table 1 lists the objective evaluation scores of the present invention and the comparative method on the MEFB dataset. Figure 3 As shown in Figure 2, the method proposed in this invention is effective in MEF-SSIM, Q C ,PSNR,CE,Q AB / F , Q MR All of these six indicators have achieved the best values. CETNet has the best performance in MEF-SSIM, Q CThe values ​​of the four indicators of , PSNR, and CE are all significantly ahead. This fully demonstrates the superior performance of the multi-exposure image fusion method based on the deep separable Transformer and CNN dense coupling network proposed in the present invention in the MEF task.

[0107] Table 1

[0108]

[0109] At the same time, in order to verify the advantages of the present invention in terms of operating efficiency, the present invention selects four comparison algorithms: SPD-MEF, MEF-GAN, U2Fusion, and SwinFusion. The comparison is made in terms of model parameter size, number of floating-point operations per second (FLOPs), and average running time (Runtime). Among them, the number of floating-point operations per second and the average running time will change with the resolution of the image to be processed, so the relevant data about these two items is the average value of 100 source image pairs in the model processing MEFB data set. The relevant results are shown in Table 2.

[0110] Table 2

[0111]

[0112] The present invention innovatively separates the input features and redesigns the self-attention calculation in two dimensions, reducing the computational complexity of obtaining the self-attention map from O(H2W2) to O(H2+W2+C2), thereby significantly reducing the three indicators for evaluating the model efficiency. Compared with SwinFusion, which also introduced the self-attention mechanism, CETNet reduced the model parameters by about 59.6% and the number of floating-point operations per second by about 67.8%, and improved the model operation efficiency by about 8.3 times.

[0113] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-exposure image fusion method based on a deep separable Transformer and CNN cascade network, characterized in that include: Construct a deep separable Transformer architecture to extract global information based on the synergy of self-attention maps between spatial and channel dimensions; Constructing a dense layer with diverse information mining capabilities, wherein the dense layer includes a cascade of a CNN cascade network and a Transformer layer, wherein multiple dense layers constitute a single dense block; Densely connect the dense blocks to construct a feature extraction network for feature extraction, and use the feature extraction network to extract feature information flow from the source image; Construct a feature reconstruction network, and use the feature reconstruction network to reconstruct the feature information flow to obtain a fused image; The feature extraction network and the feature reconstruction network are combined into a deep separable Transformer and CNN cascade network for multi-exposure image fusion; A combination of structural similarity, mean square error, and gradient loss functions is used to train a deep separable Transformer and CNN cascade network, which is used to measure the similarity between the fused image and the two source images.

2. The multi-exposure image fusion method based on a depthwise separable Transformer and a CNN cascade network according to claim 1, characterized in that: When constructing a deep separable Transformer architecture, the correlation between pixels is modeled through the synergy of self-attention maps in the spatial dimension and the channel dimension, specifically including: obtaining spatial self-attention features, calculating the cross-channel cross covariance of image features, and then obtaining the cross-channel self-attention map, normalizing the cross-channel self-attention map to generate a channel weight map, and obtaining the composite self-attention feature by guiding the weighted sum process of the spatial attention features through the channel weight map.

3. The multi-exposure image fusion method based on a depthwise separable Transformer and a CNN cascade network according to claim 2, characterized in that: The specific method for obtaining spatial self-attention features is as follows: the original input features Separate each channel to obtain C features with a channel number of 1 Where i = [1, 2, ..., C], represents the channel number of the original input feature X, and the feature mapping matrix is ​​obtained after the convolution layer and layer normalization (LN) Z i =LN(W1·Y i ) Where W1 is the convolutional layer, and the feature mapping matrix is ​​split by channel to obtain query (Q i )、key(K i ) and value(V i ), after reshaping, the sizes of the three mapping matrices are {Q i ,K i ,V i }={Re(C1(Z i )),Re(C2(Z i )),Re(C3(Z i ))} Among them C a Represents the feature Z i The features of the ath channel cut out in , Re() represents the dimension reshaping operation; The paired pixel points in the spatial dimension are explicitly modeled, and the spatial self-attention map is calculated along the height and width directions of the single channel feature Zi: in Represents the single channel feature Z i The spatial self-attention map calculated in the height direction is represents the calculated spatial self-attention map along the width direction, is a scaling parameter used to control the dot product size of K and Q, d k K i The number of channels; Multiply the two obtained spatial self-attention maps with Vi and reconstruct the dimensions to obtain the preliminary self-attention features F i1 =Re(M i1 (Q i ,K i )·V i ) F i2 =Re(M i2 (Q i ,K i )·V i T ) where F i1 and Represents the single channel feature Y i Preliminary self-attention features generated along the height and width directions that explicitly encode the global context; F i1 With F i2 After splicing, the convolutional layers are used for fusion to obtain the single-channel feature Y i The spatial self-attention feature map F i F i =W2·Concat(F i1 ,F i2 ) in W2 is a convolutional layer. The above operation is repeated on C channels, and the results are concatenated in the channel dimension to obtain the final spatial self-attention feature. F s =Concat(F1,F2,...,F c )。 4. The multi-exposure image fusion method based on a deep separable Transformer and a CNN cascade network according to claim 3 is characterized by: The cross-channel self-attention map is obtained by calculating the self-attention map of the input feature matrix along the channel dimension, and then normalizing it row by row to obtain a weight map, and guiding the spatial self-attention feature F based on the weight map. s Perform weighted calculations to strengthen the interaction between feature channels: Obtain a self-attention map based on the full channel of the original input feature X. First, pass X through a convolutional layer and layer normalization operation (LN), then split it according to the channel and reshape the dimension to obtain the query (Q) and key (K). {Q,K}={Re(C1(LN(W3·X))),Re(C2(LN(W3·X)))} in W3 is a convolutional layer with the same configuration as W2. It performs self-attention calculation based on the obtained feature mapping matrices Q and K to obtain a cross-channel self-attention map: in The channel self-attention map explicitly models the cross-correlation between channels; After normalizing the channel self-attention map row by row, we get the channel weight map Channel weight map M W Each row of represents a weight sequence, and each value of the weight sequence is assigned to F S The weight value of the corresponding channel C i (M W )=C i (M C (Q,K)) / Sum(C i (M C (Q,K))) in The sum of all the values ​​contained in it is 1, and Sum() represents the sum operation.

5. According to the multi-exposure image fusion method based on the deep separable Transformer and CNN cascade network of claim 3: according to the channel weight map M W Guided spatial self-attention feature F s Get the final weighted output feature: channel weight map M W Each row of weight value sequence is used to guide F s To interact information between channels, the calculation process of the composite self-attention feature is as follows: F complex =Concat(V i (C i (M W ))×C i (F s )) in V i represents the i-th value of the weight sequence, The composite self-attention feature is input into LN and the feed-forward neural network FFN to obtain the final output feature vector after the original input feature X is processed by the deep separable Transformer Keeping the dimension of X the same, the process is: F final =FFN(LN(F complex ))+F complex in V i Represents the i-th value of the weight sequence; The composite self-attention feature is input into LN and the feed-forward neural network FFN to obtain the final output feature vector after the original input feature X is processed by the deep separable Transformer Keeping the dimension of X the same, the process is: F final =FFN(LN(F complex ))+F complex .

6. The multi-exposure image fusion method based on a depthwise separable Transformer and a CNN cascade network according to claim 1, characterized in that: When constructing a feature extraction network, dense connections are first made between dense blocks: the input and output of a single dense block are spliced ​​in the channel dimension to become the input of the next-level dense block. The feature extraction network includes a front shallow feature extraction layer and three dense blocks, wherein the shallow feature extraction layer includes a convolution layer and an activation function, and the shallow feature extraction layer is used to fuse the spliced ​​source image sequence and extract mixed shallow features.

7. The multi-exposure image fusion method based on a depthwise separable Transformer and a CNN cascade network according to claim 1, characterized in that: The dense block includes L j dense layers, where j is the number of the densely connected block, ranging from [1-3]. For the first dense layer of each dense block, a 3×3 convolutional layer and an activation function are placed in front, and for the last dense layer, it is placed after it. The remaining dense layers are composed of depthwise separable Transformers.

Images