A multi-exposure image fusion method based on pyramid convolution

Abstract

The present invention discloses a multi-exposure image fusion method based on pyramid convolution, comprising the following steps: Step 1: splicing overexposed and underexposed images by channel and preprocessing them to obtain a system input; Step 2: sending the system input to a decorrelation module to eliminate channel-level and pixel-level correlations in the data; Step 3: performing feature fusion on the output of Step 2 through an encoder and mapping it to a high-dimensional data space; Step 4: performing dimensionality reduction on the fused high-dimensional data through a decoder, reconstructing the fused image, and completing multi-exposure fusion. The multi-exposure image fusion method of the present invention can improve halos and artifacts when fusing extreme lighting scenes, enhance the generalization ability of the multi-exposure image fusion algorithm for extreme scenes, and is widely used in scenes such as underwater and nighttime observation.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-exposure image fusion, and in particular to a multi-exposure image fusion method based on pyramid convolution. Background Art

[0002] Visual perception is the primary way robots perceive their environment. The uncertainty and high complexity of unstructured environments require visual perception to be adaptable. In low-light environments, however, imaging sensors struggle to obtain well-exposed images using fixed parameters. This information loss caused by image degradation poses a challenge to effective visual perception.

[0003] Multi-exposure image fusion technology aims to synthesize a well-exposed image by preserving the effective information from a sequence of images with different exposures. Existing multi-exposure techniques based on handcrafted features and fusion rules suffer from poor robustness. Furthermore, multi-exposure image fusion techniques based on convolutional neural networks (CNNs) have a small receptive field, making it difficult to establish dependencies between local and global features. This results in artifacts and halos near areas of extreme exposure.

[0004] Therefore, those skilled in the art are committed to providing a multi-exposure image fusion method based on pyramid convolution to reduce the impact of extreme exposure on local detail recovery. Summary of the Invention

[0005] In view of the deficiencies in the prior art, the technical problem to be solved by the present invention is how to provide a multi-exposure image fusion method that can reduce the impact of extreme exposure on local recovery.

[0006] To achieve the above object, the present invention provides a multi-exposure image fusion method based on pyramid convolution, comprising the following steps:

[0007] Step 1: Splice the overexposed image and the underexposed image by channel and obtain the system input after preprocessing;

[0008] Step 2: Send the system input to the decorrelation module to eliminate the channel-level and pixel-level correlation of the data;

[0009] Step 3: The output of step 2 is fused through the encoder and mapped to a high-dimensional data space;

[0010] Step 4: Reduce the dimensionality of the fused high-dimensional data through the decoder, reconstruct the fused image, and complete the multi-exposure fusion.

[0011] Furthermore, the overexposed image and underexposed image in step 1 are 3-channel RGB images, which are stitched together into 6-channel images.

[0012] Furthermore, the step 1 specifically includes:

[0013] Step 1.1: Randomly flip and crop the overexposed and underexposed images horizontally in a plane space to obtain an input tensor with a shape of 6×64×64.

[0014] Step 1.2: normalize the input tensor to obtain the system input with each pixel value between 0 and 1.

[0015] Furthermore, the step 3 specifically includes:

[0016] Step 3.1: Pass the output of step 2 through a pyramid convolution layer, and deepen the channel from 6 to 32. The convolution kernel size of the pyramid convolution layer is 3 and the number of groups is 1.

[0017] Step 3.2: Pass the data obtained in step 3.1 through three consecutive attention modules to extract the effective information features of the image;

[0018] Step 3.3: Through the pyramid dense convolution connection module, the effective image information is fused across scales and network layers to obtain the high-dimensional data space with a channel number of 224.

[0019] Furthermore, the step 4 specifically includes:

[0020] Step 4.1, the high-dimensional data space is passed through three pyramid convolution layers, and the number of channels is reduced from 224 to 32. The convolution kernel sizes of the pyramid convolution layers are [3, 5, 7, 9], [3, 5, 7, 9], and [3, 5, 7], respectively. The number of groups is [1, 4, 8, 16], [1, 4, 8, 16], and [1, 4, 8]. The number of output channels corresponding to each group is [32, 32, 32, 32], [16, 16, 16, 16], and [8, 8, 16], respectively.

[0021] Step 4.2: Pass the data obtained in step 4.1 through a standard convolution and Tanh activation function to reconstruct the fused image. The convolution kernel size of the standard convolution is 3, and the fused image shape is 3×64×64.

[0022] Preferably, the pyramid convolution layer includes: pyramid convolution and LeakyReLu activation function.

[0023] Furthermore, the implementation process of the attention module includes:

[0024] S1, passes through a standard convolution layer with a convolution kernel size of 3, and jumps the input and output of the convolution layer;

[0025] S2, after global average pooling, compresses the 32×64×64 data tensor into a 32×1×1 feature channel distribution by channel;

[0026] S3, after the standard convolution layer and standard convolution with a convolution kernel size of 1, and the Sigmoid activation function, the channel weight map is obtained;

[0027] S4, multiplying the result obtained in step S1 by the channel weight map obtained in step S3 through a dot product operation;

[0028] S5, after a standard convolution layer with a convolution kernel size of 1 and a standard convolution, and a Sigmoid activation function, the spatial weight map is compressed to 1×64×64;

[0029] S6. Multiply the result obtained in step S4 by the spatial weight map obtained in step S5 through a dot product operation, and jump-connect the input of S1 and the result of the dot product.

[0030] Preferably, the standard convolution layer includes: standard convolution and LeakyReLu activation function.

[0031] Furthermore, the pyramid dense convolution connection block includes 6 pyramid convolution layers, the convolution kernel size of the pyramid convolution layer is [3, 5, 7], the number of groups is [1, 4, 8], and the number of output channels corresponding to each group is [8, 8, 16].

[0032] Preferably, the goal of the multi-exposure fusion is to minimize the pixel difference and structural difference between the fused image and the input image; using the smooth L1 norm SmoothL1 as the pixel loss and the structural similarity metric SSIM as the structural loss, for a given over-exposed image I h , underexposed image I l , and the fused image I f , the loss function can be expressed as:

[0033]

[0034] Among them, L s Indicates structural loss, L p represents pixel loss, and λ is a hyperparameter that balances structural loss and pixel loss, which is set to 20.

[0035] The present invention has at least the following beneficial technical effects:

[0036] The multi-exposure image fusion method based on pyramid convolution provided by the present invention combines an attention module with a pyramid dense convolution module and a network structure. The attention module helps the system efficiently extract non-uniform features in the source image. Because the pyramid convolution grouping uses convolution kernels of different sizes in parallel, it enables the system to extract multi-scale features without increasing the number of parameters. Dense connections connect the outputs of each layer of pyramid convolution, effectively fusing information between shallow and deep layers and between different scales, enabling the system to efficiently extract and fuse non-uniform and multi-scale features, achieving multi-exposure image fusion. The multi-exposure image fusion method of the present invention significantly improves fusion performance.

[0037] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 is a schematic diagram of a multi-exposure image fusion method based on pyramid convolution according to an embodiment of the present invention;

[0039] Figure 2 is a schematic diagram of the attention module according to an embodiment of the present invention;

[0040] Figure 3 2 is a schematic diagram of a pyramid dense convolution module according to an embodiment of the present invention;

[0041] Figure 4 These are the implementation details of the pyramid convolution layer and the standard convolution layer in the embodiment of the present invention. DETAILED DESCRIPTION

[0042] The following describes preferred embodiments of the present invention to make its technical content clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the scope of protection of the present invention is not limited to the embodiments mentioned herein.

[0043] In the drawings, components with identical structures are denoted by the same reference numerals, and components with similar structures or functions are denoted by similar reference numerals. The size and thickness of each component shown in the drawings are arbitrary and are not limited by the present invention. For clarity, the thickness of components in some places in the drawings is appropriately exaggerated.

[0044] The present invention provides a multi-exposure image fusion method based on pyramid convolution, which makes effective imaging in low-light environments possible. The multi-exposure image fusion method based on pyramid convolution of the present invention constructs a multi-exposure image fusion system through a preprocessing module, an encoder and a decoder, and the fusion system realizes imaging. The preprocessing module splices the over-exposed and under-exposed images by channel, and performs channel and pixel decorrelation processing; the encoder extracts and fuses the effective features of the image pair; the decoder is used to reconstruct the fused image. Figures 1 to 4 As shown, the specific steps of the multi-exposure image fusion method of this embodiment are as follows.

[0045] Step 1: Splice the overexposed image and the underexposed image by channel and obtain the system input after preprocessing.

[0046] In this step, the overexposed and underexposed images are 3-channel RGB images, which are stitched together to form a 6-channel image. The preprocessing process specifically includes the following steps:

[0047] Step 1.1: Perform random horizontal flipping and cropping of the overexposed and underexposed images in the plane space to obtain an input tensor of shape 6×64×64. This step is only applicable during the training phase and is not required during the testing phase.

[0048] Step 1.2: Normalize the input tensor to obtain a system input with each pixel value between 0 and 1.

[0049] Step 2: Send the system input to the decorrelation module and use FastDeconv to eliminate the channel-level and pixel-level correlation of the data.

[0050] Step 3: The output of step 2 is fused through the encoder and mapped to a high-dimensional data space. The specific process of this step is as follows:

[0051] Step 3.1: Pass the output of step 2 through a pyramid convolution layer, and deepen the channel from 6 to 32. The convolution kernel size of the pyramid convolution layer is 3 and the number of groups is 1.

[0052] Step 3.2: Pass the data obtained in step 3.1 through three consecutive attention modules to extract the effective information features of the image;

[0053] Step 3.3: Through the pyramid dense convolution connection module, the effective image information is fused across scales and network layers to obtain the high-dimensional data space with a channel number of 224.

[0054] like Figure 2 As shown in Figure 2, the implementation process of the attention module is as follows:

[0055] S1, passes through a standard convolution layer with a convolution kernel size of 3, and jumps the input and output of the convolution layer;

[0056] S2, after global average pooling, compresses the 32×64×64 data tensor into a 32×1×1 feature channel distribution by channel;

[0057] S3, after the standard convolution layer and standard convolution with a convolution kernel size of 1, and the Sigmoid activation function, the channel weight map is obtained;

[0058] S4, multiplying the result obtained in step S1 by the channel weight map obtained in step S3 through a dot product operation;

[0059] S5, after a standard convolution layer with a convolution kernel size of 1 and a standard convolution, and a Sigmoid activation function, the spatial weight map is compressed to 1×64×64;

[0060] S6. Multiply the result obtained in step S4 by the spatial weight map obtained in step S5 through a dot product operation, and jump-connect the input of S1 and the result of the dot product.

[0061] Among them, the standard convolution layer includes: standard convolution and LeakyReLu activation function.

[0062] like Figure 3 As shown in Figure 1, the pyramid dense convolution connection block includes 6 pyramid convolution layers, where the convolution kernel size of the pyramid convolution layer is [3, 5, 7], the number of groups is [1, 4, 8], and the number of output channels corresponding to each group is [8, 8, 16].

[0063] Step 4: Reduce the dimensionality of the fused high-dimensional data through the decoder, reconstruct the fused image, and complete the multi-exposure fusion. The specific process of this step is as follows:

[0064] Step 4.1. Pass the high-dimensional data space through three pyramid convolution layers, and reduce the number of channels from 224 to 32. The convolution kernel sizes of the pyramid convolution layers are [3, 5, 7, 9], [3, 5, 7, 9], [3, 5, 7], and the number of groups are [1, 4, 8, 16], [1, 4, 8, 16], and [1, 4, 8]. The corresponding output channel numbers of each group are [32, 32, 32, 32], [16, 16, 16, 16], and [8, 8, 16] respectively.

[0065] Step 4.2: Pass the data obtained in step 4.1 through a standard convolution and Tanh activation function to reconstruct the fused image. The convolution kernel size of the standard convolution is 3, and the fused image shape is 3×64×64.

[0066] The pyramid convolution layer in step 3 and step 4 of this embodiment includes: pyramid convolution and LeakyReLu activation function.

[0067] During the testing phase, the weights of the spatial weight map are fixed, and the overexposed and underexposed image pairs are input into the fusion system. The fused image can be obtained by following the above steps.

[0068] The optimization goal of the encoder-decoder model of this embodiment is to minimize the pixel difference and structural difference between the fused image and the input image; the smooth L1 norm SmoothL1 is used as the pixel loss, and the structural similarity metric SSIM is used as the structural loss. For a given overexposed image I h , underexposed image I l , and the fused image I f , the loss function can be expressed as:

[0069]

[0070] Among them, L s Indicates structural loss, L p represents pixel loss, and λ is a hyperparameter that balances structural loss and pixel loss, which is set to 20.

[0071] By testing on the MEFB test set, the fusion results shown in Table 1 are obtained.

[0072] Table 1

[0073]

[0074]

[0075] It can be seen from the test results that the present invention can improve the halo and artifacts when fusing extreme lighting scenes, enhance the generalization ability of the multi-exposure image fusion algorithm for extreme scenes, and significantly improve the fusion performance, making it suitable for wide application in underwater and nighttime observation scenes.

[0076] The preferred embodiments of the present invention have been described in detail above. It should be understood that numerous modifications and variations based on the concepts of the present invention can be made by one of ordinary skill in the art without inventive effort. Therefore, those skilled in the art can arrive at the present invention through logical analysis, reasoning, or limited experimentation based on the concepts of the present invention and the prior art.

Claims

1. A multi-exposure image fusion method based on pyramid convolution, characterized in that: The following steps are involved: Step 1: splice the overexposed image and the underexposed image by channel, and obtain the system input after preprocessing; the overexposed image and the underexposed image are 3-channel RGB images, which are spliced ​​by channel to obtain a 6-channel image; Step 2: Send the system input to the decorrelation module to eliminate the channel-level and pixel-level correlation of the data; Step 3: The output of step 2 is fused through the encoder and mapped to a high-dimensional data space; The step 3 specifically includes: Step 3.1: Pass the output of step 2 through a pyramid convolution layer, and deepen the channel from 6 to 32. The convolution kernel size of the pyramid convolution layer is 3 and the number of groups is 1. Step 3.2: Pass the data obtained in step 3.1 through three consecutive attention modules to extract the effective information features of the image; Step 3.3: Using a pyramid dense convolutional connection module, cross-scale and cross-network layer fusion is performed on the effective image information to obtain the high-dimensional data space with 224 channels; Step 4: Reduce the dimensionality of the fused high-dimensional data through the decoder, reconstruct the fused image, and complete the multi-exposure fusion.

2. The multi-exposure image fusion method based on pyramid convolution according to claim 1, wherein: The step 1 specifically includes: Step 1.1: Randomly flip and crop the overexposed and underexposed images horizontally in a plane space to obtain an input tensor with a shape of 6×64×64. Step 1.2: normalize the input tensor to obtain the system input with each pixel value between 0 and 1.

3. The multi-exposure image fusion method based on pyramid convolution according to claim 1, wherein: The step 4 specifically includes: Step 4.1, the high-dimensional data space is passed through three pyramid convolution layers, and the number of channels is reduced from 224 to 32. The convolution kernel sizes of the pyramid convolution layers are [3, 5, 7, 9], [3, 5, 7, 9], and [3, 5, 7], respectively. The number of groups is [1, 4, 8, 16], [1, 4, 8, 16], and [1, 4, 8]. The number of output channels corresponding to each group is [32, 32, 32, 32], [16, 16, 16, 16], and [8, 8, 16], respectively. Step 4.2: Pass the data obtained in step 4.1 through a standard convolution and Tanh activation function to reconstruct the fused image. The convolution kernel size of the standard convolution is 3, and the fused image shape is 3×64×64.

4. The multi-exposure image fusion method based on pyramid convolution according to claim 1 or 3, wherein: The pyramid convolution layer includes: pyramid convolution and LeakyReLu activation function.

5. The multi-exposure image fusion method based on pyramid convolution according to claim 1, wherein: The implementation process of the attention module includes: S1, passes through a standard convolution layer with a convolution kernel size of 3, and jumps the input and output of the convolution layer; S2, after global average pooling, compresses the 32×64×64 data tensor into a 32×1×1 feature channel distribution by channel; S3, after the standard convolution layer and standard convolution with a convolution kernel size of 1, and the Sigmoid activation function, the channel weight map is obtained; S4, multiplying the result obtained in step S1 by the channel weight map obtained in step S3 through a dot product operation; S5, after a standard convolution layer with a convolution kernel size of 1 and a standard convolution, and a Sigmoid activation function, the spatial weight map is compressed to 1×64×64; S6. Multiply the result obtained in step S4 by the spatial weight map obtained in step S5 through a dot product operation, and jump-connect the input of S1 and the result of the dot product.

6. The multi-exposure image fusion method based on pyramid convolution according to claim 5, wherein: The standard convolution layer includes: standard convolution and LeakyReLu activation function.

7. The multi-exposure image fusion method based on pyramid convolution according to claim 1, wherein: The pyramid dense convolution connection block includes 6 pyramid convolution layers, the convolution kernel size of the pyramid convolution layer is [3, 5, 7], the number of groups is [1, 4, 8], and the number of output channels corresponding to each group is [8, 8, 16].

8. The multi-exposure image fusion method based on pyramid convolution according to claim 1, wherein: The goal of the multi-exposure fusion is to minimize the pixel difference and structural difference between the fused image and the input image; the smooth L1 norm SmoothL1 is used as the pixel loss, and the structural similarity metric SSIM is used as the structural loss. For a given over-exposed image I h , underexposed image I l , and the fused image I f , the loss function can be expressed as: Among them, L s Indicates structural loss, L p represents pixel loss, and λ is a hyperparameter that balances structural loss and pixel loss, which is set to 20.

Images