A method, system, and medium for deblurring multi-exposure image fusion

Abstract

The application discloses a deblurring multi-exposure image fusion method and system and a medium. The method comprises the following steps: constructing a deblurring multi-exposure image fusion dataset; performing pretreatment; training an initial deblurring multi-exposure image fusion network based on high and low exposure data sets with blur in a training set and a clear reference image with proper exposure; obtaining a target deblurring multi-exposure image fusion network; inputting an overexposed image with blur and an underexposed image into the target deblurring multi-exposure image fusion network for deblurring multi-exposure image fusion processing to obtain a clear image with proper exposure. The application captures frequency domain information, local information and global information in an image through the construction of a key module Res Block. The combination of image deblurring and multi-exposure image fusion is realized through the design of a network structure, and the multi-exposure image fusion problem with blur in a dynamic scene is solved. The multi-exposure image fusion effect in a dynamic scene is greatly improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of image fusion, and particularly relates to a deblurring multi-exposure image fusion method, system and medium based on a convolutional neural network. BACKGROUND

[0002] Multi-exposure image fusion extracts the details in long and short exposure images for algorithm fusion, thereby obtaining higher quality images. We apply the multi-exposure image fusion technology to automobile photography. When the visual sensor is in high-speed motion, the long and short exposure images obtained have different states. For the short exposure image, a clear image can be captured due to the short exposure time; for the long exposure image, the captured image is in a blurred state, which is not conducive to the generation of high-quality HDR images. Through literature research, it is found that there is no precedent research on the combination of fusion and deblurring at present, that is, when long and short exposure images are used to generate HDR images in a dynamic scene, the quality of the images is ensured, and there are no phenomena such as ghosting and blurring.

[0003] In the research on deblurring in a dynamic scene, the current mainstream is a deep learning model based on multi-scale (MS), multi-image block (MP) and multi-time (MT). Nah et al. [1] proposed an MS network that gradually restores clear images of different resolutions through a coarse-to-fine strategy. Zhang et al. [2] established a hierarchical deblurring model using the MP method. Park et al. [3] proposed an MT deblurring model with original spatial scale increment time training to save more high-frequency information and achieve reliable deblurring.

[0004] In the research on multi-exposure image fusion, inspired by the successful use of convolutional neural networks (CNN) in low-level image tasks such as image enhancement, super-resolution reconstruction, image reconstruction and image fusion, some scholars use CNN models to synthesize HDR images or HDR-like images to improve their visual quality. Liu [4] converted color images into grayscale images and maintained visual contrast through gradient domain modeling. Deng [5] reconstructed and fused multi-modal images through convolutional sparse coding. Xu [6] constructed a general image fusion network using dense connection modules and VGG shallow features. Prabhakar [7] optimized multi-exposure structural similarity (MEF-SSIM) to realize a multi-exposure image brightness generation network with unsupervised learning, and achieved color multi-exposure image fusion by combining weighted chrominance fusion.

[0005] When applying multi-exposure fusion technology to high-speed moving objects, the short-exposure image may appear sharp while the long-exposure image is blurred. Traditional methods require deblurring the high-exposure image using an image deblurring model, and then feeding the high- and low-exposure images into the multi-exposure fusion model for fusion. This process is time-consuming, inefficient, and ineffective.

[0006] [1]Nah S,Hyun Kim T,Mu Lee K.Deep multi-scale convolutional neural network for dynamic scene deblurring[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2017:3883-3891.

[0007] [2]Zhang H, Dai Y, Li H, et al.Deep stacked hierarchical multi-patchnetwork for image deblurring[C] / / Proceedings of the IEEE / CVF Conference onComputer Vision and Pattern Recognition.2019:5978-5986.

[0008] [3]Park D,Kang DU,Kim J,et al.Multi-temporal recurrent neuralnetworks for progressive non-uniform single image deblurring with incrementaltemporal

[0009] training[C] / / European Conference on Computer Vision.Cham:Springer

[0010] International Publishing,2020:327-343.

[0011] [4]Liu Q,Leung H.Variable augmented neural network for decolorizationand

[0012] multi-exposure fusion[J].Information Fusion,2019,46:114-127.

[0013] [5]Deng X,Dragotti P L.Deep convolutional neural network for multi-modal image restoration and fusion[J].IEEE transactions on pattern analysisand machine intelligence,2020,43(10):3333-3348.

[0014] [6]Xu H,Ma J,Le Z,et al.Fusiondn:A unified densely connected networkfor image fusion[C] / / Proceedings of the AAAI conference on artificialintelligence.2020,

[0015] 34(07):12484-12491.

[0016] [7]Ram Prabhakar K,Sai Srikar V,Venkatesh Babu R.Deepfuse:A deepunsupervised approach for exposure fusion with extreme exposure image pairs[C] / / Proceedings of the IEEE international conference on computervision.2017:4714-4722. 发明内容

[0017] The technical problem to be solved by this invention is to provide a deblurred multi-exposure image fusion method, system, and medium based on a convolutional neural network. This method enables the fused image to contain rich details while being completely free of blur. This method can solve the problem of fusion of blurred multi-exposure images in dynamic scenes, with faster processing time and greater efficiency and effectiveness.

[0018] In order to achieve the purpose of the present invention, the present invention provides a deblurred multi-exposure image fusion method based on a convolutional neural network, which includes the following steps:

[0019] S1: Collect clear pairs of images with different exposures and name them low-exposure images and high-exposure images according to the exposure levels from low to high. Perform multi-exposure image fusion on the collected low-exposure images and high-exposure images to obtain a clear reference image. Add random motion blur to the high-exposure image to generate a blurred high-exposure image. The low-exposure image and the blurred high-exposure image constitute a blurred high-low exposure data set. The blurred high-low exposure data set and the corresponding clear reference images construct a deblurred multi-exposure image fusion dataset, which is divided into training set, validation set, and test set.

[0020] S2: Data preprocessing of the deblurred multi-exposure image fusion dataset;

[0021] S3: Based on the blurred high and low exposure data groups and clear reference images in the training set, the initial deblurred multi-exposure image fusion network is trained to obtain the target deblurred multi-exposure image fusion network, wherein the deblurred multi-exposure image fusion network includes an encoder, a three-stage structure and a decoder. The encoder is used to extract features of the low exposure image to obtain low exposure features. The three-stage structure is used to extract features of the high exposure image to obtain high exposure features. The low exposure features and the high exposure features are mean-fused and input into the decoder to obtain a clear image. The first stage of the three-stage structure is composed of an encoder, and the second and third stages are composed of an encoder-decoder pair. Both the encoder and the decoder include a residual convolution module. The residual convolution module includes a frequency domain learning flow for obtaining high-frequency and low-frequency information and a convolution learning flow with channel attention for obtaining spatial domain information.

[0022] S4: Input the blurred high-exposure image and low-exposure image into the target deblurring multi-exposure image fusion network for deblurring multi-exposure image fusion processing to obtain a clear image with appropriate exposure.

[0023] Furthermore, the paired high-exposure and low-exposure images in the dataset in step S1 constitute a high- and low-exposure dataset with blur, where the data size is 800×800. The high- and low-exposure datasets with blur and the corresponding reference images with moderate exposure are divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0024] Furthermore, the step S2 includes randomly cutting the training set before entering the deblurred multi-exposure image fusion network training into a size of 256×256.

[0025] Furthermore, the deblurred multi-exposure image fusion network in step S3 is a multi-level network. For low-exposure images, the deblurred multi-exposure image fusion network extracts features by inputting the image into an encoder. For high-exposure images, the deblurred multi-exposure image fusion network extracts blurred image features by inputting the image into a three-level structure. The obtained high and low exposure features are averaged and fused and input into a decoder to produce a clear image with appropriate exposure.

[0026] Furthermore, the three-level structure within the deblurred multi-exposure image fusion network can be illustrated using a 1-2-4 model. The notation (1-2-4) represents the number of image patches from the coarsest to the finest level, i.e., the second level is vertically bisected, and the third level is horizontally and vertically quadrupled. The first level of the three-level structure consists solely of an encoder, while the second and third levels of the three-level structure consist of an encoder-decoder pair. The input to each level is generated by partitioning the original blurred image input into multiple non-overlapping image patches. The outputs of the encoder and decoder at the bottom level (the lower level) are added to the encoder at the upper level (the upper level), so that the features generated at the top level contain all the information inferred at the finer levels. This structure enables the lower levels to focus on local information, thereby generating residual information for the coarser network (obtained by concatenating convolutional features), which is ultimately aggregated into the first level of the three-level structure.

[0027] Furthermore, the encoder and decoder in the deblurred multi-exposure image fusion network have similar structures. The encoder has 6 residual convolution modules (Res Block) modules, 2 downsampling layers, and one convolution layer; the decoder has 6 residual convolution modules (Res Block) modules, 2 upsampling layers, and one convolution layer.

[0028] Furthermore, the residual convolution module (Res Block) in the encoder and decoder has the same structure.

[0029] The Residual Convolutional Module (Res Block) has two learning streams: a frequency domain learning stream that converts spatial feature maps to the frequency domain, extracts features using convolution, and then returns them to the spatial domain. This allows for the acquisition of both low-frequency and high-frequency information, improving the Res Block module's ability to extract frequency domain information. A convolutional learning stream with channel attention focuses on extracting both local and global information, improving the Res Block module's ability to extract spatial domain information. In the Res Block structure, 1×1 convolutions are implemented using standard convolutions, while 3×3 convolutions are implemented using DO-Conv convolutions.

[0030] Furthermore, the downsampling in the encoder is implemented by using a 2×2 ordinary convolution module without padding, and the upsampling in the decoder is implemented by combining a 1×1 convolution with a PixelShuttle.

[0031] Furthermore, the residual convolution module (Res Block) includes a convolutional learning flow with channel attention. This channel attention is a simplification of the attention mechanism proposed in SENet. This module extracts the essential aspects of channel attention and simplifies other aspects. First, adaptive average pooling is performed on the spatial dimension. Then, the importance weights between channels are learned through 1×1 convolution. This learned importance weight is then multiplied by the original feature to obtain the weighted features in the spatial dimension.

[0032] Furthermore, the target deblurring multi-exposure image fusion network in step S3 is deep learning, and the optimal loss value is calculated through gradient descent, and the network corresponding to the optimal loss function is saved.

[0033] Compared with the prior art, the present invention has at least the following beneficial technical effects:

[0034] (1) The present invention combines image deblurring with image fusion and proposes a network that can simultaneously realize image deblurring and image fusion. It has the dual functions of image deblurring and multi-exposure fusion, can improve the dynamic range of the image, and enhance the brightness and contrast of the image.

[0035] (2) The present invention uses a new residual convolution module (Res Block) structure. The residual convolution module (ResBlock) has two residual learning flows: a frequency domain learning flow that uses convolution to extract features after converting the spatial feature map to the frequency domain, and finally returns to the spatial domain, thereby achieving the acquisition of low-frequency information and high-frequency information, and improving the frequency domain information extraction capability of the Res Block module; a convolution learning flow with channel attention focuses on extracting local information and global information, and improves the spatial domain information extraction capability of the Res Block module, thereby making the image fusion result clearer and with better details. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 1 is a flow chart of a deblurred multi-exposure image fusion method provided by an embodiment of the present invention;

[0037] Figure 2 This is a diagram of a deblurred multi-exposure image fusion structure according to an embodiment of the present invention;

[0038] Figure 3 This is a structural diagram of an encoder and a decoder in the structural diagram of deblurred multi-exposure image fusion in an embodiment of the present invention;

[0039] Figure 4 1. The structure diagram of the residual convolution module (Res Block) (Figure (a)) and the upsampling structure diagram (Figure (b)) in the encoder and decoder structures in an embodiment of the present invention;

[0040] Figure 5 : This is a diagram of the channel attention SCA structure in the residual convolution module (Res Block) structure in an embodiment of the present invention;

[0041] Figure 6 : These are model effect diagrams in an embodiment of the present invention, where (a) is a high-exposure image with blur, (b) is a low-exposure image, and (c) is a fusion result diagram;

[0042] Figure 7 3 are model effect diagrams of an embodiment of the present invention, wherein (a) is a high-exposure image with blur, (b) is a low-exposure image, and (c) is a fusion result diagram. DETAILED DESCRIPTION

[0043] The embodiments of the present invention are described in detail below. The following embodiments are implemented based on the technical solutions of the present invention, and provide detailed implementation methods and specific operating procedures. However, the protection scope of the present invention is not limited to the following embodiments.

[0044] like Figure 1As shown, the present invention provides a deblurred multi-exposure image fusion method based on convolutional neural network, comprising the following steps:

[0045] Step S1: Construct a deblurred multi-exposure image fusion dataset.

[0046] We collected clear pairs of images at different exposure levels and named them low-exposure and high-exposure images, respectively, according to their exposure levels. Using the multi-exposure fusion model U2Fusion, we fused the high-exposure and low-exposure images to obtain a reference image with appropriate exposure. We then added random motion blur to the high-exposure image to generate a blurred high-exposure image.

[0047] Among them, appropriate exposure refers to an image with good exposure. Compared with high exposure that is too bright and low exposure that is too dark, the overall image is better and closer to the actual exposure level seen by the human eye.

[0048] In some embodiments of the present invention, a dataset of paired blurred high-exposure and low-exposure images constitutes a blurred high- and low-exposure dataset, where the dataset size is 800×800. The blurred high- and low-exposure datasets and their corresponding moderately exposed reference images are divided into a training set, a validation set, and a test set in a ratio of 8:1:1.

[0049] Step S2: performing data preprocessing on the deblurred multi-exposure image fusion dataset.

[0050] In some embodiments of the present invention, during the preprocessing, the images of the training set are cut into images of 256×256 size, which can reduce the training time.

[0051] Step S3: Based on the high and low exposure data sets with blur in the training set and the clear reference images with appropriate exposure, the initial deblurred multi-exposure image fusion network (DeblurFusion) is trained to obtain the target deblurred multi-exposure image fusion network.

[0052] The deblurred multi-exposure image fusion network is a multi-level network. Figure 2 As shown in the figure, for low-exposure images, the deblurring multi-exposure image fusion network extracts features from the image by inputting it into an encoder. For high-exposure images, the deblurring multi-exposure image fusion network extracts features from the blurred image by inputting the image into a three-level structure. The obtained low-exposure features and high-exposure features are mean-fused and input into the decoder to obtain a clear image with appropriate exposure.

[0053] The first level of the three-level structure includes an encoder, and the second and third levels of the three-level structure each include an encoder-decoder pair. The input of each level is generated by dividing the original blurred image input into multiple non-overlapping image blocks. The output of the bottom layer (the lower level) of the encoder and decoder will be added to the encoder of the upper layer (the upper level) so that the features generated by the first level contain all the information inferred from the finer levels (the second and third levels). This structure enables the lower-level structures to focus on local information, thereby generating residual information for the coarser network (obtained by connecting convolutional features), and ultimately the information is aggregated into the first level of the three-level structure.

[0054] The encoder and decoder in the deblurred multi-exposure image fusion network have similar structures, such as Figure 3 As shown, the encoder includes multiple residual convolution modules (Res Block), 2 downsampling layers ((32-64-2-2, 64-128-2-2) in the figure), and a convolution layer; the decoder includes multiple residual convolution modules (Res Block), 2 upsampling layers, and a convolution layer. In some embodiments of the present invention, there are 6 residual convolution modules (Res Block) in the encoder and decoder.

[0055] The structure of the residual convolution module (Res Block) in the encoder and decoder is the same. Figure 4 As shown in Figure (a), the Residual Convolutional Module (Res Block) has two learning streams: a frequency domain learning stream. This stream converts the spatial feature map to the frequency domain, extracts features using convolution, and then returns to the spatial domain, thereby acquiring both low-frequency and high-frequency information. This improves the Res Block module's ability to extract frequency domain information. This stream first converts the image feature map from the spatial domain to the frequency domain using RFFT. It then learns high-frequency and low-frequency information using 1×1 convolution, RELU, and 1×1 convolution. Finally, it converts the features back to the spatial domain using IRFFT. A convolutional learning stream with channel attention focuses on extracting both local and global information, improving the Res Block module's ability to extract spatial domain information. This stream extracts spatial domain information using 3×3 convolution, RELU layers, and 3×3 convolution. The resulting features are then fed into the SCA attention module. In the implementation of the Res Block structure, 1×1 convolutions are implemented using standard convolutions, while 3×3 convolutions are implemented using DO-Conv convolutions. The addition of a frequency domain learning flow to the residual convolution module improves the model's ability to extract frequency domain information; the addition of a convolutional learning flow with channel attention improves the model's ability to extract spatial domain information, resulting in better model performance.

[0056] The downsampling in the encoder is implemented by a 2×2 normal convolution module without padding, and the upsampling in the decoder is implemented by a combination of 1×1 convolution and PixelShuttle. Figure 4 As shown in (Figure b).

[0057] The residual convolution module (Res Block) contains a convolution learning flow with channel attention. This channel attention is a simplification of the attention mechanism proposed in SENet. This module extracts the essential link of channel attention and simplifies other links. First, the spatial dimension is adaptively averaged pooled, and then the importance weights between channels are learned through 1×1 convolution. Then, the learned importance weights are multiplied with the original features to obtain the features after weighting in the spatial dimension. The channel attention structure is as follows Figure 5 shown.

[0058] In some embodiments of the present invention, the three-level structure of the deblurred multi-exposure image fusion network structure can be described using a 1-2-4 model. The symbol (1-2-4) represents the number of image blocks from the coarsest to the finest level, i.e., the second level is a vertical bisection, and the third level is a horizontal and vertical quartering. Vertical or horizontal segmentation should be selected at the second level, and a different segmentation method should be selected at the third level, such as bisection at the second level and quartering at the third level. This ensures that the image size after segmentation is the same at each level, and the size is more appropriate.

[0059] The specific implementation steps of the deblurred multi-exposure image fusion network during operation in this embodiment are as follows:

[0060] In the first step, the low-exposure image input1 is input into the encoder to obtain the feature map feature1=F(input1) of the low-exposure image, where F is the encoder.

[0061] In the second step, the high-exposure image input2 is input into the three-level structure to extract the features of the blurred image. The three-level structure is divided into three levels, represented by i, i = {1, 2, 3}. The initial high-exposure image input with blur is represented as input2, and the segmented image is named input2_ij, where input2_ij is the jth patch of the i-th level, and F i and G i is the encoder and decoder of the i-th level, C ij Input2_ij passes through encoder F i Output, S ij For decoder G i The output patch.

[0062] The third step is to start from the third level of the three-level structure and move up layer by layer. First, the blurred high-exposure image input2_1 is divided into four equal horizontal and vertical image blocks, denoted by input2_3j, where j = {1...4}. The naming rule is counterclockwise, with the top left being input2_31, the bottom left being input2_32, the bottom right being input2_33, and the top right being input2_34. Each of the segmented image blocks (patch) is input into the encoder F3 to obtain the output convolutional feature C. 3j .

[0063] C 3j =F3(input2_3j),j∈{1…4} (1)

[0064] Each two adjacent features are concatenated (concat) 31 with C 34 and C 32 with C 33 Perform concat operation. The new convolution feature is used This aggregation enables the information on the third-level small patch of the tertiary structure to be transferred to the larger patch on the second level of the tertiary structure.

[0065]

[0066] in Represents the concat operation. The features obtained after concat The result output by decoder G3 is expressed as S 3j (represented by output3 in the figure), the formula is as follows

[0067]

[0068] The fourth step is to convert the output S of the third level of the three-level structure 3j The second level input input2_2j of the three-level structure (the two have the same size at this time) is superimposed, and the superimposed feature is input into the encoder F2, and the output F2 (input2_2j+S 3j ) and the features obtained by splicing the third level of the three-level structure are superimposed. As shown in the following formula, the second level feature C of the three-level structure is obtained. 2j .

[0069]

[0070] The fifth step is to concatenate the obtained features and convert C 21 with C 22Perform concat operation. The new convolution feature is used This aggregation enables the information on the second-level patch of the three-level structure to be transferred to the first-level unpatch input of the three-level structure.

[0071]

[0072] Features obtained after Concat The result output by decoder G2 is represented by S2 (represented by output2 in the figure), and the formula is as follows

[0073]

[0074] In the sixth step, the output S2 of the second level of the three-level structure is superimposed with the input input2 of the first level of the three-level structure (both are of the same size). The resulting feature is input into encoder F2. The resulting output F1 (input2 + S2) is superimposed with the concatenated features of the second level of the three-level structure. As shown below, the first-level feature C1 is obtained.

[0075]

[0076] In order to be consistent with the feature naming method in the first step, the obtained high-exposure image feature C1 is named feature2. The meaning of this part is to extract the unblurred high-exposure feature.

[0077] In the seventh step, the low-exposure image feature feature1 and the high-exposure image feature feature2 are averaged to obtain a fused feature map feature = (feature1 + feature2) / 2.

[0078] In the eighth step, the fused feature map is input into the decoder to realize image reconstruction.

[0079] S=G(feature) (8)

[0080] Where G is the decoder and S is the reconstructed image.

[0081] In the ninth step, the restored reconstructed image S and the clear reference image R are simultaneously input into the loss function loss, the optimal loss value is calculated, and the network corresponding to the optimal loss function is saved, which is the target deblurred multi-exposure image fusion network.

[0082] In some embodiments of the present invention, the loss function is a combination of Charbonnier Loss, Frequency Loss, and Edge Loss. The combination in the network is as follows:

[0083] T loss =loss char +0.01loss fft +0.05loss edge (9)

[0084] loss char 、loss fft 、loss edge They are the loss functions of Charbonnier Loss, Frequency Loss, and edge Loss respectively.

[0085] The values ​​0.01 and 0.05 in this embodiment are empirical parameters. In other embodiments, other values ​​may be used based on experience.

[0086] The specific functions of each function are as follows:

[0087] Charbonnier Loss: Approximates L1 loss to improve network performance. The gradient of values ​​close to zero is kept constant due to the presence of ε (ε is an empirical value, intended to prevent gradients from being too large or too small), thus preventing gradient vanishing. Compared to L1 and L2 loss functions, this reduces the problem of oversmoothing and enhances the perceived realism of the photo. The formula is as follows:

[0088]

[0089] Wherein S is the reconstructed image and R is the reference image. In some embodiments of the present invention, the empirical value of ε is 0.001.

[0090] Frequency Loss: Since the purpose of deblurring is to recover the lost high-frequency components, it is necessary to reduce the difference in frequency space. Frequency Loss measures the L1 distance between the reference image and the reconstructed image in the frequency domain. The specific formula is as follows:

[0091]

[0092] Where S is the reconstructed image and R is the reference image. is the FFT transform.

[0093] Edge Loss: This loss function can well consider high-frequency texture structure information and improve the detail performance of the reconstructed image. The specific formula is as follows:

[0094]

[0095] Where S is the reconstructed image and R is the reference image, Δ represents the Laplace operator, and the empirical value of ε is 0.001.

[0096] Step S4: inputting a high-exposure image with blur and a low-exposure image into a target deblurring multi-exposure image fusion network for deblurring multi-exposure image fusion processing to obtain a clear image with proper exposure.

[0097] In some embodiments of the present application, the image fusion result obtained by the method of the present application is as shown in Figure 6 and Figure 7 As can be seen from the figure, the fused image obtained based on the high-exposure image with blur and the low-exposure image removes the blur information of the high-exposure image and can integrate the brightness information of the high-exposure image and the low-exposure image to obtain a clear image.

[0098] The above embodiments of the present application construct a network capable of realizing the combination of multi-exposure image fusion and deblurring, which can generate a clear image with proper exposure based on input of a low-exposure image and a high-exposure image with blur.

[0099] The aforementioned method of the present application has wide application fields, which include but are not limited to the following fields:

[0100] Security monitoring: can be used to improve the monitoring image under dynamic links and improve the accuracy of target detection and recognition.

[0101] Autonomous driving: can improve the effect of images taken by vehicles during high-speed motion and improve the visual effect.

[0102] Computer vision and image recognition: applied to image processing, target detection, image segmentation and other tasks to improve the robustness and performance of algorithms.

[0103] In some embodiments of the present application, a deblurring multi-exposure image fusion system is provided for realizing the aforementioned method, which comprises the following modules:

[0104] An image acquisition module is used to acquire clear images with different exposures, and the images are named as low-exposure images and high-exposure images according to the exposure degree from low to high. The acquired low-exposure images and high-exposure images are subjected to multi-exposure image fusion to obtain a clear reference image. A high-exposure image with blur is generated by adding random motion blur to the high-exposure image. The low-exposure image and the high-exposure image with blur constitute a high-low exposure data set with blur. The high-low exposure data set with blur and the corresponding clear reference image construct a deblurring multi-exposure image fusion data set, and the data set is divided into a training set, a validation set and a test set.

[0105] An image processing module is used to perform data preprocessing on the deblurring multi-exposure image fusion data set.

[0106] The target deblurred multi-exposure image fusion network training module is used to train the initial deblurred multi-exposure image fusion network based on the high and low exposure data sets with blur and the clear reference image in the training set to obtain the target deblurred multi-exposure image fusion network;

[0107] The detection module inputs the blurred high-exposure image and low-exposure image into the target deblurring multi-exposure image fusion network for deblurring multi-exposure image fusion processing to obtain a clear image with appropriate exposure.

[0108] A computer-readable storage medium is also provided, storing a program, which implements the aforementioned method when executed.

[0109] As for the deblurred multi-exposure image fusion system and computer-readable storage medium disclosed in the embodiment, since they correspond to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0110] The above describes in detail the preferred embodiments of the present invention. It should be understood that numerous modifications and variations based on the concepts of the present invention can be made by those skilled in the art without inventive effort. Therefore, any technical solution that can be derived by those skilled in the art based on the concepts of the present invention through logical analysis, reasoning, or limited experimentation based on the existing technology should be within the scope of protection defined by the claims.

Claims

1. A deblurred multi-exposure image fusion method, characterized in that: The following steps are involved: S1: Collect clear pairs of images with different exposures and name them low-exposure images and high-exposure images according to the exposure levels from low to high. Perform multi-exposure image fusion on the collected low-exposure images and high-exposure images to obtain a clear reference image. Add random motion blur to the high-exposure image to generate a blurred high-exposure image. The low-exposure image and the blurred high-exposure image constitute a blurred high-low exposure data set. The blurred high-low exposure data set and the corresponding clear reference images construct a deblurred multi-exposure image fusion dataset, which is divided into training set, validation set, and test set. S2: Data preprocessing of the deblurred multi-exposure image fusion dataset; S3: Based on the blurred high and low exposure data groups and clear reference images in the training set, the initial deblurred multi-exposure image fusion network is trained to obtain the target deblurred multi-exposure image fusion network, wherein the deblurred multi-exposure image fusion network includes an encoder, a three-stage structure and a decoder. The encoder is used to extract features of the low exposure image to obtain low exposure features. The three-stage structure is used to extract features of the high exposure image to obtain high exposure features. The low exposure features and the high exposure features are mean-fused and input into the decoder to obtain a clear image. The first stage of the three-stage structure is composed of an encoder, and the second and third stages are composed of an encoder-decoder pair. Both the encoder and the decoder include a residual convolution module. The residual convolution module includes a frequency domain learning flow for obtaining high-frequency and low-frequency information and a convolution learning flow with channel attention for obtaining spatial domain information. S4: Input the blurred high-exposure image and low-exposure image into the target deblurring multi-exposure image fusion network for deblurring multi-exposure image fusion processing to obtain a clear image with appropriate exposure.

2. The deblurred multi-exposure image fusion method according to claim 1, characterized in that: In step S1, the multi-exposure image fusion is performed through the multi-exposure fusion model U2Fusion to obtain the clear reference image.

3. The deblurred multi-exposure image fusion method according to claim 1, wherein: In the deblurred multi-exposure image fusion network, the encoder has multiple Res Block modules, two downsampling layers, and one convolutional layer; the decoder has multiple Res Block modules, two upsampling layers, and one convolutional layer.

4. The deblurred multi-exposure image fusion method according to claim 3, wherein: The structure of the Res Block module in the encoder and decoder is the same. The frequency domain learning flow in the Res Block module uses convolution to extract features after converting the spatial feature map to the frequency domain, and finally returns to the spatial domain, thereby obtaining low-frequency information and high-frequency information, and improving the frequency domain information extraction capability of the Res Block module; the convolution learning flow with channel attention focuses on extracting local information and global information, and improves the spatial domain information extraction capability of the Res Block module. In the implementation process of the Res Block module, 1×1 convolution is implemented using ordinary convolution, and 3×3 convolution is implemented using DO-Conv convolution.

5. The deblurred multi-exposure image fusion method according to claim 3, characterized in that: The downsampling in the encoder is implemented by a 2×2 ordinary convolution module without padding, and the upsampling in the decoder is implemented by a combination of 1×1 convolution and PixelShuttle.

6. The deblurred multi-exposure image fusion method according to claim 3, wherein: The convolutional learning flow with channel attention in the ResBlock module first performs adaptive average pooling on the spatial dimension, then learns the importance weights between channels through 1×1 convolution, and then multiplies the importance weights with the original features to obtain the features after weighting in the spatial dimension.

7. The deblurred multi-exposure image fusion method according to claim 1, characterized in that: The three-level structure in the deblurred multi-exposure image fusion network is described using a 1-2-4 model, where the symbol 1-2-4 represents the number of image blocks from the coarsest to the finest level, i.e., the second-level equivalence vertical bisection and the third-level horizontal vertical bisection.

8. The deblurred multi-exposure image fusion method according to claim 1, characterized in that: In the three-level structure, the third level is the lowest level, and the input of each level is generated by splitting the original blurred image input into multiple non-overlapping image blocks. The output of the encoder and decoder of the next level will be added to the encoder of the upper level so that the features generated by the first level contain all the information inferred in the lower level.

9. A deblurred multi-exposure image fusion system, characterized in that: For implementing the method according to any one of claims 1 to 8, the system comprises: The image acquisition module is used to collect clear pairs of images with different exposures, and name them low-exposure images and high-exposure images according to the image exposure levels from low to high. The collected low-exposure images and high-exposure images are fused to obtain a clear reference image. A blurred high-exposure image is generated by adding random motion blur to the high-exposure image. The low-exposure images and the blurred high-exposure images constitute a blurred high-low exposure data set. The blurred high-low exposure data set and the corresponding clear reference images are used to construct a deblurred multi-exposure image fusion dataset, which is divided into a training set, a validation set, and a test set. Image processing module, used to perform data preprocessing on the deblurred multi-exposure image fusion dataset; The target deblurred multi-exposure image fusion network training module is used to train the initial deblurred multi-exposure image fusion network based on the high and low exposure data sets with blur and the clear reference image in the training set to obtain the target deblurred multi-exposure image fusion network; The detection module inputs the blurred high-exposure image and low-exposure image into the target deblurring multi-exposure image fusion network for deblurring multi-exposure image fusion processing to obtain a clear image with appropriate exposure.

10. A computer-readable storage medium storing a program, characterized in that: When the program is executed, the method according to any one of claims 1 to 8 is implemented.

Images