Image enhancement method based on multimodal loss function U-shaped coding-decoding network

Abstract

The application discloses an image enhancement method based on a multi-modal loss function U-type coding and decoding network, relates to the field of image processing, and comprises the following steps: designing a U-type coding and decoding network model, taking short exposure S0 and Dolp images as a training set, corresponding long exposure images as true value references, inputting the network for training, designing a light intensity and linear polarization degree multi-modal loss function using a YUV color space, adjusting network parameters by using a gradient descent algorithm until a maximum iteration number, and outputting a network model; inputting a short exposure test set image into the trained network model, outputting an enhanced image, and comparing and analyzing the enhanced image with a corresponding true value image. The network model improved by the application has improved peak signal-to-noise ratio, structural similarity and processing speed in color polarized image restoration.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of dark light color image enhancement, and particularly relates to an image enhancement method based on a U-shaped coding and decoding network of a multi-modal loss function. BACKGROUND

[0002] With the development of photoelectric technology and the rise of related industries, photoelectric detection technology has been widely applied and developed unprecedentedly in the fields of military detection, city security, environmental monitoring, special monitoring, etc. The traditional photoelectric imaging technology uses the electromagnetic waves reflected or radiated by the target object as a signal source to detect, identify and track the target to be identified. However, as people's requirements for imaging information continue to rise, the traditional photoelectric imaging technology has been unable to meet the growing demand: in actual use scenarios, the background clutter, light changes, obstacles and other factors in the surrounding environment may interfere with the traditional photoelectric imaging technology. For example: in military detection, if the target radiation is similar to the background radiation or the background radiation is too strong, it is difficult to accurately and efficiently identify the target in the background clutter; or the target uses optical camouflage, then the traditional photoelectric imaging technology is helpless; or the target is in water, then the scattering of water will cause great errors. Simply put, the interference in the actual detection scene will greatly increase the complexity of identification, greatly weaken the optical characteristics difference between the target and the non-target, and cause obvious gray response aliasing phenomenon, thereby causing the identification efficiency of the traditional photoelectric imaging technology to decrease significantly, and even produce serious errors.

[0003] Polarization is one of the important physical properties of light. The target on the ground or in the atmosphere will produce specific polarization information when reflecting, scattering, transmitting and radiating electromagnetic waves, and these polarization information can be used to analyze the shape, surface roughness, texture direction and physical and chemical properties of the material of the target. By combining light intensity imaging and polarization imaging, the response aliasing effect of light intensity can be overcome to some extent, the shortcomings of light intensity imaging in dark light conditions can be solved, the signal-to-noise ratio of the imaging system can be improved, and more dimensional information can be provided.

[0004] In a complex and variable optical environment, especially in dark light conditions with extremely low photon counting, polarization imaging often has great difficulty. Since the generation of polarization images is mostly obtained by calculating the intensity of polarized light, the low signal-to-noise ratio of light intensity in dark light conditions will significantly reduce the clarity of polarization images. Moreover, the transmission rate of the polarizing plate may decrease in dark light conditions, thereby causing image degradation. Traditional optical imaging devices and data processing methods often cannot obtain and process the polarization information of the image, affecting target detection and identification

[0005] Traditional low-light environment imaging enhancement denoising techniques such as NLM, CBM3D algorithm are mostly based on similar blocks for enhancement denoising. However, due to the different sensitivities of light intensity (S0), degree of linear polarization (DoLP) and angle of polarization (AoP) to noise in different scenes. The traditional enhancement scheme can only effectively enhance the color intensity of the RGB three channels in general. For optoelectronic detection systems that need to use RGB imaging technology and polarization imaging technology at the same time, these processing methods often cannot take into account the enhancement of polarization imaging, and thus a large number of under-enhancement or over-enhancement problems will be generated. This makes the traditional enhancement method lose its inherent advantages in the face of the rapidly changing imaging system, so that it may produce extremely serious errors when working in complex environments. SUMMARY

[0006] In order to overcome the shortcomings of the above-mentioned traditional technology, the present application provides a polarization image enhancement method based on deep learning to realize fast and accurate enhancement and restoration of dark light images.

[0007] The technical solution for achieving the purpose of the present application is: an image enhancement method based on a multi-modal loss function U-shaped encoding and decoding network, comprising the following steps:

[0008] Step 1, constructing an image enhancement network PolarUnet;

[0009] Step 2, constructing a loss function of the image enhancement network PolarUnet;

[0010] Step 3, initializing the parameters of the image enhancement network PolarUnet: learning rate lr, picture cropping size patch-size sent into the network per batch, picture quantity batch-size sent into the network per batch, maximum iteration number;

[0011] Step 4, inputting the training set original image and corresponding true value into the denoising network PolarUnet, training until the loss function converges, and obtaining the trained image enhancement network PolarUnet model;

[0012] Step 5, inputting the test set image to be enhanced into the image enhancement network PolarUnet trained in step 5 for image enhancement processing.

[0013] Preferably, the specific steps of constructing the image enhancement network PolarUnet are as follows:

[0014] Step 1.1, the overall structure of the network is a symmetrical feature extraction network and a feature fusion network as well as a bottom convolutional network. The data shape in the corresponding layer of the feature extraction network and the feature fusion network is consistent. The channel number of each layer of the feature extraction network is twice that of the previous layer, and the height and width are The number of channels of each layer of the feature fusion network is twice that of the next layer, and the height and width are twice that of the next layer The incoming data is a six-channel image composed of three-channel S0 images and three-channel Dolp images in the channel dimension;

[0015] Step 1.2, build a four-layer feature extraction network, each layer of which is composed of two 3x3 convolution layers and a 2x2 max pooling layer; each layer of the network is first convolved twice with 3x3, and then max-pooled with 2x2, the first three layers output to the next layer of the feature extraction network, and the last layer outputs to the bottom convolution network;

[0016] Step 1.3, build a bottom convolution network composed of two 3x3 convolution layers, and output to the first layer of the feature fusion network;

[0017] Step 1.4, build a four-layer feature fusion network, each layer of which is composed of a 2x2 up-sampling convolution layer, a feature concatenation layer, and two 3x3 convolution layers; first deconvolve with 2x2, and then fuse the features with the corresponding feature extraction layer after convolution, and then convolve twice with 3x3. The first three layers output to the next layer of the network, and the last layer outputs through a 1x1 convolution layer to restore to a 6-channel output.

[0018] Step 1.5, use LRelu activation function in convolution layer:

[0019]

[0020] Preferably, the loss function of the image enhancement network PolarUnet built in step 2 is specifically:

[0021] Loss=λ S0 ×Loss S0 +λ Dolp ×Loss Dolp

[0022]

[0023]

[0024] wherein, Loss S0 is the Loss value of the light intensity mode, Loss Dolp is the Loss value of the linear polarization degree mode. n is the Batch-size number of each training input, represents the Y channel image after Rgb-Yuv conversion of the light intensity channel of the output image after each round of network training, and its corresponding true value image; represents the UV channel image after Rgb-Yuv conversion of the light intensity channel of the output image after each round of network training, and its corresponding true value image; The output image polarization degree channel image after each round of network training and the corresponding true value image are represented.

[0025] Step 3, using the loss function function constructed in step 2, using the Adam optimizer to optimize the network parameters; the training set is cut according to the patch-size, randomly grouped according to the batch-size, and then sent into the network, and the network is trained until the maximum iteration number, and the trained network model is obtained.

[0026] Compared with the prior art, the present application has the following advantages: (1) by converting the image in YUV color space, by adjusting the linear combination coefficient of the loss function of the different Y and UV channels, the color difference can be reduced without affecting the overall brightness enhancement effect of the image; while not significantly reducing the information amount of the light intensity loss function, the influence of the linear polarization degree loss function is reduced, which indirectly improves the enhancement effect of the linear polarization degree image; (2) the model obtained by training in the present application can realize visible light image denoising quickly while ensuring the denoising performance, greatly improving the efficiency.

[0027] The present application will be further described below in conjunction with the drawings of the specification. BRIEF DESCRIPTION OF DRAWINGS

[0028] Figure 1 The flowchart of the present application is a Unet color polarization image denoising method based on a multi-modal loss function.

[0029] Figure 2 The Unet network structure diagram is shown.

[0030] Figure 3 The output light intensity image comparison chart of different algorithms in the example is shown.

[0031] Figure 4 The output linear polarization degree image comparison chart of different algorithms in the example is shown. DETAILED DESCRIPTION

[0032] As shown in Figure 1 , a U-shaped coding and decoding network based on a multi-modal loss function image enhancement method, comprising the following steps:

[0033] Step 1, combined with Figure 2 , build an image enhancement network PolarUnet, specifically as follows:

[0034] Step 1.1, build a five-layer feature extraction network, which is composed of two 3x3 convolution layers and one 2x2 max-pooling layer. Each layer of the network is first convolved twice by 3x3, and then transmitted to the next layer of the network through 2x2 max-pooling.

[0035] Step 1.2, a five-layer feature fusion network is constructed, which is composed of a 2x2 up-sampling convolution layer, a feature splicing layer and two 3x3 convolution layers. Each layer of the network first performs 2x2 up-sampling convolution, and then performs feature fusion with the feature map after convolution of the corresponding feature extraction layer, and then performs two 3x3 convolutions to pass into the next layer of the network.

[0036] Step 2, construct the loss function Loss, the specific formula is:

[0037] Loss=λ S0 ×Loss S0 +λ Dolp ×Loss Dolp

[0038]

[0039]

[0040] wherein, Loss S0 is the Loss value of the light intensity mode, Loss Dolp is the Loss value of the linear polarization degree mode. n is the Batch-size number of each training input, represents the Y channel image after Rgb-Yuv conversion of the light intensity channel of the output image after each round of network training, and its corresponding true value image; represents the UV channel image after Rgb-Yuv conversion of the light intensity channel of the output image after each round of network training, and its corresponding true value image; represents the polarization degree channel image of the output image after each round of network training, and its corresponding true value image.

[0041] Step 3, initialize network parameters: learning rate lr, picture cropping size patch-size sent into the network every batch, maximum iteration number, as follows:

[0042] Set the input picture patch-size to 64x64, the training batch-size to 16, the initial learning rate lr to 1x10 -5 , and the training epoch to 400.

[0043] Step 3, divide the LLCP dataset into training set, validation set and test set, crop the training set images to patch-size and group them according to batch-size:

[0044] Step 4, input the original images and corresponding true values of the grouped training set into the denoising network PolarUnet, and train it until the loss function converges, to obtain the trained image enhancement network PolarUnet model.

[0045] Step 5, input the image to be enhanced into the image enhancement network PolarUnet trained in step 3 for image enhancement processing, output the enhanced image and compare it with the true value image.

[0046] Step 5.1, compare the output enhanced image with the true value image to evaluate performance, get PSNR and SSIM performance indicators.

[0047] PSNR is used to measure the difference between two images, the formula is as follows:

[0048]

[0049] Where, MSE is the mean square error of two images; MaxValue is the maximum value of image pixels.

[0050] SSIM is based on the assumption that the human eye will extract structured information in the image, which is more consistent with human visual perception than traditional methods, and the formula is as follows:

[0051] SSIM(x,y)=[l(x,y)] α ×[c(x,y)] β ×[s(x,y)] γ

[0052]

[0053]

[0054]

[0055]

[0056]

[0057]

[0058] Where, N is the total number of picture samples, x i represents the output image of network training, y i represents the true value image corresponding to the output image, C1=(K1L) 2 ; C2=(K2L) 2 ; L is the maximum value of image pixels; K1 is usually taken as 0.01; K2 is usually taken as 0.03; α=β=γ=1 and to simplify the formula:

[0059]

[0060] SSIM≤1, the larger the SSIM, the more similar the two images.

[0061] Example 1

[0062] This example compares the present application with the conventional methods NLM, CBM3D and the general Unet based on deep learning on the data set LLCP. The experimental environment is Tensorflow-gpu1.14.0, the GPU used is NVIDIA Titan RTX, the CPU is 11th Gen Inter(R) Core(TM) i5-10210U 1.6Ghz, the GPU device running memory is 64GB, and the CPU device running memory is 16G.

[0063] After the 250 1024x1224 training images are preprocessed by the above step 3, they are input into the PolarUnet network for training, and the output images are evaluated, with the results as follows:

[0064] Table 1 Evaluation indicators of s0 images of each method on the LLCP data set (the optimal value is in bold)

[0065]

[0066] Table 2 Evaluation indicators of Dolp images of each method on the LLCP data set (the optimal value is in bold)

[0067]

[0068]

[0069] Table 2 Image processing time of each method on the LLCP data set (the optimal value is in bold)

[0070]

[0071] In Table 1 and Table 2, PSNR and SSIM are used to measure the denoising effect of each algorithm on noisy images. The higher the PSNR and the closer the SSIM to 1, the better the denoising performance of the algorithm on the image. The processing time Time (unit: seconds) is the average time taken by the model. The test set contains multiple images, so the data in the table is the average value of PSNR, SSIM and Time of multiple test images in the test set.

[0072] From Table 1 and Table 2, it can be seen that the application has more outstanding performance on the LLCP test set compared with other methods. On the light intensity image, compared with the better NLM method in the traditional method, the average PSNR of the application is increased by 5.555, and the average SSIM is increased by 0.038; compared with the UNet algorithm based on the deep learning method, the average PSNR of the application is increased by 3.439, and the average SSIM is increased by 0.005. On the online polarization degree image, compared with the better NLM method in the traditional method, the average PSNR of the application is increased by 3.635, and the average SSIM is increased by 0.3476; compared with the UNet algorithm based on the deep learning method, the average PSNR of the application is increased by 2.638, and the average SSIM is increased by 0.098. From Table 3, it can be seen that the application is far superior to the traditional algorithm in processing speed, and the average processing time is only 0.01s higher than that of the ordinary U-type coding and decoding network.

[0073] The application obtains the highest average value on the PSNR and SSIM of two different images, which means that, relative to the other three existing denoising algorithms, PolarUnet has better average denoising effect in an objective way. At the same time, from Table 3, it can be seen that, compared with the traditional algorithm, the application greatly improves the denoising processing time. Figure 3 、 Figure 4 The method of the application can realize fast and efficient visible light image denoising in combination with the denoising comparison effect diagram of the NLM method, the CBM3D method and the UNet method.

[0074] Figure 3 In the figure, (a) is an original S0 image; (b) is a true value S0 image; (c) is an S0 image after NLM denoising; (d) is an S0 image after CBM3D denoising; (e) is an S0 image after UNet denoising; and (f) is an S0 image after denoising of the application.

[0075] Figure 4 In the figure, (a) is an original DOLP image; (b) is a true value DOLP image; (c) is a DOLP image after NLM denoising; (d) is a DOLP image after CBM3D denoising; (e) is a DOLP image after UNet denoising; and (f) is a DOLP image after denoising of the application. (The image is presented in a pseudo-color mode).

Claims

1. An image enhancement method based on a multi-modal loss function U-type coding and decoding network, characterized in that, Comprising the following steps: Step 1, according to the characteristics of U type coding network to build image enhancement network PolarUnet, the specific steps are as follows: Step 1.1, the overall structure of the network is a symmetrical feature extraction network and a feature fusion network and an underlying convolutional network; the data shape in the corresponding layer of the feature extraction network and the feature fusion network is consistent; the number of channels of each layer of the feature extraction network is twice that of the previous layer, and the height and width are twice those of the previous layer The incoming data is a six-channel image composed of three-channel S0 images and three-channel Dolp images in the channel dimension; Step 1.2, a four-layer feature extraction network is constructed, each layer of feature extraction network is composed of two 3*3 convolution layers and a 2*2 maximum pooling layer; each layer of network is first convolved twice by 3*3, and then pooled by 2*2 maximum, the output of the first three layers of network is transmitted to the next layer of feature extraction network, and the output of the last layer is transmitted to the bottom layer convolution network; Step 1.3, the bottom layer convolution network is constructed by two 3*3 convolution layers, and the output is transmitted to the first layer of feature fusion network; Step 1.4, a four-layer feature fusion network is constructed, each layer of feature fusion network is composed of a 2*2 up-sampling convolution layer, a feature splicing layer and two 3*3 convolution layers; first, 2*2 deconvolution is performed, and then feature fusion is performed with the feature map after convolution of the corresponding feature extraction layer, and then two 3*3 convolutions are performed, the output of the first three layers is transmitted to the next layer of feature fusion network, and the output of the last layer is restored to 6 channels by 1*1 convolution layer; Step 2, the loss function of the image enhancement network PolarUnet is constructed, specifically: Loss = λ S0 × Loss S0 + λ Dolp × Loss Dolp wherein, Loss S0 is the Loss value of the light intensity mode, Loss Dolp is the Loss value of the linear polarization mode, n is the batch-size number of each training input, denote the Y channel images of the output image light intensity channel after Rgb-Yuv conversion and the corresponding ground truth images after each round of network training; denote the UV channel images of the output image light intensity channel after Rgb-Yuv conversion and the corresponding ground truth images after each round of network training, respectively; denote the polarization degree channel images of the output image and the corresponding ground truth images after each round of network training, respectively; Step 3, initialize the parameters of the image enhancement network PolarUnet; Step 4, add Gaussian noise to the true value image to obtain a training image, input the training image into the image enhancement network PolarUnet, train the image enhancement network PolarUnet until the loss function converges, and obtain the trained PolarUnet network model; Step 5, input the image to be enhanced into the PolarUnet network trained in step 5 for image denoising processing.

2. The image enhancement method based on the multi-modal loss function U-shaped coding and decoding network according to claim 1, characterized in that, The initialized parameters of the image enhancement network PolarUnet include: learning rate lr, picture cropping size patch-size sent into the network per batch, and maximum iteration number.

3. The image enhancement method based on the multi-modal loss function U-shaped coding and decoding network of claim 1, characterized in that, The network model is trained, and the steps are as follows: Step 5.1, select part of the original images and their corresponding knitted images from the total image set as the training set; Step 5.2, use the loss function constructed in step 2, use Adam optimizer to optimize the network parameters; after the training set is cropped according to patch-size and grouped according to batch-size, it is sent into the network, and the network model is trained until the maximum iteration number is reached.

Images