Brightness adjustment method for incorrectly exposed images based on unpaired training

Abstract

The present invention designs a brightness adjustment method for mis-exposed images based on unpaired training, which belongs to the field of computer vision image technology. The present invention first decouples an image of any brightness level and an image of target brightness into a content component and a brightness component that satisfies a Gaussian distribution through an encoder, or directly inputs a Gaussian distribution as the target brightness component, and then recouples the content component of a certain brightness image to be processed and the brightness component of the target through a decoder to generate an image of the target brightness. The processing of the brightness component refers to fitting the brightness components of different brightness levels into a unique corresponding Gaussian distribution and converting it into an adjustable variable. The present invention realizes the conversion of target brightness by controlling the mean of the Gaussian distribution. Experiments show that the present invention realizes the conversion between different brightness of images, can effectively restore under-exposed and over-exposed images of different degrees, and has strong generalization ability in various data sets.

Description

Technical Field

[0001] The present invention belongs to the technical field of computer vision images, and in particular to a brightness adjustment method for an erroneously exposed image based on non-paired training. Background Art

[0002] With the development of the times and the advancement of science and technology, especially the popularity of smart devices, more and more people use cameras to shoot, and the captured images are used for various purposes. In photography, overexposure and underexposure are common errors, which will lead to poor image quality. There are many reasons for exposure errors, such as measurement errors of scene light during automatic exposure, exposure errors during manual adjustment, and poor lighting conditions. The above problems caused by shooting are difficult to correct directly through a single software or operation after complex camera image signal processing. Images generated by incorrect exposure have low contrast and poor visual effects, and will also affect subsequent applications. Existing methods mainly focus on correcting underexposure or general image quality enhancement, and few methods can simultaneously restore overexposed and underexposed images to images with correct exposure. In addition, different scenes and subjective needs have different definitions of normal brightness levels, and a method that can obtain images with different brightness levels is needed. Therefore, how to simultaneously repair overexposed and underexposed images and efficiently perform adjustable brightness enhancement on low-quality images is an important but challenging research topic in the field of computer vision.

[0003] Image enhancement technology refers to a method that uses a series of technologies to improve the visual effect of an image. Specifically, the enhancement of an incorrectly exposed image refers to restoring the brightness and improving the visual effect of the incorrectly exposed image with problems such as low brightness, low contrast, and low signal-to-noise ratio. Since the image enhancement effect is subjective and the requirements for brightness levels are different under different needs, it is more universal to achieve adjustable brightness for incorrectly exposed images.

[0004] Most traditional exposure correction and contrast enhancement methods use histogram equalization to improve image quality. Histogram equalization is a classic digital image processing method that enhances image contrast by approximating the histogram distribution of an image to a uniform distribution, thereby improving visual effects. Common methods include global histogram equalization, local histogram equalization, histogram regulation, dynamic histogram regulation, and histogram partitioning methods. There are also studies that use the adjustment of brightness curves to correct images with exposure errors. In addition, many works use the Retinex theory, assuming that the image can be decomposed into reflection and illumination components, predicting the illumination map to restore the target image with good exposure, such as the low-illumination enhancement technique (LIME) that applies a structural prior on the initial illumination map as the final illumination map. The performance of traditional methods cannot be guaranteed, the processing conditions are limited, and it is difficult to be widely used. With the development of convolutional neural networks and the improvement of computing power, more and more researchers choose to use deep learning methods to solve image enhancement problems, such as image enhancement technology that learns feature mapping from mobile phones to DSLR cameras (DPED), convolutional neural networks based on Retinex theory (RetinexNet), generative adversarial networks based on unpaired data pairs (EnlightenGAN), and reference-free image enhancement methods that learn image-to-curve mapping to adjust image brightness (Zero-DCE). However, most of these methods are limited to correcting underexposure errors, and perform poorly in overexposed image processing. It is impossible to use a single model to simultaneously correct overexposed and underexposed images. They can only learn one-to-one mappings and cannot be adjusted to multiple brightness levels.

[0005] In addition, it is difficult to obtain paired low-quality images and normal lighting images, and the cost of shooting and selecting is high, and the images obtained by manual adjustment are different from the real samples. Training based on paired data sets will also lead to poor generalization of the model. Therefore, our invention aims to solve the conversion problem between different brightness levels through unpaired training. In order to achieve the mapping between images of different brightness levels, we adopt a generative model. The generative model captures the real data distribution from the training data, and then uses the learned model and distribution to generate new data. Common deep generative models include variational autoencoders (VAE) and generative adversarial networks (GAN). VAE is a generative network based on variational Bayesian (VB) reasoning. The disadvantage of VAE is that the generated samples are usually blurred due to injected noise and imperfect measurement methods. GAN includes two iteratively trained models: a generative model for generating samples and a discriminative model for distinguishing natural samples from generated samples. The disadvantage of GAN is that the model is difficult to converge during the training phase, and the generated samples are often far from natural samples. Many methods have been proposed to optimize GAN to improve the quality of generated samples. Our invention structurally combines the two, encodes the image into content and brightness, and converts the brightness encoding into a Gaussian distribution, thereby sampling the Gaussian distribution corresponding to different brightness levels into brightness encoding, and then reconstructs the image in combination with the content encoding. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and propose a brightness adjustment method for mis-exposed images with a reasonable design and no paired training based on a generative adversarial network.

[0007] The present invention solves the technical problem by adopting the following technical solutions:

[0008] A brightness adjustment method for an incorrectly exposed image based on unpaired training comprises the following steps:

[0009] Step 1: At the image input stage, the image with incorrect exposure is subjected to data augmentation such as random flipping to increase sample diversity, and the image is normalized from the pixel value range [0, 255] to [0, 1];

[0010] Step 2: Input the exposure error image and brightness provision obtained in step 1 into the encoding network and decouple them into content encoding and brightness encoding, and perform feature extraction and conversion respectively. In this stage, the two encoders output the content encoding of the exposure error image and the brightness encoding corresponding to the brightness provision through operations such as convolutional layers, residual blocks, global average pooling layers, fully connected layers, and reparameterization;

[0011] Step 3: Recombine the content coding and brightness coding obtained in step 2, and output an image with the same brightness level as the brightness provided through operations such as upsampling layers and convolution layers.

[0012] Step 4: Input the image obtained in step 3 into the discriminator and classifier, where the discriminator is used to determine whether the image is real, and the classifier determines whether the brightness level of the current image is the brightness level corresponding to the input brightness level, which is used as a loss term to optimize the network;

[0013] Step 5: Adjust the reconstructed image obtained in step 3 so that its pixel value ranges from [0, 1] to [0, 255] to obtain an image with adjusted brightness.

[0014] Furthermore, the specific method of decoupling the exposure error image and the brightness provided amount input into the encoding network in step 2 includes the following contents:

[0015] (1) If it is in the test phase, the brightness provided is the brightness level or the brightness reference image; if it is in the training phase, the brightness provided is the original image or the randomly selected brightness level; if the brightness provided is the image type, we input the brightness encoder to output a brightness code that satisfies the Gaussian distribution style; if the brightness provided is the brightness level type, we convert the level (a number) into a Gaussian distribution to obtain the brightness code;

[0016] (2) The brightness encoder contains four convolutional layers, one global average pooling layer, and two fully connected layers. First, a 7×7 convolution is performed to change the number of output channels from 3 channels to 64 channels without changing the size of the feature map; then three 4×4 convolutions are performed to upsample the size to 1 / 2, 1 / 4, and 1 / 8 of the original size, and the channels are changed to 128, 256, and 256 respectively; then global average pooling is used to convert the feature map into a 256-dimensional feature vector, and the 100-dimensional mean and variance corresponding to the brightness code are obtained through two fully connected layers respectively. Finally, the brightness code is fitted to a Gaussian distribution as the output by combining the reparameterization technique;

[0017] (3) The content encoder contains three convolutional layers and four residual blocks for downsampling and feature extraction. First, a 7×7 convolution is performed to change the number of output channels from 3 channels to 64 channels without changing the size of the feature map; then two 4×4 convolutions are performed for upsampling, the size is reduced to 1 / 2 and 1 / 4 of the original, and the channels are changed to 128 and 256; finally, four residual blocks composed of 3×3 convolutions are passed, without changing the size of the feature map and the number of channels, and the final output is used as the content code;

[0018] (4) To fit the brightness code to a Gaussian distribution, we designed a distribution consistency constraint based on the relative entropy (KL divergence) that measures the difference between the two distributions. The smaller the relative entropy, the closer the two distributions are. We calculated the relative entropy between the brightness code generated by the brightness encoder with a Gaussian distribution and the Gaussian distribution with the corresponding level and unit variance as the mean.

[0019] Further, the specific details of step (1) include the following:

[0020] ① If the brightness provided is a brightness reference image, the brightness encoder is input for feature extraction, and the obtained feature vector is fitted into a Gaussian distribution to obtain brightness coding;

[0021] ② If the brightness provided is a brightness level, the value is used as the mean to fit a Gaussian distribution (unit variance), and after dimension expansion and dimension replication operations, it becomes a code with the same size as the brightness code generated by the brightness encoder.

[0022] Furthermore, the specific method of recombining the brightness coding and the content coding in step 3 includes the following contents:

[0023] (1) Corresponding to the encoder, the content encoding is first processed through four residual blocks, and the feature map size and number of channels remain unchanged (1 / 4 of the input width and height, 256 channels); then through two upsampling layers, the size becomes 2 and 4 times of the original, and the number of channels becomes 128 and 64, that is, the width and height are the same as the input, and the number of channels is 64; finally, a 7×7 convolution is performed to generate a reconstructed 3-channel image with the same size and number of channels as the input image;

[0024] (2) The residual block uses adaptive instance normalization (AdaIN) to align the mean and variance of the content image features with the mean and variance of the brightness image. The parameters of AdaIN are dynamically generated from the brightness encoding by a multi-layer perceptron (MLP):

[0025]

[0026] Where c is the content code, μ(c) and μ(c) represent the mean and standard deviation of the content image features, respectively, and γ and β correspond to the parameters generated by the MLP for the brightness image. Therefore, the content image is first normalized, multiplied by the standard deviation of the brightness image, and then added to its mean. This makes the standard deviation and mean of the original content feature map consistent with the brightness feature map. Figure 1 To achieve brightness conversion;

[0027] (3) Calculate the self-consistency loss function;

[0028] Furthermore, the specific method for calculating the self-consistency loss function in step (3) includes the following:

[0029] ① During training, when the brightness provided in step 2 of claim 1 is the original image, the image obtained in step 3 of claim 1 is sent to a brightness encoder and a content encoder to obtain brightness encoding and content encoding of the reconstructed image;

[0030] ② Calculate the consistency constraint of the reconstructed image: the reconstructed image in ① should be consistent with the reconstructed image obtained in step 3 of claim 1, and calculate their L1 loss, SSIM loss and perceptual loss respectively;

[0031] ③ Calculate the consistency constraint of the reconstructed content component: use the content encoding in ① and the content encoding obtained in step 2 of claim 1 to calculate the L1 loss;

[0032] ④ Calculate the consistency constraint of the reconstructed brightness component: use the brightness coding in ① and the brightness coding obtained in step 2 of claim 1 to calculate the L1 loss;

[0033] Furthermore, the specific method of inputting the obtained image into the discriminator and classifier in step 4 includes the following contents:

[0034] (1) If it is in the testing phase, ignore this step; if it is in the training phase, perform the following operations;

[0035] (2) patchGAN is used to determine whether the generated image is "real" or "fake", combining local image features and global image features; the basic GAN is used as the classifier C to classify the brightness of the image and determine the brightness level (label) of the current generated image; first, four 4×4 convolutions are performed to extract features, and then the discriminator and classifier are respectively output through convolutional layers of different sizes with the number of channels being 1 and the number of brightness level categories (set to 7 in the present invention);

[0036] (3) Calculate classification and discrimination losses: The images obtained in step 3 and step 1 are sent to the discriminator and classifier to calculate the loss. The discriminator determines whether the images in step 1 and step 3 are real, and the classifier determines whether the images in step 1 and step 3 correspond to their respective brightness level labels.

[0037] The advantages and positive effects of the present invention are:

[0038] 1. The present invention is reasonably designed, meets practical needs, and realizes multi-level exposure adjustment based on non-paired training. The present invention can use one model to process underexposed and overexposed images and convert images between different brightness levels.

[0039] 2. Since the present invention does not require paired datasets, it can reduce the cost of data collection and simplify training. Due to unpaired training, it has better generalization ability and is applicable to various datasets and real-world images.

[0040] 3. The model proposed in this invention decouples the image into content space and brightness space and encodes them separately for training. The brightness encoding is fitted to a Gaussian distribution, and each brightness level label corresponds to a different Gaussian distribution, thereby realizing the conversion between different brightness levels. We use the self-regularization of image features to design consistency loss and optimize unpaired training. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 is the overall network architecture of the present invention, wherein the dotted line represents that the brightness provided can be an image or a Gaussian distribution;

[0042] Figure 2 It is a specific structural diagram of the image decoupling and reconstruction module of the present invention. DETAILED DESCRIPTION

[0043] The embodiments of the present invention are further described in detail below with reference to the accompanying drawings.

[0044] A brightness adjustment method for mis-exposed images based on unpaired training, such as Figure 1 to Figure 2 As shown, the following steps are included:

[0045] Step S1: In the image input stage, the image with incorrect exposure is subjected to data augmentation methods such as random flipping to increase sample diversity, and the image is normalized from the pixel value range [0, 255] to [0, 1];

[0046] Step S2: Input the exposure error image and the brightness provided amount obtained in step S1 into the encoding network and decouple them into content encoding and brightness encoding, and perform feature extraction and conversion respectively. In this stage, the two encoders output the content encoding of the exposure error image and the brightness encoding corresponding to the brightness provided amount through operations such as convolutional layers, residual blocks, global average pooling layers, fully connected layers and re-parameterization;

[0047] Step S3: Recombine the content coding and brightness coding obtained in step S2, and output an image with the same brightness level as the brightness provided by the upsampling layer, convolution layer and other operations.

[0048] Step S4: input the image obtained in step S3 into the discriminator and the classifier, wherein the discriminator is used to determine whether the image is real, and the classifier determines whether the brightness level of the current image is the brightness level corresponding to the input brightness supply, which is used as a loss term to optimize the network;

[0049] Step S5: Adjust the reconstructed image obtained in step S3 so that its pixel value range is from [0, 1] to [0, 255], thereby obtaining an image with adjusted brightness.

[0050] The specific implementation method of step S2 is as follows:

[0051] Step S2.1: If it is in the test phase, the brightness provided is the brightness level or the brightness reference image; if it is in the training phase, the brightness provided is the original image or the randomly selected brightness level; if the brightness provided is the image type, we input the brightness encoder to output a brightness code that satisfies the Gaussian distribution style; if the brightness provided is the brightness level type, we convert the level (a number) into a Gaussian distribution to obtain the brightness code;

[0052] Step S2.2: The brightness encoder contains four convolutional layers, one global average pooling layer, and two fully connected layers. First, a 7×7 convolution is performed to change the number of output channels from 3 channels to 64 channels without changing the size of the feature map; then three 4×4 convolutions are performed to upsample, and the size is reduced to 1 / 2, 1 / 4, and 1 / 8 of the original size, and the channels are changed to 128, 256, and 256 respectively; then global average pooling is used to convert the feature map into a 256-dimensional feature vector, and the 100-dimensional mean and variance corresponding to the brightness code are obtained through two fully connected layers respectively, and finally the brightness code is fitted to a Gaussian distribution as the output by combining the reparameterization technique;

[0053] Step S2.3: The content encoder contains three convolutional layers and four residual blocks for downsampling and feature extraction. First, a 7×7 convolution is performed to change the number of output channels from 3 channels to 64 channels without changing the feature map size; then two 4×4 convolutions are performed for upsampling, the size is reduced to 1 / 2 and 1 / 4 of the original, and the channels become 128 and 256; finally, four residual blocks composed of 3×3 convolutions are passed, without changing the feature map size and the number of channels, and the final output is used as the content encoding;

[0054] Step S2.4: To fit the brightness code to the Gaussian distribution, we designed a distribution consistency constraint based on the relative entropy (KL divergence) that measures the difference between the two distributions. The smaller the relative entropy, the closer the two distributions are. Calculate the relative entropy between the brightness code of the Gaussian distribution style generated by the brightness encoder and the Gaussian distribution with the mean of the corresponding level and unit variance.

[0055] The specific implementation method of step S2.1 is as follows:

[0056] Step S2.1.1: If the brightness provided is a brightness reference image, the brightness encoder is input for feature extraction, and the obtained feature vector is fitted into a Gaussian distribution to obtain brightness coding;

[0057] Step S2.1.2: If the brightness provided is a brightness level, a Gaussian distribution (unit variance) is fitted with this value as the mean, and after dimension expansion and dimension replication operations, it becomes a code with the same size as the brightness code generated by the brightness encoder.

[0058] The specific implementation method of step S3 is as follows:

[0059] Step S3.1: Corresponding to the encoder, firstly, four residual blocks are used to process the content encoding, and the feature map size and number of channels remain unchanged (1 / 4 of the input width and height, and 256 channels); then, two upsampling layers are used to increase the size by 2 and 4 times, and the number of channels becomes 128 and 64, that is, the width and height are the same as the input, and the number of channels is 64; finally, a 7×7 convolution is performed to generate a reconstructed 3-channel image with the same size and number of channels as the input image;

[0060] Step S3.2: The residual block uses adaptive instance normalization (AdaIN) to align the mean and variance of the content image features with the mean and variance of the brightness image. The parameters of AdaIN are dynamically generated from the brightness encoding by a multi-layer perceptron (MLP):

[0061]

[0062] Where c is the content code, μ(c) and σ(c) represent the mean and standard deviation of the content image features, respectively, and γ and β correspond to the brightness image, which are the parameters generated by the MLP. Therefore, the content image is first normalized, multiplied by the standard deviation of the brightness image, and then added to its mean. This makes the standard deviation and mean of the original content feature map consistent with the brightness feature map. Figure 1 To achieve brightness conversion;

[0063] Step S3.3: Calculate the self-consistency loss function;

[0064] The specific implementation method of step S3.3 is as follows:

[0065] Step S3.3.1: During training, when the brightness provided in step 2 of claim 1 is the original image, the image obtained in step 3 of claim 1 is sent to a brightness encoder and a content encoder to obtain brightness encoding and content encoding of the reconstructed image;

[0066] Step S3.3.2: Calculate the consistency constraint of the reconstructed image: the reconstructed image in step S3.3.1 should be consistent with the reconstructed image obtained in step 3 of claim 1, and calculate their L1 loss, SSIM loss and perceptual loss respectively;

[0067] Step S3.3.3: Calculate the consistency constraint of the reconstructed content component: Calculate the L1 loss using the content encoding in step S3.3.1 and the content encoding obtained in step S2;

[0068] Step S3.3.4: Calculate the consistency constraint of the reconstructed content component: Calculate the L1 loss using the brightness code in step S3.3.1 and the brightness code obtained in step S2;

[0069] The specific implementation method of step S4 is as follows:

[0070] Step S4.1: If it is in the testing phase, ignore this step; if it is in the training phase, perform the following operations;

[0071] Step S4.2: patchGAN is used to determine whether the generated image is "real" or "fake", combining local image features and global image features; the basic GAN is used as the classifier C to classify the brightness of the image and determine the brightness level (label) of the current generated image; first, four 4×4 convolutions are performed to extract features, and then the discriminator and classifier are respectively output through convolutional layers of different sizes with the number of channels being 1 and the number of brightness level categories (set to 7 in the present invention);

[0072] Step S4.3: Calculate classification and discrimination losses: Send the images obtained in step 3 and step 1 to the discriminator and classifier to calculate the losses, where the discriminator determines whether the images in step 1 and step 3 are real, and the classifier determines whether the images in step 1 and step 3 correspond to their respective labels.

[0073] Through the above steps, the low-light enhanced image can be obtained.

[0074] Finally, we train the network by minimizing the reconstructed image consistency constraint, content component consistency constraint, content component consistency constraint loss, and the corresponding loss of the adversarial generative network, and use PSNR (Peak Signal to Noise Ratio) and SSIM (structural similarity index) to evaluate the network performance.

[0075] Test environment: Python 3.6; Pytorch deep learning framework; Ubuntu 16.04 system; NVIDIA GTX2080ti GPU.

[0076] Test sequence:

[0077] (1) The selected dataset is 100 image sequences randomly selected from the reorganized Afifi M et al. dataset. Each sequence has 5 images. Images with the same brightness form a subset, resulting in a total of 7 datasets with different brightness levels, each with 100 images.

[0078] (2) The selected dataset is to select the middle seven images for each image sequence from the reorganized SICE Part 2 dataset. Images with the same brightness form a subset, resulting in a total of 7 datasets with different brightness levels, each with 209 images.

[0079] Testing method: The image enhancement model is obtained through training, tested using the above dataset, and the evaluation index data is calculated.

[0080] Test index: The present invention uses PSNR, SSIM and other indexes for evaluation. These index data are calculated for different popular algorithms and then the results are compared, proving that the present invention can obtain better results in the field of image enhancement.

[0081] Test results:

[0082] Table 1. Performance comparison of the present invention and other algorithms on the Afifi M et al. dataset (PSNR / SSIM)

[0083]

[0084] Table 2. Performance comparison of the present invention and other algorithms on the SICE dataset (PSNR / SSIM)

[0085]

[0086] It can be seen from the above comparison data that in terms of image enhancement of incorrectly exposed images, the present invention is superior to all other methods. It not only achieves enhancement of underexposed images, but also outperforms other methods under overexposure. The conversion between different exposure levels also shows that the present invention achieves conversion between image brightness.

[0087] Any matters not described in the present invention are applicable to the prior art.

[0088] It should be emphasized that the embodiments described in the present invention are illustrative rather than restrictive. Therefore, the present invention includes but is not limited to the embodiments described in the specific implementation manner. Any other implementation manners derived by those skilled in the art based on the technical solution of the present invention also fall within the scope of protection of the present invention.

Claims

1. A brightness adjustment method for mis-exposed images based on unpaired training, characterized in that The following steps are involved: Step 1: At the image input stage, the image with incorrect exposure is subjected to data augmentation such as random flipping to increase sample diversity, and the image is normalized from the pixel value range [0, 255] to [0, 1]; Step 2: Input the exposure error image and brightness provision obtained in step 1 into the encoding network and decouple them into content coding and brightness coding, and perform feature extraction and conversion respectively. In this stage, the two encoders output the content coding of the exposure error image and the brightness coding corresponding to the brightness provision through operations such as convolution layer, residual block, global average pooling layer, fully connected layer and reparameterization. The brightness provision can be a picture type or a brightness level type. If the brightness provision is a picture type, we input the brightness encoder to output a brightness coding that satisfies the Gaussian distribution style. If the brightness provision is a brightness level type, we convert a digital level into a Gaussian distribution to obtain the brightness coding; Step 3: Recombine the content code and brightness code obtained in step 2, and output an image with the same brightness level as the brightness provided by the brightness level through operations such as upsampling layer and convolution layer; Step 4: Input the image obtained in step 3 into the discriminator and classifier, where the discriminator is used to determine whether the image is real, and the classifier determines whether the brightness level of the current image is the brightness level corresponding to the input brightness level, which is used as a loss term to optimize the network; Step 5: Adjust the reconstructed image obtained in step 3 so that its pixel value ranges from [0, 1] to [0, 255] to obtain an image with adjusted brightness.

2. The brightness adjustment method of the mis-exposure image based on non-paired training according to claim 1, characterized in that: The specific method of decoupling the exposure error image and the brightness provided amount input into the encoding network in step 2 includes the following contents: (1) If it is in the test phase, the brightness provided is the brightness level or the brightness reference image; if it is in the training phase, the brightness provided is the original image or a randomly selected brightness level; (2) The brightness encoder contains four convolutional layers, one global average pooling layer, and two fully connected layers. First, a 7×7 convolution is performed to change the number of output channels from 3 channels to 64 channels without changing the size of the feature map. Then, three 4×4 convolutions are performed to upsample the size to 1 / 2, 1 / 4, and 1 / 8 of the original size, and the channels are changed to 128, 256, and 256 respectively. Global average pooling is then used to convert the feature map into a 256-dimensional feature vector. The 100-dimensional mean and variance corresponding to the brightness code are obtained through two fully connected layers. Finally, the brightness code is fitted to a Gaussian distribution as the output by combining the reparameterization technique. (3) The content encoder contains three convolutional layers and four residual blocks for downsampling and feature extraction. First, a 7×7 convolution is performed to change the number of output channels from 3 channels to 64 channels without changing the size of the feature map. Then, two 4×4 convolutions are performed for upsampling, reducing the size to 1 / 2 and 1 / 4 of the original, and the number of channels to 128 and 256. Finally, four residual blocks composed of 3×3 convolutions are used without changing the size of the feature map and the number of channels. The final output is used as the content code. (4) To fit the brightness code to a Gaussian distribution, we designed a distribution consistency constraint based on the relative entropy KL divergence that measures the difference between the two distributions. The smaller the relative entropy, the closer the two distributions are. We calculated the relative entropy between the brightness code of the Gaussian distribution style generated by the brightness encoder and the Gaussian distribution with the mean of the corresponding level and unit variance, and reduced its value during training.

3. The brightness adjustment method of the mis-exposure image based on non-paired training according to claim 2, characterized in that: The specific details of step (1) include the following: ① If the brightness provided is a brightness reference image, the brightness encoder is input for feature extraction, and the obtained feature vector is fitted into a Gaussian distribution to obtain the brightness encoding; ② If the brightness provided is a brightness level, the value is used as the mean to fit a unit variance Gaussian distribution. After the operations of dimension expansion and dimension replication, it becomes a code with the same size as the brightness code generated by the brightness encoder.

4. The brightness adjustment method of the mis-exposure image based on non-paired training according to claim 1, characterized in that: The specific method of recombining brightness coding and content coding in step 3 includes the following contents: (1) Corresponding to the encoder, the content encoding is first processed through four residual blocks, and the feature map size and number of channels remain unchanged, 1 / 4 of the input width and height, and the number of channels is 256; then through two upsampling layers, the size becomes 2 and 4 times of the original, and the number of channels becomes 128 and 64, that is, the same width and height as the input, and 64 channels; finally, a 7×7 convolution is performed to generate a reconstructed 3-channel image with the same size and number of channels as the input image; (2) The residual block uses adaptive instance normalization AdaIN to align the mean and variance of the content image features with the mean and variance of the brightness image; the parameters of AdaIN are dynamically generated from the brightness encoding by a multi-layer perceptron MLP: Where c is the content code, μ(c) and σ(c) represent the mean and standard deviation of the content image features, respectively, and γ and β correspond to the parameters generated by the MLP for the brightness image; therefore, the content image is first normalized, multiplied by the standard deviation of the brightness image, and then added to its mean; Make the standard deviation and mean of the original content feature map consistent with the brightness feature map to achieve brightness conversion; (3) Calculate the self-consistency loss function.

5. The specific method for recombining brightness coding and content coding according to claim 4, characterized in that: The specific method for calculating the self-consistency loss function in step (3) includes the following: ① During training, when the brightness provided in step 2 of claim 1 is the original image, the image obtained in step 3 of claim 1 is sent to a brightness encoder and a content encoder to obtain brightness encoding and content encoding of the reconstructed image; ② Calculate the consistency constraint of the reconstructed image: the reconstructed image in ① should be consistent with the reconstructed image obtained in step 3 of claim 1, and calculate their L1 loss, SSIM loss and perceptual loss respectively; ③ Calculate the consistency constraint of the reconstructed content component: use the content encoding in ① and the content encoding obtained in step 2 of claim 1 to calculate the L1 loss; ④ Calculate the consistency constraint of the reconstructed brightness component: use the internal brightness coding in ① and the brightness coding obtained in step 2 of claim 1 to calculate the L1 loss.

6. The brightness adjustment method of the mis-exposure image based on non-paired training according to claim 1, characterized in that: The specific method of inputting the obtained image into the discriminator and classifier in step 4 includes the following contents: (1) If it is in the testing phase, ignore this step; if it is in the training phase, perform the following operations; (2) PatchGAN is used to determine whether the generated image is "real" or "fake", combining local image features and global image features; The basic GAN is used as the classifier C to classify the brightness of the image and determine the brightness level label of the current generated image; first, four 4×4 convolutions are performed to extract features, and then the discriminator and the classifier respectively output the number of channels of 1 and the number of brightness level categories through convolution layers of different sizes, and the number of brightness level categories is set to 7 in the present invention; (3) Calculate classification and discrimination losses: The images obtained in step 3 and step 1 are sent to the discriminator and classifier to calculate the loss. The discriminator determines whether the images in step 1 and step 3 are real, and the classifier determines whether the images in step 1 and step 3 correspond to their respective labels.

Images