An exposure level guided low-light low-resolution image quality optimization method

Abstract

Embodiments of the present disclosure disclose a low-light low-resolution image quality optimization method based on exposure level guidance. A specific implementation of the method comprises: obtaining a low-light low-resolution degraded image and an exposure level corresponding to the degraded image, training an exposure estimation model, after the training is completed, giving a new low-light low-resolution degraded image, and predicting the exposure level corresponding to the degraded image; obtaining exposure information of the degraded image, randomly fusing RGB information and exposure level information corresponding to the RGB information of two images with the same scene and different exposure levels; constructing an image inpainting model, training the image inpainting model using the result of data enhancement, after the training of the image inpainting model is completed, inputting a low-light low-resolution image, and obtaining a normal-exposure high-resolution image result. The implementation guides the overall optimization network with more information between the degraded image and the image ground truth, thereby improving the overall image quality improvement effect.

Description

TECHNICAL FIELD

[0001] Embodiments of the present disclosure relate to the field of computer technology, in particular to an exposure level guided low-light low-resolution image quality optimization method. BACKGROUND

[0002] Digital images suffer from a variety of different degradation problems, which can be caused by environmental factors in the scene and limitations of hardware. In real life, the imaging quality of real digital images is usually affected by multiple factors at the same time, for example, imaging in low-light environment is prone to cause problems such as insufficient image exposure, low resolution, blur and noise. However, in the existing research, the model method usually solves the problems of low-light enhancement and image super-resolution separately, and there are few related researches on directly converting the degraded and low-resolution low-light image into a normal exposure high-resolution image. This task is challenging, and it is necessary to adjust the brightness and improve the resolution of the degraded image, while considering other possible degradation factors such as color distortion, blur and noise.

[0003] In recent years, with the popularity and application of intelligent computing vision systems, the research on the problems of low-light enhancement and image super-resolution of image enhancement algorithm has gradually increased. Previous methods independently solve two image processing tasks, i.e. low-light enhancement and image super-resolution. These methods guide the degradation learning process and make independent assumptions about specific problems according to each specific physical model. As a result, any single task-specific method cannot effectively cope with this joint degradation problem. Directly applying existing LLE (image low-light enhancement method) and SR (image super-resolution method) methods to the joint task often collapses, can enhance noise or fail to restore the visibility of low-light images, resulting in various brightness deficiencies and unexplained artifacts. In addition, simply combining two specific tasks cannot solve this complex degradation problem. Performing low-light enhancement first and then performing image super-resolution will cause significant amplification of noise and blur, and with various noise and artifacts. The opposite execution order will also cause brightness to be inappropriate and color saturation to be low. Existing low-light enhancement and image super-resolution methods are designed for single degradation problems and are insufficient to represent the essence of the super-resolution problem in low-light scenes. SUMMARY

[0004] This part of the disclosure is used to introduce the concepts in a brief form, which will be described in detail in the specific implementation part later. This part of the disclosure is not intended to identify the key features or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0005] Some embodiments of the present disclosure propose an exposure level guided low-light low-resolution image quality optimization method to solve one or more of the technical problems mentioned in the background section.

[0006] In a first aspect, some embodiments of the present disclosure provide an exposure level guided low-light low-resolution image quality optimization method, which comprises: obtaining a low-light low-resolution degraded image and its corresponding exposure level, training a classification network, after the training is completed, given a new low-light low-resolution degraded image, predicting the exposure level corresponding to the degraded image; obtaining the exposure information of the degraded image, randomly fusing the RGB information of two images with the same scene and different exposure levels and the corresponding exposure level information, completing data enhancement; constructing an image restoration model, training the image restoration model using the data enhancement result, after the image restoration model is trained, inputting a low-light low-resolution image, and obtaining a normally exposed high-resolution image result.

[0007] Some embodiments of the present disclosure provide an exposure level guided low-light low-resolution image quality optimization method, which uses the automatic exposure adjustment idea of a digital camera, designs a special exposure estimation module, fuses the exposure information of the degraded image into the network, fully excavates the information of the input image, guides the adjustment of the image quality of the entire network module, and solves the cross problem end to end. BRIEF DESCRIPTION OF DRAWINGS

[0008] The above and other features, advantages, and aspects of embodiments of the present disclosure will become more apparent upon consideration of the following detailed description, taken in conjunction with the accompanying drawings. Throughout the drawings, like or similar reference numerals designate identical or similar elements throughout the several views. It should be understood that the drawings are schematic and elements and features are not necessarily to scale.

[0009] Figure 1 is a flowchart of some embodiments of the exposure level guided low-light low-resolution image quality optimization method according to the present disclosure.

[0010] Figure 2 is a schematic diagram of an exposure estimation module of some embodiments of the exposure level guided low-light low-resolution image quality optimization method according to the present disclosure.

[0011] Figure 3 is a schematic diagram of exposure fusion data enhancement of some embodiments of the exposure level guided low-light low-resolution image quality optimization method according to the present disclosure.

[0012] Figure 4 is a schematic diagram of an image restoration subnetwork of some embodiments of the exposure level guided low-light low-resolution image quality optimization method according to the present disclosure. DETAILED DESCRIPTION

[0013] Embodiments of the present disclosure will be described below in greater detail with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is understood that the present disclosure can be embodied in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, these embodiments are provided so that the present disclosure will be thoroughly and completely understood. It should be understood that the drawings of the present disclosure are only for illustrative purposes and should not be used to limit the scope of protection of the present disclosure.

[0014] It should also be noted that, for the convenience of description, only the parts related to the present application are shown in the drawings. The embodiments in the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0015] It should be noted that the terms "first", "second", and the like mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not intended to limit the order or interdependence of the functions performed by these devices, modules or units.

[0016] It should be noted that the adjectives "one", "multiple" mentioned in the present disclosure are illustrative and not restrictive, and those skilled in the art should understand that unless the context clearly indicates otherwise, it should be understood as "one or more".

[0017] The names of the messages or information exchanged between the devices in the embodiments of the present disclosure are only for illustrative purposes, and are not intended to limit the scope of the messages or information.

[0018] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments.

[0019] Figure 1 Flow 100 of some embodiments of the exposure level guided low-light low-resolution image quality optimization method according to the present disclosure is shown. The exposure level guided low-light low-resolution image quality optimization method includes the following steps:

[0020] Step 101, obtain a low-light low-resolution degraded image and its corresponding exposure level, train an exposure estimation model, and after the training is completed, given a new low-light low-resolution degraded image, predict the exposure level corresponding to the degraded image.

[0021] Among them, the exposure estimation model mentioned above inputs the RGB information of the degraded image and the degree of influence of the low-light factor corresponding to each training, there are multiple exposure levels, the exposure level information corresponds to the evaluation of the overall exposure level of the image, which is taken from the ambient brightness information when the image is imaged, and the degree of influence of the low-light factor corresponding to different data sets is quantized according to the information of the data set;

[0022] The exposure estimation model is based on a convolutional neural network and a channel attention mechanism. After obtaining the output label of the module, the exposure estimation model parameters are updated according to the gradient back propagation of the cross-entropy loss function.

[0023] In step 102, the exposure information of the degraded image is obtained, the RGB information of two images with the same scene and different exposure levels and the corresponding exposure level information are randomly fused, and data enhancement is completed.

[0024] The data enhancement is the random fusion of the RGB information of two images with the same scene and different exposure levels and the corresponding exposure level information. The exposure level information is the prediction of the luminance level of the input image by the exposure estimation module. The exposure level information is a matrix with the same size as the input image. Two images with different luminance levels in the same scene are selected, a 0-1 matrix R obeying Gaussian distribution is randomly generated, and then the RGB information of the two images and their corresponding exposure level information are linearly combined according to R.

[0025] In step 103, an image inpainting model is constructed, and the image inpainting model is trained using the data enhancement result. After the training of the image inpainting model is completed, a low-light low-resolution image is input, and a normal-exposure high-resolution image result is obtained.

[0026] The image inpainting model first performs image exposure information evaluation and exposure fusion data enhancement at each training. After the data fusion is completed, the training of the final image inpainting model is performed. After the output result of the inpainting module is obtained, the corresponding image quality optimization loss is calculated, the gradient back propagation is performed according to the loss, and the image quality overall optimization model parameters are updated.

[0027] The calculation method of the image quality optimization loss is as follows:

[0028]

[0029] wherein, is a pixel loss, is a perception loss, is an adversarial loss, is a structured loss. per , λ adv , λ str are hyperparameters for balancing the weights of the loss function.

[0030] Some embodiments of the exposure level guided low-light low-resolution image quality optimization method of the present disclosure mainly include three main modules:

[0031] Exposure estimation module: input a low-light, low-resolution image, output the exposure level information corresponding to the image, used to guide the subsequent module to convert the low-light, low-resolution image with different exposure levels into a normal exposure, high-resolution image with consistent exposure levels, and can correctly weaken noise and restore details.

[0032] Exposure fusion data enhancement: randomly fuse the pixel and the corresponding exposure level information of two images with the same scene and different exposure levels, the purpose is to improve the guidance of exposure information and the robustness of the network.

[0033] Image restoration subnetwork: input the information after the above fusion, output a normal exposure, high-resolution image. The subnetwork introduces a channel attention mechanism, which can adaptively adjust the influence of exposure information.

[0034] Exposure estimation module: it is difficult to obtain exposure level information from an RGB image. In order to solve the problem of lack of exposure information, some embodiments of the present disclosure propose an exposure estimation module for estimating the exposure information of an RGB image. Unlike the strategy commonly used in existing low-light enhancement technology to estimate pixel-level exposure information, the global exposure level of the image is estimated. This process can be represented as:

[0035]

[0036] Where I LQ represents the input low-light, low-resolution image, ε represents the exposure estimation module, represents the estimated global exposure level.

[0037] The classification network is based on ResNet and channel attention mechanism, and uses cross-entropy loss function for training. The exposure estimation module of some embodiments of the exposure level guided low-light low-resolution image quality optimization method of the present disclosure is shown in Figure 2 .

[0038] Exposure fusion data augmentation: In order to better enhance the guidance of the estimated exposure information and prevent the network from directly learning the mapping from low-light, low-resolution images to normal exposure, high-resolution images, which makes the exposure information lose its effect, some embodiments of the present disclosure design a new data augmentation method, named "exposure fusion". This data augmentation method randomly fuses the pixel and its corresponding exposure level information of two images with the same scene and different exposure levels, so as to enable the network to learn the guidance of the exposure level. In addition, this pixel-by-pixel fusion strategy can also be regarded as adding "coupled noise" with the same content but lower exposure level and higher noise intensity (because the image with lower exposure level has noise itself), which helps to improve the robustness of the model and better handle the coupling problem of exposure calibration, noise removal and detail restoration. The exposure fusion data augmentation diagram of some embodiments of the exposure level guided low-light low-resolution image quality optimization method of the present disclosure is shown in FIG. 8. Figure 3

[0039] Specifically, two low-light, low-resolution images I LQ，1 and I LQ，2 with the same scene and different exposure levels are randomly selected, and the exposure levels of the two images are estimated using the exposure estimation module, respectively and The exposure level maps M e，1 and M e，2 with the size of HxWx1 are copied and expanded, where H and W represent the height and width of the input image, respectively. After that, a 0-1 matrix R obeying Gaussian distribution is randomly generated, and then the two images and their corresponding exposure level maps will be linearly weighted and combined according to R, which can be represented as:

[0040] I LQ = R·I LQ，1 +(1-R)·I LQ，2 .

[0041] M e = R·M e，1 +(1-R)·M e，2 .

[0042] Finally, the fused low-light, low-resolution image I LQ and the fused exposure level map M e are spliced at the channel level, which can be represented as

[0043]

[0044] where represents the pixel-level splicing operation, and I is the input of the subsequent image inpainting subnetwork.

[0045] ​Image inpainting subnetwork: In the process of low-light enhancement and super-resolution at the same time, the exposure information generated by the above modules and data enhancement process can help the subsequent process of converting from low-light, low-resolution images to normally exposed, high-resolution images. Here, some embodiments of the present disclosure propose an image inpainting subnetwork to complete the above process with the guidance of exposure level information. The image inpainting subnetwork of some embodiments of the exposure level guided low-light low-resolution image quality optimization method of the present disclosure is shown in FIG. 8. Figure 4

[0046] The image inpainting subnetwork is based on the excellent image super-resolution technology Real-ESRGAN, which contains a generator and a discriminator, and inherits the structure of the traditional generative adversarial network.

[0047] Generator: input low-light, low-resolution images, output normally exposed, high-resolution images. The original Real-ESRGAN uses several residual dense blocks (RRDB) to constitute the generator structure. In order to make full use of the low-light, low-resolution image I LQ after fusion and the exposure level map M e , the channel attention mechanism must be introduced into each residual structure in the generator, so that the image inpainting subnetwork can adaptively adjust the influence degree of exposure information.

[0048] Discriminator: input normally exposed, high-resolution images, and discriminate whether the image is real or synthetic. Here, the discriminator used in Real-ESRGAN is inherited.

[0049] Loss function: Real-ESRGAN is used, and the following loss function is used:

[0050] (1) Pixel loss: here, the L1 distance is used to constrain the distance between the generated normally exposed, high-resolution image I and the real normally exposed, high-resolution image I HQ :

[0051]

[0052] wherein I HQ represents the normally exposed, high-resolution image generated. represents the real normally exposed, high-resolution image. represents the 1-norm of .

[0053] (2) Perceptual loss: perceptual loss is used to constrain the visual effect of the image, and is represented as: ​

[0054]

[0055] where φ denotes the pre-trained VGG network. l denotes the number of convolutional neural network layers. φ (l) denotes the feature at the l-th layer. w l is the weight of the l-th layer. H l , W l , C l are the height, width and channel number of φ (l) , respectively. i, j, k are three pointer vectors for traversal.

[0056] (3) Adversarial loss: inherited from the original generative adversarial network, the adversarial loss is used to encourage the generator to generate a sufficiently realistic so as to deceive the discriminator D. The adversarial loss of the generator can be expressed as:

[0057]

[0058] In addition, the discriminator D is used to distinguish the generated normal exposure, high-resolution image and the real normal exposure, high-resolution image I HQ . Its loss function can be expressed as:

[0059]

[0060] In addition, the low-light, low-resolution image I LQ is usually affected by detail loss, including high-frequency information such as structural information and edge information. In order to solve this problem, structural similarity (SSIM) is also introduced as a structural loss to constrain the structure. The structural loss can be expressed as:

[0061]

[0062] where SSIM() denotes the structural similarity loss function.

[0063] The total loss function of the generator is:

[0064]

[0065] where L is the pixel loss, is the perceptual loss, is the adversarial loss, is the structural loss. λ per , λ adv , λ str are hyperparameters for balancing the weights of the loss functions.

[0066] The above description is merely exemplary of some embodiments of the present disclosure and of the principles thereof. It is to be understood that the present disclosure is not limited to the above-described technical solutions, and the scope of the application involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combinations of the technical features described above. Meanwhile, the present disclosure should also cover other technical solutions formed by any combinations of the technical features described above or equivalent features thereof without departing from the above-described inventive concept. For example, the technical solutions formed by the mutual replacement of the above-described features and the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) form technical solutions.

Claims

1. An exposure level guided low-light low-resolution image quality optimization method, comprising: obtaining a low-light low-resolution degraded image and an exposure level corresponding to the degraded image, training an exposure estimation model, and after the training is completed, given a new low-light low-resolution degraded image, predicting the exposure level corresponding to the degraded image; wherein two low-light, low-resolution images with the same scene but different exposure levels are randomly selected and and their exposure levels are estimated using an exposure estimation module and and they are copied and expanded into exposure level maps and with the size of wherein H and W represent the height and width of the input image respectively, a 0-1 matrix following a Gaussian distribution is randomly generated The two images and their corresponding exposure level maps are linearly combined according to ​ ， ， low light, low resolution image after fusion exposure level map after fusion perform channel-wise stitching: ， wherein, represents a pixel-level stitching operation, as input to a subsequent image inpainting subnetwork. obtaining exposure information of the degraded image, randomly fusing RGB information and exposure level information corresponding to the RGB information of two images with the same scene and different exposure levels, and completing data enhancement; constructing an image inpainting model, training the image inpainting model using the data enhancement result, inputting a low-light low-resolution image into the image inpainting model after the training of the image inpainting model is completed, and obtaining a normal exposure high-resolution image result.

2. The method of claim 1, wherein, The exposure estimation model inputs RGB information of a degraded image and a degree of influence of a low-light factor corresponding to the RGB information at each training, there are multiple exposure levels, the exposure level information corresponds to an evaluation of an overall exposure level of an image, is taken from environmental brightness information when the image is imaged, corresponds to degrees of influence of low-light factors of different other data sets, and is quantized according to information of the data sets; The exposure estimation model is based on a convolutional neural network and a channel attention mechanism, gradient backpropagation is performed according to a cross-entropy loss function after an output label of the obtaining module is obtained, and exposure estimation model parameters are updated.

3. The method of claim 1, wherein, The data enhancement includes: randomly fusing RGB information and exposure level information corresponding to the RGB information of two images with the same scene and different exposure levels, wherein the exposure level information is a prediction of a brightness level of an input image by an exposure estimation module, and the exposure level information is a matrix with the same size as the input image; two images of the same scene and different brightness levels are selected, a 0-1 matrix R obeying a Gaussian distribution is randomly generated, and then RGB information of the two images and exposure level information corresponding to the two images are linearly combined according to R.

4. The method of claim 1, wherein, The image inpainting model first performs image exposure information evaluation and exposure fusion data enhancement at each training; after the data fusion is completed, training of the final image inpainting model is performed; after an output result of the inpainting module is obtained, a corresponding image quality optimization loss is calculated, gradient backpropagation is performed according to the loss, and image quality overall optimization model parameters are updated.

5. The method of claim 4, wherein, The calculation method of the image quality optimization loss is: ， wherein, is a pixel loss, is a perceptual loss, is an adversarial loss, is a structured loss, , , are hyperparameters balancing the weights of the individual loss terms.

Images