Image enhancement method, device and storage medium

Abstract

The present invention provides an image enhancement method and device. The image enhancement method comprises: obtaining an original image; performing grayscale transformation on each pixel value of the original image to obtain each target pixel value of the original image; and enhancing each target pixel value of the original image using a pre-trained first image generator to obtain an enhanced image, so that the content, color and texture loss of the enhanced image is less than a predetermined threshold.

Description

Technical Field

[0001] The present invention relates to the field of deep learning, and in particular to an image enhancement method, device and storage medium based on deep learning. Background Art

[0002] Image enhancement methods mainly include traditional image enhancement methods and image enhancement methods based on deep learning. In traditional image enhancement based on histogram equalization, the contrast of noise can be increased, the contrast of useful signals can be reduced, and the grayscale level can be reduced, so the image transition is unnatural. It is only effective for grayscale images, and it will cause distortion for color images. Image enhancement methods based on deep learning include image enhancement based on the LightNet network. However, this method will cause block effects in images for low-light images with large noise. In addition, the network requires image pairs with strict corresponding high brightness and low brightness during training, and this image pair is difficult to obtain and usually needs to be generated manually. The present invention designs an image enhancement method that can combine multiple methods to enhance the image quality of images collected by end-side devices, which helps users to take more beautiful images more efficiently. Summary of the invention

[0003] According to a first aspect of the present invention, an image enhancement method is provided. The method comprises: obtaining an original image; performing grayscale transformation on each pixel value of the original image to obtain each target pixel value of the original image; and enhancing each target pixel value of the original image using a first image generator to obtain an enhanced image, and enhancing each target pixel value of the original image using a pre-trained first image generator to obtain an enhanced image, so that the content, color and texture loss of the enhanced image is less than a predetermined threshold.

[0004] In some embodiments, the grayscale transformation includes: dividing each pixel value of the original image into grayscales based on a predetermined grayscale value, expanding pixel values ​​below the predetermined grayscale value, and compressing pixel values ​​above the predetermined grayscale value.

[0005] In some embodiments, the first image generator is trained based on the following steps: learning a mapping from a set of real images of a first quality to a set of real images of a second quality, and learning a mapping from a set of real images of the second quality to a set of real images of the first quality, the second quality being higher than the first quality.

[0006] In some embodiments, the first image generator is trained based on the following steps: inputting a real image of a first quality in a real image set of a first quality into the first image generator to obtain a first generated image of a second quality, and determining a first global change loss, a first color loss, and a first texture loss between the real image of the second quality in the real image set of the second quality and the first generated image of the second quality, wherein the second quality is higher than the first quality; inputting the first generated image of the second quality into the second image generator to obtain a second generated image of the first quality, and determining a first perceptual loss between the real image of the first quality and the second generated image of the first quality; inputting the real image of the second quality into the second image generator to obtain the first generated image of the first quality, and determining a first perceptual loss between the real image of the first quality and the second generated image of the first quality. A second global change loss, a second color loss, and a second texture loss between a real image of one quality and a first generated image of the first quality; inputting the first generated image of the first quality into the first image generator to obtain a second generated image of the second quality, and determining the second perceptual loss between the real image of the second quality and the second generated image of the second quality; determining a total loss function based on the first global change loss, the first color loss, the first texture loss, the first perceptual loss, the second global change loss, the second color loss, the second texture loss, and the second perceptual loss; training the first image generator using the real images of the first quality in the real image set of the first quality and the real images of the second quality in the real image set of the second quality, so that the total loss function converges.

[0007] In some embodiments, determining a first global change loss between a real image of second quality and a first generated image of second quality includes: determining the first global change loss between the real image of second quality and the first generated image of second quality by a pretrained first convolutional neural network, and wherein determining a first perceptual loss between the real image of first quality and the second generated image of first quality includes: determining the first perceptual loss between the real image of first quality and the second generated image of first quality by a pretrained first convolutional neural network.

[0008] In some embodiments, determining a second global change loss between a real image of first quality and a first generated image of first quality includes: determining a second global change loss between the real image of first quality and the first generated image of first quality through a pretrained first convolutional neural network, and determining a second perceptual loss between a real image of second quality and a second generated image of second quality includes: determining a second perceptual loss between the real image of second quality and the second generated image of second quality through a pretrained first convolutional neural network.

[0009] In some embodiments, determining the first color loss between the real image of the second quality and the first generated image of the second quality includes: performing Gaussian noise processing on the real image of the second quality and the first generated image of the second quality to obtain the real image of the second quality after Gaussian noise and the first generated image of the second quality after Gaussian noise, and inputting the real image of the second quality after Gaussian noise and the first generated image of the second quality after Gaussian noise into a first discriminator to obtain the first color loss.

[0010] In some embodiments, determining the second color loss between a real image of first quality and a first generated image of first quality includes: performing Gaussian noise processing on the real image of first quality and the first generated image of first quality to obtain the real image of first quality after Gaussian noise and the first generated image of first quality after Gaussian noise, and inputting the real image of first quality after Gaussian noise and the first generated image of first quality after Gaussian noise into a first discriminator to obtain the second color loss.

[0011] In some embodiments, determining a first texture loss between a real image of second quality and a first generated image of second quality includes: graying the real image of second quality and the first generated image of second quality to obtain a grayed real image of second quality and a grayed first generated image of second quality, and inputting the grayed real image of second quality and the grayed first generated image of second quality into a second discriminator to obtain a first texture loss.

[0012] In some embodiments, determining the second texture loss between a real image of the first quality and a first generated image of the first quality includes: graying the real image of the first quality and the first generated image of the first quality to obtain the grayed real image of the first quality and the grayed first generated image of the first quality, and inputting the grayed real image of the first quality and the grayed first generated image of the first quality into a second discriminator to obtain the second texture loss.

[0013] In some embodiments, the method further includes: scoring the enhanced image based on a pre-trained second convolutional neural network, and displaying the scoring result.

[0014] In some embodiments, the first discriminator and the second discriminator are configured to determine whether the input image is a real image or an image generated by the first image generator or the second image generator.

[0015] According to a second aspect of the present invention, an image enhancement device is provided. The device comprises: an acquisition module configured to acquire an original image; a grayscale transformation module configured to perform grayscale transformation on each pixel value of the original image to obtain each target pixel value of the original image; and an image enhancement module configured to enhance each target pixel value of the original image using a pre-trained first image generator to obtain an enhanced image, so that the content, color and texture loss of the enhanced image is less than a predetermined threshold.

[0016] According to some embodiments of the present invention, there is provided a computer device, comprising: a processor; and a memory having instructions stored thereon, which, when executed on the processor, cause the processor to execute any one of the above methods.

[0017] According to some embodiments of the present invention, there is provided a computer-readable storage medium having instructions stored thereon, which, when executed on a processor, cause the processor to perform any one of the above methods.

[0018] The method and device provided by the present invention combine the traditional image enhancement method and the image enhancement method based on deep learning, and do not require image pairs with strict correspondence between high-quality and low-quality images to train the deep learning model, so training data is easy to obtain. The joint image enhancement method and device proposed in the present invention can be mounted on various terminals (for example, smart phones, tablet computers, etc.) and various cameras (for example, SLR cameras and other devices), and are widely used. The joint image enhancement method and device greatly reduces the difficulty of taking pictures by providing an image enhancement function, and more conveniently takes better pictures, greatly improves the user experience, and improves the efficiency of users taking pictures.

[0019] These and other aspects of the invention will be apparent from and elucidated with reference to the embodiments described hereinafter. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Further details, features and advantages of the invention are disclosed in the following description of exemplary embodiments in conjunction with the accompanying drawings, in which:

[0021] Figure 1 An example interface of a terminal provided by the present invention is shown;

[0022] Figure 2 An exemplary flow chart of a method according to an embodiment of the present invention is shown;

[0023] Figure 3 A schematic diagram of a deep learning network model for training a first image generator according to an embodiment of the present invention is shown;

[0024] Figure 4 A network schematic diagram of an image generator according to an embodiment of the present invention is shown;

[0025] Figure 5 A network schematic diagram of a discriminator according to an embodiment of the present invention is shown;

[0026] FIG6 shows a schematic diagram of the effect of image enhancement;

[0027] Figure 7 shows the structure of a convolutional neural network used to score images;

[0028] Figure 8 A schematic diagram of an image enhancement device according to an embodiment of the present invention is shown; and

[0029] Fig. 9 An example computer processing system is shown. DETAILED DESCRIPTION

[0030] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0031] Machine Learning (ML) is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies.

[0032] In recent years, machine learning (especially deep learning in machine learning) has performed very well in various scenarios of computer vision. Image enhancement methods based on deep learning include image enhancement based on the RetinexNet network. The RetinexNet network takes low-light images as input and outputs enhanced images. The RetinexNet model believes that low-light images are the result of the interaction between normal images and an illumination map, so only the illumination needs to be removed to obtain a normal image for image enhancement. This method learns the parameters of the RetinexNet network model (i.e., illumination map) and uses low-light images to perform illumination inverse operations to obtain normal images, i.e., enhanced images. However, this method will cause image block effects for low-light images with large noise. In addition, the network requires high-brightness and low-brightness image pairs during training. However, these high-brightness and low-brightness image pairs are difficult to obtain and usually need to be generated manually.

[0033] Before describing the embodiments of the present invention in detail, some related concepts are first explained:

[0034] 1. Retinex image enhancement: Any image can be decomposed into an illumination image and a reflection image. The reflection image is determined by the nature of the object itself and is the unchanging part. The illumination image is greatly affected by the outside world. The purpose of enhancing the image can be achieved by removing the illumination influence or correcting the illumination image. Retinex includes a decomposition network (Decom) for decomposition and an enhancement network (Relight) for lighting adjustment.

[0035] 2. Generative Adversarial Networks: It is a deep learning model with a generator G and a discriminator D. There are two data domains, X and Y. G is responsible for imitating the data in domain X into real data and hiding them in the real data, while D is responsible for separating fake data from real data. After the game between the two, until D can no longer distinguish whether the data is real or generated, the confrontation process reaches a dynamic balance.

[0036] 3. VGG-19: VGG is a deep convolutional neural network developed by researchers from the Visual Geometry Group at the University of Oxford and Google DeepMind. VGG-19 contains a 19-layer deep convolutional neural network.

[0037] Figure 1 An example interface 100 of a terminal provided by the present invention is schematically shown. Figure 1As shown, the image enhancement method proposed in the present invention can be embedded in various photographic devices such as mobile phones, SLR cameras, etc. Users can enhance the image by selecting the image enhancement function. First, use the camera to take a photo (i.e., take a stream) to obtain the front-end image data, and then enhance the image based on traditional logarithmic image enhancement, and then enhance the image based on deep learning image enhancement. Because the joint image enhancement method proposed in the present invention requires less computing resources, it can be quickly completed on ordinary end-side devices. In addition, by training a neural network model to score the finished image or the image in the shooting preview, the user can be guided to adjust the framing. For example, Figure 1 After shooting, the user can be notified "Tip: Score 80 points, it is recommended to increase the lighting", and the user can make corresponding adjustments to take better photos.

[0038] Figure 2 An exemplary flow chart 200 of a method according to an embodiment of the present invention is shown. The method aims to combine traditional logarithmic image enhancement with an image enhancement algorithm based on deep learning. First, in step 201, an original image is obtained. The original image can be obtained through a mobile phone, a camera or other image acquisition device. In step 202, each pixel value of the original image is gray-scale transformed to obtain each target pixel value of the original image. Thus, each pixel value of the original image is divided according to grayscale based on a predetermined grayscale value, and pixel values ​​below the predetermined grayscale value are expanded, and pixel values ​​above the predetermined grayscale value are compressed. Typically, grayscale transformation includes logarithmic transformation, image histogram equalization, etc. In one embodiment, the grayscale transformation uses logarithmic image enhancement. Logarithmic image enhancement can enhance the image by dividing each pixel value of the original image according to grayscale based on a predetermined grayscale value, expanding the part below the predetermined grayscale value, and compressing the part above the predetermined grayscale value. The formula is: S = c log2 (r + 1), where c is a constant (this method uses c = 255 / (log2 256), r is the source pixel value, and S is the target pixel value. In this way, the brightness of the entire picture can be increased. The logarithmic transformation can expand the low grayscale value part of the image to show more details of the low grayscale part, compress its high grayscale value part, and reduce the details of the high grayscale value part, thereby achieving the purpose of emphasizing the low grayscale part of the image. In step 203, each target pixel value of the original image is enhanced using a pre-trained, deep learning-based first image generator to obtain an enhanced image, so that the content, color and texture loss of the enhanced image is less than a predetermined threshold. The first image generator is based on Figure 3 The deep learning model shown in is trained.

[0039] Figure 3A schematic diagram of a deep learning network model 300 for training a first image generator according to an embodiment of the present invention is shown. The deep learning network model designed by the present invention utilizes a generative adversarial network and a convolutional neural network to achieve end-to-end image enhancement. The machine learning network model is different from other networks and is a weakly supervised image enhancement model that can complete training without strictly corresponding image pairs. At the same time, the network also defines a new loss function to ensure that the image has good performance in texture, color and smoothness.

[0040] Typically, training of network models requires strictly corresponding image pairs, such as image pairs of the same scene but under different illumination. Figure 3 The deep learning network model 300 in is a weakly supervised deep learning network model that does not require strictly corresponding image pairs. Specifically, the required data set includes a low-quality image set and a high-quality image set, but the two data sets do not need to be strictly corresponding. Here, a first-quality real image set X and a second-quality real image set Y are obtained. The first-quality real image set X includes the first-quality real images x k (k=1, 2, 3, ..., N), and the real image set Y of the second quality includes the real images y of the second quality k (k=1,2,3,…,N). Specifically, the real image of the first quality may represent the real image whose quality is lower than the predetermined condition, and the real image of the second quality may represent the real image whose quality is higher than the predetermined condition. The real image set X of the first quality may include low-quality images collected from mobile phones and low-end cameras. The real image set Y of the second quality may include high-quality images crawled from the Internet. For example, 10,000 real images of the first quality and 10,000 real images of the second quality may be obtained for this training. The two generators, the first image generator G1 301 and the second image generator G2 302, are cyclically trained, G1 mainly realizes the enhancement of the first quality image, and G2 mainly realizes the restoration of the second quality image to the first quality image. The first image generator is pre-trained to enhance each target pixel value of the original image to obtain an enhanced image, so that the content, color and texture loss of the enhanced image is less than a predetermined threshold. The first image generator is trained based on the following steps, mainly including the first and second stages. The first stage learns the mapping from the real image set of the first quality to the real image set of the second quality, and the second stage learns the mapping from the real image set of the second quality to the real image set of the first quality. Herein, the second mass is higher than the first mass.

[0041] In the first stage, the first quality real image x in the first quality real image set X is k (k=1,2,3,…,N) inputs the first image generator to obtain the second quality generated image yk ', and determine the second quality real image y in the second quality real image set Y k (k=1,2,3,…,N) and the second quality generated image y k 'The global change loss L between tv1 , color loss L c1 and texture loss L t1 ; Generate the second quality image y k 'Input the second image generator to get the first quality generated image x k ", and determine the first quality real image x k Generate image x with first quality k The perceptual loss L between p1 In one embodiment, the real image y of the second quality is determined k Generate image y with the second quality k 'The global change loss L between tv1 The method comprises: determining a real image y of the second quality through a pre-trained convolutional neural network (as understood by those skilled in the art, it can be any convolutional neural network used, such as DenseNet, ResNet, VGG-16, VGG-19, etc.) network. k Generate image y with the second quality k 'The global change loss L between tv1 In one embodiment, a first quality real image x is determined k Generate image x with first quality k The perceptual loss L between p1 The method comprises: determining a first quality real image x by a pre-trained convolutional neural network. k Generate image x with first quality k The perceptual loss L between p1 In another embodiment, the real image y of the second quality is determined k Generate image y with the second quality k 'Color loss between c1 The method comprises: performing Gaussian noise processing on the real image y of the second quality and the generated image y' of the second quality to obtain the real image y of the second quality after Gaussian noise processing. k加噪 And the generated image y of the second quality after Gaussian noise k ' 加噪 , y k加噪 and k ' 加噪 Input to the first discriminator 305 to obtain the color loss L c1 Gaussian noise processing (also called Gaussian blur) is done by Figure 3The Gaussian noise processing module 303 in the image processing module 304 mainly processes the image using the Gaussian blur method. Noise addition can remove the influence of other factors (e.g., image texture and content) on color loss. Because the second quality image source y used is different from the first quality image content, Gaussian noise addition can allow the first discriminator 305 to focus on analyzing the color distribution (including brightness, contrast, etc.) as much as possible. In another embodiment, the second quality real image y is determined. k Generate image y with the second quality k 'The texture loss between t1 Includes: the real image y of the second quality k and the second quality generated image y k ' Perform grayscale processing to obtain the second quality real image y after grayscale processing k灰度 and the grayscaled second quality generated image y k ' 灰度 , y k灰度 and k ' 灰度 Input the second discriminator D t In 306, the texture loss L is obtained t1 Grayscale (Gray) Figure 3 The grayscale processing module 304 in the embodiment of the present invention is executed to convert the real image y of the second quality into k Generate image y with the second quality k 'The grayscale image is taken to remove the influence of factors such as color.

[0042] In the second stage, the real image y of the second quality is k Input the second image generator G2 302 to obtain the first quality generated image x k ', and determine the first quality real image x k Generate image x with first quality k 'The global change loss L between tv2 , color loss L c2 and texture loss L t2 ; Generate the first quality image x k 'Input the first image generator G1 301 to obtain the second quality generated image y k ", and determine the real image y of the second quality k Generate image y with the second quality k The perceptual loss L between p2 In one embodiment, the real image y of the second quality is determined k Generate image y with the second quality k The perceptual loss L between p2Comprising: determining a second quality real image y by a pre-trained convolutional neural network k Generate image y with the second quality k The perceptual loss L between p2 In one embodiment, a first quality real image x is determined k Generate image x with first quality k 'The global change loss L between tv2 The method comprises: determining a first quality real image x by a pre-trained convolutional neural network. k Generate image x with first quality k 'The global change loss L between tv2 In another embodiment, the real image x of the first quality is determined k Generate image x with first quality k 'Color loss L c2 Includes: For the first quality real image x k Generate image x with first quality k 'Perform Gaussian noise processing to obtain the first quality real image x after Gaussian noise k加噪 And the first quality image x after Gaussian noise k ' 加噪 , x k加噪 and x k ' 加噪 Input the first discriminator D c The color loss L is obtained from c2 Gaussian noise processing (also called Gaussian blur) is done by Figure 3 The Gaussian noise processing module 303 in the image processing module 304 mainly processes the image using the Gaussian blur method. Noise addition can remove the influence of other factors (such as the texture and content of the image) on the color loss. In another embodiment, the real image x of the first quality is determined. k Generate image x with first quality k 'Texture loss L t2 Includes: For the first quality real image x k and the first quality generated image x k 'Perform grayscale processing to obtain the first quality real image x after grayscale processing k灰度 and the grayscaled first quality image x k ' 灰度 , x k灰度 and x k ' 灰度 Input the second discriminator D t In the above example, we get the texture loss L t2 Grayscale (Gray) Figure 3 The grayscale processing module 304 in is executed.

[0043] The perceptual loss function is defined to measure the low quality image x k and low quality image x k ', high quality image k and k 'Content consistency, using the feature layer of a pre-trained convolutional neural network (e.g., VGG-19) to calculate its perceptual loss L p1 and L p2 As follows, this avoids perceptual bias.

[0044]

[0045] Where C, H, W represent the channel, height and width of the VGG-19 network respectively, j represents the number of layers of the VGG-19 network, and the loss measures the amount of content loss by calculating the average L2 norm of the corresponding points of the generated image and the real image.

[0046] The definition of color loss is to achieve color discrimination. Based on the discriminator D c Output color loss L c , the input image is Gaussian blurred and then the color loss is calculated. The Gaussian function in two-dimensional space is as follows:

[0047]

[0048] where u and v represent the blur radius in two dimensions. In this paper, σ 2 is 3, then the actual discriminator D c The individual pixel values ​​of the input are:

[0049] y b (i,j)=∑ u,v y(i+u,j+v)*G u,v ,

[0050] Then L c The expression of loss is the classic GAN loss function as follows:

[0051]

[0052] Among them, G1 represents the first image generator, D c represents the first discriminator (color discriminator), G2 represents the second image generator, and m is the number of images.

[0053] The definition of texture loss is to achieve texture discrimination. Based on the discriminator D t Calculate the texture loss L t , the texture loss can be defined as follows:

[0054]

[0055] Among them, G1 represents the first image generator, D t represents the second discriminator (texture discriminator), G2 represents the second image generator, and m is the number of images.

[0056] Based on the above content, in order to make the generated enhanced image as smooth as possible, L tv Global change loss. Global change loss is often used as a regular term in the loss function, which can smooth images, remove ghosts, and eliminate noise. Global change is the sum of the degree of change of the signal, that is, the sum of the absolute values ​​of the differences between adjacent pixels. Generally, since the adjacent signals of the noise signal (image) are not smooth and the random changes are large, the global change is large, while the global change of the smooth signal is small. By minimizing the global change, noise can be removed and the signal can be smoothed.

[0057]

[0058] Where C, H, and W represent the channel, height, and width of the VGG-19 network, respectively. The square of the difference between two adjacent eigenvalues ​​in the x and y directions is compared, and i and j respectively identify the two-dimensional index of the feature map. From this, we can conclude that L tv1 and L tv2 .

[0059] Finally, based on the global change loss L tv1 , color loss L c1 , texture loss L t1 , Perceptual loss L p1 , global change loss L tv2 , color loss L c2 , texture loss L t2 and the perceptual loss L p2 To determine the total loss function. Using the first quality real image x in the first quality real image set X k The first image generator is trained with a real image yk of second quality in a set Y of real images of second quality such that the total loss function converges for each xk and yk.

[0060] To sum up, the total loss is:

[0061] L 总 =α(L p1 +L p2 )+β(L t1 +L t2 +L c1 +L c2 )+γ(L tv1 +L tv2 ),

[0062] Wherein, α, β, and γ are respectively preset constants. In one embodiment, α, β, and γ are 1, 0.002, and 100, respectively. After reaching a balanced state (i.e., the total loss function converges) through training, a first image generator G1 is obtained, thereby completing image enhancement.

[0063] Figure 4 A network diagram of an image generator (including a first image generator and a second image generator) 400 according to an embodiment of the present invention is shown. In the generator 400, the input image 401 is first input through the convolution layer 402 and the activation layer 403, and then input into four identical blocks 404 to 407 respectively. It is worth noting that the first convolution after the input image, that is, the convolution layer 402 is a full-size convolution, that is, the size of the convolution kernel is equal to the size of the input image, so as to learn global features. Global features include high-level information, such as scene classification, subject type, global brightness, etc. Figure 4 As shown in , the structure of each block 404 to 407 is the same. Taking block 404 as an example, it includes convolution layer 4041, normalization layer 4042, activation layer 4043, convolution layer 4044 and activation layer 4045 in sequence, and based on the idea of ​​ResNet, the input of block 404 is accumulated to the output end of layer 4045. Block 405 includes convolution layer 4051, normalization layer 4052, activation layer 4053, convolution layer 4054 and activation layer 4055 in sequence; block 406 includes convolution layer 4061, normalization layer 4062, activation layer 4063, convolution layer 4064 and activation layer 4065 in sequence; block 407 includes convolution layer 4071, normalization layer 4072, activation layer 4073, convolution layer 4074 and activation layer 4075 in sequence. After being processed by blocks 405 to 407 in a similar manner, the output of block 407 passes through convolution layer 408 and normalization layer 409 in sequence, and based on the idea of ​​ResNet, the output of layer 409 is added to the input of block 404. After the addition, the output is processed by convolution layer 410, normalization layer 411, convolution layer 412, and normalization layer 413 in sequence, and the generated image 414 is output. As those skilled in the art should understand, the number of blocks with the same structure as blocks 404 to 407 is not limited to 4, and may also have other suitable numbers of such structures.

[0064] Figure 5 FIG. 5 shows a network diagram of a discriminator (including a first discriminator and a second discriminator) 500 according to an embodiment of the present invention. c and the second discriminator D tIt is configured to determine whether the input image is a real image or an image generated by the first image generator or the second image generator. After the input images 501 and 502 are input into the convolution layer 503 and the activation layer 504 in turn, they are input into four blocks 505 to 508 with the same structure. Block 505 includes a convolution layer 5051, a normalization layer 5052 and an activation layer 5053. Block 506 includes a convolution layer 5061, a normalization layer 5062 and an activation layer 5063. Block 507 includes a convolution layer 5071, a normalization layer 5072 and an activation layer 5073. Block 508 includes a convolution layer 5081, a normalization layer 5082 and an activation layer 5083. The output of block 508 is fully connected twice to obtain a probability score to determine the authenticity of the image.

[0065] FIG. 6 is a schematic diagram showing the effect of image enhancement. Figure 6a , 6c , 6e are the real original images, Figure 6b , 6d 6f and 6f are the enhanced images. It can be seen that the enhancement effect is significant. The indicators for image enhancement mainly include PSNR and SSIM. PSNR is the abbreviation of "Peak Signal to Noise Ratio", which is an objective standard for evaluating images. Its calculation method is shown in the figure below:

[0066]

[0067] MSE is the mean square error between the original image and the processed image. This indicator can be used to evaluate the quality of the processed image. The higher the PSNR, the higher the image quality and the less distortion.

[0068] SSIM (structural similarity index) is an indicator to measure the similarity between two images. Its calculation method is shown in the following figure:

[0069]

[0070] where μ x is the mean value of x, μ y is the mean value of y, is the variance of x, is the variance of y, σ xy is the covariance of x and y. c1=(k1L ) 2, c1 = (k2L) 2is a constant used to maintain stability. L is the dynamic range of pixel values. k1=0.01, k2=0.03. The larger the SSIM value, the higher the structural similarity of the two images. Experiments on the DPED dataset show that this method has good performance for PSNR and SSIM indicators.

[0071] In step 204, the image enhancement method further includes: scoring the enhanced image based on a pre-trained neural network, and displaying the scoring result.

[0072] Figure 7 The structure 700 of a convolutional neural network for scoring images is shown. First, images crawled from the Internet are manually scored to form a training set. After training, an image quality scoring model is obtained. Considering the computing power of the end-side device, the model needs to ensure the effect without sacrificing speed. The model scores the enhanced image and displays the scoring result on the device display in real time, so as to guide the user to adjust the composition and obtain a better quality image. Figure 7 As shown in FIG. 7 , in one embodiment, the input image sequentially passes through convolution layer 701, activation layer 702, convolution layer 703, pooling layer 704, activation layer 705, element sum layer 706, convolution layer 707, pooling layer 708, activation layer 709, element sum layer 710, convolution layer 711, pooling layer 712, activation layer 713, element sum layer 714, and then passes through convolution layer 715, activation layer 716, element sum layer 717, activation layer 718, convolution layer 719, fully connected layer 720, activation layer 721, and fully connected layer 722. Finally, the quality score, i.e., the score of the image, is output.

[0073] In one application scenario, scoring images helps users take images more efficiently. When users take photos through terminals such as mobile phones, tablets, and smart SLR cameras, they can select the preview mode, and the display interface can prompt the score of the image in the viewfinder in real time, as well as the adjustments that should be made. For example, after selecting the image to be taken and putting it in the viewfinder, the display screen can prompt "The current image is 60 points, and the brightness needs to be increased." Therefore, before taking photos, users can use the prompts to make appropriate adjustments to the shooting environment and actions, so as to take better photos more efficiently.

[0074] In another application scenario, scoring an image helps the user to modify the image more efficiently. For example, the user inputs the image that has been taken into the image enhancement system, and the system scores the image that has been taken, and prompts on the display screen that corresponding modifications should be made. For example, after inputting the image that has been taken, the display screen displays "This image is 80 points, and you can choose to increase the contrast." In this way, the user can make more appropriate modifications and enhancements to the image that has been taken through the prompt.

[0075] Figure 8 The schematic diagram of an image enhancement device 800 according to an embodiment of the present invention is schematically shown. The image enhancement device 800 includes: an acquisition module 801, a grayscale transformation module 802 and an image enhancement module 803. The acquisition module 801 is configured to acquire an original image. The grayscale transformation module 802 is configured to perform grayscale transformation on each pixel value of the original image to obtain each target pixel value of the original image. Performing grayscale transformation includes: expanding a first grayscale value portion of each pixel value of the original image; and compressing a second grayscale value portion of each pixel value of the original image, wherein the second grayscale value is greater than the first grayscale value. The image enhancement module 803 is configured to enhance each target pixel value of the original image using a pre-trained first image generator to obtain an enhanced image, so that the content, color and texture loss of the enhanced image is less than a predetermined threshold. In one embodiment, the device further includes: a scoring module 804, which is configured to score the enhanced image based on a pre-trained neural network and display the scoring result.

[0076] Fig. 9 An example system 900 is schematically illustrated that includes an example computing device 910 that represents one or more systems and / or devices that may implement the various techniques described herein.

[0077] The computing device 910 may be, for example, a server of a service provider or any other suitable computing device or computing system, ranging from a full resource device with a large amount of memory and processor resources to a low resource device with limited memory and / or processing resources. Figure 8 The depicted image enhancement apparatus 800 may take the form of a computing device 910 .

[0078] The example computing device 910 as shown includes a processing system 911, one or more computer-readable media 912, and one or more I / O interfaces 913 that are communicatively coupled to each other. Although not shown, the computing device 910 may also include a system bus or other data and command transmission system that couples the various components to each other. The system bus may include any one or combination of different bus structures, such as a memory bus or memory controller, a peripheral bus, a universal serial bus, and / or a processor or local bus utilizing any of a variety of bus architectures.

[0079] Processing system 911 represents the functionality of performing one or more operations using hardware. Thus, processing system 911 is illustrated as including hardware elements 914 that may be configured as processors, functional blocks, and the like. This may include other logic devices implemented in hardware as application specific integrated circuits or formed using one or more semiconductors. Hardware elements 914 are not limited by the materials from which they are formed or the processing mechanisms employed therein. For example, a processor may be composed of (multiple) semiconductors and / or transistors (e.g., electronic integrated circuits (ICs)). In such a context, processor executable instructions may be electronic executable instructions.

[0080] Computer readable media 912 is illustrated as including memory / storage 919. Memory / storage 915 represents memory / storage capacity associated with one or more computer readable media. Memory / storage 915 may include volatile media (such as random access memory (RAM)) and / or non-volatile media (such as read-only memory (ROM), flash memory, optical disks, magnetic disks, etc.). Memory / storage 915 may include fixed media (e.g., RAM, ROM, fixed hard drive, etc.) and removable media (e.g., flash memory, removable hard drive, optical disk, etc.). Computer readable media 912 may be configured in various other ways as further described below.

[0081] One or more input / output interfaces 913 represent functionality that allows a user to enter commands and information into computing device 910 and also allows information to be presented to the user and / or sent to other components or devices using various input / output devices. Examples of input devices include a keyboard, a cursor control device (e.g., a mouse), a microphone (e.g., for voice input), a scanner, touch functionality (e.g., a capacitive or other sensor configured to detect physical touch), a camera (e.g., that can detect motion that does not involve touch as gestures using visible or invisible wavelengths (such as infrared frequencies)), a network card, a receiver, and the like. Examples of output devices include a display device (e.g., a projector), a speaker, a printer, a tactile response device, a network card, a transmitter, and the like.

[0082] The computing device 910 also includes an image enhancement application 916. The image enhancement application 916 may be stored as computer program instructions in the memory / storage device 919. The image enhancement application 916 may be implemented in conjunction with the processing system 911 and the I / O interface. Figure 8 The entire functions of each module of the image enhancement device 800 are described.

[0083] Various techniques may be described herein in the general context of software hardware elements or program modules. Generally, these modules include routines, programs, objects, elements, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The terms "module," "function," and "component" as used herein generally represent software, firmware, hardware, or a combination thereof. The features of the techniques described herein are platform-independent, meaning that these techniques can be implemented on various computing platforms with various processors.

[0084] An implementation of the described modules and techniques may be stored on or transmitted across some form of computer-readable media. Computer-readable media may include a variety of media accessible by computing device 910. By way of example and not limitation, computer-readable media may include “computer-readable storage media” and “computer-readable signal media.”

[0085] In contrast to a simple signal transmission, carrier wave, or signal itself, "computer-readable storage medium" refers to a medium and / or device capable of persistently storing information, and / or a tangible storage device. Therefore, a computer-readable storage medium refers to a non-signal-bearing medium. Computer-readable storage media include hardware such as volatile and non-volatile, removable and non-removable media and / or storage devices implemented in a method or technology suitable for storing information (such as computer-readable instructions, data structures, program modules, logic elements / circuits, or other data). Examples of computer-readable storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical storage device, hard disk, cassette, magnetic tape, magnetic disk storage device or other magnetic storage device, or other storage device, tangible medium, or article of manufacture suitable for storing desired information and accessible by a computer.

[0086] "Computer-readable signal media" refers to signal-bearing media that is configured to send instructions to the hardware of the computing device 910, such as via a network. Signal media typically can embody computer-readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave, data signal, or other transport mechanism. Signal media also include any information transfer media. The term "modulated data signal" refers to a signal that encodes the information in the signal in such a manner to set or change one or more of its characteristics. By way of example and not limitation, communication media include wired media such as a wired network or direct connection and wireless media such as acoustic, RF, infrared and other wireless media.

[0087] As previously described, hardware elements 914 and computer-readable media 912 represent instructions, modules, programmable device logic, and / or fixed device logic implemented in hardware form, which in some embodiments can be used to implement at least some aspects of the technology described herein. Hardware elements can include integrated circuits or systems on a chip, application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), complex programmable logic devices (CPLDs), and other implementations in silicon or components of other hardware devices. In this context, hardware elements can be used as processing devices for executing program tasks defined by instructions, modules, and / or logic embodied by hardware elements, as well as hardware devices for storing instructions for execution, such as the computer-readable storage media described previously.

[0088] The aforementioned combination may also be used to implement the various techniques and modules described herein. Therefore, software, hardware or program modules and other program modules may be implemented as one or more instructions and / or logic embodied on some form of computer-readable storage medium and / or by one or more hardware elements 914. The computing device 910 may be configured to implement specific instructions and / or functions corresponding to the software and / or hardware modules. Therefore, for example, by using a computer-readable storage medium and / or hardware element 914 of a processing system, the module may be implemented as a module executable by the computing device 910 as software, at least in part, in hardware. Instructions and / or functions may be executable / operable by one or more articles (e.g., one or more computing devices 910 and / or processing systems 911) to implement the techniques, modules and examples described herein.

[0089] The techniques described herein can be supported by these various configurations of computing device 910 and are not limited to the specific examples of the techniques described herein. The functionality of computing device 910 can also be implemented in whole or in part on the "cloud" 920 using a distributed system, such as through platform 930 as described below.

[0090] Cloud 920 includes and / or represents a platform 930 for resources 932. Platform 930 abstracts the underlying functionality of the hardware (e.g., servers) and software resources of cloud 920. Resources 932 may include applications and / or data that may be used when executing computer processing on a server remote from computing device 910. Resources 932 may also include services provided over the Internet and / or over a client network such as a cellular or Wi-Fi network.

[0091] The platform 930 may abstract resources and functionality to connect the computing device 910 with other computing devices. The platform 930 may also be used to abstract hierarchies of resources to provide hierarchies of corresponding levels of demand encountered for resources 932 implemented via the platform 930. Thus, in an interconnected device embodiment, the implementation of the functionality described herein may be distributed throughout the system 900. For example, functionality may be implemented in part on the computing device 910 and through the platform 930 that abstracts functionality of the cloud 920.

[0092] Variations to the disclosed embodiments will be understood and effected by those skilled in the art in practicing the claimed subject matter by studying the drawings, the disclosure and the appended claims. In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

Claims

1. An image enhancement method, comprising: Get the original image; Performing grayscale transformation on each pixel value of the original image to obtain each target pixel value of the original image; and Using a pre-trained first image generator to enhance each target pixel value of the original image to obtain an enhanced image, so that the content, color and texture loss of the enhanced image is less than a predetermined threshold; The first image generator is trained based on the following steps: Inputting a real image of a first quality in a real image set of a first quality into the first image generator to obtain a first generated image of a second quality, and determining a first global change loss, a first color loss, and a first texture loss between the real image of the second quality in the real image set of the second quality and the first generated image of the second quality, wherein the second quality is higher than the first quality; Inputting the first generated image of the second quality into a second image generator to obtain a second generated image of the first quality, and determining a first perceptual loss between the real image of the first quality and the second generated image of the first quality; Inputting the real image of the second quality into the second image generator to obtain a first generated image of the first quality, and determining a second global change loss, a second color loss, and a second texture loss between the real image of the first quality and the first generated image of the first quality; Inputting the first generated image of the first quality into the first image generator to obtain a second generated image of the second quality, and determining a second perceptual loss between the real image of the second quality and the second generated image of the second quality; determining a total loss function based on the first global variation loss, the first color loss, the first texture loss, the first perceptual loss, the second global variation loss, the second color loss, the second texture loss, and the second perceptual loss; Training the first image generator using real images of first quality in the real image set of the first quality and real images of second quality in the real image set of the second quality, so that the total loss function converges; and The first quality set of real images and the second quality set of real images are not in one-to-one correspondence, and the first color loss is The second color loss is G1 represents the first image generator, D c represents the first discriminator, G2 represents the second image generator, and m is the number of images.

2. The method according to claim 1, wherein the grayscale transformation comprises: Based on a predetermined grayscale value, each pixel value of the original image is divided according to grayscale, and pixel values ​​below the predetermined grayscale value are expanded, and pixel values ​​above the predetermined grayscale value are compressed.

3. The method of claim 1, wherein the first image generator is trained based on the following steps: learning a mapping from a set of real images of the first quality to a set of real images of the second quality, and learning a mapping from a set of real images of the second quality to a set of real images of the first quality, the second quality being higher than the first quality.

4. The method according to claim 1, Wherein determining a first global change loss between the real image of the second quality and the first generated image of the second quality comprises: determining a first global change loss between a real image of the second quality and a first generated image of the second quality by a pre-trained first convolutional neural network, and Wherein determining a first perceptual loss between a real image of the first quality and a second generated image of the first quality comprises: determining the first perceptual loss between the real image of the first quality and the second generated image of the first quality by means of the pre-trained first convolutional neural network.

5. The method according to claim 1, Wherein determining a second global change loss between the real image of the first quality and the first generated image of the first quality comprises: determining a second global change loss between the real image of the first quality and the first generated image of the first quality by a pre-trained first convolutional neural network, and Wherein, determining the second perceptual loss between the real image of the second quality and the second generated image of the second quality comprises: determining the second perceptual loss between the real image of the second quality and the second generated image of the second quality by the pre-trained first convolutional neural network.

6. The method of claim 1 , wherein the determining the first color loss between the real image of the second quality and the first generated image of the second quality comprises: Perform Gaussian noise processing on the real image of the second quality and the first generated image of the second quality to obtain the real image of the second quality after Gaussian noise and the first generated image of the second quality after Gaussian noise, and input the real image of the second quality after Gaussian noise and the first generated image of the second quality after Gaussian noise into a first discriminator to obtain the first color loss.

7. The method of claim 1, wherein determining a second color loss between the real image of the first quality and the first generated image of the first quality comprises: Perform Gaussian noise processing on the real image of the first quality and the first generated image of the first quality to obtain the real image of the first quality after Gaussian noise and the first generated image of the first quality after Gaussian noise, and input the real image of the first quality after Gaussian noise and the first generated image of the first quality after Gaussian noise into a first discriminator to obtain a second color loss.

8. The method of claim 1, wherein the determining the first texture loss between the real image of the second quality and the first generated image of the second quality comprises: The real image of the second quality and the first generated image of the second quality are grayed to obtain a grayed real image of the second quality and a grayed first generated image of the second quality, and the grayed real image of the second quality and the grayed first generated image of the second quality are input into a second discriminator to obtain a first texture loss.

9. The method of claim 1 , wherein the determining the second texture loss between the real image of the first quality and the first generated image of the first quality comprises: The real image of the first quality and the first generated image of the first quality are grayed to obtain a grayed real image of the first quality and a grayed first generated image of the first quality, and the grayed real image of the first quality and the grayed first generated image of the first quality are input into a second discriminator to obtain a second texture loss.

10. The method according to any one of the preceding claims, further comprising: The enhanced image is scored based on a pre-trained second convolutional neural network, and the scoring result is displayed.

11. The method as claimed in claim 6 or 7, wherein the first discriminator is configured to determine whether the input image is a real image or an image generated by the first image generator or the second image generator.

12. The method of claim 8 or 9, wherein the second discriminator is configured to determine whether the input image is a real image or an image generated by the first image generator or the second image generator.

13. An image enhancement device, comprising: An acquisition module is configured to acquire an original image; A grayscale transformation module is configured to perform grayscale transformation on each pixel value of the original image to obtain each target pixel value of the original image; and An image enhancement module is configured to enhance each target pixel value of the original image using a pre-trained first image generator to obtain an enhanced image, so that the content, color and texture loss of the enhanced image is less than a predetermined threshold; The first image generator is trained based on the following steps: Inputting a real image of a first quality in a real image set of a first quality into the first image generator to obtain a first generated image of a second quality, and determining a first global change loss, a first color loss, and a first texture loss between the real image of the second quality in the real image set of the second quality and the first generated image of the second quality, wherein the second quality is higher than the first quality; Inputting the first generated image of the second quality into a second image generator to obtain a second generated image of the first quality, and determining a first perceptual loss between the real image of the first quality and the second generated image of the first quality; Inputting the real image of the second quality into the second image generator to obtain a first generated image of the first quality, and determining a second global change loss, a second color loss, and a second texture loss between the real image of the first quality and the first generated image of the first quality; Inputting the first generated image of the first quality into the first image generator to obtain a second generated image of the second quality, and determining a second perceptual loss between the real image of the second quality and the second generated image of the second quality; determining a total loss function based on the first global variation loss, the first color loss, the first texture loss, the first perceptual loss, the second global variation loss, the second color loss, the second texture loss, and the second perceptual loss; Training the first image generator using real images of first quality in the real image set of the first quality and real images of second quality in the real image set of the second quality, so that the total loss function converges; and The first quality set of real images and the second quality set of real images are not in one-to-one correspondence, and the first color loss is The second color loss is G1 represents the first image generator, D c represents the first discriminator, G2 represents the second image generator, and m is the number of images.

14. A computer device, characterized in that: The method comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of any one of claims 1 to 12.

Images