A Semi-Supervised Learning-Based Image Enhancement Method for Full-Screen Mobile Phones

Abstract

This invention discloses a method for enhancing images of full-screen mobile phones based on semi-supervised learning. The steps include: 1) selecting or constructing a dataset, wherein each image data pair in the dataset includes images of the same scene taken by the same smartphone with and without transparent glass; 2) constructing an image enhancement model; 3) training the image enhancement model in a supervised manner using the dataset: first, enhancing the low-quality image y to obtain the enhanced image Y', and then calculating the loss function with the high-quality image Y to optimize the image enhancement model; 4) training the image enhancement model in an unsupervised manner using the dataset: first, enhancing the image y using the optimized model to obtain Y', and then calculating the loss based on Y' to optimize the model; 5) using the optimized image enhancement model to iteratively enhance the hue and saturation channels of the images taken by the full-screen mobile phone to obtain the corresponding high-quality images.

Description

Technical Field

[0001] This invention relates to a method for enhancing images on full-screen mobile phones, and more specifically to an adaptive enhancement method for full-screen mobile phone images based on semi-supervised learning. Background Technology

[0002] The pursuit of a better visual experience has driven the rise of full-screen smartphones, making under-display camera technology essential. However, in under-display camera technology, the display embedded in front of the camera affects light transmittance and direction, leading to saturation degradation, color deviation, and reduced contrast in images captured by full-screen smartphones. Therefore, achieving high saturation and vibrant colors in images captured by full-screen smartphones has become a new challenge for real-world image enhancement.

[0003] Single-image enhancement aims to improve the contrast and illumination of low-quality images. Due to various challenges related to constraints and optimization, it has become an active research area in computer vision. Many deep learning-based methods have been proposed in image enhancement. These methods extract effective and representative features, outperforming prior-based algorithms. However, deep learning-based methods consistently require a large number of labeled image pairs to learn the mapping between low-quality and high-quality images. Since labeled image pairs from real-world scenes are not readily available, most methods are trained on synthetic labeled data pairs. Due to the inherent limitations of synthetic images, methods trained on synthetic image datasets consistently fail to generalize well to image enhancement in real-world scenes. Summary of the Invention

[0004] This invention addresses the shortcomings of existing deep learning enhancement methods, such as the limited number of real-world scene datasets and poor model generalization ability, by proposing a full-screen mobile phone image enhancement method based on semi-supervised learning.

[0005] In the same scene, although the data distribution of images captured by full-screen mobile phones differs from their corresponding high-quality images, a correlation still exists. This invention analyzes the statistical characteristics of the hue, saturation, and brightness channels in the HSV color space, finding that the hue and saturation channels differ significantly between low-quality and high-quality images, while the brightness channel shows smaller differences. To fit the mapping relationship between different channels, a curve estimation method for the hue and saturation channels is proposed. A lightweight deep convolutional neural network is used to learn pixel-wise adaptive curve parameters for the hue and saturation channels. Then, the learned mapping curve parameters are used to iteratively enhance the low-quality images captured in real-world shooting to obtain the corresponding high-quality images.

[0006] The basic idea of ​​this method is:

[0007] 1. Enhancement Curve Estimation: This invention utilizes a lightweight convolutional neural network to learn the enhancement curves for the image's hue and saturation channels. According to formula (1), the enhancement curves can automatically adjust the magnitude of hue and saturation enhancement for each pixel, mapping low-quality images to high-quality images from the hue and saturation channels. The adaptive parameters depend on the input, and all operations are performed pixel-wise. Furthermore, a channel and spatial attention module is proposed, which includes channel attention and spatial attention to extract attention-aware features of low-quality input images as the network depth increases.

[0008] I n =I n-1 +μ n I n-1 (1-I n-1 (1)

[0009] Among them, I n Let I and μ represent the image after the nth enhancement. n This represents the parameters of the nth enhancement curve.

[0010] 2. Cascaded networks: By stacking curve estimation modules, network depth is increased to enhance model performance. Network models can be cascaded in various ways. Although this invention employs a relatively simple stacked architecture to cascade and enhance curve estimation modules, cascading is a general framework, and other depth enhancement methods can be stacked in a similar manner to improve their performance.

[0011] 3. Semi-supervised learning: To comprehensively consider the limited number of labeled data pairs and the model's generalization ability, this invention proposes a semi-supervised deep learning network for full-screen mobile phone image enhancement through curve estimation. The supervised branch is trained on labeled image pairs, and a referenced loss function is applied to constrain the training process. Furthermore, the unsupervised branch is trained on unlabeled images, and the training process is constrained by a non-referenced loss function such as channel prior.

[0012] The technical solution of this invention is as follows:

[0013] A method for image enhancement on full-screen mobile phones based on semi-supervised learning, comprising the following steps:

[0014] 1) Select or construct a dataset, wherein each image data pair in the dataset includes images of the same scene taken by the same smartphone covered with transparent glass and without transparent glass; wherein, the images taken by the smartphone covered with transparent glass are low-quality images, and the images taken by the smartphone without transparent glass are high-quality images;

[0015] 2) Construct an image enhancement model, which includes a cascaded network composed of multiple convolutional neural networks;

[0016] 3) The image enhancement model is trained in a supervised manner using the dataset: First, the low-quality image y is enhanced using the image enhancement model to obtain the enhanced image Y'. Then, the loss function is calculated for the labeled high-quality image Y corresponding to the low-quality image y to optimize the image enhancement model.

[0017] 4) Train the image enhancement model using the dataset in an unsupervised manner: First, use the image enhancement model optimized in step 3) to enhance the low-quality image y to obtain the enhanced image Y'. Then, calculate the saturation loss function, tone channel loss function and spatial consistency loss function based on the enhanced image Y' to optimize the image enhancement model.

[0018] 5) Using the image enhancement model optimized in step 4), the hue and saturation channels of the image taken by the full-screen mobile phone are iteratively enhanced to obtain the corresponding high-quality image.

[0019] Furthermore, the method for enhancing a low-quality image y to obtain an enhanced image Y' using the image enhancement model is as follows: The hue and saturation channel enhancement curves of the low-quality image y are learned using a convolutional neural network, and then, according to formula I... n =I n-1 +μ n I n-1 (1-I n-1 Adjust the magnitude of hue and saturation enhancement for each pixel in the low-quality image y, and map the low-quality image y from the hue and saturation channels to obtain the enhanced image Y'; where, I n Let μ represent the image after the nth iteration of enhancement of the low-quality image y. n This represents the parameters of the hue and saturation channel enhancement curves for the nth iteration. The enhanced image Y' is the image after the last iteration of enhancement on the low-quality image y.

[0020] Furthermore, in each cascade, the convolutional neural network uses the enhanced image obtained from the previous cascaded network module to learn the hue and saturation channel enhancement curves of the next cascaded network module.

[0021] Furthermore, during each cascade, the convolutional neural network extracts multi-channel curves from the resulting enhanced image through connection operations and attention mechanisms, and separates them into multiple adaptive hue and saturation channel enhancement curves, which are then used to iteratively enhance the input image.

[0022] Furthermore, during supervised and unsupervised training, a curve smoothing loss function is applied to the obtained hue and saturation channel enhancement curves to maintain the smoothness of each curve by constraining the gradients in each direction.

[0023] Furthermore, when training the image augmentation model using the dataset in a supervised manner, the mean squared error loss function is used to ensure that the augmented image Y' is close to the label image Y.

[0024] A server is characterized by comprising a memory and a processor, the memory storing a computer program configured to be executed by the processor, the computer program including instructions for performing the steps of the methods described above.

[0025] A computer-readable storage medium having a computer program stored thereon, characterized in that the computer program, when executed by a processor, implements the steps of the above-described method.

[0026] Compared with the prior art, the positive effects of the present invention are as follows:

[0027] It not only performs well in terms of structural similarity (SSIM) and peak signal-to-noise ratio (PSNR), but also achieves real-time enhancement, especially in 4K ultra-high-definition images. Attached Figure Description

[0028] Figure 1 This is an overall framework diagram of the present invention. Detailed Implementation

[0029] This invention proposes a method for enhancing images on full-screen mobile phones based on semi-supervised learning. The invention will be further described in detail below with reference to specific embodiments.

[0030] 1. Constructing a Real-World Dataset: Because collecting labeled data pairs is challenging, constructing training data pairs for real-world scenes is a major challenge in image enhancement tasks. To overcome this problem, many studies have synthesized labeled data pairs using various image processing algorithms. This invention establishes a full-screen mobile phone image enhancement dataset that can be used for specific image quality enhancement tasks. Each image data pair in the dataset consists of images of the same scene taken with and without transparent glass covering the camera, including images taken in indoor and outdoor environments. Outdoor scenes include street scenes, parks, and ancient and modern buildings. Default camera settings were used throughout the collection process.

[0031] 2. Enhancement Curve Estimation: This invention learns multiple image enhancement curves that automatically adjust the enhancement magnitude for each pixel, mapping low-quality images to enhanced images from hue and saturation channels. The adaptive parameters depend on the input, and all operations are performed pixel-by-pixel. Furthermore, a channel and spatial attention module is proposed, incorporating channel and spatial attention from the enhancement network to extract attention-aware features as the network depth increases. In the model, the low-quality input image is fed into a multi-layer convolutional neural network, and through connection operations and attention mechanisms, multi-channel curves are effectively extracted and separated into multiple adaptive hue and saturation channel enhancement curves, each used for iterative enhancement based on the original low-quality input image.

[0032] 3. Cascaded networks: Enhancement curve estimation modules are stacked to increase network depth and improve model performance. Network models can be cascaded in various ways. Although this invention uses a relatively simple stacked architecture to cascade enhancement curve estimation modules, cascading is a general framework, and other depth enhancement methods can be stacked in a similar way to improve their performance. The image enhanced by the previous curve estimation module is input into the next curve estimation module for cascaded enhancement; that is, a new enhanced curve is recalculated in the next curve estimation module, and iterative enhancement is performed again to achieve better enhancement results.

[0033] 4. Supervised branch training and optimization: Based on labeled data, the constructed cascaded network is used for model training, and the mean squared error loss function is used to ensure that the augmented image Y' is close to the labeled image Y.

[0034] 5. Unsupervised Branch Training and Optimization. Based on the model parameters obtained in step 4, unsupervised branch training is constrained using saturation loss function, hue channel loss function, and spatial consistency loss function based on unlabeled data. Furthermore, to maintain consistency between adjacent pixels and avoid artifacts, a curve smoothing loss function is used to maintain the smoothness of each curve during both supervised and unsupervised training.

[0035] 6. Using the optimized network model, the hue and saturation channels of images captured by full-screen mobile phones are iteratively enhanced to obtain corresponding high-quality images.

[0036] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A full-screen mobile phone image enhancement method based on semi-supervised learning, comprising the following steps: 1) Selecting or constructing a data set, each image data pair in the data set comprising images of the same scene taken by the same mobile phone with and without a transparent glass cover; wherein the image taken by the mobile phone with the transparent glass cover is a low-quality image, and the image taken by the mobile phone without the transparent glass cover is a high-quality image; 2) Constructing an image enhancement model comprising a cascade network of multiple convolutional neural networks; 3) Training the image enhancement model using the data set in a supervised manner: first, enhancing the low-quality image y using the image enhancement model to obtain an enhanced image Y', and then calculating a loss function with the labeled high-quality image Y corresponding to the low-quality image y to optimize the image enhancement model; 4) Training the image enhancement model using the data set in an unsupervised manner: first, enhancing the low-quality image y using the image enhancement model optimized in step 3) to obtain an enhanced image Y', and then calculating a saturation loss function, a hue channel loss function and a spatial consistency loss function based on the enhanced image Y' to optimize the image enhancement model; 5) Iteratively enhancing the hue and saturation channels of the image taken by the full-screen mobile phone using the image enhancement model optimized in step 4) to obtain a corresponding high-quality image In steps 3) and 4), the method for enhancing the low-quality image y using the image enhancement model to obtain an enhanced image Y' is as follows: learning the hue and saturation channel enhancement curves of the low-quality image y using a convolutional neural network, and then iteratively enhancing the low-quality image y using the hue and saturation channel enhancement curves to obtain a corresponding high-quality image Y'.

2. The method of claim 1, wherein, The method for enhancing the low-quality image y by using the image enhancement model is: learning the hue and saturation channel enhancement curve of the low-quality image y by using the convolutional neural network, then mapping the low-quality image y from the hue and saturation channels to obtain the enhanced image Y' according to the formula I n = I n-1 + μ n I n-1 (1-I n-1 ) adjusts the amplitude of the hue and saturation enhancement of each pixel point in the low-quality image y, and maps the low-quality image y from the hue and saturation channels to obtain the enhanced image Y'; wherein I n represents the image after the n th iteration of enhancing the low-quality image y, μ n represents the n th hue and saturation channel enhancement curve parameter, and the enhanced image Y' is the image after the last iteration of enhancing the low-quality image y.

3. The method of claim 2, wherein, In each cascade, the convolutional neural network uses the enhanced image obtained from the previous cascade network module to learn the hue and saturation channel enhancement curve of the next cascade network module.

4. The method according to claim 2 or 3, characterized in that, In each cascade, the convolutional neural network extracts a multi-channel curve graph from the enhanced image through connection operation and attention mechanism, and separates it into multiple adaptive hue and saturation channel enhancement curves for iteratively enhancing the input image.

5. The method according to claim 2 or 3, characterized in that, In the supervised and unsupervised training process, the hue and saturation channel enhancement curves obtained are subjected to a curve smoothing loss function to maintain the smoothness of each curve by constraining the gradient in each direction.

6. The method according to claim 1 or 2 or 3, characterized in that, In the supervised training process, a mean square error loss function is used to ensure that the enhanced image Y' is close to the labeled image Y.

7. A server, characterized by A computer program is stored in a memory and executed by a processor, and the computer program comprises instructions for executing each step of the method of any one of claims 1 to 6.

8. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 6.

Images