Image enhancement method, electronic device, and storage medium

Abstract

This application discloses an image enhancement method, electronic device, and storage medium. The method includes: acquiring a first image, wherein the first image is a low-light image to be enhanced; inputting the first image into a target generator for processing to obtain a second image, wherein the second image is a low-light enhanced image; wherein the target generator is trained based on a generator in a generative adversarial network, the generative adversarial network including the generator and a discriminator, the generator including a semantic segmentation network, the semantic segmentation network being used to assist in extracting low-light features at different levels in the first image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image enhancement method, electronic device, and storage medium. Background Technology

[0002] With the development and widespread use of electronic devices such as smartphones and network cameras, more and more users are choosing to use electronic devices to capture images. At the same time, users' requirements for image capture by electronic devices are also increasing. In nighttime or other poor lighting conditions, images captured by electronic devices often suffer from problems such as decreased image quality and missing scene information due to excessive darkness, affecting the user experience. Therefore, improving the image quality of low-light images has become essential.

[0003] In related technologies, low-light image enhancement is mainly based on convolutional neural networks. However, since the structure of convolutional neural networks is relatively simple, a large amount of data is required during training, and they are prone to getting stuck in local optima, so a good result cannot be obtained, resulting in poor enhancement effect of low-light images. Summary of the Invention

[0004] This application provides an image enhancement method, an electronic device, and a storage medium to solve the technical problem of poor enhancement effect for low-light images in related technologies.

[0005] According to a first aspect of this application, an image enhancement method is disclosed, the method comprising:

[0006] Acquire a first image, which is a low-light image to be enhanced;

[0007] The first image is input into the target generator for processing to obtain the second image, which is a low-light enhanced image.

[0008] The target generator is trained based on a generator in a generative adversarial network, which includes the generator and a discriminator. The generator includes a semantic segmentation network, which is used to assist in extracting dark light features at different levels in the first image.

[0009] According to a second aspect of this application, an image enhancement apparatus is disclosed, the apparatus comprising:

[0010] An acquisition module is used to acquire a first image, which is a low-light image to be enhanced;

[0011] The enhancement module is used to input the first image into the target generator for processing to obtain a second image, which is a low-light enhanced image;

[0012] The target generator is trained based on a generator in a generative adversarial network, which includes the generator and a discriminator. The generator includes a semantic segmentation network, which is used to assist in extracting dark light features at different levels in the first image.

[0013] According to a third aspect of this application, an electronic device is disclosed, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the image enhancement method as described in the first aspect.

[0014] According to a fourth aspect of this application, a computer-readable storage medium is disclosed having a computer program / instructions stored thereon, which, when executed by a processor, implements the image enhancement method as described in the first aspect.

[0015] According to a fifth aspect of this application, a computer program product is disclosed, comprising a computer program / instructions that, when executed by a processor, implement the image enhancement method as described in the first aspect.

[0016] In this embodiment of the application, a first image is obtained, which is a dark light image to be enhanced; the first image is input into a target generator for processing to obtain a second image, which is a dark light enhanced image; wherein, the target generator is obtained by training a generator based on a generative adversarial network, which includes a generator and a discriminator, and the generator includes a semantic segmentation network, which is used to assist in extracting dark light features at different levels in the first image.

[0017] As can be seen, in this embodiment, low-light image enhancement can be performed based on a target generator trained by a generative adversarial network. Since the generative adversarial network consists of two different networks instead of a single network, and the training method is adversarial, the continuous interaction between the generator and the discriminator networks can fully utilize the training data. In addition, the generator contains a semantic segmentation network that can fully mine the features of the image. Therefore, the trained target generator can learn relatively accurate prior low-light features, resulting in a better enhancement effect when enhancing low-light images based on the target generator. Attached Figure Description

[0018] Figure 1 This is a flowchart of an image enhancement method provided in an embodiment of this application;

[0019] Figure 2 This is an example diagram of the network structure of the generator in the generative adversarial network provided in the embodiments of this application;

[0020] Figure 3This is a flowchart of a model training method provided in an embodiment of this application;

[0021] Figure 4 This is a schematic diagram of the structure of an image enhancement device provided in an embodiment of this application;

[0022] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this application. Detailed Implementation

[0023] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0024] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of this application are not limited to the described order of actions, because according to the embodiments of this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the embodiments of this application.

[0025] In recent years, significant progress has been made in research on technologies based on artificial intelligence, such as computer vision, deep learning, machine learning, image processing, and image recognition. Artificial intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies, and application systems to simulate and extend human intelligence. AI is a comprehensive discipline involving numerous technologies, including chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, and neural networks. Computer vision, as an important branch of AI, specifically enables machines to recognize the world. Computer vision technologies typically include face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, object detection, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, and robot navigation and localization. With the research and advancement of artificial intelligence technology, this technology has been applied in numerous fields, such as security, urban management, traffic management, building management, park management, facial recognition access control, facial recognition attendance, logistics management, warehouse management, robotics, intelligent marketing, computational photography, mobile imaging, cloud services, smart homes, wearable devices, autonomous driving, autonomous driving, smart healthcare, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile internet, live streaming, beautification, makeup, medical aesthetics, and intelligent temperature measurement.

[0026] Taking low-light enhancement (also known as dim light enhancement) in the field of image processing as an example, low-light enhancement has a wide range of applications in different fields, including video capture, autonomous driving and computational photography.

[0027] Currently, low-light enhancement methods are divided into traditional methods and deep learning approaches. Traditional methods include histogram matching, Retinex-based methods, and fusion-based methods. Deep learning-based low-light enhancement methods primarily rely on convolutional neural networks (CNNs). However, CNNs have a relatively simple structure, require large datasets for training, and are prone to getting stuck in local optima, failing to achieve satisfactory results. Furthermore, dark areas in low-light images often conceal significant noise; when these areas are brightened, the noise is amplified, leading to a degraded image quality. Neither traditional methods nor CNNs can effectively address this issue.

[0028] To address the aforementioned issues, embodiments of this application provide an image enhancement method, an electronic device, and a storage medium.

[0029] To facilitate understanding, some concepts involved in the embodiments of this application will be introduced below.

[0030] A Generative Adversarial Network (GAN) is a combination of two networks: a generator (also called a generator network, G) responsible for generating simulated data, and a discriminator (also called a discriminator network, D) responsible for determining whether the input data is real or generated. The generator continuously optimizes its generated data to make it undetectable by the discriminator, while the discriminator also optimizes itself to make its judgments more accurate. The two networks are in an adversarial relationship, hence the name adversarial network.

[0031] Semantic segmentation networks are used to perform semantic segmentation on images. Image semantic segmentation assigns a semantic category to each pixel in the input image to obtain pixelated dense classification.

[0032] Compared to related technologies that use convolutional neural networks for low-light image enhancement, this application proposes a low-light image enhancement method based on generative adversarial networks (GANs). The GAN consists of two distinct networks, rather than a single network, and is trained adversarially. The generator and discriminator networks continuously compete to fully utilize the limited dataset and achieve optimal results. Compared to other methods, GANs can generate clearer and more realistic samples, effectively remove noise from dark areas in low-light images to obtain high-quality images, and require relatively less data. During training, the GAN model only uses backpropagation and does not require complex Markov chains, thus significantly reducing its time complexity.

[0033] The following section describes an image enhancement method provided by an embodiment of this application.

[0034] Figure 1 This is a flowchart of an image enhancement method provided in an embodiment of this application, such as... Figure 1 As shown, the method may include the following steps: step 101 and step 102;

[0035] In step 101, a first image is acquired, wherein the first image is a dark light image to be enhanced.

[0036] In this embodiment, the first image can be an image captured by a mobile device such as a smartphone or drone, or an image captured by a webcam.

[0037] In step 102, the first image is input into the target generator for processing to obtain the second image, which is the image after dark light enhancement. The target generator is trained based on the generator in the generative adversarial network. The generative adversarial network includes a generator and a discriminator. The generator includes a semantic segmentation network, which is used to help extract dark light features at different levels in the first image.

[0038] In this embodiment, high-quality ev-0 real images (GT) and ev-2 low-light data samples can be obtained using multi-frame fusion. The ev-0 and ev-2 data samples need to be paired. The data is then fed into a generative adversarial network (GAN). The generator continuously generates high-quality image samples from the ev-2 low-light images, while the discriminator continuously scores the generated image samples using the GT data. The images generated by the generator need to deceive the discriminator into believing they are high-quality images and real samples, while the discriminator tries to identify the generated images as fake samples. The two networks continuously train against each other, fully utilizing the data content to train an optimal model, i.e., the target generator.

[0039] In this embodiment of the application, a semantic segmentation network is introduced into the generator of the generative adversarial network. This network structure can enhance the full mining of data by enhancing features.

[0040] In some embodiments, the semantic segmentation network in the generator network can be a Unet network, where the Unet network can simply concatenate the feature maps of the encoder to the upsampled feature maps of the decoder at each stage, thereby forming a trapezoidal structure; through the skip-concatenation architecture, the decoder is allowed to learn the relevant features lost in the encoder pooling at each stage, resulting in more accurate and faster inference results.

[0041] For example, such as Figure 2 The diagram illustrates the network structure of the generator in a generative adversarial network (GAN). This generator adds a Unet network to the original ResNet network, where the Unet network is connected to the ResNet network and is located after the ResNet network. The network structure of the discriminator in the GAN can adopt the network structure in related technologies.

[0042] In this embodiment of the application, when the target generator processes the first image, it extracts the basic feature map of the first image through a residual network, and processes the basic feature map through a Unet network to obtain the second image.

[0043] As can be seen from the above embodiments, in this embodiment, a first image is obtained, which is a dark light image to be enhanced; the first image is input into a target generator for processing to obtain a second image, which is a dark light enhanced image; wherein, the target generator is obtained by training a generator based on a generative adversarial network, which includes a generator and a discriminator, and the generator includes a semantic segmentation network, which is used to assist in extracting dark light features at different levels in the first image.

[0044] As can be seen, in this embodiment, low-light image enhancement can be performed based on a target generator trained by a generative adversarial network. Since the generative adversarial network consists of two different networks instead of a single network, and the training method is adversarial, the continuous interaction between the generator and the discriminator networks can fully utilize the training data. In addition, the generator contains a semantic segmentation network that can fully mine the features of the image. Therefore, the trained target generator can learn relatively accurate prior low-light features, resulting in a better enhancement effect when enhancing low-light images based on the target generator.

[0045] Figure 3 This is a flowchart of a model training method provided in an embodiment of this application, such as... Figure 3 As shown, the method may include the following steps: step 301, step 302 and step 303;

[0046] In step 301, a training set is obtained, wherein the training set includes: multiple sample image pairs, and each sample image pair includes: a sample normal light image and a corresponding sample dark light image.

[0047] In this embodiment of the application, the sample images in the training set can be automatically generated images. For example, a multi-frame fusion method can be used to generate sample images in each sample image pair.

[0048] In step 302, a generative adversarial network is constructed, along with a first loss function corresponding to the generator and a second loss function corresponding to the discriminator within the generative adversarial network.

[0049] In some embodiments, to improve the quality of the generator's output image, the first loss function may consist of at least one of the following loss functions: a third loss function, a fourth loss function, and a fifth loss function; wherein, the third loss function is used to balance the enhancement degree between dark areas and other areas in the sample dark-light image; the fourth loss function is used to measure the structural similarity between the generator's output image and the corresponding sample normal-light image; and the fifth loss function is used to measure the style difference between the generator's output image and the corresponding sample normal-light image. The second loss function may employ a conventional adversarial loss function.

[0050] For example, the first loss function is Loss1, the second loss function is Loss2, the third loss function is Loss3, the fourth loss function is Loss4, and the fifth loss function is Loss5. Loss1 = Loss3*a + Loss4*b + Loss5*c, where a, b, and c are weight coefficients that can be set according to actual needs; Loss2 can use the conventional adversarial loss function V(G, D).

[0051] Optionally, the third loss function can be determined by: obtaining the brightness weight term of each pixel in the sample dark light image; calculating the weight image of the sample dark light image based on the brightness weight term of each pixel, wherein the dark light weight value of the dark light region in the weight image is greater than the dark light weight value of other regions; and determining the third loss function based on the weight image, the generator's output image, and the corresponding sample normal light image.

[0052] In this embodiment, the brightness weight term I of a pixel in the image can be calculated as follows: Using a general approach, it is assumed that the brightness of a well-exposed pixel is likely to approach 0.5. The formula for calculating I is as follows: Where Y is the pixel value of the pixel in the Y channel, and μ and δ1 are parameters set according to actual needs.

[0053] For example, the third loss function Where Gθ(lin) is the output image of the generator, lgt is the normal light image of the sample, m is the number of pixels in the image, and w1 is the dark light weight. It can be seen that the closer the brightness weight of a pixel is to 0.2, the greater the dark light weight of that pixel. δ2 is a parameter set according to actual needs.

[0054] Optionally, the fourth loss function can be determined by the following method: based on the structural similarity (SSIM) between the generator's output image and the corresponding sample normal light image.

[0055] For example, the fourth loss function is Loss4 = SSIM(lgt, Gθ(lin)).

[0056] Optionally, the fifth loss function can be determined by: extracting the first feature of the generator's output image through a style transfer network; and extracting the second feature of the corresponding sample normal light image through a style transfer network; and determining the fifth loss function based on the first and second features. The style transfer network can be a VGG19 network.

[0057] For example, the fifth loss function VGG(lgt) is the second feature of the normal light image of the sample, and VGG(Gθ(lin)) is the first feature of the output image of the generator.

[0058] In this embodiment, during the training of the generative adversarial network, in addition to the commonly used adversarial loss, a fifth loss function is introduced to obtain high-level information of the image and improve its visual quality. The fake image generated by the generator and the ground truth are fed into the style transfer network for feature extraction, and the root mean square error of the extracted features is used as a constraint (i.e., the fifth loss function). Furthermore, for the low-light image enhancement task, the low-light regions in the image should receive more attention; therefore, a third loss function is introduced to balance the enhancement level between low-light regions and other regions.

[0059] In step 303, the generative adversarial network is trained iteratively multiple times based on the training set. In each iteration, the discriminator is fixed first, and the sample dark light image is input into the generator for processing. The first loss value is calculated based on the sample normal light image corresponding to the sample dark light image, the generator's output image, and the first loss function. The parameters in the generator are updated by backpropagation based on the first loss value. Then, the generator is fixed, and the generator's output image and the corresponding sample normal light image are input into the discriminator for processing. The second loss value is calculated based on the discriminator's output and the second loss function. The parameters in the discriminator are updated by backpropagation based on the second loss value. After multiple iterations of training, the generative adversarial network converges, and the target generator is obtained.

[0060] In some embodiments, to improve training performance, the sample low-light images can be enhanced, and the enhanced sample data and ground truth (GT) data are simultaneously fed into a generative adversarial network (GAN). The generator network and discriminator network in the network then begin adversarial training. Accordingly, step 303 above may include the following steps: step 3031 and step 3032;

[0061] In step 3031, the sample dark-light images in the training set are preprocessed; wherein, the preprocessing includes at least one of the following: adding random noise, image rotation, and image cropping.

[0062] In step 3032, the generative adversarial network is trained iteratively multiple times using normal light images of samples in the training set and the corresponding preprocessed dark light images.

[0063] In one example, the discriminator is first fixed to train the generator. Underexposed, low-light images are used as input to the generator in the generative adversarial network (GAN). To extract as much information as possible from the data, a residual structure and a Unet network structure are introduced into the generator. The generator continuously generates fake samples, hoping to deceive the discriminator and achieve a high score. The generator's loss function involves the following: since the research focuses on low-light image enhancement, dark areas in the image should be given greater weight; therefore, a region loss function (the third loss function) is introduced to constrain this. The fake samples generated by the generator and the ground truth (GT) are used to calculate the SSIM loss function (the fourth loss function) to enhance the detail information of the network's output image. To obtain high-level image information and improve the visual quality, the fake images generated by the generator and the GT are input into a VGG19 network for feature extraction. The root mean square error of the extracted features is used as a constraint (the fifth loss function).

[0064] Then, the discriminator is trained with the generator fixed. The fake samples generated by the generator and the real samples with good exposure are used as the input of the discriminator. The discriminator needs to identify the fake samples as much as possible and give them low scores. The loss function of the discriminator adopts the conventional adversarial loss function of generative adversarial networks.

[0065] Compared to related technologies that use convolutional neural networks for low-light image enhancement, this application proposes a low-light image enhancement method based on generative adversarial networks (GANs). The GAN consists of two distinct networks, rather than a single network, and is trained adversarially. The generator and discriminator networks continuously compete to fully utilize the limited dataset and achieve optimal results. Compared to other methods, GANs can generate clearer and more realistic samples, effectively remove noise from dark areas in low-light images to obtain high-quality images, require relatively less data, and only use backpropagation during training, without the need for complex Markov chains, thus significantly reducing time complexity.

[0066] Figure 4 This is a schematic diagram of the structure of an image enhancement device provided in an embodiment of this application, as shown below. Figure 4 As shown, the image enhancement device 400 may include: an acquisition module 401 and an enhancement module 402;

[0067] The acquisition module 401 is used to acquire a first image, which is a low-light image to be enhanced;

[0068] Enhancement module 402 is used to input the first image into the target generator for processing to obtain a second image, wherein the second image is a low-light enhanced image;

[0069] The target generator is trained based on a generator in a generative adversarial network, which includes the generator and a discriminator. The generator includes a semantic segmentation network, which is used to assist in extracting dark light features at different levels in the first image.

[0070] As can be seen from the above embodiments, in this embodiment, a first image is obtained, which is a dark light image to be enhanced; the first image is input into a target generator for processing to obtain a second image, which is a dark light enhanced image; wherein, the target generator is obtained by training a generator based on a generative adversarial network, which includes a generator and a discriminator, and the generator includes a semantic segmentation network, which is used to assist in extracting dark light features at different levels in the first image.

[0071] As can be seen, in this embodiment, low-light image enhancement can be performed based on a target generator trained by a generative adversarial network. Since the generative adversarial network consists of two different networks instead of a single network, and the training method is adversarial, the continuous interaction between the generator and the discriminator networks can fully utilize the training data. In addition, the generator contains a semantic segmentation network that can fully mine the features of the image. Therefore, the trained target generator can learn relatively accurate prior low-light features, resulting in a better enhancement effect when enhancing low-light images based on the target generator.

[0072] Optionally, as an embodiment, the semantic segmentation network may include a Unet network.

[0073] Alternatively, as an embodiment, the target generator can be trained in the following manner:

[0074] Obtain a training set, which includes: multiple sample image pairs, each sample image pair including: a sample normal light image and a corresponding sample dark light image;

[0075] Construct the generative adversarial network, the first loss function corresponding to the generator in the generative adversarial network, and the second loss function corresponding to the discriminator;

[0076] Based on the training set, the generative adversarial network is trained iteratively multiple times. In each iteration, the discriminator is fixed, and the sample dark-light image is input into the generator for processing. A first loss value is calculated based on the sample normal-light image corresponding to the sample dark-light image, the output image of the generator, and the first loss function. The parameters in the generator are updated by backpropagation based on the first loss value. Then, the generator is fixed, and the output image of the generator and the corresponding sample normal-light image are input into the discriminator for processing. A second loss value is calculated based on the output of the discriminator and the second loss function. The parameters in the discriminator are updated by backpropagation based on the second loss value. After multiple iterations of training, the generative adversarial network converges, and the target generator is obtained.

[0077] Optionally, as an embodiment, the first loss function may be composed of at least one of the following loss functions: a third loss function, a fourth loss function, and a fifth loss function;

[0078] The third loss function is used to balance the enhancement level between dark areas and other areas in the sample dark image;

[0079] The fourth loss function is used to measure the structural similarity between the generator's output image and the corresponding sample normal light image;

[0080] The fifth loss function is used to measure the style difference between the generator's output image and the corresponding sample normal light image.

[0081] Alternatively, as an embodiment, the third loss function can be determined in the following way:

[0082] Obtain the brightness weight term of each pixel in the sample low-light image;

[0083] Based on the brightness weight term of each pixel, a weighted image of the sample dark image is calculated, wherein the dark weight value of the dark area in the weighted image is greater than the dark weight value of other areas.

[0084] The third loss function is determined based on the weighted image, the generator's output image, and the corresponding normal light image of the sample.

[0085] Alternatively, as an embodiment, the fifth loss function can be determined in the following way:

[0086] The first feature of the generator's output image is extracted using a style transfer network;

[0087] The style transfer network is used to extract the second feature of the corresponding normal light image of the sample.

[0088] The fifth loss function is determined based on the first feature and the second feature.

[0089] Optionally, as an embodiment, the step of iteratively training the generative adversarial network based on the training set may include:

[0090] Preprocess the sample low-light images in the training set;

[0091] The generative adversarial network is trained iteratively multiple times using normal light images and corresponding preprocessed dark light images from the training set.

[0092] The preprocessing includes at least one of the following: adding random noise, image rotation, and image cropping.

[0093] Any step and specific operation within any step in the embodiments of the image enhancement method provided in this application can be performed by a corresponding module in the image enhancement device. The procedures for the corresponding operations performed by each module in the image enhancement device are described in the embodiments of the image enhancement method.

[0094] As the device embodiment is basically similar to the method embodiment, the description is relatively simple, and relevant parts can be found in the description of the method embodiment.

[0095] Figure 5 This is a structural block diagram of an electronic device provided in an embodiment of this application. The electronic device includes a processing component 522, which further includes one or more processors, and memory resources represented by a memory 532 for storing instructions executable by the processing component 522, such as application programs. The application programs stored in the memory 532 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 522 is configured to execute instructions to perform the methods described above.

[0096] The electronic device may also include a power supply component 526 configured to perform power management of the electronic device, a wired or wireless network interface 550 configured to connect the electronic device to a network, and an input / output (I / O) interface 558. The electronic device may operate on an operating system stored in memory 532, such as Windows Server™, MacOS X™, Unix™, Linux™, FreeBSD™, or similar.

[0097] According to another embodiment of this application, this application also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implements the steps in the image enhancement method as described in any of the above embodiments.

[0098] According to yet another embodiment of this application, this application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps in the image enhancement method as described in any of the above embodiments.

[0099] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0100] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0101] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0102] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0103] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0104] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0105] The above provides a detailed description of the image enhancement method, electronic device, and storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. An image enhancement method, characterized in that, The method includes: Acquire a first image, which is a low-light image to be enhanced; The first image is input into the target generator for processing to obtain the second image, which is a low-light enhanced image. The target generator is trained based on a generator in a generative adversarial network (GAN). The GAN includes the generator and a discriminator. The generator includes a semantic segmentation network, which is used to assist in extracting dark light features at different levels in the first image. When training the generator, a first loss value is calculated based on a first loss function and the output image of the generator. The parameters in the generator are updated by backpropagation based on the first loss value. The first loss function is constructed based on the generator. The first loss function is composed of at least one of the following loss functions: a third loss function, a fourth loss function, and a fifth loss function; The third loss function is used to balance the enhancement level between dark areas and other areas in the sample dark image; The fourth loss function is used to measure the structural similarity between the generator's output image and the corresponding sample normal light image; The fifth loss function is used to measure the style difference between the generator's output image and the corresponding sample normal light image. The fifth loss function is determined in the following way: The first feature of the generator's output image is extracted using a style transfer network; The style transfer network is used to extract the second feature of the corresponding normal light image of the sample. The fifth loss function is determined based on the first feature and the second feature, using the root mean square error.

2. The method according to claim 1, characterized in that, The semantic segmentation network includes the Unet network.

3. The method according to claim 1 or 2, characterized in that, The target generator is trained in the following way: Obtain a training set, which includes: multiple sample image pairs, each sample image pair including: a sample normal light image and a corresponding sample dark light image; Construct the generative adversarial network, the first loss function corresponding to the generator in the generative adversarial network, and the second loss function corresponding to the discriminator; Based on the training set, the generative adversarial network is trained iteratively multiple times. In each iteration, the discriminator is fixed, and the sample dark-light image is input into the generator for processing. A first loss value is calculated based on the sample normal-light image corresponding to the sample dark-light image, the output image of the generator, and the first loss function. The parameters in the generator are updated by backpropagation based on the first loss value. Then, the generator is fixed, and the output image of the generator and the corresponding sample normal-light image are input into the discriminator for processing. A second loss value is calculated based on the output of the discriminator and the second loss function. The parameters in the discriminator are updated by backpropagation based on the second loss value. After multiple iterations of training, the generative adversarial network converges, and the target generator is obtained.

4. The method according to claim 1, characterized in that, The third loss function is determined in the following way: Obtain the brightness weight term of each pixel in the sample low-light image; Based on the brightness weight term of each pixel, a weighted image of the sample dark image is calculated, wherein the dark weight value of the dark area in the weighted image is greater than the dark weight value of other areas. The third loss function is determined based on the weighted image, the generator's output image, and the corresponding normal light image of the sample.

5. The method according to claim 3, characterized in that, The step of iteratively training the generative adversarial network based on the training set includes: Preprocess the sample low-light images in the training set; The generative adversarial network is trained iteratively multiple times using normal light images and corresponding preprocessed dark light images from the training set. The preprocessing includes at least one of the following: adding random noise, image rotation, and image cropping.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the method according to any one of claims 1-5.

7. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1-5.

8. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1-5.

Images