An Image Enhancement System and Method Based on ECO-GAN

Abstract

An image enhancement system and method based on ECO-GAN, including a generator network and a discriminator network. The generator network includes a generator feature extraction sub-module and an image restoration sub-module, which extracts features and restores the input image during the training phase and outputs generated samples that are sufficient to deceive the discriminator. The discriminator network includes a discriminator feature extraction sub-module and a classifier, which discriminates the authenticity of the generated samples during the training phase. The present invention uses a deep learning algorithm to enhance images. The algorithm is efficient and has strong real-time performance. By reusing the feature extraction network, various image enhancement tasks are achieved; it has a wide range of applications. The deep learning-based image enhancement algorithm depends on paired data training and does not rely on complex and diverse physical knowledge; moreover, it has diverse functions and is flexible to use.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and specifically to an image enhancement method based on ECO-GAN. Background Art

[0002] With the rise of deep learning and the rapid development of computer vision, it has promoted the wide application of deep learning-based computer vision technology in various fields. Since the application of deep learning requires a large amount of high-quality data, especially in the field of computer vision, a huge amount of high-quality image data is needed. To improve the performance of the already deployed visual task devices, the cost of reconstructing each device is very high. To conveniently and quickly obtain a large amount of high-quality pictures, the labor cost of using image processing software is also very high. How to conveniently and quickly obtain high-quality images and be plug-and-play on the pre-deployed visual task devices has become a very worthy question to think about. Based on these ideas, this paper has established a research on image enhancement technology of a fast image enhancement algorithm ECO-GAN based on deep learning. Generate high-quality images quickly and efficiently through ECO-GAN. Generate the corresponding generator model from the available large batch of training data sets, and make the discriminator model and the generator model confront and update asynchronously, so as to achieve the purpose of generating high-quality images.

[0003] The invention patent with the application number CN113870162A (Patent Center of Dalian University of Technology) discloses a low-light image enhancement method that fuses illumination and reflection. The method includes taking the physical model into consideration at each stage of the network and learning a propagation that fuses the physical model in a network manner, so that each step of the network is based on the physical laws of imaging. The disadvantage of this method is that it is too dependent on the physical model and each step has to be based on the physical laws of imaging; this method does not make full use of the learning ability of the neural network and does not have generalization; this method can only deal with low-light images for image enhancement and has a narrow application range.

[0004] The existing technologies have the following deficiencies:

[0005] 1. The secondary development of the image enhancement algorithm based on the physical model is difficult, and the requirements for physical knowledge are high.

[0006] 2. Combining the physical laws of imaging with the neural network and learning the physical model in a network manner narrows the application range.

[0007] 3. The image enhancement algorithm based on deep learning requires a large amount of paired data for training and has a large amount of computation.

[0008] 4. The functions of the current image enhancement algorithms are single, and the mutual inhibitory effects of multiple image enhancements are not optimized. Summary of the Invention

[0009] To solve the above problems existing in the current technology, the present invention provides an image enhancement system and method based on ECO-GAN, which can not only quickly complete various image enhancement tasks, but also effectively improve the quality of images, provide high-quality sample pictures for the research in the field of computer vision based on deep learning, and is trained end-to-end independently of other devices, being flexible and convenient, and plug-and-play.

[0010] The specific technical solution of the present invention is as follows:

[0011] An image enhancement system based on ECO-GAN, comprising a generator network and a discriminator network:

[0012] The generator network includes a generator feature extraction sub-module and an image restoration sub-module, which extract features and restore the input image during the training phase, and output a generated sample sufficient to deceive the discriminator; the discriminator network includes a discriminator feature extraction sub-module and a classifier, which discriminate the authenticity of the generated sample during the training phase.

[0013] The generator feature extraction sub-module: located at the input end of the generator network, adopts a cascaded downsampling convolutional neural network for one stage, and is used to extract features from the original image;

[0014] The image restoration sub-module: located at the output end of the generator network, adopts a cascaded upsampling convolutional neural network for N stages, and completes various types of enhancement tasks respectively according to the features extracted by the generator feature extraction sub-module, and outputs a generated sample for the discriminator to discriminate the authenticity;

[0015] The discriminator feature extraction sub-module: located at the input end of the discriminator network, adopts a cascaded VGG19 network for one stage, and extracts features from the input real image and generated sample;

[0016] The classifier: located at the output end of the discriminator network, discriminates the authenticity of the generated sample by comparing the output of the generated sample and the real image after passing through the discriminator feature extraction sub-module.

[0017] Furthermore, both the downsampling convolutional neural network and the upsampling convolutional neural network adopt ResNeSt network modules. The ResNeSt network module divides multiple channels into multiple branches, processes the attention mechanism between channels branch by branch to obtain the information hidden between channels, and then fuses and stacks the multiple branches, combined with the form of skip connection, which alleviates the phenomenon of gradient disappearance.

[0018] Both the single-stage cascaded downsampling convolutional neural network and the multi-stage cascaded upsampling convolutional neural network are composed of multiple parallel branches. Each branch of the downsampling convolutional neural network consists of multiple cascaded downsampling convolutional sub-modules, and each downsampling convolutional sub-module of each branch is composed of a convolutional layer, a batch normalization layer, and an activation layer cascaded together.

[0019] Each branch of the upsampling convolutional neural network consists of multiple cascaded upsampling convolutional sub-modules, and each upsampling convolutional sub-module of each branch is composed of a transposed convolutional layer, a batch normalization layer, and an activation layer cascaded together. The activation layer uses the LeakyReLU activation function.

[0020] Furthermore, a cross-stage fusion module is adopted in the image inpainting sub-module to reduce the mutual inhibition effect brought about when the network model processes N types of enhancement type tasks simultaneously. The cross-stage fusion module uses two convolutional layers, which respectively receive the input of the current branch and the input of the previous stage branch, stack them along the channel dimension, and then process the features from different stages through a convolutional layer to achieve cross-stage feature fusion. The cross-stage fusion module first stacks the features with the number of channels ch in the current stage and the features with the number of channels ch in the previous stage to obtain features with the number of channels 2×ch, then extracts the information therein through a convolutional layer, condenses the features, obtains features with the number of channels ch, and continues to send them to the network in the current stage.

[0021] Furthermore, the generated samples output by the generator network are discriminated by the discriminator network, and the generator network is optimized by minimizing the difference loss function between the generated samples and the real samples, so that the generator network can generate generated samples that are sufficient to deceive the discriminator.

[0022] Furthermore, the discriminator adopts Patch-GAN, and comprehensively discriminates the quality of the output samples of the generator network through a global discriminator and a local discriminator.

[0023] An image enhancement method based on ECO-GAN is as follows:

[0024] Step 1: Collect the original image, analyze the type of image enhancement required for the original image, and determine the number of segments N of the upsampling convolutional neural network.

[0025] Step 2: Use image enhancement software to make a paired dataset (original image - real image) for ECO-GAN to learn.

[0026] Step 3: Obtain generated samples through the generator network of the image enhancement system based on ECO-GAN.

[0027] Step 4: The generated samples are discriminated by the discriminator network, and the generator network is optimized by minimizing the difference loss function between the generated samples and the real samples, so that the generator network can generate generated samples that are sufficient to deceive the discriminator.

[0028] The discriminator network outputs the discrimination result of whether the generated samples belong to the real samples. By optimizing the ability of the discriminator network to distinguish true from false, the discriminator network puts forward higher requirements for the generator network. The discriminator network and the generator network promote and confront each other and make progress together.

[0029] Step 5: The enhancement of the image to be processed is completed by the trained generator network.

[0030] The original image is input into the generator network of the image enhancement system based on ECO-GAN, and a cascaded downsampling convolutional neural network in a segment of the generator feature extraction sub-module is used to extract features from the original image;

[0031] Furthermore, the specific method of Step 3 is as follows:

[0032] The downsampling convolutional neural network is composed of multiple branches, each branch includes multiple convolutional modules, and the convolutional module is a combination of a convolutional layer, a batch normalization layer and an activation layer. The activation function of all activation layers adopts LeakyReLU. The batch normalization layer normalizes the features extracted by the convolutional layer according to the minimum batch, and then sends them to the activation layer. The activation layer uses the LeakyReLU function to activate the normalized features, and then sends the image features to the next convolutional module. The downsampling convolutional neural network processes the channel attention mechanism between channels branch by branch, mines the information hidden between channels, and then fuses and stacks multiple branches, combined with the form of skip connection, which alleviates the phenomenon of gradient disappearance.

[0033] The upsampling convolutional neural network has N segments, and the image is repaired according to the image features extracted by the downsampling convolutional neural network. Each segment of the upsampling convolutional neural network corresponds to an image enhancement task. Each segment of the upsampling convolutional neural network has multiple branches, and each branch includes multiple deconvolutional modules. The deconvolutional module is a combination of a deconvolutional layer, a batch normalization layer and an activation layer. The number of deconvolutional modules is the same as that of the convolutional modules of the downsampling convolutional neural network. The activation function of all activation layers adopts LeakyReLU. The batch normalization layer normalizes the features extracted by the convolutional neural network according to the minimum batch, and then sends them to the next layer. The cross-stage fusion module is adopted to reduce the mutual inhibition effect brought by the network model when processing multiple enhancement type tasks at the same time. The image features pass through N segments of the upsampling convolutional neural network and the cross-stage fusion module, and the finally restored image is obtained through the Nth segment of the upsampling convolutional neural network, which is called the generated sample.

[0034] Furthermore, the specific method of step 4 is as follows:

[0035] Input the generated samples and real images into the discriminator network, use the discriminator feature extraction sub-module to extract features, and then input them into the classifier. The classifier determines the authenticity of the input images.

[0036] Each image enhancement task corresponds to a set of generator loss functions. The discriminator loss function is composed of the least squares loss of the content gap between the generated samples and the real samples, the local least squares loss and the global least squares loss of the discriminator's discrimination of the authenticity of the generated samples:

[0037]

[0038]

[0039] Among them, I raw and I gt respectively represent the original image and the real image. L content represents the content loss between the generated samples and the real samples. c, h, and w are the number of channels, height, and width of the output image respectively. p represents the pixel point, and G represents the generator model.

[0040] L adv represents the adversarial loss generated by the discriminator's discrimination of the authenticity of the input samples. and respectively represent the local discrimination loss and the global discrimination loss. D and G respectively represent the discriminator model and the generator model, and E represents taking the expectation of the input random variable.

[0041] By minimizing the generator loss and using the gradient descent algorithm for solution, the generator network can generate generated samples that are sufficient to deceive the discriminator.

[0042] The discriminator uses the Markov discriminator Patch-GAN, and constructs the discriminator loss function through the least squares loss of the global discriminator and the local discriminator's discrimination of the authenticity of the generated samples. By minimizing the discriminator loss and using the gradient descent algorithm for solution, the ability of the discriminator network to distinguish authenticity is improved. By continuously optimizing the ability of the discriminator network to distinguish authenticity, the discriminator network puts forward higher requirements for the generator network. The discriminator network and the generator network promote and oppose each other and make common progress.

[0043] Furthermore, the paired data sets in step 2 are made using the software Adobe Lightroom, and the format of the required data sets is uniformly.jpg images.

[0044] The beneficial effects of the present invention are:

[0045] The present invention enhances images using deep learning algorithms and has the following characteristics: 1. The algorithm is efficient and has strong real-time performance. By reusing the feature extraction network, various image enhancement tasks can be achieved; 2. It has a wide range of applications. The image enhancement algorithm based on deep learning relies on paired data training and does not depend on complex and diverse physical knowledge; 3. It has diverse functions and is flexible to use. At present, the functions of image enhancement algorithms are single, and the mutual inhibition effects brought about when multiple image enhancement algorithms are used in combination are not considered. The present invention can achieve various image enhancement tasks through only one convolutional neural network model. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 FIG. is a schematic diagram of the overall model structure of the image enhancement method based on ECO-GAN provided by the present invention.

[0047] Figure 2 FIG. is a training flow chart of the image enhancement method based on ECO-GAN provided by the present invention.

[0048] Figure 3 FIG. is a test flow chart of the image enhancement method based on ECO-GAN provided by the present invention.

[0049] Figure 4 FIG. is a schematic diagram of the overall process of the preferred embodiment of the image enhancement method based on ECO-GAN provided by the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0050] The present invention will be further described below in conjunction with specific embodiments, but the present invention is not limited to this specific embodiment. Those skilled in the art should recognize that the present invention covers all alternative solutions, improvement solutions, and equivalent solutions that may be included within the scope of the claims.

[0051] An image enhancement system based on ECO-GAN includes a generator network and a discriminator network:

[0052] The generator network includes a generator feature extraction sub-module and an image inpainting sub-module, which extract features and inpaint the input image during the training phase and output generated samples that are sufficient to deceive the discriminator; the discriminator network includes a discriminator feature extraction sub-module and a classifier, which discriminate the authenticity of the generated samples during the training phase.

[0053] The generator feature extraction sub-module: located at the input end of the generator network, adopts a cascaded downsampling convolutional neural network for extracting features from the original image;

[0054] The described image inpainting sub-module: Located at the output end of the generator network, it adopts an N-stage cascaded upsampling convolutional neural network, and completes various types of enhancement tasks according to the features extracted by the generator feature extraction sub-module, and outputs generated samples for the discriminator to distinguish authenticity;

[0055] The described discriminator feature extraction sub-module: Located at the input end of the discriminator network, it adopts a one-stage cascaded VGG19 network to extract features from the input real images and generated samples;

[0056] The described classifier: Located at the output end of the discriminator network, it makes a judgment on the authenticity of the generated samples by comparing the output of the generated samples and the real images after passing through the discriminator feature extraction sub-module.

[0057] Furthermore, both the downsampling convolutional neural network and the upsampling convolutional neural network adopt ResNeSt network modules. The ResNeSt network module divides multiple channels into multiple branches, processes the channel attention mechanism among channels branch by branch to obtain the information hidden among channels, and then fuses and stacks multiple branches, combined with the form of skip connection, which alleviates the phenomenon of gradient disappearance.

[0058] The one-stage cascaded downsampling convolutional neural network and the multi-stage cascaded upsampling convolutional neural network are both composed of multiple parallel branches. Each branch of the downsampling convolutional neural network is composed of multiple cascaded downsampling convolutional sub-modules, and each downsampling convolutional sub-module of each branch is composed of a convolutional layer, a batch normalization layer, and an activation layer in cascade.

[0059] Each branch of the upsampling convolutional neural network is composed of multiple cascaded upsampling convolutional sub-modules, and each upsampling convolutional sub-module of each branch is composed of a transposed convolutional layer, a batch normalization layer, and an activation layer in cascade. The activation layer all adopts the LeakyReLU activation function.

[0060] The described image enhancement system can process N types of image enhancement, and the network structure adopts a one-stage downsampling neural network and an N-stage upsampling network structure. Each upsampling network structure can be used to process one type of image enhancement respectively, and the computational amount can be reduced by reusing the features extracted by the downsampling neural network.

[0061] In the described image inpainting sub-module, a cross-stage fusion module is adopted to reduce the mutual inhibition effect brought about when the network model processes N types of enhancement type tasks simultaneously. The cross-stage fusion module uses two convolutional layers, which respectively receive the input of the current branch and the input of the previous stage branch, stack along the channel dimension, and then pass through a convolutional layer to process the features from different stages to achieve cross-stage feature fusion. The cross-stage fusion module first stacks the features with the channel number of ch in the current stage and the features with the channel number of ch in the previous stage to obtain features with the channel number of 2×ch, then extracts the information therein through a convolutional layer, condenses the features, obtains features with the channel number of ch, and continues to send them to the network of the current stage.

[0062] The generated samples output by the generator network are discriminated by the discriminator network. The generator network is optimized by minimizing the difference loss function between the generated samples and the real samples, so that the generator network can generate generated samples that are sufficient to deceive the discriminator.

[0063] The discriminator adopts Patch-GAN and comprehensively discriminates the quality of the output samples of the generator network through the global discriminator and the local discriminator.

[0064] The discriminator network outputs the discrimination result of whether the generated samples belong to the real samples. By optimizing the ability of the discriminator network to distinguish true from false, the discriminator network puts forward higher requirements for the generator network. The discriminator network and the generator network promote and confront each other and make progress together.

[0065] An image enhancement method based on ECO-GAN is as follows:

[0066] Step 1: Collect the original image, analyze the type of image enhancement required for the original image, and determine the number of segments N of the upsampling convolutional neural network.

[0067] Step 2: Use image enhancement software to make a paired data set (original image - real image) for ECO-GAN to learn.

[0068] Step 3: Obtain generated samples through the generator network of the image enhancement system based on ECO-GAN.

[0069] The described downsampling convolutional neural network consists of multiple branches, each branch includes multiple convolutional modules, and the convolutional module is a combination of a convolutional layer, a batch normalization layer, and an activation layer. The activation function of all activation layers uses LeakyReLU. The batch normalization layer normalizes the features extracted by the convolutional layer in the smallest batch and then sends them to the activation layer. The activation layer uses the LeakyReLU function to activate the normalized features and then sends the image features to the next convolutional module. The downsampling convolutional neural network processes the channel attention mechanism between channels branch by branch, mines the information hidden between channels, and then fuses and stacks multiple branches, combined with the form of skip connection, which alleviates the phenomenon of gradient disappearance.

[0070] The described upsampling convolutional neural network has N segments, and the image is restored according to the image features extracted by the downsampling convolutional neural network. Each segment of the upsampling convolutional neural network corresponds to an image enhancement task. Each segment of the upsampling convolutional neural network has multiple branches, and each branch includes multiple transposed convolutional modules. The transposed convolutional module is a combination of a transposed convolutional layer, a batch normalization layer, and an activation layer. The number of transposed convolutional modules is the same as that of the convolutional modules of the downsampling convolutional neural network. The activation function of all activation layers uses LeakyReLU. The batch normalization layer normalizes the features extracted by the convolutional neural network in the smallest batch and then sends them to the next layer. A cross-stage fusion module is used to reduce the mutual inhibition effect brought by the network model when processing multiple enhancement type tasks simultaneously. The image features pass through N segments of the upsampling convolutional neural network and the cross-stage fusion module, and the finally restored image is obtained through the Nth segment of the upsampling convolutional neural network, which is called the generated sample.

[0071] Step 4: The generated sample is discriminated by the discriminator network, and the generator network is optimized by minimizing the difference loss function between the generated sample and the real sample, so that the generator network generates a generated sample that is sufficient to deceive the discriminator.

[0072] The discriminator network outputs the discrimination result of whether the generated sample belongs to the real sample. By optimizing the ability of the discriminator network to distinguish true from false, the discriminator network puts forward higher requirements for the generator network. The discriminator network and the generator network promote and confront each other and make progress together.

[0073] The generated sample and the real image are input into the discriminator network, and the discriminator feature extraction sub-module is used to extract features and send them into the classifier, and the classifier discriminates the authenticity of the input image.

[0074] Each image enhancement task corresponds to a set of generator loss functions. According to the least square loss of the content gap between the generated sample and the real sample, the local least square loss and the global least square loss of the discriminator's discrimination of the authenticity of the generated sample constitute the discriminator loss function:

[0075]

[0076]

[0077] Among them, I raw and I gt respectively represent the original image and the real image. L content represents the content loss between the generated sample and the real sample. c, h, and w are the number of channels, height, and width of the output image respectively, p represents a pixel point, and G represents the generator model.

[0078] L adv represents the adversarial loss generated by the discriminator to distinguish the authenticity of the input sample. and respectively represent the local discriminant loss and the global discriminant loss. D and G respectively represent the discriminator model and the generator model, and E represents taking the expectation of the input random variable.

[0079] By minimizing the generator loss and using the gradient descent algorithm for solution, the generator network is enabled to generate generated samples that are sufficient to deceive the discriminator.

[0080] The discriminator uses the Markov discriminator Patch-GAN, and constructs the discriminator loss function through the least square loss of the global discriminator and the local discriminator to distinguish the authenticity of the generated sample. By minimizing the discriminator loss and using the gradient descent algorithm for solution, the ability of the discriminator network to distinguish authenticity is improved. By continuously optimizing the ability of the discriminator network to distinguish authenticity, the discriminator network puts forward higher requirements for the generator network. The discriminator network and the generator network promote and confront each other and make common progress.

[0081] Step 5: Complete the enhancement of the image to be processed through the trained generator network.

[0082] Input the original image into the generator network of the image enhancement system based on ECO-GAN, and use a cascaded downsampling convolutional neural network in the generator feature extraction sub-module to extract features from the original image;

[0083] The paired data sets in Step 2 are made using the software Adobe Lightroom, and the format of the required data sets is uniformly.jpg images.

[0084] The overall model structure diagram of the present invention will be introduced in detail below.

[0085] Please refer to Figure 1 , Figure 1 which is the overall model structure diagram of an image enhancement method based on ECO-GAN provided by an embodiment of the present invention:

[0086] 101. Downsampling Convolutional Neural Network

[0087] Among them, in the downsampling convolutional neural network of the present invention, the input image can be a grayscale image or a color image, and the image extracts features through a number of downsampling convolutional modules. Each convolutional module consists of a convolutional layer, a batch normalization layer, and an activation layer, and the number of convolutional modules can be determined according to the actual application scenario.

[0088] 102. Upsampling Convolutional Neural Network

[0089] Among them, in the upsampling convolutional neural network of the present invention, the extracted features can pass through N upsampling branches respectively. The features restore the image through a number of upsampling transposed convolutional modules. Each transposed convolutional module consists of a transposed convolutional layer, a batch normalization layer, and an activation layer, and the number of transposed convolutional modules is the same as the number of downsampling convolutional modules.

[0090] 103. Cross-Stage Fusion Module

[0091] Among them, in the upsampling convolutional neural network of the present invention, a cross-stage fusion module is adopted, which is located in the middle of the current stage and the previous stage. It adopts a fully convolutional form and alleviates the mutual inhibition effect between different image enhancement branches by integrating features of different stages.

[0092] 104. Global Discriminator and Local Discriminator

[0093] The discriminator discriminates whether the input image belongs to a real image, and then guides the generator to generate generated samples that are sufficiently confusing to the discriminator. Among them, in the discriminator of the present invention, a local discriminator and a global discriminator are jointly used to discriminate the input image, and thus, targeted enhancement can be achieved for both the local and global parts of the image.

[0094] Please refer to Figure 2 , Figure 2 which is the training flowchart of an image enhancement method based on ECO-GAN provided by an embodiment of the present invention, including the training dataset step 200.

[0095] In the training dataset step 200, step 201 is executed to obtain the original image, analyze the required image enhancement type, and determine the number of segments of the upsampling branch.

[0096] Step 202 is executed to make a paired dataset using image enhancement software to obtain paired original images and real images for ECO-GAN to learn.

[0097] Step 203 is executed, and the original image is input into the generator, and after various types of image enhancement, the final generated sample is obtained.

[0098] Step 204 is executed. The real image and the generated sample are input into the discriminator. After being discriminated by the discriminator, an evaluation is made on the authenticity of the generated sample.

[0099] Please refer to Figure 3 , Figure 3 which is an image enhancement method based on ECO-GAN provided by an embodiment of the present invention, and includes step 300 of the test data set.

[0100] In step 300 of the test data set, step 301 is executed to obtain the original image.

[0101] Step 302 is executed. The original image is input into the generator, and through various types of image enhancement, the final generated sample is obtained.

[0102] Preferred Embodiment 1

[0103] Please refer to Figure 4 , Figure 4 which is a schematic diagram of the overall process of an image enhancement method based on ECO-GAN provided by an embodiment of the present invention.

[0104] Step 401 is executed to obtain the original image and analyze the required type of image enhancement.

[0105] Step 402 is executed to make a paired data set using image enhancement software for ECO-GAN learning. The original image is input into the generator.

[0106] Execute step 403 to extract features from the image using a segment of downsampling convolutional neural network. This segment of downsampling convolutional neural network consists of multiple branches, and each branch is a combination of multiple convolutional layers, batch normalization layers, and activation layers. The activation function of all activation layers adopts LeakyReLU. The batch normalization layer normalizes the features extracted by the convolutional neural network in terms of the minimum batch, and then sends them to the next layer. The activation layer uses the LeakyReLU function to activate the features after batch normalization, and then sends the image features to the next layer. The downsampling convolutional neural network processes the channel attention mechanism between channels branch by branch, mines the information hidden between channels, and then fuses and stacks multiple branches, combined with the form of skip connection, alleviating the vanishing gradient phenomenon. Use three segments of upsampling convolutional neural network to repair the image according to the extracted image features. Each of these three segments of upsampling convolutional neural network corresponds to an image enhancement task, and each segment has multiple branches, and each branch is a combination of multiple deconvolution layers, batch normalization layers, and activation layers. The activation function of all activation layers adopts LeakyReLU. The batch normalization layer normalizes the features extracted by the convolutional neural network in terms of the minimum batch, and then sends them to the next layer. The image enhancement method adopts a cross-stage fusion module to reduce the mutual inhibition effect brought about by the network model when processing multiple enhancement type tasks simultaneously. The image features pass through three segments of upsampling convolutional neural network and the cross-stage fusion module, and the finally restored image is obtained from the third segment, which is called the generated sample.

[0107] Execute step 404. The generated sample is input into the discriminator, and a segment of downsampling convolutional neural network is used to extract features and then passed to the subsequent classifier. The classifier discriminates the authenticity of the input image. The model adopted by the preferred embodiment has a set of generator loss functions for each of the three image enhancement targets. The generator loss function is constructed based on the least square loss between the generated sample and the real sample and the cross-entropy loss of the discriminator for discriminating the authenticity of the generated sample. By minimizing the generator loss and using the gradient descent algorithm for solution, the generator network is enabled to generate generated samples that are sufficient to deceive the discriminator. The discriminator adopts Patch-GAN, and the discriminator loss function is constructed based on the cross-entropy loss of the global discriminator and the local discriminator for discriminating the authenticity of the generated sample. By minimizing the discriminator loss and using the gradient descent algorithm for solution, the ability of the discriminator network to distinguish authenticity is improved. By continuously optimizing the ability of the discriminator network to distinguish authenticity, the discriminator network puts forward higher requirements for the generator network. The discriminator network and the generator network promote and confront each other and make progress together.

[0108] It should be noted that the above embodiments can be freely combined as needed. The above description only details the preferred embodiments and principles of the present invention, but does not limit the present invention. For those of ordinary skill in the art, according to the idea provided by the present invention, there will be changes in the specific implementation manners, and these changes should also be regarded as the protection scope of the present invention.

Claims

1. An image enhancement system based on ECO-GAN, characterized in that, It includes a generator network and a discriminator network: The generator network includes a generator feature extraction sub-module and an image inpainting sub-module, which extract features from and inpaint the input image during the training phase and output generated samples that are sufficient to deceive the discriminator. The discriminator network includes a discriminator feature extraction sub-module and a classifier, which discriminate the authenticity of the generated samples during the training phase; The generator feature extraction sub-module: Located at the input end of the generator network, it uses a cascaded downsampling convolutional neural network to extract features from the original image; The image inpainting sub-module: Located at the output end of the generator network, it uses an N-stage cascaded upsampling convolutional neural network to complete various types of enhancement tasks respectively according to the features extracted by the generator feature extraction sub-module, and outputs generated samples for the discriminator to discriminate authenticity; The discriminator feature extraction sub-module: Located at the input end of the discriminator network, it uses a cascaded VGG19 network to extract features from the input real image and generated samples; The classifier: Located at the output end of the discriminator network, it discriminates the authenticity of the generated samples by comparing the output of the generated samples and the real image after passing through the discriminator feature extraction sub-module; Both the downsampling convolutional neural network and the upsampling convolutional neural network use ResNeSt network modules. The ResNeSt network module divides the multi-channels into multiple branches, processes the channel attention mechanism among the channels branch by branch to obtain the information hidden among the channels, and then fuses and stacks the multiple branches, combined with the form of skip connection, which alleviates the gradient disappearance phenomenon; The cascaded downsampling convolutional neural network and the multi-stage cascaded upsampling convolutional neural network are both composed of multiple parallel branches; each branch of the downsampling convolutional neural network is composed of multiple cascaded downsampling convolutional sub-modules, and each downsampling convolutional sub-module of each branch is composed of a convolutional layer, a batch normalization layer, and an activation layer in cascade; Each branch of the upsampling convolutional neural network is composed of multiple cascaded upsampling convolutional sub-modules, and each upsampling convolutional sub-module of each branch is composed of a transposed convolutional layer, a batch normalization layer, and an activation layer in cascade; The activation layers all use the LeakyReLU activation function; A cross-stage fusion module is adopted in the image inpainting sub-module to reduce the mutual inhibition effect brought by the network model when simultaneously processing N types of enhancement tasks; The cross-stage fusion module uses two convolutional layers, which respectively receive the input of the current branch and the input of the previous stage branch, stack them along the channel dimension, and then process the features from different stages through a convolutional layer to achieve cross-stage feature fusion; The cross-stage fusion module first stacks the features with the number of channels ch in the current stage and the features with the number of channels ch in the previous stage to obtain features with the number of channels 2×ch, then extracts the information therein through a convolutional layer, condenses the features, obtains features with the number of channels ch, and continues to send them to the network in the current stage; The discriminator adopts Patch-GAN, and comprehensively discriminates the quality of the output samples of the generator network through a global discriminator and a local discriminator.

2. The image enhancement system based on ECO-GAN according to claim 1, characterized in that, The generated samples output by the generator network are discriminated by the discriminator network. The generator network is optimized by minimizing the difference loss function between the generated samples and the real samples, so that the generator network can generate generated samples that are sufficient to deceive the discriminator.

3. An image enhancement method based on ECO-GAN, characterized in that, The steps are as follows: Step 1: Collect the original image, analyze the type of image enhancement required for the original image, and determine the number of segments N of the upsampling convolutional neural network. Step 2: Use image enhancement software to make a paired data set for ECO-GAN to learn. Step 3: Obtain generated samples through the generator network of the image enhancement system based on ECO-GAN. Step 4: The generated samples are discriminated by the discriminator network. The generator network is optimized by minimizing the difference loss function between the generated samples and the real samples, so that the generator network can generate generated samples that are sufficient to deceive the discriminator. The discriminator network outputs a discrimination result on whether the generated samples belong to real samples. By optimizing the ability of the discriminator network to distinguish true from false, the discriminator network puts forward higher requirements for the generator network. The discriminator network and the generator network promote and confront each other and make common progress. Step 5: Complete the enhancement of the image to be processed through the trained generator network. Input the original image into the generator network of the image enhancement system based on ECO-GAN, and use a cascaded downsampling convolutional neural network in one segment of the generator feature extraction sub-module to extract features from the original image. The specific method of Step 3 is as follows: The downsampling convolutional neural network is composed of multiple branches, and each branch includes multiple convolutional modules. The convolutional module is a combination of a convolutional layer, a batch normalization layer, and an activation layer; the activation function of all activation layers adopts LeakyReLU. The batch normalization layer normalizes the features extracted by the convolutional layer according to the minimum batch, and then sends them to the activation layer. The activation layer uses the LeakyReLU function to activate the batch-normalized features, and then sends the image features to the next convolutional module; the downsampling convolutional neural network processes the channel attention mechanism between channels branch by branch, mines the information hidden between channels, and then fuses and stacks multiple branches, combined with the form of skip connection, to alleviate the phenomenon of gradient disappearance. ​ A cross-stage fusion module is adopted to reduce the mutual inhibition effect brought about when the network model processes multiple enhancement type tasks simultaneously; the image features pass through an N-stage upsampling convolutional neural network and the cross-stage fusion module, and the finally restored image is obtained through the Nth stage upsampling convolutional neural network, which is called the generated sample; The specific method of step 4 is as follows: The generated sample and the real image are input into the discriminator network. The discriminator feature extraction sub-module is used to extract features and then passed into the classifier, and the classifier discriminates the authenticity of the input image; Each image enhancement task corresponds to a set of generator loss functions. The discriminator loss function is composed of the least squares loss of the content gap between the generated sample and the real sample, the local least squares loss and the global least squares loss of the discriminator to judge the authenticity of the generated sample: Among them, I raw and I gt respectively represent the original image and the real image; L content represents the content loss between the generated sample and the real sample. c, h, and w are the number of channels, height, and width of the output image respectively, p represents a pixel point, and G represents the generator model; L adv represents the adversarial loss generated by the discriminator to distinguish the authenticity of the input samples, and respectively represent the local discriminant loss and the global discriminant loss. D and G respectively represent the discriminator model and the generator model, and E represents taking the expectation of the input random variable; By minimizing the generator loss, the gradient descent algorithm is used for solution, so that the generator network generates generated samples that are sufficient to deceive the discriminator; The discriminator adopts the Markov discriminator Patch-GAN, and the discriminator loss function is constructed by the least squares loss of the global discriminator and the local discriminator to judge the authenticity of the generated sample; by minimizing the discriminator loss, the gradient descent algorithm is used for solution to improve the ability of the discriminator network to distinguish authenticity; by continuously optimizing the ability of the discriminator network to distinguish authenticity, the discriminator network puts forward higher requirements for the generator network, and the discriminator network and the generator network promote and oppose each other and make common progress.

4. An image enhancement method based on ECO-GAN according to claim 3, characterized in that The paired data sets in step 2 are made using the software Adobe Lightroom, and the format of the required data sets is uniformly.jpg pictures.

Images