Image enhancement method and device, electronic equipment and storage medium

Abstract

This invention provides an image enhancement method, apparatus, electronic device, and storage medium. The method includes: acquiring a target image; inputting the target image into an image enhancement model to obtain an enhanced image of the target image; the image enhancement model is obtained by unsupervised training of a generative adversarial network (GAN) based on adversarial loss and contrastive loss; the contrastive loss is obtained by contrastive learning based on a first sample image and a second sample image in a first sample image set, as well as a sample enhanced image of the first sample image output by the generator in the GAN; the adversarial loss is obtained by adversarial learning based on the sample enhanced image and a third sample image in the second sample image set. This invention ensures that the enhanced image meets high-quality standards and that the enhanced image has a strong correlation with the image content of the original image, thus enabling accurate and efficient image enhancement.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to an image enhancement method, apparatus, electronic device, and storage medium. Background Technology

[0002] In recent years, hardware imaging equipment has developed rapidly, and the high-quality images obtained through it can contain more information. However, due to the complexity of the imaging environment, the images directly obtained by imaging equipment are often degraded. For example, underexposure will produce low-light images, poor scattering will cause image blurring, and equipment errors will cause image noise. Therefore, how to provide a practical and highly generalizable image enhancement method is of great research significance.

[0003] In current technologies, deep learning has made remarkable progress in various image processing tasks. Typically, deep learning models are trained in a supervised manner based on pairs of image samples (i.e., images with identical content but different image quality). This involves constructing complex nonlinear networks to learn the mapping relationship between low-quality and high-quality sample images in the image sample pair, thereby achieving image enhancement. However, due to the complexity of the imaging environment, it is difficult to collect paired supervised data (i.e., image sample pairs), leading to insufficient training data. Therefore, it is difficult to obtain deep learning models that can accurately perform image enhancement, making it difficult to achieve accurate and efficient image enhancement. Summary of the Invention

[0004] This invention provides an image enhancement method, apparatus, electronic device, and storage medium to address the shortcomings of existing technologies where the complexity of the imaging environment makes it difficult to collect paired supervisory data, resulting in insufficient training data and difficulty in accurately and efficiently achieving image enhancement.

[0005] This invention provides an image enhancement method, comprising:

[0006] Acquire the target image;

[0007] The target image is input into the image enhancement model to obtain the enhanced image of the target image;

[0008] The image enhancement model is obtained by unsupervised training of a generative adversarial network based on adversarial loss and contrastive loss. The contrastive loss is obtained by contrastive learning based on the first and second sample images in the first sample image set, and the sample enhanced image of the first sample image output by the generator in the generative adversarial network. The adversarial loss is obtained by adversarial learning based on the sample enhanced image and the third sample image in the second sample image set. The quality of the images in the first sample image set is lower than the quality of the images in the second sample image set.

[0009] According to an image enhancement method provided by the present invention, the image enhancement model is trained based on the following steps:

[0010] The first sample image, the second sample image, and the sample enhancement image are input into the feature extraction model to obtain the feature map of the first sample image, the feature map of the second sample image, and the feature map of the sample enhancement image.

[0011] The contrastive loss is obtained by performing contrastive learning based on the feature map of the first sample image, the feature map of the second sample image, and the feature map of the sample enhancement image.

[0012] The enhanced sample image and the third sample image are input into the discriminator of the generative adversarial network to perform adversarial learning and obtain the adversarial loss.

[0013] Based on the adversarial loss and the contrastive loss, the parameters of the generative adversarial network are iteratively trained adversarially.

[0014] The image enhancement model is constructed based on the generator in the trained generative adversarial network. According to an image enhancement method provided by the present invention, the step of obtaining the contrastive loss by performing contrastive learning based on the feature maps of the first sample image, the second sample image, and the enhanced sample image includes:

[0015] Obtain a first similarity distance between the feature map of the first sample image and the feature map of the sample enhancement image, and a second similarity distance between the feature map of the second sample image and the feature map of the sample enhancement image;

[0016] The comparison loss is determined based on the first similarity distance and the second similarity distance; the comparison loss aims to minimize the first similarity distance and maximize the second similarity distance.

[0017] According to an image enhancement method provided by the present invention, the step of inputting the enhanced sample image and the third sample image into the discriminator of the generative adversarial network to perform adversarial learning and obtain the adversarial loss includes:

[0018] The enhanced sample image and the third sample image are input into the discriminator to obtain the discrimination result of the enhanced sample image and the discrimination result of the third sample image;

[0019] The adversarial loss is determined based on the discrimination results of the enhanced sample image and the third sample image.

[0020] According to an image enhancement method provided by the present invention, the step of iteratively training the parameters of the generative adversarial network based on the adversarial loss and the contrast loss includes:

[0021] For the current adversarial training, the parameters of the discriminator obtained from the previous adversarial training are fixed. With the goal of minimizing the fusion result between the adversarial loss and the contrastive loss, the parameters of the generator obtained from the previous adversarial training are trained to obtain the generator corresponding to the current adversarial training.

[0022] The generator corresponding to the current adversarial training is fixed, and the parameters of the discriminator obtained in the previous adversarial training are trained with the goal of maximizing the adversarial loss, so as to obtain the discriminator corresponding to the current adversarial training.

[0023] Based on the discriminator and generator corresponding to the current adversarial training, the next adversarial training is executed iteratively until the generative adversarial network meets the preset termination condition.

[0024] According to an image enhancement method provided by the present invention, the image enhancement model includes an encoder, an attention module, a residual module, and a decoder;

[0025] The step of inputting the target image into an image enhancement model to obtain the enhanced image of the target image includes:

[0026] The target image is input into the encoder for downsampling to obtain the first feature map of the target image;

[0027] The first feature map is input into the attention module for detail feature extraction to obtain the second feature map of the target image;

[0028] The second feature map is input into the residual module for residual operation to obtain the third feature map of the target image;

[0029] The third feature map is input into the decoder for upsampling to obtain the enhanced image of the target image.

[0030] According to an image enhancement method provided by the present invention, the attention module includes a channel attention unit and a spatial attention unit;

[0031] The step of inputting the first feature map into the attention module for detail feature extraction to obtain the second feature map of the target image includes:

[0032] The first feature map is input into the channel attention unit, the channel weights of the first feature map in the channel dimension are calculated, and the feature sub-maps in each channel dimension of the first feature map are adjusted according to the channel weights.

[0033] The adjusted first feature map is input into the spatial attention unit, the spatial position weight of the adjusted first feature map in the spatial dimension is calculated, and the feature sub-map in each spatial dimension of the adjusted first feature map is adjusted according to the spatial position weight to obtain the second feature map.

[0034] The present invention also provides an image enhancement apparatus, comprising:

[0035] The acquisition unit is used to acquire the target image;

[0036] An image enhancement unit is used to input the target image into an image enhancement model to obtain an enhanced image of the target image;

[0037] The image enhancement model is obtained by unsupervised training of a generative adversarial network based on adversarial loss and contrastive loss. The contrastive loss is obtained by contrastive learning based on the first and second sample images in the first sample image set, and the sample enhanced image of the first sample image output by the generator in the generative adversarial network. The adversarial loss is obtained by adversarial learning based on the sample enhanced image and the third sample image in the second sample image set. The quality of the images in the first sample image set is lower than the quality of the images in the second sample image set.

[0038] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the image enhancement method as described above.

[0039] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the image enhancement method as described above.

[0040] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the image enhancement method as described above.

[0041] The image enhancement method, apparatus, electronic device, and storage medium provided by this invention perform unsupervised training of the image enhancement model by combining contrastive loss and adversarial loss, so as to enhance the target image according to the image enhancement model. On the one hand, it does not require additional paired supervised data, effectively reducing the impact of the scarcity of paired supervised data, and the training efficiency and accuracy are higher. On the other hand, it can ensure that the image quality of the enhanced image reaches the high-quality standard, and also ensure that the image content of the enhanced image is strongly correlated with the image content of the original image before enhancement, thus realizing accurate and efficient image enhancement. Attached Figure Description

[0042] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0043] Figure 1 This is a schematic flowchart of the image enhancement method provided by the present invention;

[0044] Figure 2 This is one of the flowcharts illustrating the training method of the image enhancement model in the image enhancement method provided by this invention;

[0045] Figure 3 This is the second flowchart illustrating the training method of the image enhancement model in the image enhancement method provided by this invention;

[0046] Figure 4 This is a schematic diagram of the structure of the generative adversarial network in the image enhancement method provided by the present invention;

[0047] Figure 5 This is a schematic diagram of the attention module in the image enhancement method provided by the present invention;

[0048] Figure 6 This is a schematic diagram of the image enhancement device provided by the present invention;

[0049] Figure 7 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0050] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0051] In existing technologies, to enhance image quality, some researchers have proposed spatial domain-based image enhancement methods. These methods primarily rely on pixel value grayscale mapping and include image enhancement techniques such as histogram correction, grayscale transformation, and spatial domain filtering. While these methods focus on improving image contrast, the enhanced image quality often fails to meet requirements when the grayscale space distribution of the image is unreasonable.

[0052] Other researchers have proposed frequency-domain-based image enhancement methods. Frequency-domain image enhancement involves using convolution to transform the image into the frequency domain space, suppressing frequency components within a certain range while leaving other components unaffected, and finally transforming it back into the image space to achieve image enhancement. The main frequency-domain methods include low-pass or high-pass filtering, as well as homomorphic filtering. These methods aim to improve a specific defect in the image without addressing other image distortion issues such as artifacts and loss of detail.

[0053] Other scholars have proposed image enhancement methods based on prior models. By understanding the principles of image degradation, they construct models based on specific prior knowledge to restore degraded images to their original state. Commonly used models include atmospheric turbulence models, dark channel priors, and degradation models. However, the physical models constructed by these methods rely on the researchers' theoretical knowledge and require a fundamental understanding of the causes of image degradation. Furthermore, specific scenarios require specific models, and the generalization ability of these models may not meet the requirements for practicality.

[0054] To address the aforementioned issues of poor generalization ability and accuracy, some researchers have proposed deep learning-based image enhancement methods. These methods construct complex nonlinear networks to learn the mapping relationship between inputs and outputs, aiming to achieve efficient and accurate image enhancement. However, due to the complexity of the imaging environment, it is difficult to collect paired, supervised data, leading to insufficient training data. Therefore, it remains difficult to obtain deep learning models capable of accurately performing image enhancement, hindering the accurate and efficient implementation of image enhancement.

[0055] To address the aforementioned problems, this embodiment provides an image enhancement method. Figure 1 This is a flowchart illustrating the image enhancement method provided herein, as shown below. Figure 1 As shown, the method includes:

[0056] Step 101: Obtain the target image;

[0057] The target image may include images that require image enhancement, such as those requiring contrast enhancement, detail and texture enhancement, or saturation enhancement. This image may be acquired in real-time by an image acquisition device, which can be a camera, smartphone, tablet computer, or smart appliance such as a television or air conditioner. After acquiring the target image via the camera, the image acquisition device may also perform noise reduction and / or normalization processing on the target image; this embodiment does not specifically limit this.

[0058] Step 102: Input the target image into the image enhancement model to obtain the enhanced image of the target image;

[0059] The image enhancement model is obtained by unsupervised training of a generative adversarial network based on adversarial loss and contrastive loss. The contrastive loss is obtained by contrastive learning based on the first and second sample images in the first sample image set, and the sample enhanced image of the first sample image output by the generator in the generative adversarial network. The adversarial loss is obtained by adversarial learning based on the sample enhanced image and the third sample image in the second sample image set. The quality of the images in the first sample image set is lower than the quality of the images in the second sample image set.

[0060] The image enhancement model is built based on the generator in the generative adversarial network, which includes, but is not limited to, an encoder, attention module, residual module and decoder. This embodiment does not specifically limit the specific components.

[0061] The discriminator can adopt a Markov discriminator (patchGAN) structure, which outputs a two-dimensional matrix through four convolutional layers with a kernel size of 4*4 and a stride of 2.

[0062] Image enhancement models learn features from low-quality input images at multiple levels to reconstruct high-quality images, thereby achieving accurate and efficient image enhancement.

[0063] Optionally, the image augmentation model can be pre-trained before performing step 102. Specific training steps include:

[0064] First, a generative adversarial network (GAN) is created. This GAN may contain a generator G, which is used to transform the degraded image into an enhanced image (x to x). (conversion); Discriminator D, used to distinguish between the real high-quality image y and the generated image. The authenticity of the data; and the feature extraction model used to assist in comparative learning.

[0065] The generator and discriminator can be modules prepared for generative adversarial learning after parameter initialization, or they can be pre-trained modules with generative adversarial learning capabilities. Similarly, the feature extraction model can be a module prepared for feature extraction after parameter initialization, or it can be a pre-trained feature extraction model with feature extraction capabilities. This embodiment of the invention does not impose specific limitations on these aspects.

[0066] In addition, a sample image library needs to be constructed, and the degraded low-quality images in the sample image library, that is, images whose contrast, detail texture and saturation do not meet the expected quality, are composed of the first sample image set (hereinafter referred to as the X domain); the high-quality images that are expected to be obtained, that is, images whose contrast, detail texture and saturation meet the expected quality, are composed of the second sample image set (hereinafter referred to as the Y domain).

[0067] A batch of sample images is extracted from the first sample image set, including a first sample image to be enhanced and a second sample image other than the first sample image; and a third sample image is extracted from the second sample image set. It should be noted that the first sample image and the third sample image are not pre-paired, that is, the image content of the first sample image and the image content of the third sample image can be the same or different.

[0068] Assuming the batch size during training is set to n, the first sample image can be represented as x, and the corresponding augmented sample image can be represented as x. Other images in the same batch, i.e., the second sample images, can be characterized as {x1, x2, ..., x n-1}

[0069] Subsequently, the first sample image can be input into the generator of the generative adversarial network (GAN). The generator performs image enhancement on the first sample image to obtain the enhanced sample image. Then, the feature extraction model performs comparative learning on the enhanced sample image, the first sample image, and the second sample image to obtain the contrastive loss. The discriminator performs adversarial learning on the third sample image in the set of the enhanced sample image and the second sample image to obtain the adversarial loss. Based on the adversarial loss and the contrastive loss, the GAN is trained in an unsupervised manner.

[0070] Adversarial loss is used to ensure that the generated image quality is close to that of the image in the Y domain; however, because the training data is unpaired, the enhanced image lacks the constraint of the true value, so the generator G may map the input original image before enhancement to other outputs that conform to the Y domain. Therefore, this embodiment uses contrastive loss on top of adversarial loss to effectively ensure that the enhanced image and the original image before enhancement have a strong correlation in image content.

[0071] This yields a trained generative adversarial network, from which the generators are used to construct an image enhancement model.

[0072] After acquiring the target image, feature extraction and data reconstruction can be performed on the target image based on the image enhancement model, thereby automatically outputting the enhanced image of the target image, i.e., the high-quality target image.

[0073] Understandably, in response to the problem of limited paired image samples in certain scenarios, this embodiment adopts an unsupervised training method based on contrast loss and adversarial loss. This method can train the image enhancement model on unpaired data, effectively improving the practicality of the method. In principle, it can train the model on an unsupervised training set, enhancing images from multiple aspects such as contrast enhancement and detail improvement, making image enhancement flexible and generalizable.

[0074] The image enhancement method provided in this embodiment uses unsupervised training of the image enhancement model by combining contrastive loss and adversarial loss to enhance the target image. On the one hand, it eliminates the need to acquire additional paired supervised data, effectively reducing the impact of the scarcity of paired supervised data and improving training efficiency and accuracy. On the other hand, it ensures that the enhanced image meets high-quality standards and that the enhanced image has a strong correlation with the original image before enhancement, thus achieving accurate and efficient image enhancement.

[0075] In some embodiments, Figure 2 This is one of the flowcharts illustrating the training method for the image enhancement model provided in this embodiment, including the following steps:

[0076] Step 201: Input the first sample image, the second sample image, and the sample enhancement image into the feature extraction model to obtain the feature map of the first sample image, the feature map of the second sample image, and the feature map of the sample enhancement image;

[0077] Step 202: Based on the feature map of the first sample image, the feature map of the second sample image, and the feature map of the sample enhancement image, perform contrastive learning to obtain the contrastive loss;

[0078] Step 203: Input the sample augmentation image and the third sample image into the discriminator of the generative adversarial network to perform adversarial learning and obtain the adversarial loss;

[0079] Step 204: Based on the adversarial loss and the contrastive loss, iteratively perform adversarial training on the parameters of the generative adversarial network;

[0080] Step 205: Construct the image enhancement model based on the generator in the trained generative adversarial network.

[0081] The feature extraction model can be generated based on a pre-trained network of VGG-19 (Visual Geometry Group 19, a 19-layer visual geometry group).

[0082] Optionally, the training steps for the image enhancement model specifically include:

[0083] During the contrastive learning process, the image is augmented with samples from the first sample image x. As an anchor point, the first sample image x, whose data content is consistent with it, can be used as a positive sample, and the second sample images {x1, x2, ..., x...} in the same batch... n-1} is defined as a negative sample. By learning through comparison between the anchor point and positive and negative samples, the distance between the anchor point and the positive sample should become smaller and smaller, and the distance with the negative sample should be as large as possible. This achieves consistency in the image content before and after input and output without the constraint of the true value.

[0084] Therefore, the first sample image, the second sample image, and the sample enhancement image are respectively input into the feature extraction model, and the feature extraction model performs multi-scale feature extraction on the first sample image, the second sample image, and the sample enhancement image to obtain the feature map of the first sample image, the feature map of the second sample image, and the feature map of the sample enhancement image.

[0085] Then, the contrast loss is determined by the similarity distance between the feature maps of the first sample image and the sample augmentation image, and the similarity distance between the feature maps of the second sample image and the sample augmentation image.

[0086] During adversarial learning, the generator G aims to minimize the adversarial loss in order to generate high-contrast, high-quality images that closely approximate the target domain Y. The goal of discriminator D is to maximize the adversarial loss and improve the ability to distinguish between true and false.

[0087] Optionally, the sample augmented image and the third sample image are input into the discriminator, which performs adversarial learning to obtain the adversarial loss.

[0088] It should be noted that the execution order of steps 201 and 203 can be parallel or one can be executed first and the other later. This embodiment does not make a specific limitation on this.

[0089] Given the contrastive loss and adversarial loss, the generative adversarial network can be trained in an unsupervised manner to obtain the trained generative adversarial network.

[0090] This yields a trained generative adversarial network, from which the generators are used to construct an image enhancement model.

[0091] The image enhancement method provided in this embodiment performs unsupervised adversarial training on the image enhancement model by combining contrast loss and adversarial loss. This ensures that the enhanced image generated by the trained image enhancement model meets high-quality standards and has a strong correlation with the image content of the original image before enhancement, effectively ensuring the effectiveness and accuracy of image enhancement.

[0092] In some embodiments, step 202 further includes:

[0093] Obtain a first similarity distance between the feature map of the first sample image and the feature map of the sample enhancement image, and a second similarity distance between the feature map of the second sample image and the feature map of the sample enhancement image;

[0094] The comparison loss is determined based on the first similarity distance and the second similarity distance; the comparison loss aims to minimize the first similarity distance and maximize the second similarity distance.

[0095] The similarity distance can be the l-norm distance, which can be set according to actual needs.

[0096] Optionally, the difference between the feature map of the enhanced image and the feature map of the first sample image is calculated, and the l-norm distance of this difference is calculated to obtain the first similarity distance; the difference between the feature map of the enhanced image and the feature map of the second sample image is calculated, and the l-norm distance of this difference is calculated to obtain the second similarity distance. Then, the first similarity distance is divided by the sum of the second similarity distances corresponding to all the second sample images to determine the contrast loss. The specific calculation formula is as follows:

[0097]

[0098] Among them, L CL For comparison of losses, L represents the number of feature layers in the feature extraction model; w l F represents the weight coefficients of the l-th feature layer. l (·) represents the feature map output by the l-th feature layer; x, and x i These represent the first sample image, the augmented image of the first sample image, and the second sample image, respectively; ||·|| represents the l-norm distance. By minimizing L... CL Implement comparison constraints.

[0099] This embodiment achieves a strong correlation between the enhanced image generated by the generator and the original image before enhancement, without relying on paired sample image pairs, effectively ensuring the effectiveness and accuracy of image enhancement.

[0100] In some embodiments, step 203 further includes:

[0101] The enhanced sample image and the third sample image are input into the discriminator to obtain the discrimination result of the enhanced sample image and the discrimination result of the third sample image;

[0102] The adversarial loss is determined based on the discrimination results of the enhanced sample image and the third sample image.

[0103] Optionally, during adversarial learning, the generator G aims to minimize the adversarial loss in order to generate high-contrast, high-quality images that closely approximate the target domain Y. The goal of the discriminator D is to maximize the adversarial loss to improve its ability to distinguish between true and false signals. The specific calculation formula is as follows:

[0104] L GAN (G,D,x,y)=E y～f(y) [logD(y)]+E x～f(x) [log(1-D(G(x)))];

[0105] Among them, L GAN (·) represents the adversarial loss; x and y follow the true distributions f(x) and f(y) of the sample images, E(·) represents the mathematical expectation of the distribution function, D(y) represents the discrimination result for the third sample image, and D(G(x)) represents the discrimination result for the sample-enhanced image of the first sample image.

[0106] In some embodiments, such as Figure 3 The diagram shown is a second flowchart illustrating the training method for the image enhancement model provided in this embodiment. Step 204 further includes:

[0107] Step 301: For the current adversarial training, fix the parameters of the discriminator obtained from the previous adversarial training, and train the parameters of the generator obtained from the previous adversarial training with the goal of minimizing the fusion result between the adversarial loss and the contrast loss, so as to obtain the generator corresponding to the current adversarial training.

[0108] Step 302: Fix the generator corresponding to the current adversarial training, and train the parameters of the discriminator obtained in the previous adversarial training with the goal of maximizing the adversarial loss, to obtain the discriminator corresponding to the current adversarial training.

[0109] Step 303: Based on the discriminator and the generator corresponding to the current adversarial training, iteratively execute the next adversarial training, that is, return to step 301 until the generative adversarial network meets the preset termination condition.

[0110] Step 304: Obtain the trained generative adversarial network.

[0111] Optionally, in the loss function part, the discriminator D maximizes the adversarial loss L. GAN (·), optimize the model parameters to improve the discriminator D's ability to distinguish between true and false (y). The ability of generator G to minimize adversarial loss L. GAN (·), optimize model parameters to make the generated It gets closer to a high-quality image. Meanwhile, the generator G compares the loss L... CL Under the constraints, By preserving the original image content, the effectiveness and accuracy of image enhancement can be effectively improved.

[0112] For the current adversarial training session, the adversarial training process includes:

[0113] When training the generator, the parameters of the discriminator obtained from the previous adversarial training are fixed, and the parameters of the generator obtained from the previous adversarial training are trained, with the goal of minimizing the adversarial loss L. GAN (·) and the counter-loss L GAN The sum of (·) is the target.

[0114] When training the discriminator, the generator obtained from the current adversarial training is fixed, and the parameters of the discriminator obtained from the previous adversarial training are trained, with the goal of maximizing the adversarial loss L. GAN (·) is the target;

[0115] By iteratively performing the adversarial training described above, the model parameters of the generative adversarial network (GAN) are continuously optimized until the GAN meets a preset termination condition, at which point the adversarial training stops, and the trained GAN is obtained. The preset termination condition may include reaching the maximum number of iterations and / or the generator's performance meeting performance requirements, etc., but this embodiment does not specify these conditions.

[0116] In this embodiment, adversarial training is performed using adversarial loss and contrastive loss to quickly and accurately obtain an image enhancement model that can effectively and accurately perform image enhancement.

[0117] In some embodiments, the image enhancement model includes an encoder, an attention module, a residual module, and a decoder;

[0118] The step of inputting the target image into an image enhancement model to obtain the enhanced image of the target image includes:

[0119] The target image is input into the encoder for downsampling to obtain the first feature map of the target image;

[0120] The first feature map is input into the attention module for detail feature extraction to obtain the second feature map of the target image;

[0121] The second feature map is input into the residual module for residual operation to obtain the third feature map of the target image;

[0122] The third feature map is input into the decoder for upsampling to obtain the enhanced image of the target image.

[0123] like Figure 4 The diagram shown is a schematic representation of the structure of a generative adversarial network; Figure 4 As shown, the purpose of image enhancement models is to transform low-quality images into high-quality images, including but not limited to an encoder (also known as a downsampling layer) for extracting input features, a deep network for mapping transformation, namely the attention module and the residual module, and a decoder (also known as an upsampling layer) for decoding the deep features.

[0124] The network structures of the encoder, attention module, residual module, and decoder can be built based on various neural networks, and the specific configuration can be tailored to actual needs. For example, the encoder can be built based on convolutional layers and pooling layers, or only convolutional layers; the attention module can be built based on pooling layers, convolutional layers, and fully connected layers; the residual module includes multiple stacked residual units, each based on at least one set of stacked structures consisting of convolutional layers and normalization layers, such as a convolutional layer + normalization layer + ReLU (Rectified Linear Unit) activation layer + convolutional layer + normalization layer; the decoding module is built based on multiple deconvolutional layers, such as two layers. The image enhancement model constructed using the above structures can efficiently and accurately enhance images.

[0125] Optionally, step 102 further includes:

[0126] The distorted target image is read according to the image resolution and requirements; the size of the target image can be set adaptively according to the actual application scenario, such as an image size of 3*256*256, where 3 represents the three color channels of the image, namely R (red), G (green) and B (blue), and 256*256 represents the width and height of the image.

[0127] The encoder performs preliminary feature extraction on the input target image and maps it to a high-dimensional space. Specifically, the encoder can use convolution operations to increase the number of channels in the target image, such as doubling the original number of channels, and then use max pooling operations to reduce the size of the target image, such as reducing the size of the target image to half its original size, in order to downsample the target image and output the first feature map, with an image size of 256*64*64; or the encoder can use convolution operations to widen the number of channels in the target image while reducing the image size, in order to downsample the target image and output the first feature map, with an image size of 256*64*64.

[0128] To focus on more important details in the image, enhance model flexibility, and improve the generalization performance of the method, the attention module then extracts detailed features from the first feature map to extract more detailed channel features and spatial texture features from the target image, resulting in the second feature map of the target image.

[0129] Subsequently, the second feature map is subjected to multiple layers of residual operations through each residual block in the residual module to complete the feature mapping and obtain the third feature map.

[0130] Next, the decoder uses multiple deconvolution layers to restore the third feature map to the same size (256*256) as the original input target image, while reducing the number of channels to 3, and outputting an enhanced image of the target image.

[0131] In this embodiment, the image enhancement model, through an encoder, attention module, residual module, and decoder, can efficiently and accurately extract more detailed features from the target image, thereby making the image enhancement results more effective and accurate.

[0132] In some embodiments, the attention module includes a channel attention unit and a spatial attention unit;

[0133] The step of inputting the first feature map into the attention module for detail feature extraction to obtain the second feature map of the target image includes:

[0134] The first feature map is input into the channel attention unit, the channel weights of the first feature map in the channel dimension are calculated, and the feature sub-maps in each channel dimension of the first feature map are adjusted according to the channel weights.

[0135] The adjusted first feature map is input into the spatial attention unit, the spatial position weight of the adjusted first feature map in the spatial dimension is calculated, and the feature sub-map in each spatial dimension of the adjusted first feature map is adjusted according to the spatial position weight to obtain the second feature map.

[0136] like Figure 5 The diagram shown is a structural schematic of the attention module; Figure 5 As shown, the attention module is constructed from the CSAM (Channel and Spatial Attention Module) module, which includes channel attention units and spatial attention units.

[0137] Channel attention units are used to extract channel features, including but not limited to global average pooling, multi-layer fully connected layers, and feature fusion layers.

[0138] Spatial attention units are used to extract spatial detail features, including but not limited to global average pooling, convolutional layers, and feature fusion layers.

[0139] Optionally, after obtaining the first feature map, the first feature map can be passed through a channel attention unit to learn more important channel information.

[0140] First, the first feature map is passed through a global average pooling layer to output a 256*1*1 feature map. Then, it passes through two fully connected layers, with the number of channels changing from 256 to 16 and then back to 256. After the transformation, the channel weights are obtained through the Sigmoid activation function. The channel weights are then multiplied element-wise with the original input first feature map to adjust the feature submaps in each channel dimension of the first feature map, resulting in a first feature map with redistributed channel weights.

[0141] Then, the adjusted first feature map output by the channel attention unit is passed through the spatial attention unit to focus on target information and texture details that are more important in the spatial dimension.

[0142] Optionally, the adjusted first feature map is first subjected to global average pooling to obtain a 1*64*64 feature map, and then subjected to 1*1 convolution and Sigmoid activation function to obtain spatial position weights of [0-1]. The spatial position weights are then multiplied element-wise with the adjusted first feature map to adjust the feature sub-maps in each spatial dimension of the first feature map, resulting in a second feature map with different spatial position weights.

[0143] Compared to existing unsupervised image enhancement methods that do not preserve image details and textures well and are prone to losing important information, this embodiment combines spatial attention and channel attention mechanisms to better focus on texture details from both spatial and channel perspectives. This enhances the ability to perform image enhancement on the target image in both channels and in space, thereby improving the quality of image enhancement.

[0144] The image enhancement apparatus provided by the present invention will be described below. The image enhancement apparatus described below can be referred to in correspondence with the image enhancement method described above.

[0145] like Figure 6 As shown, this embodiment provides an image enhancement device, which includes an acquisition unit 601 and an image enhancement unit 602, wherein:

[0146] The acquisition unit 601 is used to acquire the target image;

[0147] The image enhancement unit 602 is used to input the target image into the image enhancement model to obtain the enhanced image of the target image;

[0148] The image enhancement model is obtained by unsupervised training of a generative adversarial network based on adversarial loss and contrastive loss. The contrastive loss is obtained by contrastive learning based on the first and second sample images in the first sample image set, and the sample enhanced image of the first sample image output by the generator in the generative adversarial network. The adversarial loss is obtained by adversarial learning based on the sample enhanced image and the third sample image in the second sample image set. The quality of the images in the first sample image set is lower than the quality of the images in the second sample image set.

[0149] In some embodiments, the device further includes a training unit for:

[0150] The first sample image, the second sample image, and the sample enhancement image are input into the feature extraction model to obtain the feature map of the first sample image, the feature map of the second sample image, and the feature map of the sample enhancement image.

[0151] The contrastive loss is obtained by performing contrastive learning based on the feature map of the first sample image, the feature map of the second sample image, and the feature map of the sample enhancement image.

[0152] The enhanced sample image and the third sample image are input into the discriminator of the generative adversarial network to perform adversarial learning and obtain the adversarial loss.

[0153] Based on the adversarial loss and the contrastive loss, the parameters of the generative adversarial network are iteratively trained adversarially.

[0154] The image enhancement model is constructed based on the generator in the trained generative adversarial network.

[0155] In some embodiments, the training unit is further configured to:

[0156] Obtain a first similarity distance between the feature map of the first sample image and the feature map of the sample enhancement image, and a second similarity distance between the feature map of the second sample image and the feature map of the sample enhancement image;

[0157] The comparison loss is determined based on the first similarity distance and the second similarity distance; the comparison loss aims to minimize the first similarity distance and maximize the second similarity distance.

[0158] In some embodiments, the training unit is further configured to:

[0159] The enhanced sample image and the third sample image are input into the discriminator to obtain the discrimination result of the enhanced sample image and the discrimination result of the third sample image;

[0160] The adversarial loss is determined based on the discrimination results of the enhanced sample image and the third sample image.

[0161] In some embodiments, the training unit is further configured to:

[0162] For the current adversarial training, the parameters of the discriminator obtained from the previous adversarial training are fixed. With the goal of minimizing the fusion result between the adversarial loss and the contrastive loss, the parameters of the generator obtained from the previous adversarial training are trained to obtain the generator corresponding to the current adversarial training.

[0163] The generator corresponding to the current adversarial training is fixed, and the parameters of the discriminator obtained in the previous adversarial training are trained with the goal of maximizing the adversarial loss, so as to obtain the discriminator corresponding to the current adversarial training.

[0164] Based on the discriminator and generator corresponding to the current adversarial training, the next adversarial training is executed iteratively until the generative adversarial network meets the preset termination condition.

[0165] In some embodiments, the image enhancement model includes an encoder, an attention module, a residual module, and a decoder;

[0166] Image enhancement unit 602 is specifically used for:

[0167] The target image is input into the encoder for downsampling to obtain the first feature map of the target image;

[0168] The first feature map is input into the attention module for detail feature extraction to obtain the second feature map of the target image;

[0169] The second feature map is input into the residual module for residual operation to obtain the third feature map of the target image;

[0170] The third feature map is input into the decoder for upsampling to obtain the enhanced image of the target image.

[0171] In some embodiments, the attention module includes a channel attention unit and a spatial attention unit;

[0172] The image enhancement unit 602 is also used for:

[0173] The first feature map is input into the channel attention unit, the channel weights of the first feature map in the channel dimension are calculated, and the feature sub-maps in each channel dimension of the first feature map are adjusted according to the channel weights.

[0174] The adjusted first feature map is input into the spatial attention unit, the spatial position weight of the adjusted first feature map in the spatial dimension is calculated, and the feature sub-map in each spatial dimension of the adjusted first feature map is adjusted according to the spatial position weight to obtain the second feature map.

[0175] Figure 7 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 7 As shown, the electronic device may include: a processor 701, a communication interface 702, a memory 703, and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704. The processor 701 can call logical instructions in the memory 703 to execute an image enhancement method, which includes: acquiring a target image; inputting the target image into an image enhancement model to obtain an enhanced image of the target image; the image enhancement model is obtained by unsupervised training of a generative adversarial network based on adversarial loss and contrast loss; the contrast loss is obtained by contrast learning based on a first sample image and a second sample image in a first sample image set, and the enhanced sample image of the first sample image output by the generator in the generative adversarial network; the adversarial loss is obtained by adversarial learning based on the enhanced sample image and a third sample image in the second sample image set; the quality of the images in the first sample image set is lower than the quality of the images in the second sample image set.

[0176] Furthermore, the logical instructions in the aforementioned memory 703 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0177] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the image enhancement methods provided by the above methods. The method includes: acquiring a target image; inputting the target image into an image enhancement model to obtain an enhanced image of the target image; the image enhancement model is obtained by unsupervised training of a generative adversarial network based on adversarial loss and contrast loss; the contrast loss is obtained by contrast learning based on a first sample image and a second sample image in a first sample image set, and a sample enhanced image of the first sample image output by the generator in the generative adversarial network; the adversarial loss is obtained by adversarial learning based on the sample enhanced image and a third sample image in the second sample image set; the quality of the images in the first sample image set is lower than the quality of the images in the second sample image set.

[0178] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the image enhancement method provided by the methods described above. The method includes: acquiring a target image; inputting the target image into an image enhancement model to obtain an enhanced image of the target image; the image enhancement model is obtained by unsupervised training of a generative adversarial network based on adversarial loss and contrastive loss; the contrastive loss is obtained by contrastive learning based on a first sample image and a second sample image in a first sample image set, and a sample enhanced image of the first sample image output by the generator in the generative adversarial network; the adversarial loss is obtained by adversarial learning based on the sample enhanced image and a third sample image in the second sample image set; the quality of the images in the first sample image set is lower than the quality of the images in the second sample image set.

[0179] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0180] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0181] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An image enhancement method, characterized in that, include: Acquire the target image; The target image is input into the image enhancement model to obtain the enhanced image of the target image; The image enhancement model is obtained by unsupervised training of a generative adversarial network based on adversarial loss and contrastive loss. The contrastive loss is obtained by contrastive learning based on the first and second sample images in the first sample image set, as well as the sample enhanced image of the first sample image output by the generator in the generative adversarial network. The adversarial loss is obtained by adversarial learning based on the sample enhanced image and the third sample image in the second sample image set. The quality of the images in the first sample image set is lower than the quality of the images in the second sample image set; The image content of the third sample image is different from that of the first sample image; the contrast loss aims to minimize the first similarity distance between the feature map of the first sample image and the feature map of the enhanced sample image, and maximize the second similarity distance between the feature map of the second sample image and the feature map of the enhanced sample image; the contrast loss is used to ensure that the enhanced image after enhancement has a strong correlation with the image content of the original image before enhancement. The first sample image and the third sample image were not pre-paired; The image enhancement model includes an encoder, an attention module, a residual module, and a decoder; The step of inputting the target image into an image enhancement model to obtain the enhanced image of the target image includes: The target image is input into the encoder for downsampling to obtain the first feature map of the target image; The first feature map is input into the attention module for detail feature extraction to obtain the second feature map of the target image; The second feature map is input into the residual module for residual operation to obtain the third feature map of the target image; The third feature map is input into the decoder for upsampling to obtain the enhanced image of the target image; The attention module includes a channel attention unit and a spatial attention unit; The step of inputting the first feature map into the attention module for detail feature extraction to obtain the second feature map of the target image includes: The first feature map is input into the channel attention unit, the channel weights of the first feature map in the channel dimension are calculated, and the feature sub-maps in each channel dimension of the first feature map are adjusted according to the channel weights. The adjusted first feature map is input into the spatial attention unit, the spatial position weight of the adjusted first feature map in the spatial dimension is calculated, and the feature sub-map in each spatial dimension of the adjusted first feature map is adjusted according to the spatial position weight to obtain the second feature map.

2. The image enhancement method according to claim 1, characterized in that, The image enhancement model is trained based on the following steps: The first sample image, the second sample image, and the sample enhancement image are input into the feature extraction model to obtain the feature map of the first sample image, the feature map of the second sample image, and the feature map of the sample enhancement image. The contrastive loss is obtained by performing contrastive learning based on the feature map of the first sample image, the feature map of the second sample image, and the feature map of the sample enhancement image. The enhanced sample image and the third sample image are input into the discriminator of the generative adversarial network to perform adversarial learning and obtain the adversarial loss. Based on the adversarial loss and the contrastive loss, the parameters of the generative adversarial network are iteratively trained adversarially. The image enhancement model is constructed based on the generator in the trained generative adversarial network.

3. The image enhancement method according to claim 2, characterized in that, The step of obtaining the contrastive loss by performing contrastive learning based on the feature maps of the first sample image, the second sample image, and the sample enhancement image includes: Obtain a first similarity distance between the feature map of the first sample image and the feature map of the sample enhancement image, and a second similarity distance between the feature map of the second sample image and the feature map of the sample enhancement image; The contrast loss is determined based on the first similarity distance and the second similarity distance.

4. The image enhancement method according to claim 2, characterized in that, The step of inputting the enhanced sample image and the third sample image into the discriminator of the generative adversarial network to perform adversarial learning and obtain the adversarial loss includes: The enhanced sample image and the third sample image are input into the discriminator to obtain the discrimination result of the enhanced sample image and the discrimination result of the third sample image; The adversarial loss is determined based on the discrimination results of the enhanced sample image and the third sample image.

5. The image enhancement method according to claim 2, characterized in that, The step of iteratively training the parameters of the generative adversarial network based on the adversarial loss and the contrastive loss includes: For the current adversarial training, the parameters of the discriminator obtained from the previous adversarial training are fixed. With the goal of minimizing the fusion result between the adversarial loss and the contrastive loss, the parameters of the generator obtained from the previous adversarial training are trained to obtain the generator corresponding to the current adversarial training. The generator corresponding to the current adversarial training is fixed, and the parameters of the discriminator obtained in the previous adversarial training are trained with the goal of maximizing the adversarial loss, so as to obtain the discriminator corresponding to the current adversarial training. Based on the discriminator and generator corresponding to the current adversarial training, the next adversarial training is executed iteratively until the generative adversarial network meets the preset termination condition.

6. An image enhancement device, characterized in that, include: The acquisition unit is used to acquire the target image; An image enhancement unit is used to input the target image into an image enhancement model to obtain an enhanced image of the target image; The image enhancement model is obtained by unsupervised training of a generative adversarial network based on adversarial loss and contrastive loss. The contrastive loss is obtained by contrastive learning based on the first and second sample images in the first sample image set, as well as the sample enhanced image of the first sample image output by the generator in the generative adversarial network. The adversarial loss is obtained by adversarial learning based on the sample enhanced image and the third sample image in the second sample image set. The quality of the images in the first sample image set is lower than the quality of the images in the second sample image set; The image content of the third sample image is different from that of the first sample image; the contrast loss aims to minimize the first similarity distance between the feature map of the first sample image and the feature map of the enhanced sample image, and maximize the second similarity distance between the feature map of the second sample image and the feature map of the enhanced sample image; the contrast loss is used to ensure that the enhanced image after enhancement has a strong correlation with the image content of the original image before enhancement. The first sample image and the third sample image were not pre-paired; The image enhancement model includes an encoder, an attention module, a residual module, and a decoder; The step of inputting the target image into an image enhancement model to obtain the enhanced image of the target image includes: The target image is input into the encoder for downsampling to obtain the first feature map of the target image; The first feature map is input into the attention module for detail feature extraction to obtain the second feature map of the target image; The second feature map is input into the residual module for residual operation to obtain the third feature map of the target image; The third feature map is input into the decoder for upsampling to obtain the enhanced image of the target image; The attention module includes a channel attention unit and a spatial attention unit; The step of inputting the first feature map into the attention module for detail feature extraction to obtain the second feature map of the target image includes: The first feature map is input into the channel attention unit, the channel weights of the first feature map in the channel dimension are calculated, and the feature sub-maps in each channel dimension of the first feature map are adjusted according to the channel weights. The adjusted first feature map is input into the spatial attention unit, the spatial position weight of the adjusted first feature map in the spatial dimension is calculated, and the feature sub-map in each spatial dimension of the adjusted first feature map is adjusted according to the spatial position weight to obtain the second feature map.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the image enhancement method as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image enhancement method as described in any one of claims 1 to 5.

Images