Dark image enhancement method and device, electronic equipment and storage medium
By using a convolutional neural network image enhancement model and fusing detail and structure-aware features, the problems of generalization ability and image quality improvement in low-light image processing are solved, and high-quality low-light image enhancement is achieved.
Patent Information
- Application Number
- CN202111138589.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-09-27
- Publication Date
- 2025-10-17
- Estimated Expiration
- 2041-09-27
AI Technical Summary
Existing low-light image processing methods have limitations in generalization ability and image quality improvement, especially unsupervised learning methods that are prone to overexposure and color shift problems.
A convolutional neural network image enhancement model is adopted, using unpaired dark-light images and normal-light images as training samples. It combines detail-aware features and structure-aware features, and performs feature fusion through a structure-aware generator, a detail-aware generator, and a feature attention fusion module to generate a lighting guidance map for image enhancement.
It effectively enhances low-light images, avoids overexposure and color shift, and generates images with sufficient brightness and naturalness.
Smart Images
Figure CN114049264B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present application relates to the technical field of image processing, and in particular to a dark-light image enhancement method and device, electronic equipment and storage medium. BACKGROUND
[0002] When users take pictures using common hardware devices such as mobile phones, they often encounter unfavorable conditions such as dark light, which results in low brightness of the images captured by the mobile phone, unclear objects and scenes, loss of details, and blurring, which not only lacks aesthetic perception but also seriously hinders subsequent visual analysis such as text recognition and object detection. Therefore, how to process dark-light images to make them have sufficient brightness and satisfactory natural characteristics has always been one of the research focuses in the fields of computational photography, computer graphics, and computer vision.
[0003] Existing deep learning methods include supervised learning and unsupervised learning. The existing supervised learning method uses a pair of images (a dark-light image I and an aligned normal-light image N) to learn the mapping relationship between I and N; however, the model trained using paired data in the supervised learning method can only produce good results for test images of the same type as the training data set, which cannot be generalized to more extensive scenarios and has great limitations. The existing unsupervised learning method uses a dark-light image and an unpaired normal-light image as training data, uses a generative adversarial network (GAN) to constrain the enhanced result to be close to a normal-light image, and uses a perception loss to constrain the enhanced result to retain the texture and content of the dark-light image; however, the existing unsupervised learning method has the problem of causing overexposure and color shift in many originally bright places. SUMMARY
[0004] In view of the problems of the prior art, the present application provides a dark-light image enhancement method, device, electronic equipment and storage medium, which realizes the use of complementary information between the detail perception features and the structure perception features of a target image to improve the enhancement effect of the target image, and avoids the problems of overexposure and color shift.
[0005] The present application provides a dark-light image enhancement method, comprising:
[0006] In the case where the illumination intensity of the target image is lower than a target threshold, an illumination guide map of the target image is determined based on the red-green-blue (RGB) three channels of the target image; wherein the illumination guide map is used to represent the degree of enhancement required at different positions in the target image.
[0007] input the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model;
[0008] The convolutional neural network image enhancement model is obtained by taking unpaired dark-light images and normal-light images as training samples and training based on the detail perception features and the structure perception features of the training samples; the convolutional neural network image enhancement model is used for image enhancement on the target image; and the illumination intensity of the enhanced image is higher than or equal to the target threshold.
[0009] According to the dark-light image enhancement method provided in the application, the illumination guide map of the target image is determined based on the RGB three channels of the target image, including:
[0010] The target image is subjected to image enhancement to obtain a first image;
[0011] The first image is subjected to normalization to obtain a second image; wherein the pixel value range of the second image includes [0, 1];
[0012] The gray-scale map of the second image is determined based on the RGB three channels of the second image;
[0013] The illumination guide map of the second image is determined based on the gray-scale map of the second image, and the illumination guide map of the second image is determined as the illumination guide map of the target image.
[0014] According to the dark-light image enhancement method provided in the application, the convolutional neural network image enhancement model includes a structure perception generator SAG, a detail perception generator DAG and a feature attention fusion module FAM.
[0015] The input of the target image and the illumination guide map into the convolutional neural network image enhancement model to obtain the enhanced image output by the convolutional neural network image enhancement model includes:
[0016] The target image and the illumination guide map are input into the SAG to obtain the structure perception features of the target image generated by the SAG;
[0017] The target image and the illumination guide map are input into the DAG to obtain the detail perception features of the target image generated by the DAG;
[0018] The structure perception features and the detail perception features are input into the FAM to obtain the fusion features of the structure perception features and the detail perception features;
[0019] input the fusion feature into the DAG to obtain an enhanced image output by the DAG.
[0020] According to the dark light image enhancement method provided by the application, the convolutional neural network image enhancement model further comprises: a pixel intensity adjustment module IAM, configured to perform pixel intensity adjustment on the enhanced image output by the DAG; and the IAM is located after the last convolutional layer of the DAG.
[0021] After the target image and the illumination guide image are input into the convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model, the following steps are further included.
[0022] The enhanced image is input into the IAM to obtain a pixel intensity adjusted enhanced image output by the IAM.
[0023] According to the dark light image enhancement method provided by the application, the target image and the illumination guide image are input into the SAG to obtain structure perception features of the target image generated by the SAG, including:
[0024] The RGB three channels of the target image are normalized to [-1, 1].
[0025] After the normalized RGB three channels of the target image and the single channel of the illumination guide image are connected in the channel dimension, the obtained four channels are input into the SAG to obtain the structure perception features of the target image generated by the SAG.
[0026] In the skip connection of the SAG, the structure consistency of the feature map is adaptively learned by using the illumination guide image; the supervision loss function adopted by the SAG comprises at least one of the following: a first global perception loss function, a first local perception loss function, a first global adversarial loss function, and a first local adversarial loss function; the first local perception loss function is used to represent that after the target image and the output image of the SAG are divided into fixed-size first image blocks, the global perception loss functions of each first image block are summed to obtain a result.
[0027] According to the dark light image enhancement method provided by the application, the target image and the illumination guide image are input into the DAG to obtain detail perception features of the target image generated by the DAG, including:
[0028] The RGB three channels of the target image are normalized to [-1, 1].
[0029] The normalized RGB three channels of the target image and the single channel of the light guide image are connected in the channel dimension, and then the obtained four-channel input is input into the DAG to obtain the detail perception feature of the target image generated by the DAG.
[0030] The DAG does not change the image resolution, and the supervised loss function adopted by the DAG includes at least one of the following: a second global perception loss function, a second local perception loss function, a second global adversarial loss function, and a second local adversarial loss function.
[0031] According to the dark light image enhancement method provided by the application, the structure perception feature and the detail perception feature are input into the FAM to obtain the fusion feature of the structure perception feature and the detail perception feature, which includes:
[0032] In the case that the structure perception feature includes the inverse at least two layers of convolution results of the SAG, and the detail perception feature includes the positive at least two layers of convolution results of the DAG, based on the FAM, the inverse at least two layers of convolution results of the SAG and the positive at least two layers of convolution results of the DAG are connected in the channel dimension, a single channel of the light guide image with the same resolution is used to guide the connected image, and the guided image is sequentially input into at least one layer of convolution kernel and hyperbolic tangent activation function for feature fusion to obtain the fusion feature of the structure perception feature and the detail perception feature.
[0033] The application further provides a dark light image enhancement device, which includes:
[0034] A determination module is configured to determine a light guide image of a target image based on red, green and blue (RGB) three channels of the target image when the light intensity of the target image is lower than a target threshold value, wherein the light guide image is used to represent the degree of enhancement required at different positions in the target image.
[0035] An enhancement module is configured to input the target image and the light guide image into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model.
[0036] The convolutional neural network image enhancement model is obtained by taking unpaired dark light images and normal light images as training samples and training based on detail perception features and structure perception features of the training samples; the convolutional neural network image enhancement model is used for image enhancement on the target image; and the illumination intensity of the enhanced image is higher than or equal to the target threshold.
[0037] The application further provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the steps of the dark light image enhancement method according to any one of the above when executing the program.
[0038] The application further provides a non-transitory computer readable storage medium, which stores a computer program, and the computer program implements the steps of the dark light image enhancement method according to any one of the above when executed by a processor.
[0039] The dark light image enhancement method, device, electronic device and storage medium provided by the application are based on an illumination guide image, and a convolutional neural network image enhancement model is used for image enhancement on a target image with low illumination intensity. Since the convolutional neural network image enhancement model is obtained by taking unpaired dark light images and normal light images as training samples and training based on detail perception features and structure perception features of the training samples, the detail perception features and structure perception features of the target image can be fused, the complementary information between the detail perception features and the structure perception features is utilized to improve the enhancement effect of the target image, and the problems of overexposure and color shift are avoided. BRIEF DESCRIPTION OF DRAWINGS
[0040] In order to more clearly illustrate the technical solutions of the present application or the prior art, the following will briefly introduce the drawings needed in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.
[0041] Figure 1 is one of the flowcharts of the dark light image enhancement method provided by the application;
[0042] Figure 2 is the second flowchart of the dark light image enhancement method provided by the application;
[0043] Figure 3 is the third flowchart of the dark light image enhancement method provided by the application;
[0044] Figure 4 is a structural schematic diagram of the dark light image enhancement device provided by the application;
[0045] Figure 5 Figure 1 is a structural schematic diagram of an electronic device provided by the present application. DETAILED DESCRIPTION
[0046] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in conjunction with the drawings in the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of protection of the present application.
[0047] The present application provides a dark light image enhancement method, comprising: in the case that the illumination intensity of a target image is lower than a target threshold, determining an illumination guide map of the target image based on red green blue (RGB) three channels of the target image; wherein the illumination guide map is used to represent the degree of enhancement required at different positions in the target image; inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; wherein the convolutional neural network image enhancement model is obtained by training based on unpaired dark light images and normal illumination images as training samples and based on the detail perception features and structure perception features of the training samples; the convolutional neural network image enhancement model is used for image enhancement of the target image; and the illumination intensity of the enhanced image is higher than or equal to the target threshold. The above method uses the convolutional neural network image enhancement model to perform image enhancement on the target image with lower illumination intensity based on the illumination guide map. Since the convolutional neural network image enhancement model is obtained by training based on unpaired dark light images and normal illumination images as training samples and based on the detail perception features and structure perception features of the training samples, the detail perception features and structure perception features of the target image can be fused, the complementary information between the detail perception features and the structure perception features is utilized to improve the enhancement effect of the target image, and thus the problems of overexposure and color shift are avoided.
[0048] The dark light image enhancement method provided by the present application can be applied to mobile phone photographing, industrial detection, background correction, automatic driving and other use scenarios. The dark light image enhancement method of the present application will be described below in conjunction with Figures 1-3
[0049] Figure 1 Figure 1 is a structural schematic diagram of an electronic device provided by the present application. Figure 1 The method comprises:
[0050] Step 101, in the case that the light intensity of a target image is lower than a target threshold, determining a light guide map of the target image based on RGB three channels of the target image; wherein the light guide map is used to represent the degree of enhancement required at different positions in the target image;
[0051] Step 102, inputting the target image and the light guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; wherein the convolutional neural network image enhancement model is obtained by training based on detail perception features and structure perception features of unpaired dark light images and normal light images as training samples; the convolutional neural network image enhancement model is used for image enhancement of the target image; and the light intensity of the enhanced image is higher than or equal to the target threshold.
[0052] Optionally, the light intensity of the dark light image is lower than the target threshold; and the light intensity of the normal light image is greater than or equal to the target threshold.
[0053] The dark light image enhancement method provided by the application is based on the light guide map, and uses the convolutional neural network image enhancement model to perform image enhancement on the target image with low light intensity. Since the convolutional neural network image enhancement model is obtained by training based on the unpaired dark light images and normal light images as training samples and the detail perception features and structure perception features of the training samples, the detail perception features and structure perception features of the target image can be fused, the complementary information between the detail perception features and the structure perception features is utilized to improve the enhancement effect of the target image, and thus the problems of overexposure and color shift are avoided.
[0054] Optionally, Figure 2 For the flowchart of the dark light image enhancement method provided by the application. As shown in the figure, the method comprises the following steps. Figure 2
[0055] Step 201, in the case that the light intensity of a target image is lower than a target threshold, performing image enhancement on the target image to obtain a first image;
[0056] Step 202, normalizing the first image to obtain a second image; wherein the pixel value range of the second image comprises [0, 1];
[0057] Step 203, determining a gray map of the second image based on RGB three channels of the second image;
[0058] Step 204, determining the light guide map of the second image based on the gray scale map of the second image, and determining the light guide map of the target image as the light guide map of the second image; wherein the light guide map is used to represent the degree of enhancement required at different positions in the target image;
[0059] Step 205, inputting the target image and the light guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; wherein the convolutional neural network image enhancement model is obtained by training on unpaired dark light images and normal light images based on the detail perception features and structure perception features of the training samples; the convolutional neural network image enhancement model is used for image enhancement of the target image; and the light intensity of the enhanced image is higher than or equal to the target threshold.
[0060] Optionally, the image enhancement operation on the target image comprises at least one of the following: an operation of enhancing brightness, an operation of enhancing chroma, an operation of enhancing contrast, and an operation of enhancing sharpness.
[0061] Optionally, based on the RGB three channels of the second image, the gray scale map of the second image Gray is calculated by using formula (1):
[0062] Gray = (0.299 * R + 0.587 * G + 0.114 * B) (1)
[0063] wherein R, G and B represent the three channels of the target image.
[0064] Optionally, based on the gray scale map of the second image, the light guide map GM of the second image is calculated by using formula (2), and then the light guide map of the second image is determined as the light guide map GM of the target image:
[0065] GM = 1 - Gray (2)
[0066] It should be noted that the description and explanation of step 205 above can refer to step 102 above and achieve the same technical effect. To avoid repetition, it will not be described here.
[0067] The dark light image enhancement method provided by the application first performs image enhancement on the target image before calculating the light guide map of the target image, then calculates the gray scale map based on the RGB three channels, and then calculates the light guide map, so that the light guide map has high distinguishability for the degree of enhancement required at different positions, and the probability of overexposure at originally bright positions due to overall image enhancement can be avoided.
[0068] Optionally, Figure 3A flowchart of a dark light image enhancement method provided by the present application. The convolutional neural network image enhancement model mentioned in the present application can include: a structure perception generator (SAG), a detail perception generator (DAG), and a feature attention fusion module (FAM). The FAM module can eliminate the difference between the high-level features of the SAG and the DAG, so as to better fuse the structure and detail perception features together. As shown in Figure 3 The dark light image enhancement method based on the convolutional neural network image enhancement model provided by the present application includes the following steps:
[0069] Step 301: In the case that the light intensity of a target image is lower than a target threshold, determining a light guide map of the target image based on the RGB three channels of the target image; wherein the light guide map is used to represent the degree of enhancement required at different positions in the target image.
[0070] Step 302: inputting the target image and the light guide map into the SAG to obtain the structure perception feature of the target image generated by the SAG;
[0071] Step 303: inputting the target image and the light guide map into the DAG to obtain the detail perception feature of the target image generated by the DAG;
[0072] Step 304: inputting the structure perception feature and the detail perception feature into the FAM to obtain the fusion feature of the structure perception feature and the detail perception feature;
[0073] Step 305: inputting the fusion feature into the DAG to obtain an enhanced image output by the DAG.
[0074] Optionally, the implementation manner of the structure perception feature of the target image generated by the SAG, obtained by inputting the target image and the illumination guide image into the SAG in step 302, comprises: normalizing the RGB three channels of the target image to [-1, 1]; connecting the normalized RGB three channels of the target image and the single channel of the illumination guide image in the channel dimension, and then inputting the obtained four channels into the SAG to obtain the structure perception feature of the target image generated by the SAG; wherein the SAG does not change the image resolution; in the skip connection of the SAG, the structure consistency of the feature map is adaptively learned by using the illumination guide image; the supervision loss function adopted by the SAG comprises at least one of the following: a first global perception loss function, a first local perception loss function, a first global adversarial loss function, and a first local adversarial loss function; the first local perception loss function is used to represent that after the target image and the output image of the SAG are segmented into fixed-size first image blocks, the global perception loss function of each first image block is summed to obtain the result.
[0075] Optionally, the first global perception loss function The first local perception loss function is represented as formula (3) The first global adversarial loss function is represented as formula (4) The first local adversarial loss function D is represented as formula (5) Ra (x,z), D Ra (z,x) is represented as formula (6):
[0076]
[0077]
[0078]
[0079]
[0080] Wherein, I, O are dark light images and SAG output results respectively, φ l represents the feature map extracted from the pre-trained VGG-16 model, W, H are the resolution of the feature map, and l is the lth layer of VGG-16; x and z in formula (5) and formula (6) represent sampling from true and false distributions respectively; σ(·) is a sigmoid function; C(·) is a non-transformed discriminant function.
[0081] Optionally, the implementation manner of inputting the target image and the light guide map into the DAG in step 303 to obtain the detail perception feature of the target image generated by the DAG can include: normalizing the RGB three channels of the target image to [-1, 1]; connecting the normalized RGB three channels of the target image and the single channel of the light guide map in the channel dimension, inputting the obtained four channels into the DAG to obtain the detail perception feature of the target image generated by the DAG; wherein the DAG does not change the image resolution; the supervised loss function adopted by the DAG includes at least one of the following: a second global perception loss function, a second local perception loss function, a second global adversarial loss function, and a second local adversarial loss function; the second local perception loss function is used to represent the result obtained by summing the global perception loss functions of each second image block after the target image and the output image of the DAG are segmented into second image blocks of a fixed size.
[0082] Optionally, the DAG adopts global and local loss functions to play a supervisory role, including perception loss and adversarial loss. Since the output forms of the DAG and the SAG are completely the same, the use of the loss functions in the DAG and the SAG is completely consistent.
[0083] Optionally, the implementation manner of inputting the structure perception feature and the detail perception feature into the FAM to obtain the fusion feature of the structure perception feature and the detail perception feature in step 304 can include: in the case that the structure perception feature includes the inverse at least two convolution results of the SAG, and the detail perception feature includes the positive at least two convolution results of the DAG, based on the FAM, connecting the inverse at least two convolution results of the SAG and the positive at least two convolution results of the DAG in the channel dimension, guiding the image obtained by the connection using the single-channel same-resolution light guide map, inputting the guided image into at least one convolution kernel and the hyperbolic tangent activation function in sequence for feature fusion to obtain the fusion feature of the structure perception feature and the detail perception feature, and the output is 2 times the input channel number.
[0084] The dark light image enhancement method provided by the application is based on a light guide map, and uses a convolutional neural network image enhancement model to perform image enhancement on a target image with low light intensity. Since the convolutional neural network image enhancement model uses unpaired dark light images and normal light images as training samples, generates structure perception features of the target image using the SAG, generates detail perception features of the target image using the DAG, and should fuse the detail perception features and the structure perception features of the target image, the complementary information between the detail perception features and the structure perception features is used to improve the enhancement effect of the target image together, thereby avoiding overexposure and color shift problems.
[0085] Optionally, the convolutional neural network image enhancement model further comprises: a pixel intensity adjustment module IAM, configured to perform pixel intensity adjustment on the enhanced image output by the DAG; the IAM is located after the last convolutional layer of the DAG. After inputting the target image and the illumination guide image into the convolutional neural network image enhancement model to obtain the enhanced image output by the convolutional neural network image enhancement model, the enhanced image is input into the IAM to obtain the pixel intensity adjusted enhanced image output by the IAM. The input of the IAM module is a 3-channel preliminary enhanced result, and the output is a 3-channel final enhanced result; the IAM is divided into an upper branch and a lower branch: one branch is based on convolution and Tanh, and the purpose is to continuously change the pixel value range to the same [-1, 1] interval as the input; the other branch is based on convolution and ReLU, and the purpose is to extract features to assist the adjustment of the pixel intensity, so that the enhanced result is finally adjusted in pixel intensity based on the IAM to obtain the pixel intensity adjusted enhanced image.
[0086] The dark light image enhancement method provided by the application can process dark light images with low light intensity in an unsupervised learning manner, and obtain real and natural normal light images through image enhancement. The application adjusts an illumination guide image, combines perception features in two aspects of details and structures, and fuses different types of perception features, so that a better dark light image enhancement effect is obtained.
[0087] The dark light image enhancement device provided by the application is described below. The dark light image enhancement device described below can be correspondingly referred to the dark light image enhancement method described above.
[0088] The application provides a dark light image enhancement device, Figure 4 A structure diagram of the dark light image enhancement device provided by the application is shown in the figure. Figure 4 As shown in the figure, the dark light image enhancement device 400 comprises:
[0089] A determination module 401, configured to determine an illumination guide image of a target image based on RGB three channels of the target image when the light intensity of the target image is lower than a target threshold; wherein the illumination guide image is used to represent the degree of enhancement required in different positions of the target image.
[0090] An enhancement module 402, configured to input the target image and the illumination guide image into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model.
[0091] The convolutional neural network image enhancement model is obtained by taking unpaired dark light images and normal light images as training samples and training based on detail perception features and structure perception features of the training samples; the convolutional neural network image enhancement model is used for image enhancement of the target image; and the illumination intensity of the enhanced image is higher than or equal to the target threshold.
[0092] The dark light image enhancement device provided by the application uses a convolutional neural network image enhancement model to perform image enhancement on a target image with low illumination intensity based on an illumination guide image. The convolutional neural network image enhancement model is obtained by taking unpaired dark light images and normal light images as training samples and training based on detail perception features and structure perception features of the training samples. The convolutional neural network image enhancement model can fuse the detail perception features and the structure perception features of the target image, improve the enhancement effect of the target image by using complementary information between the detail perception features and the structure perception features, and avoid overexposure and color shift.
[0093] Optionally, the determining module 401 is specifically configured to:
[0094] perform image enhancement on the target image to obtain a first image;
[0095] perform normalization on the first image to obtain a second image; wherein a pixel value range of the second image includes [0, 1];
[0096] determine a gray image of the second image based on RGB three channels of the second image;
[0097] determine an illumination guide image of the second image based on the gray image of the second image, and determine the illumination guide image of the second image as an illumination guide image of the target image.
[0098] Optionally, the convolutional neural network image enhancement model includes a structure perception generator SAG, a detail perception generator DAG, and a feature attention fusion module FAM.
[0099] The enhancement module 402 is specifically configured to:
[0100] input the target image and the illumination guide image into the SAG to obtain structure perception features of the target image generated by the SAG;
[0101] input the target image and the illumination guide image into the DAG to obtain detail perception features of the target image generated by the DAG;
[0102] input the structure perception features and the detail perception features into the FAM to obtain fused features of the structure perception features and the detail perception features.
[0103] inputting the fusion feature into the DAG to obtain an enhanced image output by the DAG.
[0104] Optionally, the convolutional neural network image enhancement model further comprises a pixel intensity adjustment module IAM, configured to perform pixel intensity adjustment on the enhanced image output by the DAG; the IAM is located after the last convolutional layer of the DAG.
[0105] The enhancement module 402 is specifically configured to:
[0106] inputting the enhanced image into the IAM to obtain a pixel intensity adjusted enhanced image output by the IAM.
[0107] Optionally, the enhancement module 402 is specifically configured to:
[0108] normalizing the RGB three channels of the target image to [-1, 1];
[0109] connecting the normalized RGB three channels of the target image and the single channel of the illumination guide map in the channel dimension, inputting the obtained four channels into the SAG to obtain the structure perception feature of the target image generated by the SAG;
[0110] In the skip connection of the SAG, the structure consistency of the feature map is adaptively learned by using the illumination guide map; the supervision loss function adopted by the SAG comprises at least one of the following: a first global perception loss function, a first local perception loss function, a first global adversarial loss function, and a first local adversarial loss function; the first local perception loss function is used to represent that after the target image and the output image of the SAG are segmented into fixed-size first image blocks, the global perception loss functions of the first image blocks are summed to obtain a result.
[0111] Optionally, the enhancement module 402 is specifically configured to:
[0112] normalizing the RGB three channels of the target image to [-1, 1];
[0113] connecting the normalized RGB three channels of the target image and the single channel of the illumination guide map in the channel dimension, inputting the obtained four channels into the DAG to obtain the detail perception feature of the target image generated by the DAG;
[0114] The DAG does not change the image resolution, and the supervised loss function adopted by the DAG comprises at least one of the following: a second global perception loss function, a second local perception loss function, a second global adversarial loss function, and a second local adversarial loss function; and the second local perception loss function is used to represent that after the target image and the output image of the DAG are segmented into second image blocks of a fixed size, the global perception loss functions of the second image blocks are summed to obtain a result.
[0115] Optionally, the enhancement module 402 is specifically configured to:
[0116] In the case that the structure perception feature comprises the inverse at least two-layer convolution result of the SAG, and the detail perception feature comprises the positive at least two-layer convolution result of the DAG, based on the FAM, the inverse at least two-layer convolution result of the SAG and the positive at least two-layer convolution result of the DAG are connected in the channel dimension, the connected image is guided by the single-channel same-resolution light guide image, the guided image is sequentially input to at least one convolution kernel and a hyperbolic tangent activation function for feature fusion, and the fusion feature of the structure perception feature and the detail perception feature is obtained.
[0117] The dark light image enhancement device in the application can be a device, a device with an operating system or an electronic equipment such as a mobile phone, and can also be a component, an integrated circuit or a chip in an electronic equipment.
[0118] The dark light image enhancement device provided by the application can realize Figures 1 to 3 The method embodiment realizes each process and achieves the same technical effect, and thus details are not repeated here.
[0119] Figure 5 is a schematic diagram of the entity structure of the electronic equipment provided by the application, such as Figure 5As shown, the electronic device 500 can include a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 complete mutual communication through the communications bus 540. The processor 510 can invoke a logic instruction in the memory 530 to execute a dark light image enhancement method, which includes: in the case that the illumination intensity of a target image is lower than a target threshold, determining an illumination guide map of the target image based on RGB three channels of the target image; wherein the illumination guide map is used to represent the degree of enhancement required at different positions in the target image; inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; wherein the convolutional neural network image enhancement model is obtained after training based on unpaired dark light images and normal illumination images as training samples and detailed perception features and structural perception features of the training samples; the convolutional neural network image enhancement model is used for image enhancement of the target image; and the illumination intensity of the enhanced image is higher than or equal to the target threshold.
[0120] In addition, the logic instruction in the memory 530 described above can be implemented in the form of a software function unit and sold or used as an independent product, and can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.
[0121] In another aspect, the present application also provides a computer program product, which comprises a computer program stored on a non-transitory computer-readable storage medium, and the computer program comprises program instructions, when the program instructions are executed by a computer, the computer can execute the dark-light image enhancement method provided by the above-mentioned methods, and the method comprises: in the case that the illumination intensity of a target image is lower than a target threshold, determining an illumination guide map of the target image based on the RGB three channels of the target image; wherein the illumination guide map is used to represent the degree of enhancement required at different positions in the target image; inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; wherein the convolutional neural network image enhancement model is obtained by training based on the detail perception features and the structure perception features of unpaired dark-light images and normal illumination images as training samples; the convolutional neural network image enhancement model is used for image enhancement of the target image; and the illumination intensity of the enhanced image is higher than or equal to the target threshold.
[0122] In another aspect, the present application also provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the dark-light image enhancement method provided by the above-mentioned methods, and the method comprises: in the case that the illumination intensity of a target image is lower than a target threshold, determining an illumination guide map of the target image based on the RGB three channels of the target image; wherein the illumination guide map is used to represent the degree of enhancement required at different positions in the target image; inputting the target image and the illumination guide map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; wherein the convolutional neural network image enhancement model is obtained by training based on the detail perception features and the structure perception features of unpaired dark-light images and normal illumination images as training samples; the convolutional neural network image enhancement model is used for image enhancement of the target image; and the illumination intensity of the enhanced image is higher than or equal to the target threshold.
[0123] The device embodiments described above are only schematic, wherein the units shown as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the present embodiment scheme. Those skilled in the art can understand and implement without creative labor.
[0124] Those skilled in the art can clearly understand the technical solutions of the various embodiments from the above description of the embodiments, and the various embodiments can be implemented by means of software with the necessary general hardware platforms, and of course, can also be implemented by hardware. Based on such understanding, the above technical solutions, essentially or in other words, the part of the prior art that makes a contribution, can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, and the like, and includes a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.
[0125] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for some technical features therein; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.
Claims
1. A dark light image enhancement method, characterized in that: include: When the illumination intensity of the target image is lower than a target threshold, determining an illumination guidance map of the target image based on the red, green, and blue (RGB) channels of the target image; wherein the illumination guidance map is used to indicate the degree to which different locations in the target image need to be enhanced; Inputting the target image and the illumination guidance map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; The convolutional neural network image enhancement model includes: a structure perception generator SAG, a detail perception generator DAG, and a feature attention fusion module FAM. The target image and the illumination guidance map are input into the convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model, including: inputting the target image and the illumination guidance map into the SAG to obtain the structure perception features of the target image generated by the SAG; inputting the target image and the illumination guidance map into the DAG to obtain the detail perception features of the target image generated by the DAG; inputting the structure perception features and the detail perception features into the FAM to obtain a fusion feature of the structure perception features and the detail perception features; and inputting the fusion feature into the DAG to obtain the enhanced image output by the DAG. Among them, the convolutional neural network image enhancement model is obtained after training with unpaired dark light images and normal light images as training samples and based on the detail perception features and structure perception features of the training samples; the convolutional neural network image enhancement model is used to perform image enhancement on the target image; the light intensity of the enhanced image is higher than or equal to the target threshold.
2. The dark light image enhancement method according to claim 1, characterized in that: The determining of the illumination guidance map of the target image based on the RGB three channels of the target image includes: Performing image enhancement on the target image to obtain a first image; Normalizing the first image to obtain a second image; wherein the pixel value range of the second image includes [0, 1]; Determine a grayscale image of the second image based on the RGB three channels of the second image; An illumination guidance map of the second image is determined based on the grayscale map of the second image, and the illumination guidance map of the second image is determined as the illumination guidance map of the target image.
3. The dark light image enhancement method according to claim 1, characterized in that: The convolutional neural network image enhancement model further includes: a pixel intensity adjustment module IAM, configured to adjust the pixel intensity of the enhanced image output by the DAG; the IAM is located after the last convolution layer of the DAG; After inputting the target image and the illumination guidance map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model, the method further comprises: The enhanced image is input into the IAM to obtain an enhanced image output by the IAM after pixel intensity adjustment.
4. The dark light image enhancement method according to claim 1, wherein: Inputting the target image and the illumination guidance map into the SAG to obtain the structural perception features of the target image generated by the SAG includes: Normalize the RGB channels of the target image to between [-1, 1]; After concatenating the normalized RGB three channels of the target image and the single channel of the illumination guidance map in the channel dimension, the resulting four channels are input into the SAG to obtain the structural perception features of the target image generated by the SAG; In which, in the skip connection of the SAG, the structural consistency of the feature map is adaptively learned using the illumination guidance map; the supervised loss function adopted by the SAG includes at least one of the following: a first global perceptual loss function, a first local perceptual loss function, a first global adversarial loss function, and a first local adversarial loss function; the first local perceptual loss function is used to represent the result obtained by summing the global perceptual loss functions of each first image block after the target image and the output image of the SAG are divided into first image blocks of a fixed size.
5. The dark light image enhancement method according to claim 1, characterized in that: Inputting the target image and the illumination guidance map into the DAG to obtain detail perception features of the target image generated by the DAG includes: Normalize the RGB channels of the target image to between [-1, 1]; After concatenating the normalized RGB three channels of the target image and the single channel of the illumination guidance map in the channel dimension, the resulting four channels are input into the DAG to obtain the detail perception features of the target image generated by the DAG; The DAG does not change the image resolution; the supervised loss function adopted by the DAG includes at least one of the following: a second global perceptual loss function, a second local perceptual loss function, a second global adversarial loss function, and a second local adversarial loss function; the second local perceptual loss function is used to represent the result of summing the global perceptual loss functions of each second image block after segmenting the target image and the output image of the DAG into second image blocks of a fixed size.
6. The dark light image enhancement method according to claim 1, characterized in that: Inputting the structure-aware feature and the detail-aware feature into the FAM to obtain a fusion feature of the structure-aware feature and the detail-aware feature includes: When the structure perception feature includes the reciprocal convolution results of at least two layers of the SAG and the detail perception feature includes the positive convolution results of at least two layers of the DAG, based on the FAM, the reciprocal convolution results of at least two layers of the SAG and the positive convolution results of at least two layers of the DAG are connected in the channel dimension, and the connected image is guided by the single-channel illumination guidance map with the same resolution. The guided image is sequentially input into at least one layer of convolution kernel and hyperbolic tangent activation function for feature fusion to obtain the fusion feature of the structure perception feature and the detail perception feature.
7. A low-light image enhancement device, characterized in that: include: a determination module, configured to determine, when the illumination intensity of the target image is lower than a target threshold, an illumination guidance map of the target image based on the red, green, and blue (RGB) channels of the target image; wherein the illumination guidance map is used to indicate the degree to which different locations in the target image need to be enhanced; An enhancement module is configured to input the target image and the illumination guidance map into a convolutional neural network image enhancement model to obtain an enhanced image output by the convolutional neural network image enhancement model; the convolutional neural network image enhancement model includes: a structure perception generator SAG, a detail perception generator DAG, and a feature attention fusion module FAM, wherein the target image and the illumination guidance map are input into the SAG to obtain structure perception features of the target image generated by the SAG; the target image and the illumination guidance map are input into the DAG to obtain detail perception features of the target image generated by the DAG; the structure perception features and the detail perception features are input into the FAM to obtain fusion features of the structure perception features and the detail perception features; and the fusion features are input into the DAG to obtain the enhanced image output by the DAG; Among them, the convolutional neural network image enhancement model is obtained after training with unpaired dark light images and normal light images as training samples and based on the detail perception features and structure perception features of the training samples; the convolutional neural network image enhancement model is used to perform image enhancement on the target image; the light intensity of the enhanced image is higher than or equal to the target threshold.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the low-light image enhancement method according to any one of claims 1 to 6 are implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the low-light image enhancement method according to any one of claims 1 to 6 are implemented.
Citation Information
Patent Citations
Wavelet change weak light image enhancement method
CN119338684A
Deep learning based image enhancement
US20220019861A1