Low-illumination image enhancement method based on self-guided learning framework

Abstract

The invention relates to a low-illumination image enhancement method based on a self-guided learning framework. The method comprises the following steps: acquiring a low-illumination image to be enhanced; carrying out graying processing on the to-be-enhanced low-illumination image, and carrying out inversion processing on the to-be-enhanced low-illumination image after the graying processing to obtain an inverted grayscale image; inputting the to-be-enhanced low-illumination image and the inverted grayscale image into the low-illumination image enhancement network to obtain an enhanced image; the low-illumination image enhancement network includes a generator; the generator comprises a feature extraction module which is used for carrying out feature extraction on an input image in a mode of combining short-range convolution and long-range convolution; and the feature recovery module is used for performing feature reconstruction on the features extracted by the feature extraction module by adopting a mode of combining short-range convolution and long-range convolution to obtain an enhanced image. According to the invention, the effect of the existing low-illumination image enhancement algorithm can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a low-illumination image enhancement method based on a self-guided learning framework. Background Art

[0002] Low-light image enhancement is a common and challenging computer vision task. In real scenes, it is inevitable that images are taken in undesirable environments, such as insufficient ambient light, uneven lighting, backlight, etc. The images taken under these conditions often have problems such as insufficient brightness, poor contrast, low signal-to-noise ratio, and loss of details. Such images not only have poor visual effects for the human eye, but also affect the effects of subsequent advanced visual tasks such as target detection, target recognition, and semantic segmentation. These tasks are often applied in mobile phones, cameras, various surveillance cameras, and autonomous driving.

[0003] The goal of low-light image enhancement is to improve the quality of such low-light images so that the human eye can obtain better visual effects, while helping to improve the accuracy of subsequent advanced visual tasks. Currently, researchers have proposed many low-light image enhancement algorithms based on deep learning, which can be mainly divided into three categories: supervised learning, unsupervised learning, and semi-supervised learning. At present, algorithms based on supervised learning can already obtain good visual effects in specified data sets, while algorithms based on unsupervised learning usually have faster inference speeds, smaller parameters, or better generalization performance.

[0004] However, the current algorithms often encounter the following problems when pursuing indicators and computational efficiency: Algorithms based on supervised learning require corresponding reference images during training, so their performance is greatly affected by the data set. To achieve better results in other unseen data sets, the network needs to be fine-tuned, but in reality, it is very difficult to obtain paired low-light images and normal-light images. Existing algorithms with high computational efficiency often use simple network designs and loss functions set according to experience to obtain a low-light image enhancement network with fast inference speed and small number of parameters. However, such networks are difficult to achieve good results due to problems such as simple networks and insufficient constraints on loss functions. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a low-light image enhancement method based on a self-guided learning framework, which can effectively improve the effect of the existing low-light image enhancement algorithm.

[0006] The technical solution adopted by the present invention to solve the technical problem is: to provide a low-light image enhancement method based on a self-guided learning framework, comprising the following steps:

[0007] Acquire a low-light image to be enhanced;

[0008] grayscale the low-light image to be enhanced, and invert the grayscale image to be enhanced to obtain an inverted grayscale image;

[0009] Inputting the low-light image to be enhanced and the inverted grayscale image into a low-light image enhancement network to obtain an enhanced image; the low-light image enhancement network includes a generator;

[0010] The generator comprises:

[0011] A feature extraction module is used to extract features from an input image by combining short-range convolution and long-range convolution;

[0012] The feature recovery module is used to reconstruct the features extracted by the feature extraction module by combining short-range convolution and long-range convolution to obtain an enhanced image.

[0013] The feature extraction module includes M groups of long- and short-range feature extraction modules and maximum pooling layers connected in sequence; the feature recovery module includes M groups of bicubic linear interpolation layers and long- and short-range feature detail reconstruction modules connected in sequence; the M groups of long- and short-range feature extraction modules and the M groups of long- and short-range feature detail reconstruction modules are jump-connected.

[0014] The long- and short-range feature extraction module comprises:

[0015] The first short-range convolution operation part includes a convolution unit, a SELU activation layer unit and a batch normalization layer unit which are arranged in sequence, and is used to perform a short-range convolution operation on the input features to obtain local feature information;

[0016] The first long-range convolution operation part includes a focus module unit, a convolution unit, an upsampling operation unit, a SELU activation layer unit, and a batch normalization layer unit, which are arranged in sequence, and are used to perform a long-range convolution operation on the input features to obtain non-local feature information;

[0017] A first fusion part, used for fusing the local feature information with the non-local feature information;

[0018] The sorting and optimization part is used to sort and optimize the fused local feature information and non-local feature information.

[0019] The long-range and short-range feature detail reconstruction module comprises:

[0020] The feature reconstruction part is used to reconstruct the input features to obtain the reconstructed features;

[0021] The second short-range convolution operation part includes a convolution unit, a SELU activation layer unit and a batch normalization layer unit which are arranged in sequence, and is used to connect the reconstructed features and the output features of the long-short-range feature extraction module of the symmetric position, and perform a short-range convolution operation on the connected features to obtain short-range features;

[0022] The second long-range convolution operation part includes a focus module unit, a convolution unit, an upsampling operation unit, a SELU activation layer unit, and a batch normalization layer unit, which are arranged in sequence, and are used to perform a long-range convolution operation on the reconstructed features to obtain long-range features;

[0023] A second fusion part, used for fusing the short-range features with the long-range features;

[0024] The adjustment part is used to adjust the fused short-range features and long-range features.

[0025] The low-light image enhancement network also includes a discriminator; the discriminator includes:

[0026] Global discriminator and local discriminator are used to ensure that the generated image is closer to the normal illumination image;

[0027] The VGG16 model is used to construct feature loss to ensure that the generated image retains the characteristics of the original image.

[0028] The low-light image enhancement network includes the following steps during training:

[0029] Performing two random data augmentation operations on the low-light training image to obtain a first input image and a second input image, wherein the first input image is similar to but different from the second input image;

[0030] Performing grayscale processing on the first input image and the second input image respectively, and performing inversion processing on the grayscaled first input image and the second input image to obtain a first inverted grayscale input image and a second inverted grayscale input image;

[0031] Inputting the first input image and the first inverted grayscale input image into a generator to obtain a training enhanced image;

[0032] Inputting the second input image and the second inverted grayscale input image into a generator to obtain a reference image;

[0033] The VGG16 model is used to extract feature maps of the training enhanced image, the reference image and the corrected image respectively; the corrected image is an image obtained by performing gamma correction on the low-illumination training image;

[0034] Using feature preservation loss to ensure that the training enhanced image and the corrected image retain the detail features of the low-light training image, and using feature consistency loss to ensure that the features of the training enhanced image and the reference image are consistent;

[0035] Using the global discriminator to determine whether the training enhanced image and the reference image are true normal illumination images;

[0036] After randomly cropping the training enhanced image and the reference image, training enhanced image slices and reference image slices are obtained respectively, and the training enhanced image slices and the reference image slices are input into the VGG16 model to obtain training enhanced image local features and reference image local features;

[0037] Using the local discriminator to determine whether the local features of the training enhanced image and the local features of the reference image are true features of normal illumination;

[0038] The generator training is guided by the outputs of the global discriminator, the local discriminator and the VGG16 model.

[0039] The random data enhancement operation refers to performing random brightness adjustment on the low-light training image with a probability of 50% to obtain a first image; performing color adjustment on the first image with a probability of 50% to obtain a second image; performing noise addition processing on the second image with a probability of 50% to obtain a third image; and adding a mask to the third image with a probability of 50% to obtain an input image.

[0040] At the beginning of training, the coverage rate of the mask is 75%, and the pixel value of the point covered by the mask is the average of the four neighborhoods of the pixel at the point; after each preset number of trainings, the coverage rate of the mask is reduced by 25% until the coverage rate is 0.

[0041] The technical solution adopted by the present invention to solve its technical problem is: to provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of the low-light image enhancement method based on the self-guided learning framework are implemented.

[0042] The technical solution adopted by the present invention to solve its technical problem is: providing a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, the steps of the low-light image enhancement method based on the self-guided learning framework are implemented.

[0043] Beneficial Effects

[0044] Due to the adoption of the above-mentioned technical scheme, the present invention has the following advantages and positive effects compared with the prior art: the generator part of the present invention is composed of a feature extraction module composed of multiple long- and short-range feature extraction modules and multiple maximum pooling layers, and a feature extraction module composed of multiple bicubic linear interpolation layers and multiple long- and short-range detail reconstruction modules. By adopting long-range and short-range operations to extract information of different receptive fields, it is possible to use non-local information to improve the information extraction capability while preserving local details, ensuring that useful information can be extracted in areas with low signal-to-noise ratio. In the feature reconstruction stage, the long-range convolution operation with a larger receptive field guides the short-range convolution to better reconstruct features, so that the enhanced low-illumination image has higher image quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 is a flow chart of a low-light image enhancement method based on a self-guided learning framework according to a first embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of the structure of a low-light image enhancement network in the first embodiment of the present invention;

[0047] Figure 3 It is a schematic diagram of adding a mask in the first embodiment of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0049] The first embodiment of the present invention relates to a low-light image enhancement method based on a self-guided learning framework, such as Figure 1 As shown, the following steps are included:

[0050] Step 1: Obtain a low-light image to be enhanced.

[0051] Step 2, graying the low-light image to be enhanced, and inverting the grayed low-light image to be enhanced to obtain an inverted gray image;

[0052] Step 3: input the low-light image to be enhanced and the inverted grayscale image into a low-light image enhancement network to obtain an enhanced image.

[0053] like Figure 2 As shown, the low-light image enhancement network in this embodiment includes a generator, and the generator includes:

[0054] A feature extraction module is used to extract features from an input image by combining short-range convolution and long-range convolution;

[0055] The feature recovery module is used to reconstruct the features extracted by the feature extraction module by combining short-range convolution and long-range convolution to obtain an enhanced image.

[0056] The generator of this embodiment adopts a 10-layer U-shaped structure with jump connections, and its input is a low-light image and its inverted grayscale image. Among them, the feature extraction module includes 4 groups of long- and short-range feature extraction modules and a maximum pooling layer connected in sequence; the feature recovery module includes 4 groups of bicubic linear interpolation layers and a long- and short-range feature detail reconstruction module connected in sequence; the 4 groups of long- and short-range feature extraction modules and the 4 groups of long- and short-range feature detail reconstruction modules are jump-connected to alleviate the information loss caused by downsampling through jump connections.

[0057] The feature extraction part of the traditional Unet is usually composed of a 3*3 convolution with a step size of 1 and a padding value of 1, combined with an activation function and a normalization layer to extract features and increase the number of channels, and then a set of the same combination is used to further optimize the extracted features, and finally a pooling layer is used to reduce the feature size. Due to the limitations of the convolution operation, such a configuration can only extract short-range features. When performing feature extraction, in places with high signal-to-noise ratios, short-range convolution operations can extract sufficient feature information, but in places with low signal-to-noise ratios, it is difficult to extract effective features by relying solely on short-range convolutions. Therefore, a long short-range feature extraction (LSFE) module is used in this embodiment. The short- and long-range feature extraction module includes: a first short-range convolution operation part, including a convolution unit, a SELU activation layer unit and a batch normalization layer unit arranged in sequence, for performing a short-range convolution operation on the input feature to obtain local feature information; a first long-range convolution operation part, including a focus module unit, a convolution unit, an upsampling operation unit, a SELU activation layer unit and a batch normalization layer unit arranged in sequence, for performing a long-range convolution operation on the input feature to obtain non-local feature information; a first fusion part, for fusing the local feature information and non-local feature information; and a sorting and optimization part, for sorting and optimizing the fused local feature information and non-local feature information.

[0058] like Figure 2The LSFE module in the LSFE module first divides the features transmitted by the previous layer into two operations. The first part is a short-range convolution operation part composed of a 3*3 convolution with a step size of 1 and a padding value of 1, a SELU activation function and a batch normalization layer to extract local feature information; the second part uses a long-range convolution operation part to extract non-local feature information, where the long-range convolution operation part is mainly composed of a focus module, a 3*3 convolution, an upsampling operation, a SELU activation layer and a batch normalization layer. The focus module connects (concat) all pixels on each adjacent 2*2 block to the channel to obtain a feature with a size reduced by half and the number of channels*4, and then uses a 1*1 convolution to convolve the number of channels back to the original size. By borrowing the focus operation to simply simulate the blocking and encoding operations of the transformer, this implementation method believes that the features in adjacent 4*4 blocks are similar, so the focus operation can reduce redundant information and increase the receptive field for subsequent convolution operations. After the focus module, a 3*3 convolution is used to extract features. Due to the existence of the focus module, the convolution here has a larger receptive field than the short-range convolution. After feature extraction, bilinear interpolation is used to double upsample to increase the feature size for subsequent information fusion. Finally, the obtained features are activated by a SELU and a batch normalization layer to complete the long-range convolution operation. After the above feature extraction, a 1*1 convolution is used to fuse the extracted local feature information (short-range information) and non-local feature information (long-range information). Finally, a 3*3 convolution with a step size of 1 and a fill value of 1 is used, combined with a SELU activation function and a batch normalization layer to further organize and optimize the extracted features.

[0059] Because the brightness of the input low-light image is generally inconsistent, and the signal-to-noise ratio of each area is also different, the number of effective pixels extracted in the feature extraction stage is also inconsistent. If the same operation is performed on all areas, the area with a high signal-to-noise ratio is well reconstructed, but the area with a low signal-to-noise ratio is directly blurred. Therefore, this embodiment proposes a long short-range feature reconstruction (LSFR) module, which uses long-range convolution operations to obtain non-local information and guide short-range convolution operations to perform local feature reconstruction. The long- and short-range feature detail reconstruction module includes: a feature reconstruction part, which is used to reconstruct the input feature to obtain the reconstructed feature; a second short-range convolution operation part, which includes a convolution unit, a SELU activation layer unit and a batch normalization layer unit arranged in sequence, which is used to connect the reconstructed feature with the output feature of the symmetrical long- and short-range feature extraction module, and perform a short-range convolution operation on the connected feature to obtain the short-range feature; a second long-range convolution operation part, which includes a focus module unit, a convolution unit, an upsampling operation unit, a SELU activation layer unit and a batch normalization layer unit arranged in sequence, which is used to perform a long-range convolution operation on the reconstructed feature to obtain the long-range feature; a second fusion part, which is used to fuse the short-range feature and the long-range feature; and an adjustment part, which is used to adjust the fused short-range feature and the long-range feature.

[0060] like Figure 2 In the LSFR module in , the input features are first upsampled twice by bilinear interpolation, and then a 3*3 convolution with a step size of 1 and a padding value of 1 is used to reconstruct the features to obtain the reconstructed features. Then the reconstructed features are respectively subjected to short-range convolution and long-range convolution operations. When performing the short-range convolution operation, the reconstructed features are connected to the output of the symmetric LSFE module, and a 3*3 convolution with a step size of 1 and a padding value of 1 is used to extract local features. By connecting the output of the symmetric LSFE module, the network can be re-contacted with shallow information, avoiding the loss of some detailed information as the number of network layers increases; when performing the long-range convolution operation, the output reconstructed features are directly subjected to long-range convolution operations. Because the long-range convolution operation obtains non-local information, it is not necessary to connect the output features of the symmetric LSFE module like the short-range convolution operation, which can reduce the amount of calculation and the amount of parameters. This residual connection operation can reduce the pressure of network learning. Then the extracted long-range features and short-range features are fused using 1*1 convolution. Finally, a set of 3*3 convolutions with a step size of 1 and a padding value of 1, SELU activation layers and batch normalization functions are used to adjust the fusion features for better fusion.

[0061] In the process of training an enhanced network using a generative adversarial model, a single data-enhanced image is usually used as the input of the network, and then the discriminator can guide the network to generate an image of normal illumination. However, the enhancement effect of the network is still poor in areas with low local brightness and low signal-to-noise ratio. The main reason is that such areas account for a small proportion of the entire image or the entire training data set. For the discriminator, the main basis for judging whether it belongs to a normal illumination image may be obvious features such as image brightness and contrast. At the same time, in order to ensure that the restored image does not lose the features of the original image, the feature loss requires that the VGG features of the input image and the VGG features of the generated image are consistent, which may include noise. Therefore, if there is no paired reference image as a guide, it is often difficult for the network to obtain a good result. Directly using supervised learning with corresponding reference images successfully avoids this problem by guiding the network to generate images that are as consistent as possible with the reference image. However, in the supervised learning process, the network learns a one-to-one mapping between the input image and the reference image, so the generalization ability of the model is poor, and it is difficult to achieve the same effect on a new data set.

[0062] To solve this problem, Figure 2 As shown, the low-illumination image enhancement network in this embodiment includes two generators and a discriminator; the discriminator includes: a global discriminator and a local discriminator, which are used to ensure that the generated image is closer to the normal illumination image; a VGG16 model is used to construct a feature loss to ensure that the generated image maintains the characteristics of the original image; wherein the weights of the two generators are shared to generate a reference image for constructing a consistency loss, thereby ensuring that the network learns the essential characteristics of the image.

[0063] The low-light image enhancement network in this embodiment includes the following steps during training:

[0064] First, the low-light training image is subjected to two random data enhancement operations to obtain a first input image inputA and a second input image inputA0, wherein the first input image inputA and the second input image inputA0 are similar but different; the random data enhancement operation is as follows: the low-light training image is subjected to random brightness adjustment using a gamma function with a probability of 50% to obtain a first image, wherein the parameter value range of the gamma function is 1 to 1.7; then the first image is subjected to color adjustment with a probability of 50% to obtain a second image; then, a Gaussian noise with a mean of 0 and a standard deviation of 0 to 5 is added to the second image with a probability of 50% to obtain a third image; finally, a mask is superimposed on the third image with a probability of 50%, and the pixel values ​​of the points covered by the mask become the mean of the surrounding 4*4 area, such as Figure 3As shown. The coverage of the mask is 75% at the beginning of training, and it decreases by 25% every 50 epochs until it reaches 0. Among them, in order to guide the network to generate clearer and more obvious features, only one of the first input image inputA and the second input image inputA0 will add a mask. Therefore, the two images obtained may be randomly adjusted in brightness, randomly added with noise, randomly changed in color, and randomly added with masks. This ensures that the two images are similar but not the same, and also ensures that the data sets composed of the first input image inputA and the second input image inputA0 have similar distributions, which is conducive to the stability of training.

[0065] Next, grayscale processing is performed on the first input image inputA and the second input image inputA0 respectively, and the grayscale processed first input image and the second input image are inverted to obtain a first inverted grayscale input image grayA and a second inverted grayscale input image grayA0;

[0066] Then, the first input image inputA and the first inverted grayscale input image grayA are input into the generator to obtain the training enhanced image fake_B; the second input image inputA0 and the second inverted grayscale input image grayA0 are input into the generator to obtain the reference image fake_B0; wherein, the gradients of the second input image inputA0, the second inverted grayscale input image grayA0 and the reference image fake_B0 are not calculated.

[0067] Then, the VGG16 model is used to extract feature maps of the training enhanced image fake_B, the reference image fake_B0 and the corrected image gamma_A respectively; the corrected image gamma_A is an image obtained after gamma correction is performed on the low-illumination training image.

[0068] Afterwards, feature preservation loss is used to ensure that the training enhanced image and the corrected image retain the detailed features of the low-light training image, and feature consistency loss is used to ensure that the features of the training enhanced image fake_B and the reference image fake_B0 are consistent. Since the corrected image gamma_A has a better illumination condition than the low-light training image, the features extracted by the VGG16 model are more significant. Using the corrected image gamma_A as a reference for feature preservation loss can make the brightness of the training enhanced image fake_B more uniform. The loss function of the VGG16 model includes the following expression:

[0069]

[0070] Among them, L SPF represents the feature preservation loss, represents the feature consistency loss of the VGG16 model, W i,j Indicates the width of the extracted feature map, H i,j Indicates the height of the extracted feature map, I Ag Represents the original low-light image I A The corrected image (i.e., the corrected image gamma_A) is expressed as: Ag =I g ×I A +(1-I g )×(I A ) 0.5 , I g is the original low-light image I A Grayscale image of I fB and I fB0 is the output image of the generator, namely the training enhanced image fake_B and the reference image fake_B0. ij (·) represents the pre-trained VGG16 model, i represents the i-th pooling, and j represents the j-th convolutional layer after the i-th pooling.

[0071] The global discriminator is used to determine whether the training enhanced image fake_B and the reference image fake_B0 are true normal illumination images; wherein the global discriminator is composed of 7 4*4 convolutions and 6 LeakyReLu activation functions.

[0072] After randomly cropping the training enhanced image fake_B and the reference image fake_B0, 5 32*32 training enhanced image slices fakeB_pacth and reference image slices fakeB0_pacth are obtained respectively, and the training enhanced image slices fakeB_pacth and the reference image slices fakeB0_pacth are input into the VGG16 model to obtain the local features of the training enhanced image and the local features of the reference image;

[0073] The local discriminator is used to determine whether the local features of the training enhanced image and the local features of the reference image are true normal illumination features; wherein the local discriminator is composed of 5 4*4 convolutions, 4 LeakyReLu activation functions and 1 Sigmoid function. For the global discriminator and the local discriminator, the closer the score of the discriminator is to the normal illumination image, the closer it is to 1, and the closer the score of the discriminator is to the normal illumination image, the closer it is to 0. The loss of the generative adversarial network part in this embodiment is as follows:

[0074]

[0075] in, represents the global discriminator loss, represents the local discriminator loss, LG represents the generator loss, represents the discriminator feature consistency loss, Represents x r is the probability of true data, Represents x f is the probability of generating data, Represents x r is the probability of real shard data, Represents x f is the probability of generating shard data, Represents x r is the probability of true data and x f is the probability of generating data, Represents x r is the probability of real shard data and x f is the probability of generating shard data, Represents x f is the probability of generating data and x f0 is the probability of generating data, Represents x f is the probability of generating shard data and x f0 is the probability of generating shard data, D G and D L are the global discriminator and the local discriminator respectively, x r 、x f and x f0 They are respectively the sampling of the real image realB distribution, the sampling of the training enhanced image fake_B distribution and the sampling of the reference image fake_B0 distribution.

[0076] Finally, the generator training is guided by the outputs of the global discriminator, the local discriminator and the VGG16 model. In order to allow the low-light image enhancement network to ignore the influence of the brightness and noise of the input image and learn the most essential features of the input image, this implementation uses information consistency loss, which constrains the low-light image enhancement network so that the enhancement results of two images with different brightness signal-to-noise ratios, the extracted features and the final results are consistent, which can be expressed as:

[0077]

[0078] Among them, L con represents feature consistency loss, L1() represents L1 norm loss, L2() represents L2 norm loss, represents the feature map of the i-th layer of the generator of fake_B, Represents the feature map of the i-th layer of the generator of fake_B0.

[0079] All the losses L of the generator in this implementation can be expressed as:

[0080]

[0081] This implementation uses EnlightenGAN as the basic training data set, and makes some additions and subtractions to the data. It deletes low-light images that appear too many times or reference images with normal light that have poor visual effects. During training, the image size is randomly cropped to 320*320, and the input images are randomly flipped up and down and left and right. Tests are conducted on datasets such as LOL, LOL-v2, DICM, LIME, MEF, VV, and NPE. Based on Pytorch, the Adam optimizer (β1=0.9, β 12 =0.999). The training was performed for 200 epochs, with the learning rate kept constant at 0.0001 for the first 100 epochs and linearly reduced to 0 for the next 100 epochs. During training, the training ratio of the generator and the discriminator was 1:1, and the generator and the discriminator used the same optimizer and learning rate.

[0082] It is not difficult to find that the generator part of the present invention is composed of multiple long-range and short-range feature extraction modules and multiple long-range and short-range detail reconstruction modules. By adopting long-range and short-range operations to extract information of different receptive fields, it is possible to use non-local information to improve the information extraction capability while preserving local details, thereby ensuring that useful information can be extracted in areas with low signal-to-noise ratio. In the feature reconstruction stage, the long-range convolution operation with a larger receptive field guides the short-range convolution to better reconstruct the features, so that the enhanced low-light image has higher image quality.

[0083] A second embodiment of the present invention relates to an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the low-light image enhancement method based on a self-guided learning framework of the first embodiment when executing the computer program.

[0084] A third embodiment of the present invention relates to a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of the low-illumination image enhancement method based on a self-supervised learning framework of the first embodiment are implemented.

[0085] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) that contain computer-usable program code.

[0086] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0087] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction method, which is implemented in the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0088] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0089] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A low-light image enhancement method based on a self-guided learning framework, It is characterized in that The following steps are involved: Acquire a low-light image to be enhanced; grayscale the low-light image to be enhanced, and invert the grayscale image to be enhanced to obtain an inverted grayscale image; Inputting the low-light image to be enhanced and the inverted grayscale image into a low-light image enhancement network to obtain an enhanced image; The low-light image enhancement network includes a generator; The generator comprises: A feature extraction module is used to extract features from an input image by combining short-range convolution and long-range convolution; The feature recovery module is used to reconstruct the features extracted by the feature extraction module by combining short-range convolution and long-range convolution to obtain an enhanced image.

2. The low-light image enhancement method based on the self-guided learning framework according to claim 1, It is characterized in that The feature extraction module includes M groups of long-range and short-range feature extraction modules and maximum pooling layers connected in sequence; the feature recovery module includes M groups of bicubic linear interpolation layers and long-range and short-range feature detail reconstruction modules connected in sequence; the long-range and short-range feature extraction modules are jump-connected to the M groups of long-range and short-range feature detail reconstruction modules.

3. The low-light image enhancement method based on the self-guided learning framework according to claim 2, It is characterized in that The long- and short-range feature extraction module comprises: The first short-range convolution operation part includes a convolution unit, a SELU activation layer unit and a batch normalization layer unit which are arranged in sequence, and is used to perform a short-range convolution operation on the input features to obtain local feature information; The first long-range convolution operation part includes a focus module unit, a convolution unit, an upsampling operation unit, a SELU activation layer unit, and a batch normalization layer unit, which are arranged in sequence, and are used to perform a long-range convolution operation on the input features to obtain non-local feature information; A first fusion part, used for fusing the local feature information with the non-local feature information; The sorting and optimization part is used to sort and optimize the fused local feature information and non-local feature information.

4. The low-light image enhancement method based on the self-guided learning framework according to claim 2, It is characterized in that The long-range and short-range feature detail reconstruction module comprises: The feature reconstruction part is used to reconstruct the input features to obtain the reconstructed features; The second short-range convolution operation part includes a convolution unit, a SELU activation layer unit and a batch normalization layer unit which are arranged in sequence, and is used to connect the reconstructed features and the output features of the long-short-range feature extraction module of the symmetric position, and perform a short-range convolution operation on the connected features to obtain short-range features; The second long-range convolution operation part includes a focus module unit, a convolution unit, an upsampling operation unit, a SELU activation layer unit, and a batch normalization layer unit, which are arranged in sequence, and are used to perform a long-range convolution operation on the reconstructed features to obtain long-range features; A second fusion part, used for fusing the short-range features with the long-range features; The adjustment part is used to adjust the fused short-range features and long-range features.

5. The low-light image enhancement method based on the self-guided learning framework according to claim 1, It is characterized in that The low-light image enhancement network also includes a discriminator; the discriminator includes: Global discriminator and local discriminator are used to ensure that the generated image is closer to the normal illumination image; The VGG16 model is used to construct feature loss to ensure that the generated image retains the characteristics of the original image.

6. The low-light image enhancement method based on the self-guided learning framework according to claim 5, It is characterized in that The low-light image enhancement network includes the following steps during training: Performing two random data augmentation operations on the low-light training image to obtain a first input image and a second input image, wherein the first input image is similar to but different from the second input image; Performing grayscale processing on the first input image and the second input image respectively, and performing inversion processing on the grayscaled first input image and the second input image to obtain a first inverted grayscale input image and a second inverted grayscale input image; Inputting the first input image and the first inverted grayscale input image into a generator to obtain a training enhanced image; Inputting the second input image and the second inverted grayscale input image into a generator to obtain a reference image; The VGG16 model is used to extract feature maps of the training enhanced image, the reference image and the corrected image respectively; the corrected image is an image obtained by performing gamma correction on the low-illumination training image; Using feature preservation loss to ensure that the training enhanced image and the corrected image retain the detailed features of the low-illuminance training image, and using feature consistency loss to ensure that the features of the training enhanced image and the reference image are consistent; using the global discriminator to determine whether the training enhanced image and the reference image are true normal illumination images; After randomly cropping the training enhanced image and the reference image, training enhanced image slices and reference image slices are obtained respectively, and the training enhanced image slices and the reference image slices are input into the VGG16 model to obtain training enhanced image local features and reference image local features; Using the local discriminator to determine whether the local features of the training enhanced image and the local features of the reference image are true features of normal illumination; The generator training is guided by the outputs of the global discriminator, the local discriminator and the VGG16 model.

7. The low-light image enhancement method based on the self-guided learning framework according to claim 6, It is characterized in that The random data enhancement operation refers to performing random brightness adjustment on the low-light training image with a probability of 50% to obtain a first image; performing color adjustment on the first image with a probability of 50% to obtain a second image; performing noise addition processing on the second image with a probability of 50% to obtain a third image; and adding a mask to the third image with a probability of 50% to obtain an input image.

8. The low-light image enhancement method based on the self-guided learning framework according to claim 7, It is characterized in that At the beginning of training, the coverage rate of the mask is 75%, and the pixel value of the point covered by the mask is the average of the four neighborhoods of the pixel at the point; After each preset number of trainings, the coverage of the mask is reduced by 25% until the coverage is 0.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the low-illumination image enhancement method based on the self-guided learning framework as described in any one of claims 1-8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, It is characterized in that When the computer program is executed by a processor, the steps of the low-illumination image enhancement method based on a self-guided learning framework as described in any one of claims 1 to 8 are implemented.

Images