Image enhancement method and device

Abstract

An embodiment of the present application provides an image enhancement method and device. In this method, multiple non-uniform brightness images are determined based on a second illumination image and multiple images of different brightness obtained by performing multiple brightness transformations on a first illumination image. Then, an image enhancement model is trained based on the first illumination image and the multiple non-uniform brightness images. The first illumination image refers to an image captured in an environment with an illumination greater than a set light intensity, and the second illumination image refers to an image captured in an environment with an illumination not greater than a set light intensity. When performing image enhancement, the image enhancement model completed based on this training can take into account the complexity of illumination, better simulate real illumination, improve the image enhancement effect, and improve the robustness of image enhancement. In addition, the embodiment of the present application does not require the capture of paired light and dark images, so the generalization can be improved, and the present application is interpretable and meets the trustworthiness characteristic.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an image enhancement method and device. Background Art

[0002] In low-light conditions at night, the images captured by cameras are often dim and their clarity is far below what the naked eye can see. Currently, image enhancement solutions can be used to improve video quality and clarity. These solutions enhance images captured in low-light conditions, making them clearer or closer to what the naked eye can see.

[0003] Commonly used image enhancement schemes include supervised algorithms. Supervised algorithms require simultaneously acquiring paired daytime and nighttime images of the same scene. However, the lighting in actual scenes is complex and changeable. This method cannot simulate real lighting, resulting in poor enhancement effects. Summary of the Invention

[0004] The embodiments of the present application provide an image enhancement method and device to solve the problem of poor enhancement effect in the prior art.

[0005] In a first aspect, an embodiment of the present application provides an image enhancement method, the method comprising:

[0006] Acquire an image captured by an image acquisition device;

[0007] Inputting the image into a pre-trained image enhancement model, the image enhancement model is trained based on a first illumination image and a plurality of non-uniform brightness images, the plurality of non-uniform brightness images are determined based on a second illumination image and a plurality of images of different brightness obtained by performing multiple brightness transformations on the first illumination image, the first illumination image is an image captured under an environment with an illumination greater than a set light intensity, and the second illumination image is an image captured under an environment with an illumination no greater than the set light intensity;

[0008] Based on the image enhancement model, an enhanced image of the image is output.

[0009] In a second aspect, an embodiment of the present application provides an electronic device, which includes at least a processor and a memory, and the processor is used to implement the steps of the image enhancement method as described in any one of the above items when executing a computer program stored in the memory.

[0010] In a third aspect, an embodiment of the present application provides an image enhancement device, comprising:

[0011] An acquisition module, used to acquire images acquired by an image acquisition device;

[0012] an input module, configured to input the image into a pre-trained image enhancement model, the image enhancement model being trained based on a first illumination image and a plurality of non-uniform brightness images, the plurality of non-uniform brightness images being determined based on a second illumination image and a plurality of images of different brightness obtained by performing multiple brightness transformations on the first illumination image, the first illumination image being an image captured in an environment having an illumination greater than a set light intensity, and the second illumination image being an image captured in an environment having an illumination no greater than the set light intensity;

[0013] An output module is used to output an enhanced image of the image based on the image enhancement model.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the image enhancement method as described in any one of the above items.

[0015] In an embodiment of the present application, an image captured by an image acquisition device is obtained; the image is input into a pre-trained image enhancement model, the image enhancement model is trained based on a first illumination image and a plurality of non-uniform brightness images, the plurality of non-uniform brightness images are determined based on a second illumination image and a plurality of images of different brightness obtained by performing multiple brightness transformations on the first illumination image; based on the image enhancement model, an enhanced image of the image is output. In this method, the first illumination image refers to an image captured in an environment with an illumination greater than a set light intensity, the second illumination image refers to an image captured in an environment with an illumination not greater than a set light intensity, a plurality of non-uniform brightness images are determined based on the second illumination image and a plurality of images of different brightness obtained by performing multiple brightness transformations on the first illumination image, and then the image enhancement model is trained based on the first illumination image and the plurality of non-uniform brightness images. When performing image enhancement based on the image enhancement model completed by this training, the complexity of illumination can be taken into account, the real illumination can be better simulated, and the image enhancement effect can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0017] Figure 1 A schematic diagram of an image enhancement process provided for some embodiments of the present application;

[0018] Figure 2 A schematic diagram of an image enhancement process provided for some embodiments of the present application;

[0019] Figure 3 A schematic diagram of a process for extracting a daytime background area image provided in some embodiments of the present application;

[0020] Figure 4 A schematic diagram of a process for performing brightness analysis on nighttime images provided in some embodiments of the present application;

[0021] Figure 5 A schematic diagram of grayscale analysis of an image provided in some embodiments of the present application;

[0022] Figure 6 A schematic diagram of a process for synthesizing a non-uniform brightness image provided in some embodiments of the present application;

[0023] Figure 7 A schematic structural diagram of an image enhancement model provided in some embodiments of the present application;

[0024] Figure 8 A schematic diagram of an image enhancement process provided for some embodiments of the present application;

[0025] Figure 9 Schematic diagram of the RepVGG model structure provided for some embodiments of the present application;

[0026] Figure 10A A schematic diagram of a conventional CBR module structure provided for some embodiments of the present application;

[0027] Figure 10B A schematic diagram of the RepVGG-CBR module structure provided for some embodiments of the present application;

[0028] Figure 11 A schematic diagram of the structure of a RepVGG-pooling module provided for some embodiments of the present application;

[0029] Figure 12A A schematic diagram of the current contract module structure provided for some embodiments of the present application;

[0030] Figure 12B A schematic diagram of the structure of an improved RepVGG-contraction module provided for some embodiments of the present application;

[0031] Figure 13 A schematic structural diagram of an image enhancement model provided in some embodiments of the present application;

[0032] Figure 14 A schematic diagram of the attention structure of a first enhancement network provided in some embodiments of the present application;

[0033] Figure 15 A schematic diagram of the attention structure of a second enhancement network provided in some embodiments of the present application;

[0034] Figure 16 A schematic diagram of an image enhancement process provided for some embodiments of the present application;

[0035] Figure 17 A schematic structural diagram of an image enhancement device provided in some embodiments of the present application;

[0036] Figure 18 A schematic structural diagram of an electronic device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0037] To make the objectives, technical solutions, and advantages of this application more clear, this application will be further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0038] In order to better simulate real lighting and improve image enhancement effects, the present application provides an image enhancement method and device. Figure 1 As shown, in an embodiment of the present application, the acquired image is input into a pre-trained image enhancement model, and based on the image enhancement model, an enhanced image of the image is output, wherein the image enhancement model is trained based on a first illumination image and multiple non-uniform brightness images, and the multiple non-uniform brightness images are determined based on a second illumination image and multiple images of different brightness obtained by performing multiple brightness transformations on the first illumination image. The first illumination image refers to an image captured in an environment with an illumination greater than the set light, and the second illumination image refers to an image captured in an environment with an illumination not greater than the set light. Therefore, when performing image enhancement, the image enhancement model can take into account the complexity of illumination, better simulate real illumination, and improve the image enhancement effect.

[0039] Some embodiments of the present application provide an image enhancement process such as Figure 2 As shown, the following steps are included:

[0040] S201: Acquire an image captured by an image acquisition device.

[0041] The image enhancement method provided in the embodiments of the present application is applied to an electronic device, which may be an image acquisition device, a user device, or a server. The image acquisition device may include a security monitoring device in a security monitoring scenario, and the user device may include but is not limited to a mobile phone, a computer, a wearable device, or a home appliance.

[0042] The image acquisition device can capture images or video streams. If the electronic device is an image acquisition device, the electronic device can directly obtain the images or video streams captured by itself. If the electronic device is not an image acquisition device, the electronic device obtains the images or video streams it captures from the image acquisition device. Taking the video stream as an example, the electronic device first decodes the video stream to obtain an image sequence, and then the electronic device can perform image enhancement on each frame of the image sequence, or perform image enhancement on part of the images in the image sequence. For example, when performing image enhancement on part of the images in the image sequence, the electronic device can perform frame skipping processing on the image sequence, for example, selecting one frame of image every other frame (for example only) to obtain the final image sequence image1, image2,...imagen... This can improve the efficiency of the algorithm.

[0043] In one implementation, the electronic device may further detect whether the acquired image is to be enhanced, and only execute the following S202 if it is determined that the image enhancement is to be performed.

[0044] S202: Input the image into a pre-trained image enhancement model, where the image enhancement model is trained based on a first illumination image and multiple non-uniform brightness images, where the multiple non-uniform brightness images are determined based on a second illumination image and multiple images of different brightness obtained by performing multiple brightness transformations on the first illumination image.

[0045] Among them, the first illumination image refers to an image captured in an environment with an illumination greater than the set light illumination, and the first illumination image can be understood as a high illumination image or a normal illumination image. Generally, the light illumination of the daytime environment is greater than the set light illumination, and the first illumination image can be a daytime image. The second illumination image refers to an image captured in an environment with an illumination not greater than the set light illumination, and the first illumination image can be understood as a low illumination image. Generally, the light illumination of the nighttime environment is not greater than the set light illumination, and the second illumination image can be a nighttime image. In the embodiment of the present application, there is no restriction on the value of the set light illumination.

[0046] The electronic device stores a pre-trained image enhancement model, which can be trained by one of the following implementation methods:

[0047] Implementation method 1: The image enhancement model is trained through supervised learning. For example, the image enhancement model first performs image enhancement on multiple uneven brightness images, and then trains the image enhancement model based on a daytime image (an example of a first illumination image) and multiple enhanced uneven brightness images.

[0048] Implementation method 2: The image enhancement model is trained through supervised and unsupervised learning. For example, the image enhancement model first performs image enhancement on multiple uneven brightness images, then performs supervised training on a daytime image (an example of a first illumination image) and the enhanced multiple uneven brightness images. Alternatively, the image enhancement model first performs image enhancement on a nighttime image (an example of a second illumination image), then performs unsupervised training on the enhanced nighttime image.

[0049] The multiple non-uniform brightness images are determined based on the second illumination image and multiple images of different brightness obtained by performing multiple brightness transformations on the first illumination image. Therefore, the multiple non-uniform brightness images can reflect the complexity of illumination. The image enhancement model obtained based on this training can better simulate real illumination.

[0050] S203: Based on the image enhancement model, output an enhanced image of the image.

[0051] Since the image enhancement model is pre-trained, an enhanced image of the image can be output based on the image enhancement model.

[0052] In an embodiment of the present application, multiple non-uniform brightness images are determined based on the second illumination image and multiple images of different brightness obtained by performing multiple brightness transformations on the first illumination image. An image enhancement model is then trained based on the first illumination image and the multiple non-uniform brightness images. This trained image enhancement model can take into account illumination complexity when performing image enhancement, better simulating real-world illumination, improving image enhancement effects, and enhancing the robustness of image enhancement. Furthermore, the embodiment of the present application does not require capturing paired light and dark images, thereby improving generalization and interpretability.

[0053] Based on the above embodiment, for the first implementation of S202, the training process of the image enhancement model includes:

[0054] Acquire a training set, where the training set includes a first illumination image acquired by an image acquisition device;

[0055] An image enhancement model is trained according to the first illumination image and the plurality of non-uniform brightness images.

[0056] Among them, multiple non-uniform brightness images are determined based on the second illumination image and multiple images of different brightness. Therefore, in addition to the first illumination image, the training set can also include a second illumination image captured by the same image acquisition device. The set including multiple non-uniform brightness images can be prepared in advance or generated in each round of training iteration. For example, the electronic device performs multiple brightness transformations on the first illumination image to obtain multiple images of different brightness, and then determines multiple non-uniform brightness images based on the second illumination image and the multiple images of different brightness. There is no limitation on the implementation method of the brightness transformation here, for example, but not limited to, performing multiple brightness transformations on the first illumination image by adjusting different transformation coefficients.

[0057] The image enhancement model in this embodiment may include one or more enhancement networks.

[0058] On the basis of the above embodiment, in the embodiment of the present application, based on the image enhancement model, the enhanced image of the output image includes:

[0059] Determining a first enhanced feature map of the image based on a first enhancement network in the image enhancement model;

[0060] Determine, based on a second enhancement network in the image enhancement model, a second enhancement feature map for the background area image of the image;

[0061] An enhanced image is output according to the first enhanced feature map and the second enhanced feature map.

[0062] In this embodiment, the image enhancement model includes multiple enhancement networks, including a first enhancement network and a second enhancement network. The first enhancement network can enhance the brightness of the image. For example, the enhancement effect of the first enhancement network can more closely resemble the lighting in a real scene. The second enhancement network can enhance the brightness of the background area of ​​the image. For example, the enhancement effect of the second enhancement network on the background area of ​​the image can improve color cast.

[0063] Then, the first enhanced feature map output by the first enhancement network and the second enhanced feature map output by the second enhancement network are fused to obtain a final enhanced image for the image.

[0064] In an embodiment of the present application, performing image enhancement on an image based on the first enhancement network and the second enhancement network in the image enhancement model can improve the image enhancement effect.

[0065] Based on the above embodiment, for the second implementation of S202, the training process of the image enhancement model includes:

[0066] Acquire a training set, where the training set includes a first illumination image and a second illumination image acquired by the same image acquisition device;

[0067] Training a first enhancement network in an image enhancement model according to the first illumination image and the plurality of non-uniform brightness images;

[0068] The second enhancement network in the image enhancement model is trained according to the first background area image and the second illumination image in the first illumination image.

[0069] The first enhancement network can be trained in a supervised manner based on the first illumination image and the multiple non-uniform brightness images. For example, the first enhancement network first performs image enhancement on the multiple non-uniform brightness images, and then supervised training is performed on the first enhancement network based on the first illumination image and the multiple enhanced non-uniform brightness images.

[0070] The second enhancement network can be trained unsupervised based on the first background region image (e.g., the daytime background region image) and the second illumination image in the first illumination image. For example, the second enhancement network first performs image enhancement on the second illumination image, then further enhances the enhanced second illumination image using the first background region image, and then unsupervisedly trains the second enhancement network based on the multiple enhanced second illumination images.

[0071] The background area usually refers to the fixed area in the image, such as buildings, trees, lawns, etc., while vehicles, pedestrians, etc. in the image belong to the foreground area. For any fixed point image, the background area only changes in brightness over a period of time, while the content does not change. Therefore, these areas of normal daytime images can be used to enhance nighttime images. Figure 3 As shown, the electronic device can use a background extraction algorithm to extract the first background area image from the first illumination image. When the electronic device extracts the first background area image from the first illumination image, it can use a frame difference method, or it can use an image segmentation method (such as yolov8), etc., and the embodiments of the present application do not limit this. The electronic device can also fuse the first background area images extracted from multiple frames of the first illumination image to determine the final first background area image, thereby improving the training accuracy. In addition, considering that the background content changes very little, the electronic device may not extract the first background area image frame by frame in real time, but may extract the first background area image at set intervals to reduce the amount of calculation. The electronic device can extract the first background area image in advance before training, or it can extract it in each round of training iteration.

[0072] In a possible implementation, after extracting the first background area image, the electronic device may further determine a background area template according to the first background area image. For the role of the background area template, please refer to the description of subsequent embodiments.

[0073] In an embodiment of the present application, by training the first enhancement network and the second enhancement network in the image enhancement model, the enhancement effect of subsequent images can be improved.

[0074] Based on the above embodiments, in an embodiment of the present application, training a first enhancement network in an image enhancement model according to a first illumination image and a plurality of non-uniform brightness images includes:

[0075] Inputting a plurality of uneven brightness images into a first enhancement network, and outputting a plurality of first feature maps based on the first enhancement network;

[0076] determining a first loss value according to the first illumination image, the plurality of first feature maps, and the first loss function;

[0077] The first enhancement network is trained according to the first loss value.

[0078] When training the first enhancement network, supervised training is performed using multiple first feature maps (the first feature maps may be enhanced feature maps obtained by enhancing the non-uniform brightness image) and the first illumination image. The first loss function may be a reference loss function, such as, but not limited to, a structural similarity index (SSIM) loss function and / or a mean-square error (MSE) function.

[0079] The electronic device can calculate corresponding multiple first loss values ​​based on the first illumination image and the multiple first feature maps, and then train the first enhancement network based on the multiple first loss values.

[0080] On the basis of the above embodiments, in an embodiment of the present application, the above method further includes:

[0081] determining a brightness region image according to a brightness region having a brightness value greater than a set brightness in the second illumination image;

[0082] The brightness area image and multiple images with different brightness are fused to determine multiple non-uniform brightness images.

[0083] For example, the second illumination image is a night image, see Figure 4 The electronic device can perform brightness analysis on the night image and then perform brightness division according to the brightness analysis result. Taking the second-order division as an example, the electronic device can decompose the night image Id into a brightness area image and a dark area image, wherein the brightness area image is the brightness area with a brightness value greater than the set brightness, that is, the high-brightness area image Ih, and the dark area image is the brightness area with a brightness value not greater than the set brightness, that is, the low-brightness area image Il.

[0084] Electronic devices can use grayscale histogram statistics to analyze the brightness of night images to obtain high-brightness areas (usually areas with lights) and low-brightness areas; or electronic devices can convert night images from red, green, blue (RGB) space to hue saturation value (HSV) space, and then extract the brightness data of the V channel to obtain high-brightness areas and low-brightness areas. Take the grayscale histogram statistics method to analyze the brightness of night images as an example, Figure 5 As shown, Figure 5 (a) in the figure shows the grayscale analysis of the daytime image. Figure 5 (b) in the figure shows the grayscale analysis diagram of the night image, where the horizontal axis is the grayscale range of the pixel value from 0 to 255, and the total axis is the number of pixels for each grayscale value. It can be seen that the pixel distribution of the night image is usually more concentrated and mainly distributed in the low-brightness area, while the grayscale distribution of the day image is usually more balanced. Therefore, the brightness analysis of the night image can be performed based on the brightness distribution.

[0085] When electronic devices perform brightness division based on brightness analysis results, they can be divided into multiple brightness levels from bright to dark based on the brightness analysis results. The more brightness levels are divided, the more layers of illumination areas are extracted, and the closer the synthesized image illumination is to the real scene, the better the effect. In the embodiment of this application, the two levels of high brightness and low brightness are used as an example, and the division results in a high brightness area image Ih and a low brightness area image Il. It should be understood that the embodiment of this application does not limit the number of brightness division levels or the division method.

[0086] In one implementation, the electronic device may also determine a brightness area template based on the highlight area image, that is, Figure 4 The highlight region template Imask in the image can be used as the attention image for the subsequent enhancement network. The low-brightness region in the highlight region template Imask can take a set value (for example, but not limited to 0), or the low-brightness region can be normalized according to the actual pixel value.

[0087] The second illumination image is also described as a night image. After determining the brightness area image in the night image in the above manner, the electronic device can fuse the brightness area image with multiple images of different brightness to synthesize multiple non-uniform brightness images. Figure 6As shown, the first illumination image is a daytime image. The electronic device performs multiple brightness transformations on the daytime image Io to obtain multiple images of varying brightness Ic. The highlighted area image Ih is then fused with the multiple images Ic of varying brightness to determine multiple uneven brightness images Im, where the daytime and nighttime images are captured at the same point. For example, during the fusion process, the electronic device can iterate over each pixel in the highlighted area image Ih. For each pixel, if the pixel value is greater than 0 (for example only), the electronic device copies the pixel value to the corresponding position in image Ic. Otherwise, the pixel value at image Ic is saved. After performing the corresponding processing on each pixel, the uneven brightness image Im is obtained. This approach ensures that during subsequent training and inference, highlighted pixel areas are de-emphasized, while non-highlighted pixels are enhanced. The fusion process can select either a portion of the highlighted area of ​​the highlighted area image Ih or the entire highlighted area, meaning that the selected area is not limited.

[0088] Because nighttime scenes have complex lighting conditions, and lighting conditions vary significantly across different scenes, simply exposing the daytime image to different low-brightness images would result in significantly different synthetic lighting from real lighting, failing to simulate the complexity of real lighting. However, in the present embodiment, the lighting corresponding to the second illumination image (e.g., nighttime lighting) is fused onto the generated low-brightness image to create a synthetic unevenly illuminated image, which increases the lighting complexity of the generated image.

[0089] On the basis of the above embodiments, in an embodiment of the present application, training the second enhancement network in the image enhancement model according to the first background area image and the second illumination image in the first illumination image includes:

[0090] Inputting the second illumination image into a second enhancement network, and outputting a second feature map based on the second enhancement network;

[0091] Fusing the background area template with the second feature map to obtain a fused second feature map;

[0092] Determining a second background area image in the fused second feature map;

[0093] determining a second loss value according to the first background area image, the second background area image, and the second loss function;

[0094] Determine a third loss value according to the fused second feature map and the third loss function;

[0095] The second enhanced network is trained according to the second loss value and the third loss value.

[0096] When training the second enhancement network, supervised training is performed using the first background area image (for example, the first background area image is a daytime background area image) and the second background area image (for example, the second background area image can be an enhanced background area image determined in the fused second feature map, wherein the second feature map can be a night-time enhanced image obtained by enhancing the night-time image), and unsupervised training is performed using the fused second feature map. The second loss function can be a reference loss function, for example, but not limited to, the second loss function includes an SSIM loss function and / or an MSE function, etc. The third loss function can be a non-reference loss function, for example, but not limited to, the third loss function includes a perceptual loss function and / or a total variation loss function.

[0097] The electronic device can calculate a corresponding second loss value based on the first background area image and the second background area image, calculate a corresponding third loss value based on the fused second feature map, and then train the second enhancement network based on the second loss value and the third loss value. In an embodiment of the present application, the advantages of supervised learning and unsupervised learning can be combined to achieve unsupervised learning of real scene images without obtaining paired samples, so that the generalization is better in real scenes. At the same time, combined with the advantages of supervised learning, supervised training using reference images can improve the accuracy and reliability of image enhancement, improve robustness, and have interpretability.

[0098] like Figure 7A schematic diagram of an image enhancement model provided by an embodiment of the present application is shown, assuming that the first illumination image is a daytime image and the second illumination image is a nighttime image. The training set of the image enhancement model includes a daytime image set and a nighttime image set. The images in each set are labeled with points, such as names. During each iterative training, a daytime image Io and a nighttime image Id of the same point are taken from the two image sets. Multiple uneven brightness images Im are then synthesized, along with a daytime background image Ibo of the daytime image Io, as training input. The image enhancement network includes a first loss function, a second loss function, and a third loss function. The first and second loss functions can be reference loss functions, while the third loss function can be a non-reference loss function. The electronic device inputs the multiple uneven brightness images Im into the first enhancement network, obtaining multiple enhanced feature maps Ime. The first loss value of the first loss function is then calculated based on the multiple enhanced feature maps Ime and the daytime image Io. Because the multiple uneven brightness images Im are related to highlight image templates, the highlight image templates can serve as attention maps to guide the first enhancement network to focus on darker areas in the image during training. The electronic device inputs the night image Id into the second enhancement network to obtain a night-enhanced image Ie. The night-enhanced image Ie is further enhanced using a background region template to obtain a fused night-enhanced image. A second loss value of a second loss function is then calculated based on the enhanced background region image Ibe in the fused night-enhanced image. A third loss value of a third loss function is then calculated based on the fused night-enhanced image. Finally, the electronic device can train the image enhancement model based on the first, second, and third loss values.

[0099] Whether in the training process or in the inference process, that is, in the image enhancement process, the electronic device can fuse the enhanced feature map Ime output by the first enhancement network and the fused night-time enhanced image output by the second enhancement network to obtain the final enhanced image. Figure 8 As shown, the enhanced feature map Ime output by the first enhancement network (an example of the above-mentioned first feature map) and the fused night-time enhanced image output by the second enhancement network (an example of the above-mentioned second feature map) are subjected to two convolutions, that is, the convolution layer-BN regularization-ReLU nonlinear activation (Conv-BN-ReLU, CBR) module and the 1*1 convolution layer (Conv) are processed to obtain the final enhanced image.

[0100] In one implementation, the image enhancement model is implemented using the UNET network (a network structure for semantic segmentation). UNet uses a fully convolutional neural network. The architecture consists of three parts: contraction, bottleneck, and expansion. The contraction part consists of multiple contraction blocks. Each block consists of two 3x3 convolutional layers and a 2x2 max pooling layer. After each block, the number of kernels or feature maps doubles, allowing the architecture to effectively learn complex structures. The bottom layer lies between the contraction and expansion layers. It uses two 3x3 convolutional neural network (CNN) layers, followed by a 2x2 upconvolution layer. The expansion part consists of multiple expansion blocks. Each block consists of two 3x3 convolutional layers and a 2x2 upsampling layer. The number of feature maps is halved after each convolution to maintain symmetry. The upsampling layer allows a low-resolution image containing high-level abstract features to be converted to high resolution while preserving these high-level abstract features. This image is then concatenated with the high-resolution image containing low-level surface features on the left. The number of expansion blocks is the same as the number of contraction blocks. Afterwards, the feature map is generated and classified using two 1x1 convolution kernels to produce two heatmaps. For example, the first heatmap represents the score for the first category, and the second heatmap represents the score for the second category. These are then used as input to the normalized exponential (softmax) function to calculate the softmax with the highest probability. Loss and backpropagation are then performed.

[0101] Based on the above embodiments, in the embodiments of the present application, the image enhancement model includes a RepVGG network structure. RepVGG is a fusionable network that runs multiple branches in parallel during the model training phase, each branch using convolution kernels of different sizes to increase the model's representational capabilities. During the model inference phase, the multi-branch model is converted into a single-branch structure, thereby speeding up inference, reducing memory usage, and facilitating actual deployment.

[0102] One way to implement this is Figure 9 As shown, Figure 9 (a) in the figure shows the structure diagram of the residual network (ResNet). RepVGG can use the residual edge of ResNet. The difference is that RepVGG uses residual connection in each layer, 3x3 convolution is connected to a feature (identity) and 1x1 convolution, for example Figure 9(b) in the figure represents the model structure of RepVGG during training. Figure 9 (c) in the figure represents the model structure of RepVGG during inference. In the training model, a batch normalization (BN) layer is added before the output of each branch.

[0103] For example, in an embodiment of the present application, the Unet network structure is improved, and the convolution of the encoder in the Unet network structure is replaced with RepVGG, thereby improving the model feature extraction capability. Figure 10A The current conventional CBR module structure is shown, including a Conv2d function and a BN layer. The output of the BN layer is connected to the linear rectification function (Relu). Figure 10B The RepVGG-CBR module structure is shown, which includes three branches, two of which include Conv2d functions and BN layers respectively, one branch includes a BN layer, and the outputs of the three branches are Relu. Figure 11 The RepVGG-pooling module structure is shown, which includes two branches, one of which includes a Conv2d function (parameters are k3, s2, p1) and a BN layer, and the other branch includes a Conv2d function (parameters are k1, s2, p0) and a BN layer. The outputs of the two branches are connected to Relu. Figure 12A The current contraction module structure is shown, including two CBR modules and a maximum pooling module. Figure 12B The improved RepVGG-contraction module structure is shown, which includes two RepVGG-CBR modules and one RepVGG-pooling module.

[0104] In an embodiment of the present application, the RepVGG network structure is introduced into the image enhancement model, which can speed up the inference speed and improve the image enhancement efficiency.

[0105] Based on the above embodiments, in an embodiment of the present application, the first enhancement network includes an attention network structure of a luminance area template, and the luminance area template is determined based on the luminance area image in the second illuminance image; and / or the second enhancement network includes an attention network structure of a background area template.

[0106] like Figure 13The figure shows a schematic diagram of the image enhancement model structure including the attention network structure. The image enhancement model encoder is composed of the above-mentioned multiple RepVGG-contraction modules. The upsampling and convolution in the network decoder stage adopt conventional operations. Figure 13 The black arrows in the middle are improvements of the embodiment of the present application, and the black bold arrows correspond to Figure 11 The RepVGG-pooling module shown, the black non-bold arrow corresponds to Figure 10B In the RepVGG-CBR module shown, the gray bold arrow represents up-conv 2*2, the gray non-bold arrow represents conv 3*3, and the activation function uses Relu.

[0107] The first enhancement network includes an attention network structure of the brightness area template, which can guide the first enhancement network to pay attention to the darker areas in the image during training. In one possible implementation, Figure 14 As shown in FIG, the first enhancement network includes a channel attention network structure and a highlight template attention network structure (i.e., the attention network structure of the brightness area template).

[0108] The second enhancement network includes an attention network structure of the background area template, which can improve the color cast problem caused by the lack of a reference image in unsupervised learning by supervising the brightness enhancement of the night background area (an example of the background area in the second illumination image). In one possible implementation, Figure 15 As shown, the second enhancement network includes a channel attention network structure and a background template attention network structure (i.e., the attention network structure of the background area template).

[0109] Based on the above embodiments, in an embodiment of the present application, before inputting the image into the pre-trained image enhancement model, the method further includes:

[0110] Perform grayscale statistics on the image to determine the brightness distribution of the image;

[0111] Determine whether the brightness distribution result conforms to the grayscale distribution law of the low-light image;

[0112] If yes, it is determined to perform image enhancement on the image.

[0113] After acquiring an image, the electronic device can first detect whether the image needs image enhancement based on the brightness distribution result of the image, that is, whether the image is a low-light image. Figure 16As shown, the electronic device acquires an image, performs brightness distribution analysis on the image, and determines whether the image is a low-light image based on the brightness distribution result. If so, the image is input into a pre-trained image enhancement model for image enhancement and outputs an enhanced image. If not, it returns to start re-acquiring the image.

[0114] In one implementation, the grayscale distribution law of the low-light image includes the grayscale distribution law of the night image. The electronic device uses a grayscale statistical method to determine the brightness distribution result of the image. Figure 5 It can be seen that there is a certain regularity in the grayscale distribution of daytime images and nighttime images. Therefore, electronic equipment can determine whether the brightness distribution result of the image conforms to the grayscale distribution regularity of the nighttime image. This method has a higher calculation speed and can improve the efficiency of image enhancement.

[0115] It is understandable that in some implementations, electronic devices may also use deep learning algorithms to classify acquired images into categories such as daytime images and nighttime images, where nighttime images are low-light images.

[0116] In an embodiment of the present application, the electronic device determines whether an image is a low-light image by grayscale statistics, and performs image enhancement on the low-light image, thereby improving the image enhancement efficiency.

[0117] In combination with the above embodiments, it can be seen that the present application can automatically analyze the brightness of the image, and then call the image enhancement model to perform image enhancement on the low-light image, which can effectively solve the overexposure problem and color cast problem after the enhancement of the image with uneven lighting. At the same time, it analyzes the internal pixel distribution of the image to improve the naturalness of the image itself, effectively improve the quality of the low-light image, and thus provide support for subsequent business scenario analysis.

[0118] Based on the above embodiments, the present application provides an image enhancement device. Figure 17 A schematic structural diagram of an image enhancement device provided in some embodiments of the present application, the device comprising:

[0119] An acquisition module 1701 is used to acquire an image captured by an image acquisition device;

[0120] Input module 1702 is used to input an image into a pre-trained image enhancement model, where the image enhancement model is trained based on a first illumination image and multiple non-uniform brightness images, where the multiple non-uniform brightness images are determined based on a second illumination image and multiple images of different brightness obtained by performing multiple brightness transformations on the first illumination image. The first illumination image refers to an image captured in an environment with an illumination greater than a set light intensity, and the second illumination image refers to an image captured in an environment with an illumination no greater than the set light intensity.

[0121] The output module 1703 is configured to output an enhanced image based on the image enhancement model.

[0122] In one possible embodiment, the output module 1703 is specifically used to determine a first enhanced feature map of the image based on the first enhancement network in the image enhancement model; determine a second enhanced feature map for the background area image of the image based on the second enhancement network in the image enhancement model; and output an enhanced image based on the first enhancement feature map and the second enhancement feature map.

[0123] In one possible embodiment, the device also includes a training module for obtaining a training set, the training set including a first illumination image and a second illumination image captured by the same image acquisition device; training a first enhancement network in the image enhancement model based on the first illumination image and multiple uneven brightness images; and training a second enhancement network in the image enhancement model based on the first background area image and the second illumination image in the first illumination image.

[0124] In one possible embodiment, the training module is specifically used to input multiple uneven brightness images into a first enhancement network, and output multiple first feature maps based on the first enhancement network; determine a first loss value based on the first illumination image, the multiple first feature maps and the first loss function; and train the first enhancement network based on the first loss value.

[0125] In a possible implementation, the training module is further configured to determine a brightness region image based on a brightness region in the second illumination image having a brightness value greater than a set brightness; and to fuse the brightness region image with a plurality of images of different brightness to determine a plurality of non-uniform brightness images.

[0126] In one possible embodiment, the training module is specifically used to input the second illumination image into the second enhancement network, and output the second feature map based on the second enhancement network; fuse the background area template and the second feature map to obtain a fused second feature map; determine the second background area image in the fused second feature map; determine the second loss value based on the first background area image, the second background area image and the second loss function; determine the third loss value based on the fused second feature map and the third loss function; and train the second enhancement network based on the second loss value and the third loss value.

[0127] In one possible implementation, the image enhancement model includes a RepVGG network structure.

[0128] In one possible embodiment, the first enhancement network includes an attention network structure of a luminance area template, where the luminance area template is determined based on a luminance area image in the second illumination image; and / or the second enhancement network includes an attention network structure of a background area template.

[0129] In one possible embodiment, the device includes a determination module for performing grayscale statistics on an image to determine a brightness distribution result of the image; judging whether the brightness distribution result conforms to the grayscale distribution law of a low-light image; and if so, determining to perform image enhancement on the image.

[0130] Based on the above embodiments, the present application further provides an electronic device, Figure 18 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 18 As shown, it includes: a processor 1801, a communication interface 1802, a memory 1803 and a communication bus 1804, wherein the processor 1801, the communication interface 1802, and the memory 1803 communicate with each other through the communication bus 1804;

[0131] The memory 1803 stores a computer program. When the program is executed by the processor 1801, the processor 1801 performs the following steps:

[0132] Acquire an image captured by an image acquisition device;

[0133] Input the image into a pre-trained image enhancement model, the image enhancement model is trained based on a first illumination image and multiple non-uniform brightness images, the multiple non-uniform brightness images are determined based on a second illumination image and multiple images of different brightness obtained by performing multiple brightness transformations on the first illumination image, the first illumination image refers to an image captured in an environment with an illumination greater than a set light intensity, and the second illumination image refers to an image captured in an environment with an illumination no greater than the set light intensity;

[0134] Based on the image enhancement model, an enhanced image of the image is output.

[0135] In a possible implementation, the processor 1801 is specifically configured to:

[0136] Determining a first enhanced feature map of the image based on a first enhancement network in the image enhancement model;

[0137] Determine, based on a second enhancement network in the image enhancement model, a second enhancement feature map for the background area image of the image;

[0138] An enhanced image is output according to the first enhanced feature map and the second enhanced feature map.

[0139] In a possible implementation, the processor 1801 is further configured to:

[0140] Acquire a training set, where the training set includes a first illumination image and a second illumination image acquired by the same image acquisition device;

[0141] Training a first enhancement network in an image enhancement model according to the first illumination image and the plurality of non-uniform brightness images;

[0142] The second enhancement network in the image enhancement model is trained according to the first background area image and the second illumination image in the first illumination image.

[0143] In a possible implementation, the processor 1801 is specifically configured to:

[0144] Inputting a plurality of uneven brightness images into a first enhancement network, and outputting a plurality of first feature maps based on the first enhancement network;

[0145] determining a first loss value according to the first illumination image, the plurality of first feature maps, and the first loss function;

[0146] The first enhancement network is trained according to the first loss value.

[0147] In a possible implementation, the processor 1801 is further configured to:

[0148] determining a brightness region image according to a brightness region having a brightness value greater than a set brightness in the second illumination image;

[0149] The brightness area image and multiple images with different brightness are fused to determine multiple non-uniform brightness images.

[0150] In a possible implementation, the processor 1801 is specifically configured to:

[0151] Inputting the second illumination image into a second enhancement network, and outputting a second feature map based on the second enhancement network;

[0152] Fusing the background area template with the second feature map to obtain a fused second feature map;

[0153] Determining a second background area image in the fused second feature map;

[0154] determining a second loss value according to the first background area image, the second background area image, and the second loss function;

[0155] Determine a third loss value according to the fused second feature map and the third loss function;

[0156] The second enhanced network is trained according to the second loss value and the third loss value.

[0157] In one possible implementation, the image enhancement model includes a RepVGG network structure.

[0158] In one possible embodiment, the first enhancement network includes an attention network structure of a luminance area template, where the luminance area template is determined based on a luminance area image in the second illumination image; and / or the second enhancement network includes an attention network structure of a background area template.

[0159] In a possible implementation, the processor 1801 is further configured to:

[0160] Perform grayscale statistics on the image to determine the brightness distribution of the image;

[0161] Determine whether the brightness distribution result conforms to the grayscale distribution law of the low-light image;

[0162] If yes, it is determined to perform image enhancement on the image.

[0163] The communication bus mentioned in the electronic device mentioned above may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, the figure shows only one thick line, but this does not mean that there is only one bus or only one type of bus.

[0164] The communication interface 1802 is used for communication between the electronic device and other devices.

[0165] The memory may include RAM (Random Access Memory) or NVM (Non-Volatile Memory), such as at least one disk storage. Alternatively, the memory may be at least one storage device located away from the processor.

[0166] The above-mentioned processor can be a general-purpose processor, including a central processing unit, NP (Network Processor), etc.; it can also be a DSP (Digital Signal Processing), a dedicated integrated circuit, a field programmable gate array or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, etc.

[0167] Based on the above embodiments, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program executable by an electronic device. When the program is executed on the electronic device, the electronic device performs the following steps:

[0168] Acquire an image captured by an image acquisition device;

[0169] Input the image into a pre-trained image enhancement model, the image enhancement model is trained based on a first illumination image and multiple non-uniform brightness images, the multiple non-uniform brightness images are determined based on a second illumination image and multiple images of different brightness obtained by performing multiple brightness transformations on the first illumination image, the first illumination image refers to an image captured in an environment with an illumination greater than a set light intensity, and the second illumination image refers to an image captured in an environment with an illumination no greater than the set light intensity;

[0170] Based on the image enhancement model, an enhanced image of the image is output.

[0171] In one possible implementation, based on the image enhancement model, outputting an enhanced image of the image includes:

[0172] Determining a first enhanced feature map of the image based on a first enhancement network in the image enhancement model;

[0173] Determine, based on a second enhancement network in the image enhancement model, a second enhancement feature map for the background area image of the image;

[0174] An enhanced image is output according to the first enhanced feature map and the second enhanced feature map.

[0175] In one possible implementation, the training process of the image enhancement model includes:

[0176] Acquire a training set, where the training set includes a first illumination image and a second illumination image acquired by the same image acquisition device;

[0177] Training a first enhancement network in an image enhancement model according to the first illumination image and the plurality of non-uniform brightness images;

[0178] The second enhancement network in the image enhancement model is trained according to the first background area image and the second illumination image in the first illumination image.

[0179] In one possible implementation, training a first enhancement network in an image enhancement model according to the first illumination image and the plurality of non-uniform brightness images includes:

[0180] Inputting a plurality of uneven brightness images into a first enhancement network, and outputting a plurality of first feature maps based on the first enhancement network;

[0181] determining a first loss value according to the first illumination image, the plurality of first feature maps, and the first loss function;

[0182] The first enhancement network is trained according to the first loss value.

[0183] In a possible implementation, the method further includes:

[0184] determining a brightness region image according to a brightness region having a brightness value greater than a set brightness in the second illumination image;

[0185] The brightness area image and multiple images with different brightness are fused to determine multiple non-uniform brightness images.

[0186] In a possible implementation, training the second enhancement network in the image enhancement model according to the first background area image and the second illumination image in the first illumination image includes:

[0187] Inputting the second illumination image into a second enhancement network, and outputting a second feature map based on the second enhancement network;

[0188] Fusing the background area template with the second feature map to obtain a fused second feature map;

[0189] Determining a second background area image in the fused second feature map;

[0190] determining a second loss value according to the first background area image, the second background area image, and the second loss function;

[0191] Determine a third loss value according to the fused second feature map and the third loss function;

[0192] The second enhanced network is trained according to the second loss value and the third loss value.

[0193] In one possible implementation, the image enhancement model includes a RepVGG network structure.

[0194] In one possible embodiment, the first enhancement network includes an attention network structure of a luminance area template, where the luminance area template is determined based on a luminance area image in the second illumination image; and / or the second enhancement network includes an attention network structure of a background area template.

[0195] In one possible implementation, before inputting the image into the pre-trained image enhancement model, the method further includes:

[0196] Perform grayscale statistics on the image to determine the brightness distribution of the image;

[0197] Determine whether the brightness distribution result conforms to the grayscale distribution law of the low-light image;

[0198] If yes, it is determined to perform image enhancement on the image.

[0199] Since the principle of solving the problem by the above computer-readable storage medium is similar to that of the image enhancement method, the implementation of the above computer-readable storage medium can refer to the embodiment of the method, and the repeated parts will be omitted.

[0200] The above-mentioned computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor in the electronic device, including but not limited to magnetic storage such as floppy disks, hard disks, magnetic tapes, MO (magneto-optical disks), etc., optical storage such as CDs, DVDs, BDs, HVDs, etc., and semiconductor storage such as ROMs, EPROMs, EEPROMs, NAND FLASH (non-volatile memory), SSDs (solid-state drives), etc.

[0201] Those skilled in the art will appreciate that the embodiments of the present application can be provided as methods, systems, or computer program products. Therefore, the present application can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0202] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0203] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0204] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0205] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. An image enhancement method, characterized in that: The method comprises: Acquire an image captured by an image acquisition device; Inputting the image into a pre-trained image enhancement model, the image enhancement model is trained based on a first illumination image and a plurality of non-uniform brightness images, the plurality of non-uniform brightness images are determined based on a second illumination image and a plurality of images of different brightness obtained by performing multiple brightness transformations on the first illumination image, the first illumination image is an image captured under an environment with an illumination greater than a set light intensity, and the second illumination image is an image captured under an environment with an illumination no greater than the set light intensity; outputting an enhanced image of the image based on the image enhancement model; The training process of the image enhancement model includes: Acquire a training set, where the training set includes a first illumination image and a second illumination image captured by the same image capture device; Training a first enhancement network in the image enhancement model according to the first illumination image and the multiple non-uniform brightness images; Training a second enhancement network in the image enhancement model according to a first background area image in the first illumination image and the second illumination image; The method further comprises: determining a brightness region image according to a brightness region having a brightness value greater than a set brightness in the second illumination image; The brightness area image and the multiple images with different brightness are fused to determine the multiple non-uniform brightness images.

2. The method according to claim 1, wherein Outputting an enhanced image of the image based on the image enhancement model includes: Determining a first enhanced feature map of the image based on a first enhancement network in the image enhancement model; Determining a second enhanced feature map for a background area image of the image based on a second enhancement network in the image enhancement model; The enhanced image is output according to the first enhanced feature map and the second enhanced feature map.

3. The method according to claim 1, wherein The training of the first enhancement network in the image enhancement model according to the first illumination image and the plurality of non-uniform brightness images includes: Inputting the plurality of uneven brightness images into the first enhancement network, and outputting a plurality of first feature maps based on the first enhancement network; determining a first loss value according to the first illumination image, the plurality of first feature maps, and a first loss function; The first enhancement network is trained according to the first loss value.

4. The method according to claim 1, wherein The training of the second enhancement network in the image enhancement model according to the first background area image in the first illumination image and the second illumination image includes: Inputting the second illumination image into the second enhancement network, and outputting a second feature map based on the second enhancement network; Fusing the background area template and the second feature map to obtain a fused second feature map; Determining a second background area image in the fused second feature map; determining a second loss value according to the first background area image, the second background area image, and the second loss function; Determining a third loss value according to the fused second feature map and the third loss function; The second enhancement network is trained according to the second loss value and the third loss value.

5. The method according to claim 1, wherein The image enhancement model includes a RepVGG network structure.

6. The method according to claim 1, wherein The first enhancement network includes an attention network structure of a luminance region template, wherein the luminance region template is determined according to the luminance region image in the second illumination image; and / or The second enhancement network includes an attention network structure of the background area template.

7. The method according to claim 1 or 2, wherein: Before inputting the image into the pre-trained image enhancement model, the method further includes: Performing grayscale statistics on the image to determine a brightness distribution result of the image; Determining whether the brightness distribution result conforms to the grayscale distribution law of the low-light image; If yes, determine to perform image enhancement on the image.

8. An electronic device, characterized in that: The electronic device includes at least a processor and a memory, and the processor is configured to implement the steps of the image enhancement method according to any one of claims 1 to 7 when executing a computer program stored in the memory.

Images