Video image enhancement method in low-light environment
By generating video image enhancement methods in low-light environments led by adversarial networks and thermal image, combined with local-global discriminator, the problem of overexposure and insignificance in low-light image enhancement is solved, and more natural and detailed images are achieved. Enhance the effect.
Patent Information
- Application Number
- CN202411267570.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-12-31
AI Technical Summary
The existing low-light image enhancement technology has problems such as overexposed, lack of obvious details and poor results. The traditional method is simple to calculate but not natural, and deep learning-based methods require continuous adjustment of parameters and are prone to distortion.
Using a video image enhancement method in a low-light environment based on a generative adversarial network, low-light images and thermal images are acquired through local-global discriminator and thermal image guidance, feature extraction and enhancement are performed, and images are reconstructed using attention modules and generators. Integrate global-local discriminator to ensure enhanced effect.
It achieves more natural and detailed low-light image enhancement, avoids overexposed, improves visual effects, and is suitable for diverse image scenes.
Smart Images
Figure CN119228661A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of image enhancement, and in particular relates to a video image enhancement method in a low-light environment. Background Art
[0002] In real life, due to factors such as the equipment used to obtain images and the environment, we often get some low-quality images. For example, in dimly lit streets and alleys at night, in places where cameras are facing light sources, and in rooms with uneven lighting, these images are generally dark overall, with poor contrast, unclear details, and accompanied by noise, distortion, etc., which affect the subjective visual effect of the human eye and the intuitive acquisition of information from the image, thereby affecting people's observation of the image content and subsequent use of the image, such as image classification, image segmentation, image recognition and other tasks. Brightening low-light video images, improving image quality, and improving the visual effect of images are of great practical significance in the fields of image processing and computer vision applications.
[0003] Currently, low-light image sharpening algorithms are mainly divided into two categories: methods based on traditional image enhancement and methods based on deep learning.
[0004] Traditional image enhancement methods are usually histogram equalization, gamma correction, filter enhancement, and Retinex theoretical models. Traditional methods are relatively simple to calculate, and sometimes the enhancement effect often does not look natural enough, and some areas are prone to over-enhancement; on the other hand, traditional image enhancement methods often require manual selection of parameters or models, and are not suitable for diverse image scenes. These methods may not be able to capture complex image features and relationships.
[0005] Methods based on deep learning use computers to learn the characteristics of low-light images. They can learn more features of such images and enhance their weak parts, thereby restoring images with normal lighting. Low-light image enhancement based on convolutional neural networks and generative adversarial networks is a current research hotspot. Such algorithms are mainly based on the extraction and enhancement of various features, and achieve better results through attention maps or multi-layer decomposition. However, some methods in the prior art require continuous adjustment of parameters, and some are not good for image enhancement, are prone to distortion, or brightness enhancement is too high or insufficient, resulting in the details and effects of the enhanced image not being very good. Summary of the invention
[0006] In order to solve the above technical problems, the present invention proposes a video image enhancement method in a low-light environment, which guides visible light image reconstruction through a local-global discriminator and a thermal image to solve the problems existing in the above-mentioned prior art.
[0007] To achieve the above object, the present invention provides a method for enhancing video images in a low-light environment, comprising:
[0008] Acquire low-light and thermal images;
[0009] Extracting features from the low-light image using an encoder to obtain low-light features;
[0010] Extracting features from the thermal image to obtain thermal features;
[0011] Performing feature enhancement on the low-light feature based on the thermal feature to obtain a thermal guidance enhancement feature;
[0012] Performing feature enhancement on the thermal guidance enhancement feature through an attention module to obtain an attention enhancement feature, and reconstructing the attention enhancement feature through a generator to obtain a reconstructed image;
[0013] The reconstructed image is discriminated by a discriminator, and an enhanced image is obtained according to the discrimination result.
[0014] Optionally, the encoder includes several ResnetD blocks connected in sequence, wherein the ResnetD block includes a first branch and a second branch connected in parallel, and the sum of the outputs of the first branch and the second branch is used as the output of the ResnetD block, wherein the first branch includes a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence, the second branch includes a maximum pooling layer and a fourth convolutional layer, the first convolutional layer, the third convolutional layer, and the fourth convolutional layer are 1*1 convolutional layers, and the third convolutional layer is a 3*3 convolutional layer with a step size of 2.
[0015] Optionally, the process of extracting features from the thermal image includes:
[0016] Performing shallow feature extraction on the thermal image through a shallow extraction module to obtain shallow features, wherein the shallow extraction module includes a convolution layer, a normalization layer and a ReLU layer connected in sequence;
[0017] Extract edge texture features of the shallow features at different scales through a plurality of parallel-connected center difference convolutions and a convolution layer connected to the plurality of parallel-connected center difference convolutions to obtain thermal features;
[0018] The thermal features and low-light features are extracted and reshaped by channel convolution and cross-channel convolution to obtain enhanced features.
[0019] Optionally, the process of performing feature enhancement on the thermal guidance feature by the attention module includes:
[0020] The thermal guidance feature is processed through a local channel to generate a first intermediate feature, a second intermediate feature, and a third intermediate feature, the first intermediate feature and the second intermediate feature are calculated through a softmax function to obtain a weight map of the third intermediate feature, a dot product calculation is performed on the weight map and the third intermediate feature, and the dot product calculation structure result is added to the thermal guidance feature to obtain a first feature map;
[0021] Processing the thermal guidance feature through a global channel to obtain a second feature map;
[0022] The first feature map and the second feature map are fused to obtain an attention enhancement feature.
[0023] Optionally, the generator includes a plurality of ResnetD blocks using deconvolution connected in sequence;
[0024] The objective function of the generator is:
[0025]
[0026] Among them, P Z (Z) is the distribution of noise Z, G(z) is the data generated by the generator given the noise Z, D(G(z)) is the probability that the data generated by the discriminator given the generator is real data, represents the expectation of the noise variable (Z), and V(D,G) represents the loss function of the generator.
[0027] Optionally, the discriminator includes a global discriminator and a local discriminator, wherein the global discriminator is used to discriminate the entire area of the reconstructed image, and the local discriminator is used to discriminate the local area of the reconstructed image.
[0028] Optionally, the standard function of the discriminator is:
[0029]
[0030]
[0031] Where C represents the discriminator network, X f and X r are the true distribution and the false distribution respectively, σ represents the sigmoid function, D Ra represents the global discriminator, Represents the expected distribution of real data.
[0032] Optionally, the process of acquiring the enhanced image includes:
[0033] When the discriminator outputs a result that the reconstructed image is a real image, the current reconstructed image is output as an enhanced image, otherwise the attention module and the generator are retrained until the output result of the discriminator is a real image.
[0034] Compared with the prior art, the present invention has the following advantages and technical effects:
[0035] This paper proposes a thermal-guided low-light enhancement algorithm based on a generative adversarial network. The main innovations are: first, the local-global discriminator is integrated to compensate for the overexposure problem through the training of the generative adversarial network; second, the features of the thermal image are used as a guide for the enhancement of the visible light features to further enhance the details of the low-light image and achieve better visual effects. BRIEF DESCRIPTION OF THE DRAWINGS
[0036] The drawings constituting a part of the present application are used to provide a further understanding of the present application. The illustrative embodiments and descriptions of the present application are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:
[0037] Figure 1 The figure is a schematic diagram of a method flow of an embodiment of the present invention. DETAILED DESCRIPTION
[0038] It should be noted that, in the absence of conflict, the embodiments and features in the embodiments of the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.
[0039] It should be noted that the steps shown in the flowcharts of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and that, although a logical order is shown in the flowcharts, in some cases, the steps shown or described can be executed in an order different from that shown here.
[0040] Based on the above-mentioned shortcomings of the prior art, the present invention proposes a thermal guidance low-light enhancement algorithm based on a generative adversarial network. The main innovations are twofold: first, the local-global discriminator is integrated to compensate for the overexposure problem through the training of a generative adversarial network; second, the features of thermal images are used as a guide for visible light feature enhancement to further enhance the details of low-light images and achieve better visual effects. Figure 1 As shown, the steps are as follows:
[0041] Acquire a low-light image and a thermal image; extract features from the low-light image through an encoder to obtain low-light features; extract features from the thermal image to obtain thermal features; enhance the low-light features based on the thermal features to obtain thermal guidance enhancement features; enhance the thermal guidance enhancement features through an attention module to obtain attention enhancement features, reconstruct the attention enhancement features through a generator to obtain a reconstructed image; identify the enhanced image through a discriminator, and obtain an enhanced image based on the identification result.
[0042] The relevant network structure and specific operation process required for the above content are described in detail:
[0043] For low-light images, i.e. visible light images, a generative adversarial network is used to extract features of the input image and generate reconstruction.
[0044] First, the features of the input image are extracted through the encoder: for any visible light image X / Y, feature encoding and extraction are performed through an encoder composed of multiple ResnetD blocks to extract features.
[0045] The encoder includes a plurality of sequentially connected ResnetD blocks, wherein the ResnetD block includes a first branch and a second branch connected in parallel, the input data of the first branch and the second branch are both visible light images, and the sum of the outputs of the first branch and the second branch is used as the output of the ResnetD block, wherein the first branch includes a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence, the second branch includes a maximum pooling layer and a fourth convolutional layer, the first convolutional layer, the third convolutional layer and the fourth convolutional layer are 1*1 convolutional layers, and the third convolutional layer is a 3*3 convolutional layer with a step size of 2. At the same time, before using the encoder, the convolutional layer can be used to perform relevant convolution operations on the visible light image to extract its features and adjust the size of the visible light to adapt to the encoder.
[0046] For thermal images, namely thermal infrared images, feature extraction of the thermal images is performed through a thermal guidance correlation algorithm to obtain thermal features.
[0047] 1) Extract shallow features
[0048] First, the features of the thermal image are extracted through different convolution modules to obtain shallow features. The convolution module includes a convolution layer, a normalization layer and a ReLU layer connected in sequence. A thermal image is input to extract shallow features, and then different convolution kernels are used to perform convolution calculations on the input thermal image in the process of extracting thermal features to generate a new feature map, namely the thermal feature. Each convolution kernel in the thermal feature will extract different features of the image. Exemplarily, it includes 3 parallel convolution modules to extract the thermal image separately, and the results are defined as the first shallow feature, the second shallow feature and the third shallow feature.
[0049] 2) Extract thermal features
[0050] The shallow features obtained are then used to extract thermal features. First, the center difference convolution is used to emphasize the extraction of edge / texture features by calculating the weighted sum of the differences between the adjacent sampled pixel values and the center pixel value. Then, different convolution kernels are used to achieve multiple receptive fields from local to global, effectively extracting thermal features at different scales.
[0051] Taking the above-mentioned different shallow features as an example, the above-mentioned shallow features are calculated through different central difference convolutions, wherein the convolution kernel size, step size and padding of the central difference convolution are different. For the first shallow feature, the central difference convolution with a convolution kernel size of 5, a step size of 1 and a padding of 2 is used for feature extraction to obtain the first scale feature. For the second shallow feature, the central difference convolution with a convolution kernel size of 3, a step size of 1 and a padding of 1 is used for feature extraction to obtain the second scale feature. For the third shallow feature, the central difference convolution with a convolution kernel size of 1, a step size of 1 and a padding of 0 is used for feature extraction to obtain the third scale feature. The first scale feature, the second scale feature and the third scale feature are spliced with the original thermal image, and the spliced features are processed through a 1*1 convolution layer to obtain the thermal feature.
[0052] 3) Thermal features enhance visible light features
[0053] After obtaining the above thermal features, feature extraction and reshaping operations are performed on the thermal features and low-light features output by the encoder through channel convolution and cross-channel convolution to perform feature enhancement. This step is achieved through two stages: channel convolution and cross-channel convolution to enhance visible light features, mainly using spatial variation kernels and reshaping operations to complete feature enhancement.
[0054] For channel convolution, first extract the thermal features Then, the thermal features after expansion are obtained through 1*1 expansion convolution. Then the result of the thermal conductivity kernel (thermal convolution module) τ t,n With visible light
[0055] The pre-enhanced visible light feature F is obtained by multiplying the e,M , some of the formula expressions are as follows:
[0056]
[0057] Where [·][·] represents a cascade operation. represents a standard convolutional layer with a kernel size of 1, are the output features of different kernel sizes, F v,M Represents the visible light characteristics, F t,M and Represents the input and output thermal features in the Mth hot convolution module, in which the hot convolution module performs The local receptive field area is sampled and then expanded on the input feature map. Is Features obtained through a standard convolutional layer with a kernel size of 1.
[0058] For cross-channel convolution, firstly, the visible features F v,M With thermal boot feature F e,M Perform a reshaping operation, and then reshape the two results to obtain the final enhanced visible feature F e,M . Some of the formula expressions are as follows:
[0059]
[0060] in and Represents the maximum pooling and shallow feature extraction operations, RS represents the reshaping operation, which mainly reshapes the dimension and shape of the tensor to adjust the size or shape of the feature to be compatible with the matching content. It is an operation in pytorch and will not be described here.
[0061] Attention-enhanced features obtained through attention channels
[0062] For the thermal guidance enhancement features, the attention channel is put into local-global enhancement, and the enhanced features are fused to obtain the reconstructed enhanced image. First, the obtained features are input into two channels, and three intermediate features are generated from the features in the local channel. The two features are used to obtain the weight map of another feature through the softmax function, and the dot product is performed to obtain the feature map. Then, the feature map obtained by another global channel is fused with it and reconstructed to obtain the enhanced image.
[0063] Specifically, the thermal guidance feature is processed through a local channel to generate a first intermediate feature, a second intermediate feature, and a third intermediate feature, the first intermediate feature and the second intermediate feature are calculated through a softmax function to obtain a weight map of the third intermediate feature, and a dot product calculation is performed on the weight map and the third intermediate feature to obtain a first feature map;
[0064] The local channel includes several parallel 1*1 convolutional layers. In the example, the number of convolutional layers in the local channel is 3. The thermal guidance enhancement features are processed by three convolutional layers. The output of the first convolutional layer is reshaped and transposed to generate the first intermediate feature. The output of the second convolutional layer is reshaped to generate the second intermediate feature. The output of the third convolutional layer is directly used as the third intermediate feature. The first intermediate feature and the second intermediate feature are calculated by the softmax function to obtain a weight map of the third intermediate feature. The weight map and the third intermediate feature are dot-producted, and the dot product calculation structure result is added to the thermal guidance feature to obtain the first feature map.
[0065] The thermal guidance feature is processed through a global channel to obtain a second feature map, wherein the global channel performs global average pooling, compression, reconstruction, and reweighting on the thermal guidance feature, performs dot product processing on the reweighted result and the thermal guidance feature to obtain a second feature map, and the first feature map and the second feature map are added, i.e., fused, to obtain an attention enhancement feature.
[0066] Generate the reconstructed image through the generator:
[0067] The generator is constructed by deconvolution ResnetD to enhance the input low-light image to a normal-light image. The generator mainly decodes the relevant features extracted by the encoder to generate the corresponding enhanced image, i.e., the reconstructed image. The objective function expression of the generator is:
[0068]
[0069] Among them, P Z (Z) is the distribution of noise Z, G(z) is the data generated by the generator given the noise Z, D(G(z)) is the probability that the data generated by the discriminator given the generator is real data, represents the expectation of the noise variable (Z), and V(D,G) represents the loss function of the generator.
[0070] The reconstructed image is repeatedly enhanced through the discriminator:
[0071] The obtained enhanced images are placed in the discriminator to learn to distinguish whether they are real (from real images) or fake (from enhanced output). In order to adaptively enhance local areas and improve global light, the present invention uses a global-local discriminator structure, where the global discriminator and the local discriminator have the same structure, and the convolutional layers inside are all 4*4 convolutional kernels. The difference is that the global discriminator has one more 4*4 convolutional layer with an input and output of 512*512 than the local one. The structure of the discriminator should be an ordinary convolutional network. The global discriminator structure is 7 convolutional layers, and a relu layer is added after each convolutional layer. The global discriminator structure is specifically 7 convolutional layers connected in sequence, and its dimensions are 3*64, 64*128, 128*256, 256*512, 512*512, 512*512, and 512*1 respectively. The local discriminator is a 6-layer convolutional layer, each convolutional layer is followed by a relu layer. The local discriminator structure is specifically 6 convolutional layers connected in sequence, and their dimensions are 3*64, 64*128, 128*256, 256*512, 512*512, and 512*1, respectively. The discriminator includes a global discriminator and a local discriminator. The global discriminator discriminates the entire area of the reconstructed image, and the local discriminator is used to discriminate the local area of the reconstructed image. The global discriminator and the local discriminator adopt the structure of the PatchGAN discriminator, in which the global discriminator discriminates the entire reconstructed image, and the local discriminator randomly crops the same area of the reconstructed image and the real image, and compares the cropped content.
[0072] Such a structure ensures that all local patches of the enhanced image look like real normal light. The discriminator standard function is as follows:
[0073]
[0074]
[0075] Where C represents the discriminator network, X f and X r are the true distribution and the false distribution respectively, σ represents the sigmoid function, D Ra represents the global discriminator, Represents the expected distribution of real data.
[0076] When the discriminator outputs that the reconstructed image is a real image or the identification probability output by the discriminator is greater than a certain threshold, the reconstructed image is considered to be a real image, and the current reconstructed image is output as an enhanced image, otherwise the attention module and the generator are retrained until the output result of the discriminator is a real image.
[0077] This paper proposes a thermal-guided low-light enhancement method based on a generative adversarial network. The main innovations are twofold: first, the local-global discriminator is integrated to compensate for the overexposure problem through the training of a generative adversarial network; second, the features of thermal images are used as a guide for visible light feature enhancement to further enhance the details of low-light images and achieve better visual effects.
[0078] The above are only preferred specific implementations of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A video image enhancement method in a low-light environment, characterized in that: include: Acquire low-light and thermal images; Extracting features from the low-light image using an encoder to obtain low-light features; Extracting features from the thermal image to obtain thermal features; Performing feature enhancement on the low-light feature based on the thermal feature to obtain a thermal guidance enhancement feature; Performing feature enhancement on the thermal guidance enhancement feature through an attention module to obtain an attention enhancement feature, and reconstructing the attention enhancement feature through a generator to obtain a reconstructed image; The reconstructed image is discriminated by a discriminator, and an enhanced image is obtained according to the discrimination result.
2. The method according to claim 1, characterized in that The encoder includes a plurality of ResnetD blocks connected in sequence, wherein the ResnetD block includes a first branch and a second branch connected in parallel, and the sum of the outputs of the first branch and the second branch is used as the output of the ResnetD block, wherein the first branch includes a first convolutional layer, a second convolutional layer and a third convolutional layer connected in sequence, the second branch includes a maximum pooling layer and a fourth convolutional layer, the first convolutional layer, the third convolutional layer and the fourth convolutional layer are 1*1 convolutional layers, and the third convolutional layer is a 3*3 convolutional layer with a step size of 2.
3. The method according to claim 1, characterized in that The process of extracting features from the thermal image includes: Performing shallow feature extraction on the thermal image through a shallow extraction module to obtain shallow features, wherein the shallow extraction module includes a convolution layer, a normalization layer and a ReLU layer connected in sequence; Extract edge texture features of the shallow features at different scales through a plurality of parallel-connected center difference convolutions and a convolution layer connected to the plurality of parallel-connected center difference convolutions to obtain thermal features; The thermal features and low-light features are extracted and reshaped by channel convolution and cross-channel convolution to obtain enhanced features.
4. The method according to claim 1, characterized in that: The process of enhancing the thermal guidance feature by the attention module includes: The thermal guidance feature is processed through a local channel to generate a first intermediate feature, a second intermediate feature, and a third intermediate feature, the first intermediate feature and the second intermediate feature are calculated through a softmax function to obtain a weight map of the third intermediate feature, a dot product calculation is performed on the weight map and the third intermediate feature, and the dot product calculation structure result is added to the thermal guidance feature to obtain a first feature map; Processing the thermal guidance feature through a global channel to obtain a second feature map; The first feature map and the second feature map are fused to obtain an attention enhancement feature.
5. The method according to claim 1, characterized in that The generator includes a plurality of ResnetD blocks using deconvolution connected in sequence; The objective function of the generator is: Among them, P Z (Z) is the distribution of noise Z, G(z) is the data generated by the generator given the noise Z, D(G(z)) is the probability that the data generated by the discriminator given the generator is real data, represents the expectation of the noise variable (Z), and V(D,G) represents the loss function of the generator.
6. The method according to claim 1, characterized in that The discriminator includes a global discriminator and a local discriminator. The global discriminator is used to discriminate the entire area of the reconstructed image, and the local discriminator is used to discriminate the local area of the reconstructed image.
7. The method according to claim 1, characterized in that The standard function of the discriminator is: Where C represents the discriminator network, X f and X r are the true distribution and the false distribution respectively, σ represents the sigmoid function, D Ra represents the global discriminator, Represents the expected distribution of real data.
8. The method according to claim 1, characterized in that: The process of acquiring enhanced images includes: When the discriminator outputs a result that the reconstructed image is a real image, the current reconstructed image is output as an enhanced image, otherwise the attention module and the generator are retrained until the output result of the discriminator is a real image.