4k dark light video enhancement method and system based on denoising

Abstract

The application discloses a 4K dark-light video enhancement method and system based on denoising. The method comprises the following steps: acquiring a plurality of continuous original images from a 4K dark-light video to be processed; inputting each original image into a convolution network respectively, extracting a corresponding single-channel image and brightness feature of each original image through the convolution network; performing low-light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain an enhanced image corresponding to each original image; performing denoising processing on the enhanced image to obtain a denoised image corresponding to each original image; and arranging and splicing a plurality of denoised images corresponding to the original images in time sequence to obtain an enhanced denoised dark-light video. The application has the beneficial effects of effectively reducing the influence of noise on image recovery and solving the flickering problem of the dark-light video after denoising and enhancement.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a 4K dark light video enhancement method and system based on denoising. BACKGROUND

[0002] The progress of science and technology is constantly pushing the limits of video image quality, making the clarity and detail performance of video a hot spot of public attention. However, when shooting in a dimly lit environment, the video often faces challenges such as insufficient contrast, many noise points, and unclear details. In order to deal with these problems, low-light video enhancement technology has emerged, which not only plays an important role in basic visual applications such as video image target detection and tracking, but also provides new possibilities for the further development and application of video images by improving the visibility of low-light images.

[0003] Existing low-light video enhancement technology improves the visual performance of video images under adverse lighting conditions, enhancing the practicality and application range of images. When restoring dark light videos, existing methods sometimes fail to balance the relationship between the enhancement effect of the video image and noise suppression. On the one hand, the brightness and contrast of the video are improved; on the other hand, unnatural artifacts and uneven lighting effects may be introduced. However, when the trade-off is not appropriate, the video image may lose its original naturalness and realism in terms of vision.

[0004] In view of the shortcomings of the prior art, the present application provides a 4K dark light video enhancement method and system based on denoising. SUMMARY

[0005] The purpose of the present application is to overcome the shortcomings of the prior art and provide a 4K dark light video enhancement method based on denoising, which restores texture information more ideally and has better noise reduction effect, and solves the problem of flicker after dark light video denoising and enhancement.

[0006] The present application provides a 4K dark light video enhancement method based on denoising, comprising:

[0007] Obtaining a plurality of continuous original images from a 4K dark light video to be processed;

[0008] Respectively inputting each original image into a convolution network, and extracting a corresponding single-channel image and brightness feature of each original image through the convolution network;

[0009] Performing low-light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain an enhanced image corresponding to each original image;

[0010] Performing denoising processing on the enhanced image to obtain a denoised image corresponding to each original image;

[0011] The multiple denoised images corresponding to the original images are arranged in time sequence to splice to obtain an enhanced denoised dark light video.

[0012] Further, the denoised 4K dark light video enhancement method further comprises: after obtaining the multiple continuous original images, pre-processing the original images to obtain pre-processed original images before inputting the original images into the convolution network; the pre-processing of the original images comprises:

[0013] down-sampling the original images to obtain multiple sub-images;

[0014] flipping the multiple sub-images twice to obtain multiple flipped sub-images;

[0015] adding noise to the multiple flipped sub-images to obtain multiple first sub-images;

[0016] splicing the multiple first sub-images to obtain the pre-processed original images.

[0017] Further, the extracting of the corresponding single-channel image and the brightness feature of each original image by the convolution network comprises:

[0018] channel separation is performed on the original images, and a feature image of each channel is obtained;

[0019] a 3*3 convolution filter is used to obtain the brightness feature of the feature image of each channel;

[0020] a 1*1 convolution kernel is used to combine the multiple channel feature images into a single-channel feature image, a second brightness feature of the single-channel feature image is obtained, and the first image brightness feature is obtained by fusing and superimposing the second brightness feature and the brightness feature of the feature image of each channel.

[0021] Further, the low light enhancement cycle processing of the single-channel feature image according to the brightness feature to obtain the enhanced image corresponding to each original image comprises:

[0022] feature information of the brightness feature and the single-channel feature image is extracted by multiple first convolution layers respectively, and multiple first feature maps are obtained according to the feature information respectively;

[0023] a second convolution layer is used to splice the multiple first feature maps to obtain a spliced image, and feature extraction is performed on the spliced image to obtain a second feature map;

[0024] an activation layer is used to map the second feature to the same number of channels as the original images to obtain the enhanced image.

[0025] Further, the low light enhancement cycle processing of the single-channel feature image according to the brightness feature to obtain the enhanced image corresponding to each original image further comprises:

[0026] When the plurality of first feature maps are input into the second convolutional layer, the plurality of first feature maps are scaled to the same input size of the second convolutional layer by using a bilinear interpolation algorithm;

[0027] When the second feature is input into the activation layer, the spliced image is scaled to the same input size of the activation layer by using a bilinear interpolation algorithm.

[0028] Further, the low-light enhancement cycle processing of the single-channel feature image according to the brightness feature to obtain the enhanced image further comprises:

[0029] When the first feature map is input into the second convolutional layer, the first feature map is subjected to nonlinear transformation;

[0030] When the second feature is input into the activation layer, the spliced image is subjected to nonlinear transformation;

[0031] The nonlinear transformation formula is as follows:

[0032]

[0033] Where x a-1 is the output image of the last iteration, x r is an enhanced factor obtained by convolution calculation.

[0034] Further, the denoising processing of the enhanced image to obtain a denoised image corresponding to each original image comprises:

[0035] The enhanced image is convolved by the denoising convolutional layer to obtain a noise map;

[0036] The noise map is up-sampled by the up-sampling layer to obtain an up-sampled noise feature map;

[0037] The up-sampled noise feature map, the enhanced image and the original image are compared to obtain a denoising parameter;

[0038] The enhanced image is subtracted from the noise map by the denoising parameter to obtain a denoised image.

[0039] The application also provides a 4K dark-light video enhancement system based on denoising, comprising a frame taking module, a preprocessing module, a feature extraction module, a low-light enhancement module, a denoising module and a splicing module.

[0040] The frame taking module is used to acquire a plurality of continuous original images in a 4K dark-light video to be processed;

[0041] The preprocessing module is used to down-sample and flip the original images to obtain preprocessed original images;

[0042] The feature extraction module is configured to acquire a single-channel feature image and a brightness feature of the original image.

[0043] The low-light enhancement module is configured to enhance the single-channel feature image according to the brightness feature to acquire an enhanced image.

[0044] The denoising network module is configured to denoise the enhanced image to acquire a denoised denoised image.

[0045] The splicing module is configured to arrange and splice a plurality of denoised images corresponding to the original image in time sequence to acquire an enhanced denoised low-light video.

[0046] Further, the low-light enhancement module comprises:

[0047] A plurality of first convolutional layers are configured to extract feature information of the brightness feature and the single-channel feature image, and acquire a plurality of first feature maps according to the feature information.

[0048] A second convolutional layer is configured to splice the plurality of first feature maps to acquire a spliced image, and acquire a second feature of the spliced image.

[0049] An activation layer is configured to map the second feature to a same number of channels as the original image to acquire the enhanced image.

[0050] Further, the loss function of the denoising network module comprises a spatial consistency loss function, a color consistency loss function, an exposure control loss function and a noise control loss function.

[0051] The present application has the following advantages:

[0052] (1) The present application establishes a nonlinear function when performing image enhancement, so that when the value of the pixel point does not change, fixed enhancement is performed, and a series of input images x and parameter enhancement factors xr related nonlinear transformations are performed to enhance the input image. For the first feature map or the spliced image, when the value of the pixel point does not change, the enhancement factor is a fixed parameter, i.e., the enhancement effect is fixed, so when low-light enhancement is performed, almost no inter-frame flicker occurs; thus, the video frames after image splicing do not have inter-frame flicker, and synchronous processing of 4K low-light video restoration, noise reduction and flicker removal is achieved.

[0053] (2) The present application designs a network structure and introduces a denoising network module, which significantly improves the effect of neural network video picture enhancement in a dark environment; this innovation not only effectively reduces the interference of noise on video restoration quality, but also establishes a nonlinear function to obtain an enhancement factor, so that the denoising network also performs well in video flicker removal. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 A method step diagram of the present application;

[0055] Figure 2 An image enhancement model structure framework flow diagram;

[0056] Figure 3 Contrast experimental data are shown. DETAILED DESCRIPTION

[0057] The present application will be further described below in conjunction with the accompanying drawings, but the scope of protection of the present application is not limited to the following description.

[0058] It should be noted that the directions or positional relationships indicated by "left", "right", etc. are based on the directions or positional relationships shown in the drawings, or the directions or positional relationships in which the product of the present application is usually placed, or the directions or positional relationships commonly understood by those skilled in the art. Such terms are only used to facilitate the description of the present application and simplify the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application.

[0059] It should be noted that the embodiments in the present application and the features and technical solutions in the embodiments can be combined with each other without conflict.

[0060] First aspect

[0061] As Figure 1 shown, the present application provides a 4K dark light video enhancement method based on denoising, comprising:

[0062] S1, obtaining a plurality of continuous original images from a 4K dark light video to be processed; the main acquisition scene of the original images is a city street or a wild field, and the resolution of the original images is 3840*2160;

[0063] S2, pre-processing the original images to obtain pre-processed original images;

[0064] S3, inputting each original image into a convolution network respectively, and extracting a corresponding single-channel image and a brightness feature of each original image through the convolution network;

[0065] S4, performing low light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain an enhanced image corresponding to each original image;

[0066] S5, performing denoising processing on the enhanced image to obtain a denoised image corresponding to each original image;

[0067] S6, arranging and splicing a plurality of denoised images corresponding to the original images in time sequence to obtain an enhanced denoised dark light video.

[0068] The application effectively reduces the interference of noise on video recovery quality by denoising the image, combines image enhancement and denoising technology, and significantly improves the effect of video picture enhancement in a dark light environment; and the use of an enhancement factor in image enhancement makes the enhanced image quality higher, solving the problems of image interference by noise, noise amplification after traditional algorithm enhancement, edge feature loss and smoothing of details, and flicker of video continuous frames when directly denoising dark light images.

[0069] Further, the preprocessing of the original image comprises:

[0070] S201, down-sampling the original image to obtain a plurality of sub-images;

[0071] By down-sampling the original image to obtain a plurality of sub-images, the original image with a resolution of 3840*2160 is down-sampled and randomly cropped into 8 sub-images with a resolution of 608*608;

[0072] S202, flipping the plurality of sub-images twice to obtain a plurality of flipped sub-images;

[0073] S203, adding noise to the plurality of flipped sub-images to obtain a plurality of first sub-images;

[0074] S204, splicing the plurality of first sub-images to obtain the preprocessed original image.

[0075] By preprocessing the original image, the diversity of the image can be effectively increased, the robustness and the subsequent recognition ability of image processing can be improved.

[0076] Further, the corresponding single-channel image and brightness feature of each original image extracted by the convolution network comprises:

[0077] S301, separating the original image by channel and obtaining a feature image of each channel;

[0078] S302, using a 3*3 convolution filter to obtain the brightness feature of the feature image of each channel; the separation processing of each channel helps the convolution filter to obtain the spatial details and brightness changes in each channel, thereby obtaining accurate brightness features;

[0079] S303, combining a plurality of channel feature images into a single-channel feature image by a 1*1 convolution kernel, obtaining a second brightness feature of the single-channel feature image, and obtaining the brightness feature of the first image by fusing and superimposing the second brightness feature and the brightness feature of the feature image of each channel;

[0080] The second brightness feature is obtained by 1*1 convolution layer cross-channel fusion, which is the brightness information of different color channels being enhanced to obtain an enhanced brightness feature, facilitating the calculation of the enhancement factor.

[0081] Further, the low-light enhancement cycle processing of the single-channel feature image according to the brightness feature to obtain an enhanced image corresponding to each original image comprises:

[0082] S401, extracting feature information of the brightness feature and the single-channel feature image through a plurality of first convolution layers, and obtaining a plurality of first feature maps according to the feature information;

[0083] S402, obtaining a spliced image by splicing the plurality of first feature maps through a second convolution layer, and extracting features of the spliced image to obtain a second feature map;

[0084] S403, mapping the second feature to a channel with the same number of original images through an activation layer to obtain an enhanced image.

[0085] In some specific embodiments, the low-light enhancement cycle processing of the single-channel feature image according to the brightness feature to obtain an enhanced image corresponding to each original image further comprises:

[0086] When the plurality of first feature maps are input into the second convolution layer, the plurality of first feature maps are scaled to the same input size of the second convolution layer by using a bilinear interpolation algorithm;

[0087] When the second feature is input into the activation layer, the spliced image is scaled to the same input size of the activation layer by using a bilinear interpolation algorithm;

[0088] When the first feature map is input into the second convolution layer, the first feature map is subjected to a nonlinear transformation;

[0089] When the second feature is input into the activation layer, the spliced image is subjected to a nonlinear transformation;

[0090] The nonlinear transformation formula is as follows:

[0091]

[0092] where x a-1 is the output image of the last iteration, x r is an enhancement factor obtained by convolution calculation.

[0093] The input image is enhanced by a nonlinear transformation related to a series of input images x and a parameter enhancement factor xr, when the value of the first feature map or the spliced image at the pixel point does not change, the enhancement factor is a fixed parameter, that is, the enhancement effect is fixed, so when low light enhancement is performed, almost no interframe flicker occurs, and by adding the nonlinear transformation formula, the problems of noise affecting video picture quality and video flicker in the traditional method are successfully solved.

[0094] Further, the denoising processing on the enhanced image to obtain a denoised image corresponding to each original image comprises:

[0095] S501, the enhanced image is convolved by a denoising convolution layer to obtain a noise map;

[0096] S502, the noise map is up-sampled by an up-sampling layer to obtain an up-sampled noise feature map;

[0097] S503, the up-sampled noise feature map, the enhanced image and the original image are compared to obtain a denoising parameter;

[0098] S504, the enhanced image is subtracted from the noise map by the denoising parameter to obtain a denoised image.

[0099] Based on the characteristics of the dark light image, the dark light image has many noise points, and direct denoising will cause the loss of many detail information, therefore, the image is convolved to obtain a noise map, and then the noise image is up-sampled to retain the detail information, then the denoising parameter is obtained by comparison and analysis, the enhanced image is compared with the difference of the noise map according to the denoising parameter to obtain a denoised image, the denoised image obtained in this way is not only accurate in noise point elimination, but also retains the detail information of the image.

[0100] The second aspect

[0101] The application also provides a 4K dark light video enhancement system based on denoising, comprising a frame taking module, a preprocessing module, a feature extraction module, a low light enhancement module, a denoising module and a splicing module.

[0102] The frame taking module is used for acquiring a plurality of continuous original images in a 4K dark light video to be processed.

[0103] The preprocessing module is used for down-sampling and flipping the original image to obtain a preprocessed original image.

[0104] The feature extraction module is used for acquiring a single-channel feature image and a brightness feature of the original image.

[0105] The low light enhancement module is used for enhancing the single-channel feature image according to the brightness feature to obtain an enhanced image.

[0106] The denoising network module is configured to denoise the enhanced image to obtain a denoised image.

[0107] The splicing module is configured to arrange and splice the plurality of denoised images corresponding to the original image in time sequence to obtain an enhanced denoised low-light video.

[0108] The present application introduces a denoising network module by designing a network structure, which significantly improves the effect of video picture enhancement in low-light environment by neural network, and effectively reduces the interference of noise on video recovery quality.

[0109] Referring to Figure 2 , the low-light enhancement module comprises:

[0110] A plurality of first convolutional layers are configured to extract feature information of luminance features and single-channel feature images, and obtain a plurality of first feature maps according to the feature information.

[0111] A second convolutional layer is configured to splice the plurality of first feature maps to obtain a spliced image, and obtain second features of the spliced image.

[0112] An activation layer is configured to map the second features to the same number of channels as the original image to obtain an enhanced image.

[0113] The present application designs a network structure, so that the low-light enhancement module can effectively cope with changes in lighting conditions, and performs well in preserving image detail information; the network structure of the low-light enhancement module can capture global information of the image, and obtain feature information through a plurality of first convolutional layers, which is crucial for restoring features in the image.

[0114] Further, the loss function of the denoising convolutional layer comprises a spatial consistency loss function, a color consistency loss function, an exposure control loss function and a noise control loss function.

[0115] The present application is trained in an unsupervised manner, and has good generalization; as Figure 3 shown, through a comparative experiment, using peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) as evaluation indexes for the quality of an image, it can be seen that the values of PSNR and SSIM are the largest when the present application is used (it should be noted that the higher the PSNR value, the higher the image quality, and the higher the SSIM value, the closer to the clear data).

[0116] The above embodiments only express the preferred embodiments, and the description is more specific and detailed, but it cannot be understood as limiting the scope of the present application. It should be noted that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are within the scope of protection of the present application.

Claims

1. A method for 4K dim light video enhancement based on denoising, characterized in that: The method comprises the following steps: obtaining a plurality of continuous original images from 4K low-light video to be processed; inputting each original image into a convolution network respectively, and extracting a corresponding single-channel image and brightness feature of each original image through the convolution network; performing low-light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain an enhanced image corresponding to each original image; performing denoising processing on the enhanced image to obtain a denoised image corresponding to each original image; arranging and splicing a plurality of denoised images corresponding to the original images in time sequence to obtain an enhanced denoised low-light video. The method further comprises the following steps: extracting feature information of the brightness feature and the single-channel feature image through a plurality of first convolution layers respectively, and obtaining a plurality of first feature maps according to the feature information; splicing the plurality of first feature maps through a second convolution layer to obtain a spliced image, and extracting features of the spliced image to obtain a second feature map; mapping the second feature to a same number of channels as the original images through an activation layer to obtain the enhanced image; The method further comprises the following steps: performing nonlinear transformation on the first feature map when the first feature map is input into the second convolution layer; performing nonlinear transformation on the spliced image when the second feature is input into the activation layer; The nonlinear transformation formula is as follows: wherein is the output image of the last iteration, is an enhancement factor resulting from a convolution calculation; wherein the enhancement factor is a fixed parameter when the value of the pixel point of the first feature map or the spliced image does not change; After obtaining the plurality of continuous original images, the original images are preprocessed to obtain preprocessed original images before being input into the convolution network; the preprocessing of the original images comprises the following steps: down-sampling the original image to obtain a plurality of sub-images; flipping the plurality of sub-images twice to obtain a plurality of flipped sub-images; adding noise to the plurality of flipped sub-images to obtain a plurality of first sub-images; splicing the plurality of first sub-images to obtain the preprocessed original image. The method further comprises the following steps: performing channel separation on the original image, and obtaining a feature image of each channel; obtaining brightness features of the feature image of each channel by using a 3*3 convolution filter; combining a plurality of channel feature images into a single-channel feature image by using a 1*1 convolution kernel, obtaining second brightness features of the single-channel feature image, and superimposing the second brightness features and the brightness features of the feature image of each channel to obtain enhanced brightness features.

2. The denoising based 4K dim light video enhancement method according to claim 1, characterized in that: The method further comprises the following steps: when the plurality of first feature maps are input into the second convolution layer, the plurality of first feature maps are scaled to the same input size of the second convolution layer by using a bilinear interpolation algorithm; when the second feature is input into the activation layer, the spliced image is scaled to the same input size of the activation layer by using a bilinear interpolation algorithm.

3. The denoising based 4K dim light video enhancement method according to claim 1, characterized in that: The method further comprises the following steps: The noise map is obtained by convolving the enhanced image through a de-noising convolution layer; The noise feature map after upsampling is obtained by upsampling the noise map through an upsampling layer; The de-noising parameter is obtained by comparing the noise feature map after upsampling, the enhanced image and the original image; The de-noised image is obtained by subtracting the noise map from the enhanced image through the de-noising parameter.

4. A 4K low light video enhancement system based on denoising, characterized in that: It comprises a frame taking module, a feature extraction module, a low-light enhancement module, a de-noising module and a splicing module; The frame taking module is used for acquiring a plurality of continuous original images from a 4K dark-light video to be processed; The feature extraction module is used for inputting each original image into a convolution network respectively, and extracting a corresponding single-channel image and a brightness feature of each original image through the convolution network; The low-light enhancement module is used for performing low-light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain an enhanced image corresponding to each original image; The de-noising module is used for de-noising the enhanced image to obtain a de-noised image corresponding to each original image; The splicing module is used for arranging and splicing a plurality of de-noised images corresponding to the original images in time sequence to obtain an enhanced de-noised dark-light video. The low-light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain an enhanced image corresponding to each original image comprises: Feature information of the brightness feature and the single-channel feature image is extracted through a plurality of first convolution layers respectively, and a plurality of first feature maps are obtained according to the feature information respectively; A splicing image is obtained by splicing the plurality of first feature maps through a second convolution layer, and a second feature map is obtained by performing feature extraction on the splicing image; The second feature map is mapped to a same number of channels as the original image through an activation layer to obtain the enhanced image; The low-light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain the enhanced image further comprises: When the first feature map is input into the second convolution layer, the first feature map is subjected to nonlinear transformation; When the second feature is input into the activation layer, the splicing image is subjected to nonlinear transformation; The nonlinear transformation formula is as follows: wherein is the output image of the last iteration, is an enhancement factor resulting from a convolution calculation; When the value of the first feature map or the splicing image at a pixel point does not change, the enhancement factor is a fixed parameter; After acquiring the plurality of continuous original images, the original images are preprocessed to obtain preprocessed original images before being input into the convolution network; the preprocessing of the original images comprises: The original images are down-sampled to obtain a plurality of sub-images; The plurality of sub-images are flipped twice to obtain a plurality of flipped sub-images; Noise is added to the plurality of flipped sub-images to obtain a plurality of first sub-images; The plurality of first sub-images are spliced to obtain the preprocessed original images; The convolution network extracts the corresponding single-channel image and the brightness feature of each original image, which comprises: The original images are subjected to channel separation, and the feature images of each channel are obtained; A 3*3 convolution filter is used to obtain the brightness feature of the feature image of each channel; A plurality of channel feature images are combined into a single-channel feature image through a 1*1 convolution kernel, and a second brightness feature of the single-channel feature image is obtained, and the enhanced brightness feature is obtained by fusing and superimposing the second brightness feature and the brightness feature of each channel feature image.

5. The de-noising based 4K low light video enhancement system of claim 4, wherein: The low-light enhancement module comprises: a plurality of first convolutional layers for extracting feature information of a luminance feature and a single-channel feature image, and obtaining a plurality of first feature maps according to the feature information; a second convolutional layer for splicing the plurality of first feature maps to obtain a spliced image, and obtaining a second feature of the spliced image; an activation layer for mapping the second feature to a same number of channels as that of the original image to obtain an enhanced image.

6. The de-noising based 4K low light video enhancement system of claim 5, wherein: The loss function of the denoising module comprises a spatial consistency loss function, a color consistency loss function, an exposure control loss function and a noise control loss function.

Images