Denoising-based 4K dark light video enhancement method and system
By using a combination of convolutional network and denoising module in 4K dark light video, low-light enhancement and denoising processing is performed, the noise suppression and inter-frame flickering problems during dark light video enhancement in the prior art are solved, and the more Ideal texture recovery and noise reduction improve the quality and naturalness of the video picture.
Patent Information
- Application Number
- CN202411066331.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-05
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-08-05
AI Technical Summary
When the existing low-light video enhancement technology restores dark-light video, it is difficult to balance the enhancement effect and noise suppression, which may lead to unnatural artifacts and uneven light, losing the natural and realistic feeling of the video, and it is easy to be in the process of denoising. Causes inter-frame flickering.
Using a 4K dark-light video enhancement method based on denoising, the single-channel and brightness characteristics of the image are extracted through the convolutional network, low-light enhancement cycle processing is performed, and combined with denoising processing, using a nonlinear transformation formula and denoising convolution layer. To ensure that image enhancement does not change pixel values and reduce inter-frame flickering, a system consisting of frame fetching, preprocessing, feature extraction, low-light enhancement and denoising modules are designed.
It realizes more ideal texture information recovery and noise reduction in 4K dark light video, significantly improving the enhancement effect of the video picture, avoiding inter-frame flickering, maintaining the natural and realistic video, and effectively reducing the noise. Recovery of quality interference.
Smart Images

Figure CN119048384A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing technology, and in particular to a 4K low-light video enhancement method and system based on denoising. Background Art
[0002] Technological progress continues to push the limits of video image quality, making video clarity and detail a hot topic of public concern. However, when shooting in dimly lit environments, video images often face challenges such as insufficient contrast, high noise, and unclear details. In order to address these issues, low-light video enhancement technology has emerged. It not only plays an important role in basic visual applications, such as object detection and tracking in video images, but also provides new possibilities for the further development and application of video images by improving the visibility of low-light images.
[0003] Existing low-light video enhancement technologies enhance the practicality and application scope of video images by improving the visual performance of video images under adverse lighting conditions. Existing methods sometimes fail to strike a good balance between video enhancement and noise suppression when restoring low-light videos. On the one hand, the brightness and contrast of the video are improved; on the other hand, unnatural artifacts and uneven lighting effects may be introduced. However, an improper balance may cause the video image to lose its original naturalness and realism visually.
[0004] In view of the deficiencies in the prior art, the present invention provides a 4K low-light video enhancement method and system based on denoising. Summary of the invention
[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a 4K low-light video enhancement method based on denoising with more ideal texture information restoration and better noise reduction effect, so as to solve the flickering problem of low-light video after denoising and enhancement while combining the enhancement and noise reduction processing of 4K low-light images.
[0006] The present invention provides a 4K dark light video enhancement method based on denoising, comprising:
[0007] Get multiple frames of continuous original images from the 4K dark-light video to be processed;
[0008] Input each original image into the convolutional network respectively, and extract the corresponding single-channel image and brightness features of each original image through the convolutional network;
[0009] According to the brightness feature, the single-channel feature image is subjected to low-light enhancement cycle processing to obtain an enhanced image corresponding to each original image;
[0010] Perform denoising on the enhanced image to obtain a denoised image corresponding to each original image;
[0011] Arrange multiple denoised images corresponding to the original image in chronological order and splice them to obtain the enhanced denoised dark-light video.
[0012] Furthermore, the 4K low-light video enhancement method based on denoising further includes: after acquiring multiple frames of continuous original images, before inputting the original images into the convolutional network, preprocessing the original images to obtain preprocessed original images; the preprocessing of the original images includes:
[0013] Down-sample the original image to obtain multiple sub-images;
[0014] Flipping the multiple sub-images twice to obtain multiple flipped sub-images;
[0015] Adding noise to the multiple flipped sub-images to obtain multiple first sub-images;
[0016] The multiple first sub-images are stitched together to obtain a preprocessed original image.
[0017] Furthermore, extracting the corresponding single-channel image and brightness features of each original image through a convolutional network includes:
[0018] Separate the channels of the original image and obtain the feature image of each channel;
[0019] A 3*3 convolution filter is used to obtain the brightness characteristics of the feature image of each channel;
[0020] The multiple channel feature images are synthesized into a single-channel feature image through a 1*1 convolution kernel, and the second brightness feature of the single-channel feature image is obtained. The brightness feature of the first image is obtained by fusing and superimposing the second brightness feature with the brightness feature of the feature image of each channel.
[0021] Furthermore, performing low-light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain an enhanced image corresponding to each original image includes:
[0022] Extracting the brightness feature and the feature information of the single-channel feature image through a plurality of first convolutional layers, and obtaining a plurality of first feature maps according to the feature information;
[0023] Splicing the multiple first feature maps through the second convolutional layer to obtain a spliced image, and extracting features from the spliced image to obtain a second feature map;
[0024] The second feature is mapped to the same number of channels as the original image through the activation layer to obtain an enhanced image.
[0025] Furthermore, the performing low-light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain an enhanced image corresponding to each original image also includes:
[0026] When the multiple first feature maps are input into the second convolutional layer, the multiple first feature maps are scaled to the same size as the input size of the second convolutional layer using a bilinear interpolation algorithm;
[0027] When the second feature is input into the activation layer, the stitched image is scaled to the same size as the input of the activation layer using a bilinear interpolation algorithm.
[0028] Furthermore, performing low-light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain an enhanced image also includes:
[0029] When the first feature map is input into the second convolutional layer, a nonlinear transformation is performed on the first feature map;
[0030] When the second feature is input into the activation layer, a nonlinear transformation is performed on the spliced image;
[0031] The nonlinear change formula is as follows:
[0032]
[0033] where x a-1 is the output image of the previous iteration, x r It is an enhancement factor calculated by convolution.
[0034] Furthermore, the denoising process of the enhanced image to obtain a denoised image corresponding to each original image includes:
[0035] Convolve the enhanced image through the denoising convolution layer to obtain the noise map;
[0036] The noise map is upsampled through the upsampling layer to obtain an upsampled noise feature map;
[0037] The upsampled noise feature map, enhanced image and original image are compared to obtain denoising parameters;
[0038] The enhanced image is obtained by subtracting the denoising parameters from the noise map to obtain the denoised image.
[0039] The present invention also provides a 4K low-light video enhancement system based on denoising, comprising a frame acquisition module, a preprocessing module, a feature extraction module, a low-light enhancement module, a denoising module and a splicing module;
[0040] The frame acquisition module is used to acquire multiple frames of continuous original images in the 4K dark light video to be processed;
[0041] The preprocessing module is used to downsample and flip the original image to obtain the preprocessed original image;
[0042] The feature extraction module is used to obtain a single-channel feature image and brightness features of the original image;
[0043] The low-light enhancement module is used to enhance the single-channel feature image according to the brightness feature to obtain an enhanced image;
[0044] The denoising network module is used to denoise the enhanced image to obtain a denoised image after denoising;
[0045] The splicing module is used to arrange and splice multiple denoised images corresponding to the original image in time sequence to obtain an enhanced denoised dark-light video.
[0046] Furthermore, the low-light enhancement module includes:
[0047] A plurality of first convolutional layers, used to extract feature information of brightness features and single-channel feature images, and obtain a plurality of first feature maps according to the feature information;
[0048] A second convolutional layer is used to splice multiple first feature maps to obtain a spliced image and obtain a second feature of the spliced image;
[0049] The activation layer is used to map the second feature to the same number of channels as the original image to obtain an enhanced image.
[0050] Furthermore, the loss function of the denoising network module includes a spatial consistency loss function, a color consistency loss function, an exposure control loss function and a noise control loss function.
[0051] The present invention has the following advantages:
[0052] (1) When performing image enhancement, the present invention establishes a nonlinear function so that when the value of the pixel point does not change, fixed enhancement is performed. The input image is enhanced through a series of nonlinear transformations related to the input image x and the parameter enhancement factor xr. When the value of the pixel point of the first feature map or the spliced image does not change, the enhancement factor is a fixed parameter, that is, the enhancement effect is fixed, so when performing low-light enhancement, inter-frame flickering almost does not occur; thereby ensuring that inter-frame flickering does not occur between the video frames after the image is spliced, and realizing the simultaneous processing of restoration, denoising and flicker removal for 4K low-light video;
[0053] (2) The present invention introduces a denoising network module by designing a network structure, which significantly improves the effect of the neural network on video image enhancement in a dark light environment. This innovation not only effectively reduces the interference of noise on the quality of video restoration, but also obtains the enhancement factor by establishing a nonlinear function, so that the denoising network also performs well in video flicker removal. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] Figure 1 It is a diagram of the method steps of the present invention;
[0055] Figure 2 This is a schematic diagram of the image enhancement model structure framework flow chart;
[0056] Figure 3 The experimental data are presented for comparison. DETAILED DESCRIPTION
[0057] The present invention is further described below in conjunction with the accompanying drawings, but the protection scope of the present invention is not limited to the following description.
[0058] It should be noted that the directions or positional relationships indicated by "left", "right", etc. are based on the directions or positional relationships shown in the accompanying drawings, or are the directions or positional relationships in which the inventive product is usually placed when in use, or are the directions or positional relationships commonly understood by those skilled in the art. Such terms are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore cannot be understood as a limitation on the present invention.
[0059] It should be noted that, in the absence of conflict, the embodiments of the present invention and the features and technical solutions in the embodiments may be combined with each other.
[0060] First aspect
[0061] like Figure 1 As shown, the present invention provides a 4K dark light video enhancement method based on denoising, comprising:
[0062] S1. Obtain multiple frames of continuous original images from the 4K dark-light video to be processed; the main acquisition scenes of the original images are urban streets and the wild, and the resolution of the original images is 3840*2160;
[0063] S2, preprocessing the original image to obtain a preprocessed original image;
[0064] S3, inputting each original image into the convolutional network respectively, and extracting the corresponding single-channel image and brightness features of each original image through the convolutional network;
[0065] S4, performing low-light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain an enhanced image corresponding to each original image;
[0066] S5, performing denoising processing on the enhanced image to obtain a denoised image corresponding to each original image;
[0067] S6. Arrange and stitch multiple denoised images corresponding to the original image in chronological order to obtain an enhanced denoised dark-light video.
[0068] The present invention denoises the image and combines image enhancement and denoising techniques, thereby effectively reducing the interference of noise on the quality of video restoration and significantly improving the effect of video picture enhancement in a low-light environment. In addition, an enhancement factor is used in image enhancement to make the enhanced image quality higher, thereby solving the problems of the image being easily disturbed by noise when directly denoising the low-light image, the noise being amplified after enhancement by the traditional algorithm, the edge features being lost, and the flickering of continuous frames of the video whose details are smoothed.
[0069] Furthermore, the preprocessing of the original image includes:
[0070] S201, down-sampling the original image to obtain multiple sub-images;
[0071] Multiple sub-images are obtained by downsampling the original image; the original image with a resolution of 3840*2160 is downsampled and randomly cropped into 8 sub-images of 608*608;
[0072] S202, flipping the multiple sub-images twice to obtain multiple flipped sub-images;
[0073] S203, adding noise to the multiple flipped sub-images to obtain multiple first sub-images;
[0074] S204 , stitching multiple first sub-images to obtain a pre-processed original image.
[0075] Preprocessing the original image can effectively increase the diversity of the image, improve the robustness and recognition ability of subsequent image processing.
[0076] Furthermore, the corresponding single-channel image and brightness features of each original image are extracted through the convolutional network, including:
[0077] S301, performing channel separation on the original image and obtaining a feature image of each channel;
[0078] S302, using a 3*3 convolution filter to obtain the brightness characteristics of the feature image of each channel; the separation processing of each channel helps the convolution filter to obtain the spatial details and brightness changes in each channel to obtain accurate brightness characteristics;
[0079] S303, synthesizing the multiple channel feature images into a single-channel feature image through a 1*1 convolution kernel, and obtaining a second brightness feature of the single-channel feature image, and obtaining a brightness feature of the first image by fusing and superimposing the second brightness feature with the brightness feature of the feature image of each channel;
[0080] The second brightness feature is obtained through cross-channel fusion of the 1*1 convolution layer. The brightness information between different color channels is mutually enhanced to obtain the enhanced brightness feature, which is convenient for the subsequent calculation of the enhancement factor.
[0081] Furthermore, performing low-light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain an enhanced image corresponding to each original image includes:
[0082] S401, respectively extracting brightness features and feature information of a single-channel feature image through a plurality of first convolutional layers, and obtaining a plurality of first feature maps according to the feature information;
[0083] S402, splicing the plurality of first feature maps through a second convolutional layer to obtain a spliced image, and performing feature extraction on the spliced image to obtain a second feature map;
[0084] S403, mapping the second feature to the same number of channels as the original image through an activation layer to obtain an enhanced image.
[0085] In some specific embodiments, performing low-light enhancement loop processing on a single-channel feature image according to brightness features to obtain an enhanced image corresponding to each original image further includes:
[0086] When the multiple first feature maps are input into the second convolutional layer, the multiple first feature maps are scaled to the same size as the input size of the second convolutional layer using a bilinear interpolation algorithm;
[0087] When the second feature is input into the activation layer, the stitched image is scaled to the same size as the input of the activation layer using a bilinear interpolation algorithm;
[0088] When the first feature map is input into the second convolutional layer, a nonlinear transformation is performed on the first feature map;
[0089] When the second feature is input into the activation layer, a nonlinear transformation is performed on the spliced image;
[0090] The nonlinear change formula is as follows:
[0091]
[0092] where x a-1 is the output image of the previous iteration, x r It is an enhancement factor calculated by convolution.
[0093] The input image is enhanced through a series of nonlinear transformations related to the input image x and the parameter enhancement factor xr. When the pixel value of the first feature map or the spliced image does not change, the enhancement factor is a fixed parameter, that is, the enhancement effect is fixed, so when performing low-light enhancement, there is almost no inter-frame flickering. In this way, by adding a nonlinear transformation formula, the problem of noise affecting video picture quality and video flickering in traditional methods is successfully solved.
[0094] Furthermore, the denoising process of the enhanced image to obtain a denoised image corresponding to each original image includes:
[0095] S501, convolving the enhanced image through a denoising convolution layer to obtain a noise map;
[0096] S502, upsampling the noise map through an upsampling layer to obtain an upsampled noise feature map;
[0097] S503, comparing the upsampled noise feature map, the enhanced image and the original image to obtain denoising parameters;
[0098] S504: Subtract the enhanced image from the noise map using the denoising parameters to obtain a denoised image.
[0099] The present invention is based on the characteristics of dark-light images. Dark-light images have many noise points, and direct denoising will cause a lot of detail information to be lost. Therefore, the image is first convolved to obtain a noise map, and then the noise image is upsampled to retain the detail information; then, denoising parameters are obtained through comparative analysis, and the enhanced image is compared with the difference in the noise map based on the denoising parameters to obtain a denoised image; the denoised image obtained in this way not only has accurate noise removal, but also retains the detail information of the image.
[0100] Second aspect
[0101] The present invention also provides a 4K low-light video enhancement system based on denoising, comprising: a frame acquisition module, a preprocessing module, a feature extraction module, a low-light enhancement module, a denoising module and a splicing module;
[0102] The frame acquisition module is used to acquire multiple frames of continuous original images in the 4K dark light video to be processed;
[0103] The preprocessing module is used to downsample and flip the original image to obtain the preprocessed original image;
[0104] The feature extraction module is used to obtain a single-channel feature image and brightness features of the original image;
[0105] The low-light enhancement module is used to enhance the single-channel feature image according to the brightness feature to obtain an enhanced image;
[0106] The denoising network module is used to denoise the enhanced image to obtain a denoised image after denoising;
[0107] The splicing module is used to arrange and splice multiple denoised images corresponding to the original image in time sequence to obtain an enhanced denoised dark-light video.
[0108] The present invention introduces a denoising network module by designing a network structure, which significantly improves the effect of the neural network on video picture enhancement in a dark light environment and effectively reduces the interference of noise on the quality of video restoration.
[0109] See also Figure 2 , the low light enhancement module comprises:
[0110] A plurality of first convolutional layers, used to extract feature information of brightness features and single-channel feature images, and obtain a plurality of first feature maps according to the feature information;
[0111] A second convolutional layer is used to splice multiple first feature maps to obtain a spliced image and obtain a second feature of the spliced image;
[0112] The activation layer is used to map the second feature to the same number of channels as the original image to obtain an enhanced image.
[0113] Through the design of the network structure, the present invention enables the low-light enhancement module to effectively cope with changes in lighting conditions while performing well in retaining image detail information; the network structure of the low-light enhancement module can capture the global information of the image and obtain feature information through multiple first convolutional layers, which is crucial for restoring features in the image.
[0114] Furthermore, the loss function of the denoising convolution layer includes a spatial consistency loss function, a color consistency loss function, an exposure control loss function and a noise control loss function.
[0115] The present invention adopts an unsupervised training method, which has good generalization performance. Figure 3 As shown, through comparative experiments, the peak signal-to-noise ratio (PSNR) and structural similarity (SSIM) are used as evaluation indicators to evaluate the quality of an image. It can be seen that when the present invention is adopted, the values of PSNR and SSIM are the largest (it should be noted that the higher the PSNR value, the higher the image quality, and the higher the SSIM value, the closer it is to the clear data).
[0116] The above embodiments only express preferred implementation modes, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, and these all belong to the protection scope of the present invention.
Claims
1. A 4K low-light video enhancement method based on denoising, characterized by: include: Get multiple frames of continuous original images from the 4K dark-light video to be processed; Input each original image into the convolutional network respectively, and extract the corresponding single-channel image and brightness features of each original image through the convolutional network; According to the brightness feature, the single-channel feature image is subjected to low-light enhancement cycle processing to obtain an enhanced image corresponding to each original image; Perform denoising on the enhanced image to obtain a denoised image corresponding to each original image; Arrange multiple denoised images corresponding to the original image in chronological order and splice them to obtain the enhanced denoised dark-light video.
2. The 4K low-light video enhancement method based on denoising according to claim 1, further comprising: After acquiring multiple frames of continuous original images, before inputting the original images into the convolutional network, the original images are preprocessed to obtain preprocessed original images; The preprocessing of the original image comprises: Down-sample the original image to obtain multiple sub-images; Flipping the multiple sub-images twice to obtain multiple flipped sub-images; Adding noise to the multiple flipped sub-images to obtain multiple first sub-images; The multiple first sub-images are stitched together to obtain a preprocessed original image.
3. The 4K low-light video enhancement method based on denoising according to claim 2, characterized in that: The extracting of the corresponding single-channel image and brightness features of each original image through the convolutional network includes: Separate the channels of the original image and obtain the feature image of each channel; A 3*3 convolution filter is used to obtain the brightness characteristics of the feature image of each channel; The multiple channel feature images are synthesized into a single-channel feature image through a 1*1 convolution kernel, and the second brightness feature of the single-channel feature image is obtained. The brightness feature of the first image is obtained by fusing and superimposing the second brightness feature with the brightness feature of the feature image of each channel.
4. The 4K low-light video enhancement method based on denoising according to claim 1, characterized in that: The step of performing low-light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain an enhanced image corresponding to each original image includes: Extracting the brightness feature and the feature information of the single-channel feature image through a plurality of first convolutional layers, and obtaining a plurality of first feature maps according to the feature information; Splicing the multiple first feature maps through the second convolutional layer to obtain a spliced image, and extracting features from the spliced image to obtain a second feature map; The second feature is mapped to the same number of channels as the original image through the activation layer to obtain an enhanced image.
5. The 4K low-light video enhancement method based on denoising according to claim 4, characterized in that: The step of performing low-light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain an enhanced image corresponding to each original image further includes: When the multiple first feature maps are input into the second convolutional layer, the multiple first feature maps are scaled to the same size as the input size of the second convolutional layer using a bilinear interpolation algorithm; When the second feature is input into the activation layer, the stitched image is scaled to the same size as the input of the activation layer using a bilinear interpolation algorithm.
6. The 4K low-light video enhancement method based on denoising according to claim 4, characterized in that: The performing low-light enhancement cycle processing on the single-channel feature image according to the brightness feature to obtain the enhanced image also includes: When the first feature map is input into the second convolutional layer, a nonlinear transformation is performed on the first feature map; When the second feature is input into the activation layer, a nonlinear transformation is performed on the spliced image; The nonlinear change formula is as follows: where x a-1 is the output image of the previous iteration, x r It is an enhancement factor calculated by convolution.
7. The 4K low-light video enhancement method based on denoising according to claim 1, characterized in that: The denoising process of the enhanced image to obtain a denoised image corresponding to each original image comprises: Convolve the enhanced image through the denoising convolution layer to obtain the noise map; The noise map is upsampled through the upsampling layer to obtain an upsampled noise feature map; The upsampled noise feature map, enhanced image and original image are compared to obtain denoising parameters; The enhanced image is obtained by subtracting the denoising parameters from the noise map to obtain the denoised image. 8.4K low-light video enhancement system based on denoising, characterized by: It includes frame acquisition module, preprocessing module, feature extraction module, low-light enhancement module, denoising module and splicing module; The frame acquisition module is used to acquire multiple frames of continuous original images in the 4K dark light video to be processed; The preprocessing module is used to downsample and flip the original image to obtain the preprocessed original image; The feature extraction module is used to obtain a single-channel feature image and brightness features of the original image; The low-light enhancement module is used to enhance the single-channel feature image according to the brightness feature to obtain an enhanced image; The denoising network module is used to denoise the enhanced image to obtain a denoised image after denoising; The splicing module is used to arrange and splice multiple denoised images corresponding to the original image in time sequence to obtain an enhanced denoised dark-light video.
9. The 4K low-light video enhancement system based on denoising according to claim 8, characterized in that: The low light enhancement module comprises: A plurality of first convolutional layers, used to extract feature information of brightness features and single-channel feature images, and obtain a plurality of first feature maps according to the feature information; A second convolutional layer is used to splice multiple first feature maps to obtain a spliced image and obtain a second feature of the spliced image; The activation layer is used to map the second feature to the same number of channels as the original image to obtain an enhanced image.
10. The 4K low-light video enhancement method based on denoising according to claim 9, characterized in that: The loss function of the denoising network module includes a spatial consistency loss function, a color consistency loss function, an exposure control loss function and a noise control loss function.
Citation Information
Patent Citations
Image processing method and device, electronic equipment and readable storage medium
CN113674159A
Image processing method and device
CN114022402A
Low-light full-color video image adaptive enhancement method and system based on deep learning
CN115393227A
Lightweight infrared and visible light image fusion method based on convolutional neural network
CN116681636A
Haptic Device and Haptic Module
KR1020230154154A
Cited By
Image enhancement method, controller, imaging device and storage medium
CN120807298A
Video processing method and device, computer equipment and readable storage medium
CN121616466A
