A low-light enhancement method and device based on a dynamic filter and a storage medium
By extracting multi-scale features and performing image enhancement using a dynamic filter method, the problem of image quality degradation under low light conditions is solved, image details and contrast are improved, computational complexity is reduced, and different spatial variations are adapted.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHONGKE HUIJIN DIGITAL TECH BEIJING
- Filing Date
- 2023-08-30
- Publication Date
- 2026-06-26
AI Technical Summary
Images taken in low light conditions often suffer from a decline in visual quality, including blurred details, reduced contrast, and noise issues. Existing technologies struggle to effectively improve image quality.
A low-light enhancement method based on dynamic filters is adopted. Multi-scale feature information is extracted through residual downsampling module and pyramid convolutional pooling module. The filter generation network and decoder are combined for image enhancement and blur removal. CurveNLU module is used for nonlinear brightness adjustment and feature enhancement.
It improves image quality under low-light conditions, enhances image details and contrast, reduces computational complexity, adapts to different spatial variations, preserves image structure and texture information, and improves the performance of image processing algorithms.
Smart Images
Figure CN117132497B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, specifically to a low-light enhancement method, apparatus, and storage medium based on dynamic filters. Background Technology
[0002] With societal progress and the widespread adoption of smart electronic and multimedia devices, multimedia information such as images and videos are increasingly used in daily life. However, in many applications, low-light images are both unavoidable and in high demand. For example, when shooting with a mobile phone at night, the shutter speed is kept open for a relatively long time to capture light and details in the image, but this often results in uneven brightness and blurry images. Urban surveillance imaging provides a wealth of information for personnel searches, vehicle management, and evidence collection of violations, but it is heavily limited by ambient light. While it can provide clear, color-accurate images during the day, the image quality drops significantly in low light or at night. In remote sensing, underwater, and other specific scenarios, it is often difficult to achieve good shooting results by adjusting equipment parameters in a timely manner, often resulting in low-brightness images. Under certain conditions, due to limitations of the electronic equipment itself and the shooting environment, obtaining clear images that meet human visual requirements is quite difficult. For example, images taken in rainy weather are often too dark and blurry. In certain scenarios, low-light images are often obtained. If long exposure is used to capture the image, blurring will occur. Moreover, it is difficult for non-professionals to improve the image quality in low-light environments by adjusting equipment parameters.
[0003] Low-light images are a very common phenomenon during shooting. Insufficient lighting can greatly reduce the visual quality of images, causing problems such as blurred details, reduced contrast, and noise. It can also affect the performance of evaluation metrics in many computer vision tasks. Therefore, low-light image enhancement has high research and application value. Summary of the Invention
[0004] To address the shortcomings of existing technologies, this invention provides a low-light enhancement method, apparatus, and storage medium based on dynamic filters, which solves the problem that insufficient lighting can greatly reduce the visual quality of images, causing blurred details, reduced contrast, and noise.
[0005] To achieve the above objectives, the present invention provides the following technical solution: a low-light enhancement method based on dynamic filters, comprising the following steps: S1, in the encoder, feature extraction is performed on the input low-light image using a residual downsampling module and a pyramid convolutional pooling module to obtain multi-scale image feature information; S2, the image feature information at different scales is input into a filter generation network to obtain various local dynamic filters at the decoding end; S3, in the decoder, the feature information obtained by the encoder is decoded using various local dynamic filters and a pixel recombination upsampling module to achieve low-light enhancement and blur removal.
[0006] Further, the specific steps in S1 for feature extraction from the input low-light image using the residual downsampling module and the pyramid convolutional pooling module include: S10, receiving a low-light image with dimensions H×W×C, where H and W are the height and width of the original low-light image, and C is the number of channels in the original low-light image; S20, performing feature extraction through three different scale residual downsampling modules and pyramid convolutional pooling modules to generate three scales of low-resolution feature maps D1, D2, and D3, with dimensions respectively.
[0007] Further, the specific steps of S20, which involves feature extraction using three different scale residual downsampling modules and pyramid convolutional pooling modules to generate low-resolution feature maps at three scales, include: S201, performing residual downsampling on the input data to obtain a preliminary low-resolution feature map; S202, applying a pyramid convolutional pooling operation to the low-resolution feature map obtained from residual downsampling to extract hierarchical global prior information at different scales; S203, applying a convolutional layer to aggregate the low-resolution feature map obtained in S201 and the global prior information obtained in S202; S204, repeating S201, S202, and S203 three times to perform residual downsampling and pyramid convolutional pooling operations at three different scales to obtain corresponding feature maps D1, D2, and D3.
[0008] Furthermore, after obtaining the low-resolution feature map in S20, a CurveNLU module is used to further extract features and enhance the structural and texture information of the image. The specific steps are as follows: S11, perform three convolution operations on the input low-resolution feature map; S21, normalize the convolution result to improve the stability and convergence of the model; S31, after normalization, apply the nonlinear activation function Sigmoid to introduce a nonlinear transformation to enhance the expressive power of the features.
[0009] Furthermore, the S2 filter generation network consists of a group of three 3×3 convolutional layers and one 1×1 convolutional layer. For the three low-resolution feature maps D1, D2, and D3 generated by S1, the corresponding set of convolutional operators is used to form the filter generation network, generating three different local dynamic filters at the decoding end. The size of each local dynamic filter is D∈d×d, where d is set to 5, which is used for filtering transformation at the decoding end to expand the feature dimension.
[0010] Further, in the decoder, the specific steps of S3 for decoding the feature information obtained by the encoder using local dynamic filters of different scales and pixel reconstruction upsampling modules are as follows: S30, the feature map D3 generated in S20 is processed by three residual modules and then used as the input feature map; S31, the input feature map is filtered and transformed using a local dynamic filter of size d×d to obtain the output feature map; S32, the output feature map is processed by two residual modules and then used as the input of the pixel reconstruction upsampling module in S3 for upsampling operation to obtain the feature map as the input feature map of the next local dynamic filter and pixel reconstruction upsampling module; S33, S31 and S32 are repeated three times to generate the result image.
[0011] A low-light enhancement device based on dynamic filters includes an encoding module, a filter generation module, and a decoding module. The encoding module extracts features from the input low-light image using a residual downsampling module and a pyramid convolutional pooling module in the encoder to obtain multi-scale image feature information. The filter generation module inputs the image feature information at different scales into a filter generation network to obtain local dynamic filters for the decoding module. The decoding module decodes the feature information obtained by the encoding module using the local dynamic filters and a pixel recombination upsampling module to achieve low-light enhancement and blur removal.
[0012] A computer-readable storage medium for storing a program that, when executed by a processor, implements the aforementioned low-light enhancement method based on a dynamic filter.
[0013] The present invention has the following beneficial effects:
[0014] This low-light enhancement method based on dynamic filters, through pyramid convolutional pooling operations, enables the dynamic filter module to perform multi-scale feature extraction, reduce computational complexity, enhance features, adapt to spatial variations, and preserve structure in low-light enhancement and blur removal tasks. This helps improve the performance and effectiveness of image processing algorithms, achieves enhanced low-light image enhancement, and solves the problem that insufficient illumination can greatly reduce the visual quality of images, causing blurred details, reduced contrast, and noise.
[0015] Of course, any product implementing this invention does not necessarily need to achieve all of the advantages described above at the same time. Attached Figure Description
[0016] Figure 1 This is a flowchart of the low-light enhancement method based on dynamic filters of the present invention;
[0017] Figure 2 This is a flowchart illustrating the feature extraction process of the low-light enhancement method based on dynamic filters according to the present invention.
[0018] Figure 3 This is a flowchart illustrating the CurveNLU module of the low-light enhancement method based on dynamic filters in this invention for further feature extraction.
[0019] Figure 4 This is a diagram of the generation network structure of the dynamic filter in the low-light enhancement method based on the dynamic filter of the present invention.
[0020] Figure 5 This is a flowchart of the encoder decoding process in the low-light enhancement method based on dynamic filters of the present invention;
[0021] Figure 6 This is a network structure diagram of the low-light enhancement method based on dynamic filters of the present invention. Detailed Implementation
[0022] This application provides a low-light enhancement method, apparatus, and storage medium based on dynamic filters, which improves the enhancement effect of low-light images and solves the problem that insufficient lighting will greatly reduce the visual quality of images, causing blurred details, reduced contrast, and noise.
[0023] The technical solution in this application embodiment is to solve the problem of the aforementioned low-light enhancement method based on dynamic filters. The overall idea is as follows:
[0024] To address the problem of blurred edges and textures in low-light images, a low-light image enhancement and deblurring network is proposed. From the perspective of structural and textural information, a dynamic local filter is designed to perform convolutional filtering on local texture information. Inspired by edge detection algorithms, an edge loss function is designed for low-light blurred images to enhance the structural and textural information in the image. The edge loss function extracts some textures and object boundaries, and then L1 constraints are used to preserve the extracted information, thus enhancing the low-light image from the perspectives of structural and textural enhancement and deblurring.
[0025] Please see Figure 1 and Figure 6As shown, this embodiment of the invention provides a technical solution: a low-light enhancement method based on dynamic filters, comprising the following steps: S1, in the encoder, using a residual downsampling module and a pyramid convolutional pooling module to extract features from the input low-light image to obtain multi-scale image feature information; S2, inputting the image feature information of different scales into a filter generation network to obtain various local dynamic filters at the decoding end; S3, in the decoder, using various local dynamic filters and a pixel recombination upsampling module to decode the feature information obtained by the encoder to achieve low-light enhancement and blur removal.
[0026] Specifically, such as Figure 2 As shown, the specific steps in S1 for feature extraction from the input low-light image using the residual downsampling module and the pyramid convolutional pooling module include: S10, receiving a low-light image with dimensions H×W×C, where H and W are the height and width of the original low-light image, and C is the number of channels in the original low-light image; S20, performing feature extraction through the residual downsampling module and the pyramid convolutional pooling module at three different scales to generate low-resolution feature maps D1, D2, and D3 at three scales, with dimensions respectively.
[0027] The specific steps of S20, which extracts features from the input data using three different scale residual downsampling modules and pyramid convolutional pooling modules to generate low-resolution feature maps at three scales, include: S201, performing residual downsampling on the input data to obtain a preliminary low-resolution feature map; S202, applying a pyramid convolutional pooling operation to the low-resolution feature map obtained from residual downsampling to extract hierarchical global prior information at different scales; S203, applying a convolutional layer to aggregate the low-resolution feature map obtained in S201 and the global prior information obtained in S202; S204, repeating S201, S202, and S203 three times to perform residual downsampling and pyramid convolutional pooling operations at three different scales to obtain the corresponding feature maps D1, D2, and D3.
[0028] In this implementation, rich contextual information is captured by performing convolution and pooling operations on the feature map at different scales. For the input feature map, multiple scales are set, such as S scales. For each scale, a convolutional layer is applied, and the feature map is convolved using an appropriate kernel size and number. This allows feature information at different scales to be captured.
[0029] Apply pooling operations, such as max pooling or average pooling, to downsample the convolutional feature map to reduce its size while retaining important feature information. Repeat the above steps to perform pyramid convolution and pooling operations for each scale.
[0030] The feature maps obtained through the pyramid convolutional pooling strategy are fused to obtain a pyramid feature map with features at multiple scales. Fusion can be achieved through simple concatenation or convolutional operations. After processing by the pyramid convolutional pooling module, feature maps D1, D2, and D3 with features at multiple scales are obtained, with dimensions of [missing data].
[0031] Specifically, such as Figure 3 As shown, after obtaining the low-resolution feature map in S20, a CurveNLU module is used to further extract features and enhance the structural and texture information of the image. The specific steps are as follows: S11, perform three convolution operations on the input low-resolution feature map; S21, normalize the convolution result to improve the stability and convergence of the model; S31, after normalization, apply the nonlinear activation function Sigmoid to introduce a nonlinear transformation to enhance the expressive power of the features.
[0032] In this implementation, CurveNLU is a module for non-linear grayscale mapping. By learning the grayscale curve mapping of the input image, it performs non-linear brightness adjustment on low-light images. The CurveNLU module can improve the contrast and brightness of the image and enhance detail information to improve the visualization effect of the image under low-light conditions.
[0033] By reusing the CurveNLU module in the encoder, we can extract features multiple times and enhance the structural and textural information of the image. This results in an enhanced feature map E with higher expressive power and richer feature representation, providing better input for subsequent processing steps such as filter generation and decoder operations.
[0034] Specifically, such as Figure 4 As shown, the S2 filter generation network consists of a group of three 3×3 convolutional layers and one 1×1 convolutional layer. For the three low-resolution feature maps D1, D2, and D3 generated by S1, the corresponding set of convolutional operators is used to form the filter generation network, generating three different local dynamic filters at the decoding end. The size of each local dynamic filter is D∈d×d, where d is set to 5, which is used for filtering transformation at the decoding end to expand the feature dimension.
[0035] In this implementation, a dynamic filter generation network can generate filters with higher dimensions, thereby expanding the dimension of the feature map and helping to extract richer feature information. This allows the decoder to more accurately reconstruct the original image. The dynamic filter generation network uses three 3×3 convolutional layers and one 1×1 convolutional layer. This design can capture contextual information at different scales and receptive fields. By introducing a larger receptive field, it can better understand the structural and semantic information in the image and improve the decoder's ability to understand the feature map.
[0036] The filtering transformation operation convolves the feature map D with a dynamic filter. By applying a corresponding convolution kernel to each element, the feature map obtained by the decoder can be further refined. This helps to remove noise, enhance details, and improve the quality of image reconstruction. In this step, the kernel size d is set to 5. A larger kernel size helps to capture a wider range of contextual information and better preserves image details.
[0037] Specifically, such as Figure 5 As shown, in the decoder, the specific steps of S3 for decoding the feature information obtained by the encoder using local dynamic filters of different scales and pixel reconstruction upsampling modules are as follows: S30, the low-resolution feature map D3 generated in S20 is processed by three residual modules and the resulting feature map is used as the input feature map; S31, the input feature map is filtered and transformed using a local dynamic filter of size d×d to obtain the output feature map; S32, the output feature map is processed by two residual modules and used as the input of the pixel reconstruction upsampling module in S3 for upsampling operation to obtain the feature map as the input feature map of the next local dynamic filter and pixel reconstruction upsampling module; S33, S31 and S32 are repeated three times to generate the result image.
[0038] In this implementation, the residual module performs a convolution operation on the filtered feature map, which further extracts and enhances features. This helps capture details and contextual information in the image, thereby improving the quality of image reconstruction. The introduction of the residual module can improve the feature extraction capability of the decoder, making the reconstructed image more accurate and clear.
[0039] A low-light enhancement device based on dynamic filters includes an encoding module, a filter generation module, and a decoding module. The encoding module extracts features from the input low-light image using a residual downsampling module and a pyramid convolutional pooling module in the encoder to obtain multi-scale image feature information. The filter generation module inputs image feature information at different scales into a filter generation network to obtain local dynamic filters for the decoding module. The decoding module decodes the feature information obtained by the encoding module using the local dynamic filters and a pixel reassembly upsampling module to achieve low-light enhancement and blur removal.
[0040] In this implementation, the pyramid convolutional pooling module performs convolution and pooling operations on the feature map at different scales, which can capture multi-scale feature information of the image and help extract rich contextual information, thereby improving the image processing effect. At the same time, the pyramid convolutional pooling module can reduce the size of the feature map, thereby reducing computational complexity and storage requirements. The smaller feature map can be operated on more efficiently in subsequent processing, accelerating the running speed of the model.
[0041] The dynamic filter module is adaptive, capable of dynamically adjusting to spatial variations in the image. This allows the model to better adapt to feature transformations at different locations and scales, improving its ability to handle complex image scenes.
[0042] The residual connection mechanism in the residual module can preserve the structural information of the input feature map, avoiding information loss and error accumulation. This helps to maintain the details and edge information of the image and improve the quality of image reconstruction.
[0043] The technical solutions described in the embodiments of this application have at least the following technical effects or advantages:
[0044] The pyramid convolutional pooling operation and dynamic filter module have multi-scale feature extraction, reduced computational complexity, feature enhancement, spatial variation adaptability and structure preservation in low-light enhancement and blur removal tasks. They help improve the performance and effect of image processing algorithms, realize the enhancement effect of low-light images, and solve the problem that insufficient lighting will greatly reduce the visual quality of images, causing image detail blurring, reduced contrast and noise.
[0045] A computer-readable storage medium for storing a program, which, when executed by a processor, implements a low-light enhancement method based on a dynamic filter.
[0046] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0047] This invention is described with reference to flowchart illustrations and / or block diagrams of systems, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0048] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0049] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0050] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.
[0051] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.
Claims
1. A low-light enhancement method based on dynamic filters, characterized in that, Includes the following steps: S1. In the encoder, the residual downsampling module and the pyramid convolutional pooling module are used to extract features from the input low-light image to obtain multi-scale image feature information; S2. Input image feature information at different scales into the filter generation network to obtain local dynamic filters at the decoding end; S3. In the decoder, the feature information obtained by the encoder is decoded using local dynamic filters and pixel recombination upsampling modules to achieve low-light enhancement and blur removal. The specific steps in S1 for feature extraction of the input low-light image using the residual downsampling module and the pyramid convolutional pooling module include: S10. Receive a low-light image, the size of which is... ,in and It is the height and width of the original low-light image. This is the number of channels in the original low-light image; S20. Feature extraction is performed using three residual downsampling modules at three different scales and a pyramid convolutional pooling module to generate low-resolution feature maps at three scales. , and Their dimensions are respectively , , ; The specific steps for S20 to extract features through three residual downsampling modules and pyramid convolutional pooling modules at three different scales, generating low-resolution feature maps at three scales, include: S201. For the input data, perform residual downsampling to obtain a preliminary low-resolution feature map; S202. Apply a pyramid convolutional pooling operation to the low-resolution feature map obtained by residual downsampling to extract hierarchical global prior information at different scales. S203. For the low-resolution feature map obtained in S201 and the global prior information obtained in S202, a convolutional layer is applied to aggregate them. S204. Repeat S201, S202, and S203 three times to perform residual downsampling and pyramid convolutional pooling operations at three different scales, obtaining the corresponding feature maps. , , ; The filter generation network in S2 consists of a group of three 3×3 convolutional layers and one 1×1 convolutional layer, which are used to generate the three low-resolution feature maps generated in S1. , , Each set of convolution operators is used to construct a filter generation network, generating three different local dynamic filters for the decoding end. The size of each local dynamic filter is [missing information]. It is used for filtering and transformation at the decoding end to expand the feature dimension.
2. The low-light enhancement method based on a dynamic filter according to claim 1, characterized in that: After obtaining the low-resolution feature map in step S20, a CurveNLU module is used to further extract features and enhance the structural and texture information of the image. The specific steps are as follows: S11. Perform three convolution operations on the input low-resolution feature map; S21. Normalize the convolution results to improve the stability and convergence of the model; S31. After normalization, the nonlinear activation function Sigmoid is applied to introduce a nonlinear transformation, which enhances the expressive power of the features.
3. The low-light enhancement method based on a dynamic filter according to claim 2, characterized in that: In the decoder, the specific steps of S3 for decoding the feature information obtained from the encoder using local dynamic filters of different scales and a pixel reconstruction upsampling module are as follows: S30, The low-resolution feature map generated in S20 The feature map after processing by three residual modules is the input feature map; S31. Utilize the input feature map The output feature map is obtained by performing a filtering transformation on a local dynamic filter of varying size. S32. The output feature map, after passing through two residual modules, is used as the input to the pixel reconstruction upsampling module in S3 for upsampling operation, and the resulting feature map is used as the input feature map for the next local dynamic filter and pixel reconstruction upsampling module; S33. Repeat S31 and S32 three times to generate the resulting image.
4. A low-light enhancement device based on a dynamic filter, used to implement the low-light enhancement method based on a dynamic filter as described in any one of claims 1 to 3, characterized in that, It includes an encoding module, a filter generation module, and a decoding module, wherein: The encoding module is used in the encoder to extract features from the input low-light image using the residual downsampling module and the pyramid convolutional pooling module to obtain multi-scale image feature information. The filter generation module is used to input image feature information at different scales into the filter generation network to obtain the local dynamic filters of the decoding module. The decoding module is used to decode the feature information obtained by the encoding module using local dynamic filters and pixel recombination upsampling module to achieve low light enhancement and blur removal. The specific steps for feature extraction from the input low-light image using the residual downsampling module and the pyramid convolutional pooling module include: S10. Receive a low-light image, the size of which is... ,in and It is the height and width of the original low-light image. This is the number of channels in the original low-light image; S20. Feature extraction is performed using three residual downsampling modules at three different scales and a pyramid convolutional pooling module to generate low-resolution feature maps at three scales. , and Their dimensions are respectively , , ; The specific steps for S20 to extract features through three residual downsampling modules and pyramid convolutional pooling modules at three different scales, generating low-resolution feature maps at three scales, include: S201. For the input data, perform residual downsampling to obtain a preliminary low-resolution feature map; S202. Apply a pyramid convolutional pooling operation to the low-resolution feature map obtained by residual downsampling to extract hierarchical global prior information at different scales. S203. For the low-resolution feature map obtained in S201 and the global prior information obtained in S202, a convolutional layer is applied to aggregate them. S204. Repeat S201, S202, and S203 three times to perform residual downsampling and pyramid convolutional pooling operations at three different scales, obtaining the corresponding feature maps. , , ; The filter generation network consists of a group of three 3×3 convolutional layers and one 1×1 convolutional layer, used to generate the low-resolution feature map. , , Each set of convolution operators is used to construct a filter generation network, generating three different local dynamic filters for the decoding end. The size of each local dynamic filter is [missing information]. It is used for filtering and transformation at the decoding end to expand the feature dimension.
5. A computer-readable storage medium for storing a program, characterized in that, When the program is executed by the processor, it implements the low-light enhancement method based on dynamic filters as described in any one of claims 1 to 3.