A method for solving pseudo edges and chessboard based on MBLLEN improved enhanced network
By employing average pooling, bilinear interpolation, and feature map concatenation in a deep learning low-light enhancement network, the pseudo-edge and checkerboard effects are resolved, improving both image enhancement performance and computational efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEFEI JUNZHENG TECH CO LTD
- Filing Date
- 2022-06-16
- Publication Date
- 2026-05-12
AI Technical Summary
Existing deep learning low-light enhancement networks suffer from false edges and checkerboard effects in image processing, leading to image quality degradation and failing to meet the engineering requirements of real-world scenarios.
We employ a combination of average pooling and bilinear interpolation for downsampling and upsampling, optimize expressive power through 1*1 convolution, and concatenate feature maps to leverage the model's adaptive capabilities to acquire necessary information.
It effectively solves the problems of false edges and checkerboard effect, enhances the expressive power of the network, reduces the amount of computation, and improves the quality of image enhancement.
Smart Images
Figure CN117291794B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of low-light enhancement technology in low-level vision tasks of deep learning neural networks, and particularly to a method for solving pseudo-edges and checkerboard patterns based on an improved enhancement network using MBLLEN. Background Technology
[0002] Existing supervised deep learning low-light enhancement methods, since both input and output are images, have a direct positive or negative impact on the results due to the choice of network structure. Most models are based on U-shaped networks or other fully convolutional structures. In real-world applications, the input and output images are often high-resolution 1080p or even 4K images, requiring extremely high computational power. Therefore, downsampling and upsampling restoration operations are necessary. Common downsampling methods include using the stride and padding of convolutions, or pooling operations. Upsampling is often achieved through deconvolution or interpolation. The choice of different structures will have a very direct and obvious impact, as the model's output is also an image.
[0003] The original enhancement structure of the MBLLEN network chose to use padding for valid light downsampling, and then perform upsampling through deconvolution.
[0004] However, downsampling using convolution padding only results in a slightly smaller feature map than the original image size, failing to significantly reduce computational cost. Furthermore, upsampling this feature map leads to noticeable false edges, which, while improved through training, cannot perfectly resolve the inherent structural flaws. Upsampling using deconvolution suffers from the checkerboard effect inherent in its computational method, again imperfect even with training, significantly impacting the final image quality. This results in a severe degrade in the quality of enhanced low-light images, failing to meet the demands of real-world engineering applications.
[0005] In addition, the commonly used terms in the prior art are as follows:
[0006] Bilinear interpolation: In mathematics, bilinear interpolation is a linear interpolation extension of an interpolation function with two variables. Its core idea is to perform linear interpolation in two directions respectively. It is widely used in signal processing, digital image and video processing, etc.
[0007] Checkerboard effect: Suppose an image generated by deconvolution contains a black object. The pixel colors of the black object should have a smooth transition. Or, to put it extremely, the entire body should be black. However, in the actual generated image, this part is composed of near-black squares of varying shades, much like a checkerboard pattern. This is the so-called checkerboard effect.
[0008] False edges: These are obvious traces that appear at the edges of an image. If the image is segmented and then merged, it often presents a grid-like phenomenon.
[0009] Downsampling and upsampling: Downsampling is a process of shrinking the feature map, while upsampling is a process of enlarging the feature map. Summary of the Invention
[0010] To address the aforementioned issues, the purpose of this application is to construct a new enhanced network branch structure based on MBLLEN, thereby resolving the pseudo-edge and checkerboard effects present in the network, while simultaneously enhancing the network's expressive power and achieving significant improvements in denoising and model performance.
[0011] Specifically, this invention provides a method for solving pseudo-edges and chessboard patterns based on an improved MBLLEN network, the method comprising the following steps:
[0012] S1. Downsampling structure selection:
[0013] For the denoising part in the low-light enhancement task, average pooling with denoising properties is selected as the downsampling structure, and the downsampling factor is 2. This operation reduces the size of the feature map by half.
[0014] S2. Downsampling structure optimization:
[0015] The property of enhancing expressive power through 1*1 convolution is used to form an optimized structure;
[0016] S3. Secondary downsampling:
[0017] By downsampling the image twice, the size of the image's feature map will become 1 / 4 of the original size;
[0018] S4. Upsampling structure selection:
[0019] We choose bilinear interpolation, with an upsampling factor of 2.
[0020] S5. Upsampling structure optimization:
[0021] The operation involves adding 3x3 and 1x1 convolutions before and after the optimized structure in step S2.
[0022] S6. Connect the upsampling result in S5 with the downsampling result in S2 by considering the context feature map information;
[0023] S7. Secondary upsampling:
[0024] Perform the same steps as in step S5 again;
[0025] S8. Again, connect the context feature map information and concatenate the result of the second upsampling with the result without downsampling, that is, concatenate the original image without S1 with the result of S7.
[0026] The optimized structure in step S2 further includes:
[0027] S2.1, passed through the first convolutional layer with a kernel of 3 and an output channel of 64;
[0028] S2.2, the average pooling layer after 2x downsampling;
[0029] S2.3, passed through a second convolutional layer with a kernel of 1 and an output channel of 32.
[0030] In step S3, the image undergoes secondary downsampling: This secondary downsampling is performed by combining maximum pooling of high-frequency information with average pooling of low-frequency information, and further includes:
[0031] S3.1, 3*3*64 convolution (conv);
[0032] S3.2, Max pooling and average pooling respectively:
[0033] S3.2.1, double the maximum pooling (MaxPool);
[0034] Perform a 1*1*32 convolution (conv);
[0035] S3.2.2, 2x average pooling (AvgPool);
[0036] Perform a 1*1*32 convolution (conv);
[0037] S3.3, concat: The feature map is a three-dimensional matrix (h, w, c), where h represents the length, w represents the width, and c represents the channel. Concatenation means that two feature maps with the same h and w are concatenated together along the c dimension.
[0038] In step S5, 3*3 and 1*1 convolutions are added before and after the optimized structure in step S2, respectively;
[0039] S5.1, passed through the first convolutional layer with a kernel of 3 and an output channel of 64;
[0040] S5.2, after passing through a second convolutional layer with 3 kernels and 64 output channels;
[0041] S5.3, an average pooling layer with 2x downsampling;
[0042] S5.4, passed through a third convolutional layer with a kernel of 1 and an output channel of 32;
[0043] S5.5 is the fourth convolutional layer with a kernel of 1 and 32 output channels.
[0044] Step S7 further includes;
[0045] S7.0, Upsampling Structure Selection: Select bilinear interpolation, with an upsampling factor of 2;
[0046] S7.1, perform 3x3 convolution (conv);
[0047] S7.2, perform 3*3*64 convolution (conv);
[0048] S7.3, perform 2x average pooling (AvgPool);
[0049] S7.4, perform a 1*1*32 convolution (conv);
[0050] S7.5, perform 1*1 convolution (conv).
[0051] Step S8, which involves stitching the original image (without S1) with the result of S7, further includes: features Figure 1 Typically, it is a three-dimensional matrix (h, w, c), where h represents the length, w represents the width, and c represents the channel. Concatenation means that two feature maps with the same h and w are concatenated together along the c dimension.
[0052] Steps S6 and S8 use a concatenation method to obtain the necessary information for the desired effect of the model in the labeled dataset by leveraging the model's own adaptive capability. The adaptive capability refers to the ability of deep learning supervised learning to gradually reduce the gap between the model output and the labeled dataset during model training by minimizing the loss function between the model output and the labeled dataset, and to train weight information that adapts to this change.
[0053] The necessary information includes brightness, contrast, and edge information.
[0054] Therefore, the advantage of this application is:
[0055] 1. Due to the inherent limitations of the MBLLEN network structure, it is inevitable that false edges and checkerboard effects will occur in general real-world image enhancement. Through the modifications in this application, on the one hand, the inherent defects of the original structure are resolved, avoiding false edges and checkerboard effects, and on the other hand, the expressive power of the network is enhanced, enabling it to converge well even under conditions where the exposure range of the training set is not fixed.
[0056] 2. Reducing the feature map size decreases the computational power required by the model;
[0057] 3. Consider the context to avoid information loss in a purely linear structure. Attached Figure Description
[0058] The accompanying drawings, which are provided to further illustrate the invention and form part of this application, are not intended to limit the scope of the invention.
[0059] Figure 1 This is a flowchart illustrating the method of the present invention.
[0060] Figure 2 This is a schematic diagram of the structure in step S2 of the method of the present invention.
[0061] Figure 3 This is a schematic diagram of the structure in step S3 of the method of the present invention.
[0062] Figure 4 This is a schematic diagram of the structure in steps S6 and S8 of the method of the present invention.
[0063] Figure 5 This is a schematic diagram of step S2 in the method of the present invention.
[0064] Figure 6 Step S of the method of the present invention
[0065] Figure 7 This is a schematic diagram of the original MBLLEN enhanced branch structure.
[0066] Figure 8 This is a schematic diagram of the improved enhanced branch structure.
[0067] Figure 9 Based on Figure 8 Display and Figure 1 A diagram illustrating the relationship between the steps. Detailed Implementation
[0068] To better understand the technical content and advantages of the present invention, the present invention will now be described in further detail with reference to the accompanying drawings.
[0069] This invention relates to a method for structural optimization based on MBLLEN network enhanced branches, in order to solve the problem of pseudo edges and checkerboard patterns in the original structure.
[0070] This application's MBLLEN network enhancement structure uses padding for light downsampling, followed by upsampling through deconvolution. A method for resolving pseudo-edges and checkerboard patterns using an improved MBLLEN enhancement network is also provided. Figure 1 The flowchart of this method is shown below:
[0071] S1. Downsampling structure selection:
[0072] For the denoising part in the low-light enhancement task, average pooling with denoising properties is selected as the downsampling structure, and the downsampling factor is 2. This operation reduces the size of the feature map by half.
[0073] S2. Downsampling structure optimization:
[0074] The property of enhancing expressive power through 1*1 convolution is used to form an optimized structure;
[0075] S3. Secondary downsampling:
[0076] By downsampling the image twice, the size of the image's feature map will become 1 / 4 of the original size;
[0077] S4. Upsampling Structure Selection:
[0078] We choose bilinear interpolation, with an upsampling factor of 2.
[0079] S5. Upsampling structure optimization:
[0080] The operation involves adding 3x3 and 1x1 convolutions before and after the optimized structure in step S2.
[0081] S6. Connect the upsampling result in S5 with the downsampling result in S2 by considering the context feature map information;
[0082] S7. Secondary upsampling:
[0083] Perform the same steps as in step S5 again;
[0084] S8. Again, connect the context feature map information and concatenate the result of the second upsampling with the result without downsampling, that is, concatenate the original image without S1 with the result of S7.
[0085] Specifically, the implementation steps are described below:
[0086] Step S1. Selection of downsampling structure
[0087] For the denoising part of the low-light enhancement task, average pooling, which has natural denoising properties, is chosen as the downsampling structure, with a downsampling factor of 2. This operation can reduce the size of the feature map by half.
[0088] Step S2. Downsampling Structure Optimization
[0089] While average pooling has denoising properties, it can also easily cause the model to become overly blurry. To avoid this, 1x1 convolutions can be used to enhance expressive power, structuring the model as follows: Figure 2 The structure shown;
[0090] The optimized structure in step S2 further includes:
[0091] S2.1, perform 3*3*64 convolution (conv);
[0092] Convolution is a fundamental operation in deep learning. It can be simply understood as multiplying a 3x3x64 matrix by the input feature map matrix in a sliding manner, as follows: Figure 5 The diagrams shown are similar to those of avgpool, but the window size is different, the calculation method is not to calculate the mean within the window but to multiply it by the weight of the window, and the step size is also different, but the overall idea is similar.
[0093] S2.2, perform double average pooling (AvgPool);
[0094] Average pooling is a fundamental operation in deep learning. Double average pooling is equivalent to sliding a 2x2 window across the input feature map, calculating the mean within each window. The final result will be half the size of the feature map, such as... Figure 5 As shown;
[0095] The dark gray area represents the size of the sliding window, with a step size of 2 for each slide. The left side shows the input feature map and the sliding window, while the right side shows the result after 2x average pooling.
[0096] S2.3, perform a 1*1*32 convolution (conv).
[0097] The optimization approach for the first downsampling structure in step S2 is as follows:
[0098] For the denoising part in the low-light enhancement task, average pooling, which has denoising properties, is selected as the downsampling structure, with a downsampling factor of 2. This operation reduces the size of the feature map by half. Although average pooling has denoising properties, it can also easily cause the model to become overly blurred. To avoid this phenomenon, the expressive power of 1*1 convolution can be enhanced to optimize it.
[0099] Step S3. Secondary downsampling
[0100] A common operation in image signal processing is to use methods such as mean filtering to denoise the model and then sharpening to restore details. Max pooling itself has a prominent role in processing high-frequency information in images, so it can be used in various ways. Figure 3 This method performs a second downsampling on the image, at which point the size of the image's feature map will become 1 / 4 of the original size;
[0101] The processing structure for the second downsampling in step S3 is as follows:
[0102] A common operation in image signal processing is to use methods such as mean filtering to denoise the model and then sharpening to restore details. Max pooling itself has a prominent role in processing high-frequency information in images, so secondary downsampling can be performed by combining the structure of max pooling for high-frequency information with average pooling for low-frequency information.
[0103] In step S3, the image is downsampled a second time. This second downsampling is performed by combining the structure of max pooling for high-frequency information with average pooling for low-frequency information. Further steps include:
[0104] S3.1, 3*3*64 convolution, can be simply understood as multiplying the input feature map with a 3*3*64 matrix in a sliding manner;
[0105] S3.2, Max pooling and average pooling respectively:
[0106] S3.2.1, 2x MaxPool: (e.g.) Figure 6 As shown, this is the maximum pooling layer after 2x downsampling.
[0107] Perform a 1*1*32 convolution (conv);
[0108] S3.2.2, 2x average pooling (AvgPool);
[0109] Perform a 1*1*32 convolution (conv);
[0110] S3.3, concat concatenation: feature Figure 1 Typically, it's a three-dimensional matrix (h, w, c), where h represents length, w represents width, and c represents channels. The concatenation refers to joining two feature maps with the same h and w along the c-dimensional axis. For example: Feature... Figure 1 The dimensions are (1080, 1920, 16), and the features are... Figure 2 The dimensions of the first feature map are (1080, 1920, 16), and the dimensions of the new feature map obtained after concatenation are (1080, 1920, 32). This concatenation operation is as follows: Figure 4 The concat shown.
[0111] Step S4. Upsampling Structure Selection
[0112] To avoid the checkerboard effect caused by deconvolution, we choose another upsampling method. Here, we choose bilinear interpolation. Since deconvolution only upsamples the feature map size and does not take into account the relationship between pixels well, bilinear interpolation can solve this problem well. Here, the upsampling factor is 2.
[0113] Step S5. Upsampling Structure Optimization
[0114] This operation is similar to step S2, adding 3*3 and 1*1 convolutions before and after the structure;
[0115] Step S6. Connect the context feature map information and concatenate the upsampling result from S5 with the downsampling result from S2, such as... Figure 4 As shown.
[0116] Step S7. Secondary upsampling
[0117] This operation is the same as step S5;
[0118] Step S8. Again, consider the contextual feature map information and concatenate the result of the second upsampling with the result of the unsampling, such as... Figure 4 As shown. Features Figure 1 Typically, it's a three-dimensional matrix (h, w, c), where h represents the length, w represents the width, and c represents the channels. Concatenation involves joining two feature maps with the same h and w along the c-dimensional axis. For example: Feature... Figure 1 The dimensions are (1080, 1920, 16), and the features are... Figure 2 The dimensions of the first feature map are (1080, 1920, 16), and the dimensions of the new feature map obtained after concatenation are (1080, 1920, 32). This concatenation operation is as follows: Figure 4 The concat in the context.
[0119] The logic behind steps S6 and S8 is as follows:
[0120] Simple linear structures easily lose information as the number of network layers increases when processing image enhancement tasks, making contextual association crucial. Additive residual structures are too straightforward, so using a concatenation approach leverages the model's own adaptive capabilities to effectively acquire the necessary information.
[0121] Steps S6 and S8 use a concatenation method to obtain necessary information by leveraging the model's own adaptive capabilities. In deep learning supervised learning, during model training, the gap between the model output and the labeled dataset is gradually reduced by decreasing the loss between the two datasets, training weight information that adapts to this change—this is the adaptive capability. The necessary information refers to the desired effect the model should achieve in the labeled dataset, such as brightness, contrast, and edge information.
[0122] In summary, such as Figure 7 The diagram shows the enhanced branch structure of the original MBLLEN network, including:
[0123] 1) 3x3 convolution (conv), Valid;
[0124] 2), 5x5 convolution (conv), Valid;
[0125] 3) 5x5 convolution (conv), Valid;
[0126] 4) 5x5 deconvolution, Valid;
[0127] 5), 5x5 deconvolution, Valid;
[0128] In this context, a valid convolution means using only the valid part of each convolution.
[0129] And such Figure 8 , 9 As shown, the improved enhanced branch structure: where, Figure 9 The display is based on Figure 8 and Figure 1 The correspondence between the steps: Step S7 can be considered as a repetition of s4 and s5.
[0130] 1) 3x3 convolution (conv);
[0131] 2), AvgPool with 2x average pooling;
[0132] 3), 1*1 convolution (conv);
[0133] 4) 3x3 convolution (conv);
[0134] 5) Further, simultaneously perform 2x max pooling (MaxPool) and 2x average pooling (AvgPool);
[0135] Further, perform 1*1 convolutions (conv) on each of them;
[0136] 6) Concatenate the two results using `concat`.
[0137] 7) Perform a 3x3 convolution (conv);
[0138] 8) Perform double-line interpolation (2x).
[0139] 9) Combine the results from 3 and 8 to perform a 1*1 convolution;
[0140] 10), 3*3 convolution (conv);
[0141] 11), 2x bilinear interpolation;
[0142] 12), 1*1 convolution (conv);
[0143] 13), the above 12) is further combined with the original image to obtain the result.
[0144] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations can be made to the embodiments of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A method for solving pseudo-edges and chessboard patterns based on an improved MBLLEN network, characterized in that, The method includes the following steps: S1. Downsampling structure selection: For the denoising part in the low-light enhancement task, average pooling with denoising properties is selected as the downsampling structure, with a downsampling factor of 2, reducing the size of the feature map by half; S2. Downsampling structure optimization: The property of enhancing expressive power through 1*1 convolution is used to form an optimized structure; S3. Secondary downsampling: By performing a second downsampling on the image, the size of the image's feature map will become 1 / 4 of the original size; the second downsampling is performed by combining the structure of max pooling of high-frequency information and average pooling of low-frequency information. S4. Upsampling structure selection: Choose bilinear interpolation with an upsampling factor of 2; S5. Upsampling structure optimization: Based on the optimized structure in step S2, add 3*3 and 1*1 convolutions before and after it; S6. Connect the upsampling result in S5 with the downsampling result in S2 by considering the contextual feature map information; S7. Secondary upsampling: Perform the same steps as in step S5 again; S8. Again, by referring to the context feature map information, the result of the second upsampling is combined with the result without downsampling, that is, the original image without S1 is combined with the result of S7.
2. The method for solving pseudo-edges and chessboard patterns based on an improved MBLLEN network according to claim 1, characterized in that, The optimized structure in step S2 includes: S2.1, passed through the first convolutional layer with a kernel of 3 and an output channel of 64; S2.2, the average pooling layer after 2x downsampling; S2.3, passed through a second convolutional layer with a kernel of 1 and an output channel of 32.
3. The method for solving pseudo-edges and chessboard patterns based on an improved MBLLEN network according to claim 1, characterized in that, Step S3 involves performing a second downsampling on the image, including: S3.1, 3*3*64 convolution (conv); S3.2, Max pooling and average pooling respectively: S3.2.1, double the maximum pooling (MaxPool); Perform a 1*1*32 convolution (conv); S3.2.2, 2x average pooling (AvgPool); Perform a 1*1*32 convolution (conv); S3.3, concat: The feature map is a three-dimensional matrix (h, w, c), where h represents the length, w represents the width, and c represents the channel. Concatenation means that two feature maps with the same h and w are concatenated together along the c dimension.
4. The method for solving pseudo-edges and chessboard patterns based on an improved MBLLEN network according to claim 1, characterized in that, In step S5, 3x3 and 1x1 convolutions are added before and after the optimized structure in step S2, including: S5.1, passed through the first convolutional layer with a kernel of 3 and an output channel of 64; S5.2, after passing through a second convolutional layer with 3 kernels and 64 output channels; S5.3, an average pooling layer with 2x downsampling; S5.4, passed through a third convolutional layer with a kernel of 1 and an output channel of 32; S5.5 is the fourth convolutional layer with a kernel of 1 and 32 output channels.
5. The method for solving pseudo-edges and chessboard patterns based on an improved MBLLEN network according to claim 1, characterized in that, Step S7 includes: S7.0, Upsampling Structure Selection: Select bilinear interpolation, upsampling factor is 2; S7.1, perform a 3x3 convolution (conv); S7.2, perform 3*3*64 convolution (conv); S7.3, perform double average pooling (AvgPool); S7.4, perform a 1*1*32 convolution (conv); S7.5, perform 1*1 convolution (conv).
6. The method for solving pseudo-edges and chessboard patterns based on an improved MBLLEN network according to claim 1, characterized in that, In step S8, the original image without S1 is concatenated with the result of S7. This includes: the feature map is a three-dimensional matrix (h, w, c), where h represents length, w represents width, and c represents channels. Concatenation means that two feature maps with the same h and w are concatenated together along the c dimension.
7. The method for solving pseudo-edges and chessboard patterns based on an improved MBLLEN network according to claim 1, characterized in that, Steps S6 and S8 use a concatenation method to obtain the necessary information for the desired effect of the model in the labeled dataset by leveraging the model's own adaptive capability. The adaptive capability refers to the ability of deep learning supervised learning to gradually reduce the gap between the model output and the labeled dataset during model training by minimizing the loss function between the model output and the labeled dataset, and to train weight information that adapts to this change. The necessary information includes brightness, contrast, and edge information.