Dark-light image enhancement method based on cross-scale embedding and wavelet transform
Through cross-scale embedding and wavelet transformation technology, the problems of feature scale incompleteness and excessive noise emphasis in dark light image enhancement are solved, efficient detail recovery and noise reduction of the image are achieved, and the visual quality of the image is improved.
Patent Information
- Application Number
- CN202411239964.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-05
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2044-09-05
AI Technical Summary
The existing dark light image enhancement technology has the problem of not being able to fully understand the characteristics of different sizes, ignoring high-frequency details and excessive noise emphasis in image processing, resulting in a lack of clarity and detail in images.
Deep learning model based on cross-scale embedding and wavelet transform is adopted to extract different scale features through the cross-scale embedding module, combine wavelet transform and multi-head self-attention mechanism to separate and enhance high and low frequency information to reduce noise impact.
It significantly improves the clarity and detail recovery effect of the image, enhances the visual quality and texture details of the image, reduces the impact of noise, and improves the image enhancement effect.
Smart Images
Figure CN118781036B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of image processing, and in particular to a dark light image enhancement method based on cross-scale embedding and wavelet transform. Background Art
[0002] Dark-light image enhancement is an important research direction in the field of image processing, which aims to improve the quality of images taken under low lighting conditions. Images taken at night or in low-light environments are usually affected by noise, low contrast, and color distortion, which seriously affect the usability and visual effects of the images. The core role of dark-light image enhancement is to restore and improve the quality of these images captured under adverse lighting conditions through technical means, making their texture closer to images taken under ideal lighting conditions. Enhancement processing can significantly improve the visibility and information richness of images, which is crucial for a variety of application scenarios such as nighttime monitoring, medical imaging, and personal photography. Effective dark-light image enhancement not only optimizes the visual experience, but also improves the accuracy and efficiency of subsequent image analysis and processing tasks such as object recognition and scene understanding. The development of this technology enables high-quality image output even in extreme low-light conditions, greatly expanding the application boundaries of modern imaging technology.
[0003] Under the deep learning framework, dark-light image enhancement technology mainly relies on powerful convolutional neural networks (CNNs) to automatically learn and simulate complex lighting adjustment processes. These networks are trained with a large number of image pairs under dark and normal lighting conditions to learn how to recover details and colors from dark-light images. For example, autoencoders are widely used for this task, which compress image content through an encoder and then reconstruct the enhanced image through a decoder, effectively restoring image details and reducing noise. Although dark-light image enhancement technology has made significant progress, there are still some defects, such as:
[0004] In computer vision tasks, images usually contain feature information of different sizes, while traditional Transformers can only extract features of fixed patch sizes and cannot fully understand the image.
[0005] In previous image enhancement methods, high-frequency detail information is often easily ignored or lost, resulting in a lack of clarity and detail in the final image.
[0006] The self-attention mechanism in the traditional Transformer considers all regions of the image equally, including those containing noise. If the features of these regions are given higher weights in the self-attention calculation, the noise features of these regions may be over-emphasized during the image enhancement process.
[0007] In view of this, the present invention is conceived in depth to address the above-mentioned problems existing in the existing low-light image enhancement technology, and actively studies and improves the technology to develop the present invention. Summary of the invention
[0008] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a dark light image enhancement method based on cross-scale embedding and wavelet transform which can improve the image enhancement effect.
[0009] In order to achieve the above object, the solution of the present invention is:
[0010] A dark light image enhancement method based on cross-scale embedding and wavelet transform, using a deep learning model to enhance image data, the model includes an encoder and a decoder, the encoder includes a cross-scale embedding module, a transformer module and a downsampling module, the decoder includes a transformer module and an upsampling module, the transformer module includes a separation and fusion module for displaying high and low frequency information and an attention mechanism with wavelet transform;
[0011] The dark-light image with noise first enters the cross-scale embedding module of the encoder. The cross-scale embedding module samples the input dark-light image by using multiple convolution kernels of different sizes, and then embeds and connects these image feature blocks of different scales into tokens, so that small image blocks learn small-scale features and large image blocks learn large-scale features, thereby introducing the explicit cross-scale features in the token, so that the model can fully understand the dark-light image. Subsequently, the spatial dimension of the image is gradually reduced through the transformer module and the downsampling module, while retaining the key feature information. The output features of the encoder can capture the core content and structural information of the image and remove most of the noise;
[0012] The image processed by the encoder is input into the decoder, and the image is reconstructed from the compressed feature representation provided by the encoder. The transformer module and the upsampling module are used to gradually increase the spatial dimension of the image, while refining the details and quality of the image, and finally a clear output image is obtained. The separation and fusion module of the high- and low-frequency information in the transformer module of the decoder separates and enhances the high-frequency information in the image, captures the detailed information of the image, and retains the low-frequency information during the image enhancement process; the attention mechanism with wavelet transform decomposes the image into components of different frequencies, processes these components independently, reduces the impact of noise, and reduces the impact of the noise area on the attention mechanism by inputting the wavelet transform coefficients as query values into the multi-head self-attention mechanism, thereby improving the image enhancement effect.
[0013] Furthermore, the cross-scale embedding module of the encoder allows the model to adaptively adjust the embedding dimension according to different convolution kernel sizes, using lower dimensions for large-sized kernels and higher dimensions for small-sized kernels.
[0014] Furthermore, the specific process of the cross-scale embedding module is:
[0015] Assume the input is , the shape is [C,H,W], C is the number of channels of the input feature, H and W are the height and width of the input feature map, respectively.
[0016] First, according to the size of the input dark light image, four image blocks of different sizes are defined , , , :
[0017]
[0018]
[0019]
[0020]
[0021] Subsequently, according to , , , The sizes of the four image patches determine the embedding dimensions , , , , the stride in the convolution operation , , , and convolution padding , , , :
[0022]
[0023]
[0024]
[0025]
[0026]
[0027]
[0028]
[0029]
[0030]
[0031] Next, after determining the parameters of the convolutional layer, input the feature It will go through 4 convolution modules and perform feature reshaping to obtain the features , , , ,
[0032]
[0033]
[0034]
[0035]
[0036] in, represents the convolution module, and They are all feature reshaping operations. Indicates that the features are expanded along the second dimension. Indicates exchanging the first and second dimension information of the feature.
[0037] Finally, the features are concatenated and layer normalized to obtain the output of the cross-scale embedding module. ,
[0038]
[0039] in, represents feature concatenation, Representation layer normalization.
[0040] Furthermore, in the converter module of the decoder, the specific process of the separation and fusion module of the high and low frequency information is as follows:
[0041] Assume that the decoder input is First, input features It will go through the pooling module to obtain low-frequency features , the input feature Subtract the low-frequency features after upsampling , we can get the high frequency features ,
[0042]
[0043]
[0044] in, represents the pooling module, represents the upsampling module;
[0045] Next, high-frequency features After the custom convolution module is used to further extract information, high-frequency features are obtained. , low frequency characteristics After a custom convolution module, the information is further extracted to obtain low-frequency features. ,
[0046]
[0047]
[0048] in, Represents a custom convolution module;
[0049] Finally, the processed low-frequency features After the upsampling module, the high-frequency features after processing Perform feature splicing. The spliced features pass through the convolution module, attention module and convolution module in turn to obtain the fused output features. ,
[0050]
[0051] in, represents feature concatenation, represents the convolution module, represents the attention module.
[0052] Furthermore, the specific process of the attention mechanism with wavelet transform in the transformer module of the decoder is:
[0053] Assume that the input features are First, input features After the wavelet transform module and the convolution module respectively, the wavelet transform module is used to obtain queries related to the wavelet transform, and the feature , feature separation is performed after the convolution module to obtain the feature , used to get keys and values,
[0054]
[0055]
[0056] in, represents wavelet transform, represents the convolution module, Indicates feature separation;
[0057] Next, the features , , Reshape the features to get , , .right , Perform L2 normalization to obtain features , ,
[0058]
[0059]
[0060]
[0061]
[0062]
[0063] in, Represents feature reshaping, represents L2 normalization;
[0064] Afterwards, and Perform matrix multiplication and softmax processing to obtain ,
[0065]
[0066] in, represents matrix multiplication, represents softmax processing, Indicates the information of the last two dimensions of the exchange feature;
[0067] Finally, and After matrix multiplication, the final output can be obtained through convolution module and feature reshaping. ,
[0068]
[0069] in, represents matrix multiplication, represents the convolution module, Indicates feature reshaping.
[0070] Furthermore, during model training, the model input is a noisy image that has been randomly flipped and normalized. The present invention uses the L1 loss function for supervision, and the calculation formula of the loss function is as follows:
[0071]
[0072] in, represents the loss function, represents a clear image of the prediction, Represents a clear and real image. represents the L1 loss function.
[0073] The training cycle of the model is 500 epochs (when the model is trained to 500 epochs, the model loss has converged), the batch size is 4, the initial learning rate is 0.0001, the optimizer is Adamw, the betas is [0.9, 0.999], and the optimizer adjustment strategy is Cosine annealing.
[0074] After adopting the above structure, compared with the existing technical solutions, the dark light enhancement method based on cross-scale embedding and wavelet transform of the present invention uses a cross-scale embedding module to sample the input image, embeds and connects image blocks of different scales to form tokens, so that small-sized image blocks focus on small-scale features and large-scale image blocks learn large-scale features. By explicitly separating high-frequency and low-frequency information, high-frequency details can be enhanced in a targeted manner, which helps to restore the clarity and texture details of the image. In addition, by inputting the wavelet transform coefficients as query values into the multi-head attention mechanism, components of different frequencies can be processed independently and the influence of noise areas on attention calculations can be reduced, thereby improving the image enhancement effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0075] Figure 1 It is a schematic diagram of the overall framework of the present invention.
[0076] Figure 2 Schematic diagram of the structure of the cross-scale embedding module of the present invention.
[0077] Figure 3 This is a schematic diagram of the structure of the separation and fusion module of the present invention showing high and low frequency information.
[0078] Figure 4 Schematic diagram of the structure of the custom convolution module of the present invention.
[0079] Figure 5 Schematic diagram of the structure of the attention module of the present invention.
[0080] Figure 6 Schematic diagram of the attention mechanism structure with wavelet transform of the present invention.
[0081] Figure 7 It is a structural schematic diagram of the wavelet transform module of the present invention. DETAILED DESCRIPTION
[0082] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.
[0083] like Figure 1 As shown, the present invention discloses a dark light image enhancement method based on cross-scale embedding and wavelet transform, which uses a deep learning model to enhance image data. The model includes an encoder and a decoder. The encoder includes three basic components: a cross-scale embedding module, a transformer module, and a downsampling module. The decoder includes two basic components: a transformer module and an upsampling module. The transformer module includes a separation and fusion module for displaying high and low frequency information and an attention mechanism with wavelet transform.
[0084] The noisy dark image (input image) passes through the encoder and decoder in sequence. The encoder's task is to extract and compress the information in the image. It first uses a cross-scale embedding module to extract information at different scales. Subsequently, a series of transformer modules and downsampling modules are used to gradually reduce the spatial dimension of the image while retaining key feature information. The output features of the encoder can capture the core content and structural information of the image and remove most of the noise. The decoder's task is to reconstruct the image from the compressed feature representation provided by the encoder, using transformer modules and upsampling modules to gradually increase the spatial dimension of the image while refining the details and quality of the image, and finally obtain a clear output image.
[0085] like Figure 2 As shown in the figure, the cross-scale embedding module samples the input image by using multiple convolution kernels of different sizes, and then embeds and connects these image feature blocks of different scales into tokens, so that small image blocks learn small-scale features and large image blocks learn large-scale features, thereby introducing explicit cross-scale features in the token and enabling the model to fully understand the image. During the embedding process, in order to control the computational cost of the cross-scale embedding module, the cross-scale embedding module allows the model to adaptively adjust the embedding dimension according to different convolution kernel sizes, using lower dimensions (number of channels) for large-sized kernels and higher dimensions for small-sized kernels. This strategy saves computational costs without significantly affecting model performance.
[0086] Assume the input is , the shape is [C,H,W], C is the number of channels of the input feature, H and W are the height and width of the input feature map respectively.
[0087] First, according to the size of the input feature map, four different patch sizes are defined, where patch is an image block.
[0088]
[0089]
[0090]
[0091]
[0092] Then, the embedding dimension, convolution kernel size, convolution step size and convolution padding are determined according to the patch size. Dim represents the dimension, stride is the stride in the convolution operation, and pad is the convolution padding.
[0093]
[0094]
[0095]
[0096]
[0097]
[0098]
[0099]
[0100]
[0101]
[0102] Next, after determining the parameters of the convolutional layer, input the feature It will go through 4 convolution modules and perform feature reshaping to obtain the features , , , ,
[0103]
[0104]
[0105]
[0106]
[0107] in, represents the convolution module, and They are all feature reshaping operations. Indicates that the features are expanded along the second dimension. Indicates exchanging the first and second dimension information of the feature.
[0108] Finally, the features are concatenated and layer normalized to obtain the output of the cross-scale embedding module. ,
[0109]
[0110] in, represents feature concatenation, Representation layer normalization.
[0111] like Figure 3 As shown in the figure, high-frequency information usually contains the texture and detail features of the image. By separating and enhancing high-frequency information, the detail information of the image can be effectively captured. Low-frequency information is usually related to the smooth area of the image and contains the main structural information of the image. In the image enhancement process, retaining low-frequency information helps reduce noise while avoiding excessive blurring of the image.
[0112] Assume that the decoder input is First, input features It will go through the pooling module to obtain low-frequency features . Input features Subtract the low-frequency features after upsampling , we can get the high frequency features ,
[0113]
[0114]
[0115] in, represents the pooling module, Represents an upsampling module.
[0116] like Figure 4 As shown, next, the high-frequency features After the custom convolution module is used to further extract information, high-frequency features are obtained. , low frequency characteristics After a custom convolution module, the information is further extracted to obtain low-frequency features. ,
[0117]
[0118]
[0119] in, Represents a custom convolution module;
[0120] like Figure 5 As shown, finally, the processed low-frequency features After the upsampling module, the high-frequency features after processing Perform feature splicing. The spliced features pass through the convolution module, attention module and convolution module in turn to obtain the fused output features. ,
[0121]
[0122] in, represents feature concatenation, represents the convolution module, represents the attention module.
[0123] In the attention mechanism with wavelet transform, wavelet transform can decompose the image into components of different frequencies, and these components can be processed independently to reduce the impact of noise. By inputting the wavelet transform coefficients as query values into the multi-head self-attention mechanism, the impact of the noise area on the attention mechanism can be effectively reduced, thereby improving the image enhancement effect.
[0124] The structural diagram of the attention mechanism with wavelet transformation is as follows Figure 6 As shown, assuming that the input feature is First, the input features are passed through the wavelet transform module and the convolution module respectively. The wavelet transform module is used to obtain queries related to wavelet transform. The features are obtained after the wavelet transform module. , feature separation is performed after the convolution module to obtain the feature , used to get keys and values,
[0125]
[0126]
[0127] in, represents wavelet transform, represents the convolution module, Indicates feature separation.
[0128] The specific structure of the wavelet transform module is as follows Figure 7 As shown;
[0129] Next, the features , , Reshape the features to get , , .right , Perform L2 normalization to obtain features , ,
[0130]
[0131]
[0132]
[0133]
[0134]
[0135] in, Represents feature reshaping, represents L2 normalization.
[0136] Afterwards, and Perform matrix multiplication and softmax processing to obtain ,
[0137]
[0138] in, represents matrix multiplication, represents softmax processing, Indicates exchanging the information of the last two dimensions of the feature.
[0139] Finally, and After matrix multiplication, the final output can be obtained through convolution module and feature reshaping. ,
[0140]
[0141] in, represents matrix multiplication, represents the convolution module, Indicates feature reshaping.
[0142] During model training, the input of the model is a noisy image that has been randomly flipped and normalized. The present invention uses the L1 loss function for supervision. The calculation formula of the loss function is as follows:
[0143]
[0144] in, represents the loss function, represents a clear image of the prediction, Represents a clear and real image. represents the L1 loss function.
[0145] The training cycle of the model is 500 epochs (when the model is trained to 500 epochs, the model loss has converged), the batch size is 4, the initial learning rate is 0.0001, the optimizer is Adamw, the betas is [0.9, 0.999], and the optimizer adjustment strategy is Cosine annealing.
[0146] The present invention proposes a cross-scale embedding module, which enables each token to not only capture feature information of a single scale, but also integrate features from different scales, significantly enhancing the model's ability to extract features of different sizes in the image. In addition, the cross-scale embedding module can dynamically adjust the embedding dimension according to the size of the image block, effectively balancing performance and computational cost, and optimizing the efficiency and flexibility of the model.
[0147] The present invention introduces a separation and fusion module for displaying high and low frequency information to specifically process high and low frequency information, which can separate and specifically enhance the high frequency details of the image, thereby helping to better restore the texture and details of the image. At the same time, the separation and fusion module for displaying high and low frequency information uses a custom convolution module to effectively fuse information at different frequency levels, ensuring that both the high and low frequency information of the image are processed in a balanced manner, further improving the visual clarity and realism of the image.
[0148] The present invention also combines the attention mechanism of wavelet transform and utilizes the advantages of wavelet transform in multi-scale feature extraction and noise reduction, which can more accurately process and understand the noise and details in the image and improve the effect of image enhancement.
[0149] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.
Claims
1. A dark light image enhancement method based on cross-scale embedding and wavelet transform, characterized by: Use a deep learning model to enhance image data. The model includes an encoder and a decoder. The encoder includes a cross-scale embedding module, a transformer module, and a downsampling module. The decoder includes a transformer module and an upsampling module. The transformer module of the decoder includes a separation and fusion module for displaying high- and low-frequency information and an attention mechanism with wavelet transform. The dark-light image with noise first enters the cross-scale embedding module of the encoder. The cross-scale embedding module samples the input dark-light image by using multiple convolution kernels of different sizes, and then embeds and connects these image feature blocks of different scales into tags, so that small image blocks learn small-scale features and large image blocks learn large-scale features, thereby introducing the explicit cross-scale features in the tags, so that the model can fully understand the dark-light image. Subsequently, the spatial dimension of the image is gradually reduced through the transformer module and the downsampling module, while retaining the key feature information. The output features of the encoder can capture the core content and structural information of the image and remove most of the noise; The image processed by the encoder is input into the decoder, and the image is reconstructed from the compressed feature representation provided by the encoder. The transformer module and the upsampling module are used to gradually increase the spatial dimension of the image, while refining the details and quality of the image, and finally a clear output image is obtained. The separation and fusion module of the high- and low-frequency information in the transformer module of the decoder separates and enhances the high-frequency information in the image, captures the detailed information of the image, and retains the low-frequency information during the image enhancement process; the attention mechanism with wavelet transform decomposes the image into components of different frequencies, processes these components independently, and reduces the impact of noise. By inputting the wavelet transform coefficients as query values into the multi-head self-attention mechanism, the impact of the noise area on the attention mechanism is reduced, thereby improving the image enhancement effect; The specific process of the cross-scale embedding module is: Assume the input is , the shape is [C,H,W], C is the number of channels of the input feature, H and W are the height and width of the input feature map, respectively. First, according to the size of the input dark light image, four image blocks of different sizes are defined , , , : Subsequently, according to , , , The sizes of the four image patches determine the embedding dimensions , , , , the stride in the convolution operation , , , and convolution padding , , , : Next, after determining the parameters of the convolutional layer, input the feature It will go through 4 convolution modules and perform feature reshaping to obtain the features , , , , in, , , represents the convolution module, and They are all feature reshaping operations. Indicates that the features are expanded along the second dimension. Indicates exchanging the first and second dimension information of the feature; Finally, the features are concatenated and layer normalized to obtain the output of the cross-scale embedding module. , in, represents feature concatenation, Representation layer normalization.
2. The dark light image enhancement method based on cross-scale embedding and wavelet transform as claimed in claim 1, characterized in that: The encoder’s cross-scale embedding module allows the model to adaptively adjust the embedding dimension according to different convolution kernel sizes, using lower dimensions for large-sized kernels and higher dimensions for small-sized kernels.
3. The dark light image enhancement method based on cross-scale embedding and wavelet transform as claimed in claim 1, characterized in that: In the converter module of the decoder, the specific process of the separation and fusion module showing high and low frequency information is: Assume that the decoder input is First, input features It will go through the pooling module to obtain low-frequency features , the input feature Subtract the low-frequency features after upsampling , we can get the high frequency features , in, represents the pooling module, represents the upsampling module; Next, high-frequency features After the custom convolution module is used to further extract information, high-frequency features are obtained. , low frequency characteristics After a custom convolution module, the information is further extracted to obtain low-frequency features. , in, Represents a custom convolution module; Finally, the processed low-frequency features After the upsampling module, the high-frequency features after processing Perform feature splicing. The spliced features pass through the convolution module, attention module and convolution module in turn to obtain the fused output features. , in, represents feature concatenation, represents the convolution module, represents the attention module.
4. The dark light image enhancement method based on cross-scale embedding and wavelet transform as claimed in claim 1, characterized in that: The specific process of the attention mechanism with wavelet transform in the transformer module of the decoder is: Assume that the input features are First, input features After the wavelet transform module and the convolution module respectively, the wavelet transform module is used to obtain queries related to the wavelet transform, and the feature , feature separation is performed after the convolution module to obtain the feature , used to get keys and values, in, represents wavelet transform, represents the convolution module, Indicates feature separation; Next, the features , , Reshape the features to get , , ,right , Perform L2 normalization to obtain features , , in, Represents feature reshaping, represents L2 normalization; Afterwards, and Perform matrix multiplication and softmax processing to obtain , in, represents matrix multiplication, represents softmax processing, Indicates the information of the last two dimensions of the exchange feature; Finally, and After matrix multiplication, the final output can be obtained through convolution module and feature reshaping. , in, represents matrix multiplication, represents the convolution module, Indicates feature reshaping.
5. The dark light image enhancement method based on cross-scale embedding and wavelet transform as claimed in claim 1, characterized in that: During model training, the input of the model is a noisy image that has been randomly flipped and normalized. The L1 loss function is used for supervision. The calculation formula of the loss function is as follows: in, represents the loss function, represents a clear image of the prediction, Represents a clear and real image. represents the L1 loss function.
6. The dark light image enhancement method based on cross-scale embedding and wavelet transform as claimed in claim 1, characterized in that: The training cycle of the model is 500 epochs, the batch size is 4, the initial learning rate is 0.0001, the optimizer is Adamw, the betas are [0.9, 0.999], and the optimizer adjustment strategy is Cosine annealing.