Dark light image enhancement method based on deep learning model

By adopting a deep learning model with mixed feature compensation and sparse attention in dark light image enhancement, the shortcomings of CNN and Transformer in processing global information and detail recovery are solved, and efficient image denoising and color rendering are achieved, suitable for extremes Image processing under lighting conditions.

CN118710537BActive Publication Date: 2024-11-15RECONOVA TECH CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411206290.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-15
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

Traditional CNNs are difficult to process global information and long-distance dependence in dark-light image enhancement. Transformer's self-attention mechanism performs poorly when restoring image details and has high computational complexity, resulting in noise interference and waste of computing resources.

Method used

Using a deep learning model based on mixed feature compensation and sparse attention, multi-scale features are extracted through feature extraction module and feature compensation module combined with multiple subnets, and dynamically adjust feature representations using sparse self-attention mechanism and gated feedforward mechanism to reduce noise Interfere with and improve image detail recovery.

Benefits of technology

It effectively improves image denoising performance and color rendering effect, improves the model's focus ability on important information, reduces the computational complexity, and is suitable for image processing under extreme lighting conditions.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118710537B_ABST
    Figure CN118710537B_ABST
Patent Text Reader

Abstract

The present invention discloses a dark light image enhancement method of a deep learning model. The raw image will first undergo a hybrid feature compensation mechanism to enhance the model's processing of image details and texture features; then, the encoder is responsible for capturing the key feature information and noise information of the image, providing a basis for subsequent image denoising and color rendering; then, the raw decoder processes the features extracted by the encoder to achieve image denoising, and uses the noise processable characteristics of the raw image to remove noise in the image, retaining important details and textures of the image; finally, the sRGB decoder with sparse attention and gated feedforward mechanism is responsible for color space conversion and color enhancement, ensuring that the final image is visually closer to the lighting conditions of the real world so that it can be presented on various display devices. The present invention can effectively improve the image quality in low-light environments by introducing a hybrid feature compensation mechanism, a sparse attention mechanism, and a gated feedforward mechanism.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of computer image processing, and in particular to a dark light image enhancement method based on a deep learning model of hybrid feature compensation and sparse attention. Background Art

[0002] Dark-light image enhancement is an important research direction in the field of image processing, which aims to improve the quality of images taken under low lighting conditions. Images taken at night or in low-light environments are usually affected by noise, low contrast, and color distortion, which seriously affect the usability and visual effects of the images. The core role of dark-light image enhancement is to restore and improve the quality of these images captured under adverse lighting conditions through technical means, so that their texture is closer to that of images taken under ideal lighting conditions. Enhancement processing can significantly improve the visibility and information richness of images, which is crucial for a variety of application scenarios such as night monitoring, medical imaging, and personal photography. Effective dark-light image enhancement not only optimizes the visual experience, but also improves the accuracy and efficiency of subsequent image analysis and processing tasks such as object recognition and scene understanding. The development of this technology enables high-quality image output even in extreme low-light conditions, greatly expanding the application boundaries of modern imaging technology.

[0003] Under the deep learning framework, dark-light image enhancement technology mainly relies on powerful convolutional neural networks (CNNs) to automatically learn and simulate complex lighting adjustment processes. These networks are trained with a large number of image pairs under dark and normal lighting conditions to learn how to recover details and colors from dark-light images. For example, autoencoders are widely used for this task, which compress the image content through the encoder and then reconstruct the enhanced image through the decoder, effectively restoring image details and reducing noise. Generative adversarial networks (GANs) also play an important role in dark-light image enhancement. In this framework, the generator tries to produce increasingly realistic enhanced images, while the discriminator strives to distinguish the difference between the generated images and the real bright images. This adversarial process continuously improves the performance of the generator and is ultimately able to generate high-quality images. Although dark-light image enhancement technology has made significant progress, there are still some defects, such as:

[0004] Traditional CNNs often have difficulty processing global information and long-range dependencies in images (there may be important semantic associations between distant regions in an image) due to their inherent local receptive fields, which limits their effectiveness in dealing with severe low-light conditions;

[0005] Although Transformers perform well in simulating global information, they do not perform well in restoring image details (such as edges, textures, etc.). This is mainly because the self-attention mechanism in Transformers fails to effectively simulate the advantages of CNNs in dealing with local invariance and cannot effectively capture local features and details;

[0006] Although the self-attention mechanism in Transformer can process the global information of the image, its calculation mode is based on the interaction between all input features. This means that each feature is compared with all other features to calculate the similarity between them. This intensive calculation mode may amplify those relatively small similarity weights, especially when the actual correlation between features is not large. Such amplification may lead to the introduction of noise in the feature aggregation process, because even less relevant features may have an impact on the final aggregation result, thereby interfering with the accurate restoration of details in the image. In addition, the self-attention mechanism of Transformer needs to calculate the similarity between Q and K, which is an intensive calculation process with high computational complexity.

[0007] In view of this, the present invention is deeply conceived to solve the many deficiencies and inconveniences caused by the defects of the dark light image enhancement technology, and actively studies and improves the trial production to develop the present invention. Summary of the invention

[0008] The purpose of the present invention is to provide a dark-light image enhancement method based on a deep learning model of hybrid feature compensation and sparse attention, which can effectively improve the image quality in low-light environments.

[0009] In order to achieve the above object, the solution of the present invention is:

[0010] A dark light image enhancement method based on a deep learning model, which uses a deep learning algorithm to enhance image data. The model includes a hybrid feature compensation mechanism, an encoder, a raw decoder, and an sRGB decoder.

[0011] The hybrid feature compensation mechanism includes a feature extraction module and a feature compensation module. The feature extraction module extracts features from the input raw image, and the feature compensation module compensates features from the input raw image and the raw image after feature extraction. It extracts and fuses multi-scale features by integrating multiple sub-networks, and uses the self-attention mechanism to dynamically adjust the contribution of each feature representation to enhance the processing of image details and texture features.

[0012] Based on the hybrid feature compensation mechanism, the encoder is responsible for capturing and distinguishing the key feature information and noise information of the image, providing a basis for subsequent image denoising and color rendering;

[0013] The raw decoder is responsible for processing the features extracted by the encoder to achieve image denoising, and uses the noise processability characteristics of the raw image to remove noise from the image and retain important details and textures of the image;

[0014] The sRGB decoder includes a feedback feature fusion module and a decoding module. The feedback feature fusion module fuses the output of the raw decoder with the features in the output of the encoder. The decoding module uses a sparse self-attention mechanism to retain the most useful feature information and reduce the interference of noise or irrelevant information, so that the network can focus more on important information. The decoding module also uses a gated feedforward mechanism to perform color correction and color space conversion on the denoised and enhanced features of the encoder and raw decoder, and maps the raw domain image to the sRGB domain to generate an enhanced image with accurate color and high quality.

[0015] Furthermore, assume that the input of the feature extraction module is , the shape is [C1,H,W], C1 is the number of channels of the input feature, H and W are the height and width of the input feature map, first, the input feature It will go through the convolution module and the pooling module to get the features , to extract spatial features and compression features, and Perform reshaping operations to obtain features , which is convenient for subsequent processing and the reshaped features The shape is [C2],

[0016]

[0017]

[0018] in, represents the convolution module, represents the pooling module, Represents a feature reshaping operation;

[0019] Then, the features It will pass through the fully connected layer 1, the nonlinear activation layer and the fully connected layer 2 in turn to obtain the output features , the output feature shape becomes [n],

[0020]

[0021] in, , represents the fully connected layer, is a nonlinear activation function, Represents the number of subnetworks in the feature compensation mechanism.

[0022] Furthermore, the feature compensation module has two inputs, one of which is , which is the same input as the feature extraction module, with a shape of [C1,H,W]; the other is the output of the feature extraction module , the shape is [n], n is 7,

[0023] First, for the first input feature , will pass through the convolution module and nonlinear activation function in turn to obtain the feature , the shape remains unchanged,

[0024]

[0025] in, represents the convolutional layer, is a nonlinear activation function;

[0026] Then, Will be input into 7 sub-networks respectively, and get 7 outputs , , , , , , ; Subnetwork 1 is composed of 1×1 convolution modules, subnetwork 2 is composed of 3×3 convolution modules, subnetwork 3 is composed of 5×5 convolution modules, subnetwork 4 is composed of 3×3 dilated convolution modules, subnetwork 5 is composed of 5×5 dilated convolution modules, subnetwork 6 is composed of 3×3 average pooling modules, and subnetwork 7 is composed of 5×5 average pooling modules.

[0027]

[0028]

[0029]

[0030]

[0031]

[0032]

[0033]

[0034] in, Indicates subnetwork 1~7;

[0035] Next, for the second input That is, the output of the feature extraction module will first separate its features and obtain 7 weights , the 7 weights And the corresponding 7 sub-network outputs , , , , , , Multiply them together to get 7 weighted features , , , , , , After the 7 weighted features are concatenated, they are passed through the convolution module and the nonlinear activation function in turn to obtain the output of the feature compensation module. , the shape is [C1,H,W],

[0036]

[0037]

[0038]

[0039]

[0040]

[0041]

[0042]

[0043]

[0044]

[0045] in, It represents feature separation. Indicates feature splicing processing, Indicates concatenation of features along the channel dimension, represents the convolution module, is a non-linear activation function.

[0046] Furthermore, assuming that the encoder input is , the shape is [C,H,W], the encoder has 2 outputs, the first output is the final output of the encoder, the second output Used to save the intermediate features of the encoder inner loop;

[0047] First, input features It goes through the depthwise separable convolution module, convolution module, nonlinear activation, convolution module and downsampling module in sequence, and needs to be cycled three times. The output of each cycle is the input of the next cycle. , , The input feature is the output of the convolution module, the convolution module, the nonlinearly activated convolution module, and then downsampled. The output of the first cycle is , the output of the second loop is , the output of the third loop is ,

[0048]

[0049]

[0050]

[0051]

[0052]

[0053]

[0054]

[0055] in, represents the deep convolution module, represents the convolution module, is a nonlinear activation function, represents the downsampling module, The second output of the encoder is used to save , and ;

[0056] Then, after the number of cycles is met, the feature with the shape [8C, H / 8, W / 8] After passing through the deep convolution module, convolution module, nonlinear activation and convolution module in turn, the final output is obtained, which is the first output of the encoder.

[0057] .

[0058] Furthermore, the raw decoder has two inputs, which are assumed to be , , the two inputs of the raw decoder correspond to the two outputs of the encoder, and the output of the raw decoder has 2, which are assumed to be , ;

[0059] First, invert the second input The feature sequence order is obtained The first input It will go through the deep convolution module, convolution module, nonlinear activation, convolution module, upsampling module and feature fusion module in sequence. This process is a cycle, and the output of each cycle is the input of the next cycle. , , It is the feature obtained after the input feature passes through the deep convolution module, convolution module, nonlinear activation, and convolution module. , , For the output of each loop,

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066]

[0067] in, represents the deep convolution module, represents the convolution module, is a nonlinear activation function, represents the upsampling module, represents the feature fusion module, represents the xth element in the feature sequence, The second output of the raw decoder is used to save , and ;

[0068] Then, the output after the cycle is passed through the depth convolution module, convolution module, nonlinear activation function and convolution module in turn to obtain the first output of the raw decoder. , the output For the predicted clear raw image,

[0069] .

[0070] Furthermore, the feedback fusion module of the sRGB decoder has two inputs, namely the second output of the encoder and the second output of the raw decoder , respectively defined as and ,

[0071] First, input two , The features in the feature sequence are separated, and the first input feature After feature separation, we get , the second input feature After feature separation, we get , the separated features are passed through different convolution modules to obtain features , , , , , , then , , Add the features, Add the features, and then concatenate the added features to get the features ,

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081] in, It represents feature separation. represents the convolution module, represents feature addition, Represents feature splicing;

[0082] Then, the features After the convolution module, deep convolution module and feature separation, the feature and ,

[0083]

[0084] in, It represents feature separation. represents the deep convolution module, Represents a convolutional module;

[0085] Then, nonlinear activation functions and convolution modules are used to adaptively select and fuse useful detail information and denoising priors.

[0086] .

[0087] Furthermore, the input of the decoding module is the output of the feedback feature fusion module. Assume that the input of the decoding module is First, input features It will go through the sparse self-attention mechanism, gated feedforward mechanism and upsampling module in sequence. This process is a cycle. The output of the previous cycle is used as the input of the next cycle. The default number of cycles is 3. The outputs of the three cycles are , , ,

[0088]

[0089]

[0090]

[0091] in, represents the sparse self-attention mechanism, represents the gated feedforward mechanism, represents the upsampling module;

[0092] After completing 3 cycles, the characteristics The features will first be obtained through the sparse self-attention mechanism and the gated feedforward mechanism , and then pass through the convolution module, nonlinear activation function and convolution module to obtain the feature , and finally through the pixel rearrangement module, the restored sRGB image is obtained ,

[0093]

[0094]

[0095]

[0096] in, represents the sparse self-attention mechanism, represents the gated feedforward mechanism, represents the convolution module, represents a nonlinear activation function, Indicates pixel rearrangement processing.

[0097] Furthermore, the sparse self-attention mechanism selects the largest k similarity scores between the query and the key for self-attention calculation, retaining the most useful information and reducing the interference of irrelevant features, so that feature aggregation can focus more on important information. Assume that the input of the sparse self-attention mechanism is ,

[0098] First, input features After layer normalization, the features are obtained by passing through the point convolution module and the depth convolution module in turn. ,feature After feature separation by feature separation module, we get , , ,

[0099]

[0100]

[0101] in, Representation layer normalization processing, represents the point convolution module, represents the deep convolution module, Indicates feature separation processing;

[0102] Then, the three features , , Reshape the features to get , , ,feature , , The shapes of are changed from [C, H, W] to [N, C / N, H×W],

[0103]

[0104]

[0105]

[0106] in, is the number of feature map channels, is the feature map height, is the feature map width, N is the number of heads in the multi-head self-attention module, Indicates feature reshaping;

[0107] Next, the features Perform L2 normalization and get , for the features Perform L2 normalization and transposition to obtain ,

[0108]

[0109]

[0110] in, represents L2 normalization processing, Indicates transposing the elements of the last two dimensions of the feature;

[0111] Then, the features and Perform matrix multiplication to obtain similarity scores, and select the first 1 / 2, 1 / 3, and 1 / 4 similarity scores (respectively expressed as , , ),right , , After softmax processing, they are respectively compared with the features Perform matrix multiplication to obtain , , ,

[0112]

[0113]

[0114]

[0115]

[0116]

[0117]

[0118] in, represents matrix multiplication processing, It means taking the first k similarity scores, Indicates that softmax processing is performed on the features along the last dimension;

[0119] Next, , , Perform feature weighting, the weights are , , , these three weights are learnable, and we get the features ,

[0120]

[0121] Finally, reshape the features The shape is passed through the convolution module to obtain the output of the sparse self-attention mechanism.

[0122]

[0123] in, Represents feature reshaping, Represents a convolutional module.

[0124] Furthermore, the input of the gated feedforward mechanism is the output of the sparse self-attention mechanism. Assume that the input of the gated feedforward mechanism is expressed as First, input features It will be processed by layer normalization and convolution module to obtain features ,

[0125]

[0126] in, Representation layer normalization processing, Represents a convolutional module;

[0127] Next, the features Will be processed by two dilated convolution modules to obtain features and ,feature It will go through a nonlinear activation function, introduce nonlinearity, and get the feature ,

[0128]

[0129]

[0130]

[0131] in, represents the dilated convolution module, is a nonlinear activation function;

[0132] Finally, and After feature multiplication, it passes through the convolution module to obtain the output of the gated feedforward mechanism. ,

[0133]

[0134] in, Represents a convolutional module.

[0135] Furthermore, during model training, the model input is the raw image after random flipping and normalization, and the model output has two, which are the first output of the raw decoder and the output of the sRGB decoder , the L1 loss function is used to supervise the raw domain and sRGB domain. The calculation formula of the loss function is as follows:

[0136]

[0137] in, represents the loss function, Represents the output 1 of the raw decoder, Indicates a clear raw image. represents the output of the sRGB decoder, represents a true sRGB image, represents the L1 loss function.

[0138] Furthermore, the training cycle of the model is 300 epochs (when the model is trained to 300 epochs, the model loss has converged), the batch size is 8, the initial learning rate is 0.0002, the optimizer is Adamw, the betas is [0.9, 0.999], and the optimizer adjustment strategy is Cosine annealing.

[0139] After adopting the above scheme, the dark light image enhancement method of the deep learning model of the present invention has the following advantages:

[0140] The hybrid feature compensation mechanism provides additional compensation for the features extracted by the encoder and decoder, thereby effectively improving the denoising performance and color rendering effect of the image. The hybrid feature compensation mechanism can also dynamically select the importance of different feature representations based on the input, and more accurately restore the color details in the image, which is not available in previous methods.

[0141] The present invention proposes a novel network structure - feature compensation module. Each sub-network in the feature compensation module is responsible for extracting features of different scales in the image. This structure allows the model to adaptively select and integrate features.

[0142] The present invention develops an adaptive feature selection mechanism, which uses self-attention technology to dynamically adjust the weights of different sub-networks to adapt to different image features and image denoising requirements.

[0143] The sparse attention mechanism proposed in the invention can effectively filter and retain the most useful feature information while reducing the interference of noise or irrelevant information. The sparse attention mechanism not only improves the network's ability to focus on important information, but also significantly reduces the amount of calculation, thereby improving the overall computational efficiency of the model.

[0144] The present invention also proposes a gated feedforward mechanism, which can effectively capture long-distance dependencies by introducing nonlinear processing, further enhancing the color rendering effect of the image. This mechanism provides a new way to enhance the functionality and efficiency of image processing networks.

[0145] The present invention can replace the traditional image signal processing (ISP) process, and is particularly suitable for images taken under extreme lighting or adverse weather conditions, and can significantly improve the processing effect and quality of images under these conditions.

[0146] Compared with the existing technical solutions, the present invention uses a hybrid feature compensation mechanism to enhance the model's ability to extract and restore image details and texture features. This mechanism allows the model to adaptively adjust the importance of hybrid features to better render images. The present invention selects the largest k similarity scores between queries and keys through a sparse attention mechanism for self-attention calculation, retains the most useful information in the image and reduces the interference of irrelevant features, so that feature aggregation is more focused on important information. In addition, the present invention introduces dilated convolutions and gating units in the feedforward mechanism. The former increases the receptive field, allowing the network to capture dependencies between pixels at a longer distance, and the latter enhances the network's nonlinear expression capabilities, further improving the rendering effect of image colors. More importantly, compared with traditional large networks such as Transformer, the present invention is based on sparse attention and feedforward mechanisms, has low requirements for computing resources, and can run in resource-constrained devices or scenarios.

[0147] The dark-light image enhancement method proposed in the present invention can replace the traditional ISP to solve the problem of poor processing effect of images taken under extreme lighting or weather conditions.

[0148] The hybrid feature compensation mechanism proposed in the present invention provides additional feature compensation for the features extracted by the encoder and decoder, which helps to improve the effects of image denoising and color rendering. In addition, the hybrid feature compensation mechanism can dynamically select the importance of different feature representations according to the input, and better restore the color detail information of the image. The gated feedforward mechanism proposed in the present invention can introduce nonlinearity while capturing long-distance dependencies, further improving the color rendering effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0149] Figure 1 A simplified flow chart of the present invention.

[0150] Figure 2 It is the overall framework diagram of the present invention.

[0151] Figure 3 Schematic diagram of the overall framework of the hybrid feature compensation mechanism of the present invention.

[0152] Figure 4 Schematic diagram of the network structure of the feature extraction module of the present invention.

[0153] Figure 5 Schematic diagram of the network structure of the characteristic compensation module of the present invention.

[0154] Figure 6 It is a schematic diagram of the flow chart of the encoder of the present invention.

[0155] Figure 7 It is a schematic diagram of the flow chart of the raw decoder of the present invention.

[0156] Figure 8 It is a flow chart of the feedback feature fusion module of the present invention.

[0157] Fig. 9 It is a flowchart diagram of the decoding module of the present invention.

[0158] Fig.10 Schematic diagram of the process of the sparse attention mechanism of the present invention.

[0159] Fig.11 It is a flow chart of the gated feedforward mechanism of the present invention. DETAILED DESCRIPTION

[0160] In order to further explain the technical solution of the present invention, the present invention is described in detail below through specific embodiments.

[0161] The present invention discloses a dark-light image enhancement model based on hybrid feature compensation and sparse attention, which uses a deep learning algorithm to enhance image data, so that clearer and brighter images can be obtained when shooting in dark light or at night, thereby improving the image quality in low-light environments.

[0162] like Figure 1 and Figure 2As shown, the present invention discloses a dark light enhancement method based on a deep learning model of hybrid feature compensation and sparse attention. The overall idea is: the raw image (unprocessed or compressed image data format) as input will first pass through a hybrid feature compensation mechanism to enhance the model's processing of image details and texture features; next, the encoder is responsible for capturing the key feature information and noise information of the image, providing a basis for subsequent image denoising and color rendering; then, the raw decoder is responsible for processing the features extracted by the encoder to achieve image denoising. It uses the noise processable characteristics of the raw image to remove noise in the image and retain important details and textures of the image; finally, the sRGB decoder with sparse attention and gated feedforward mechanism is responsible for color space conversion and color enhancement, ensuring that the final image is visually closer to the lighting conditions of the real world so that it can be presented on various display devices. By introducing a hybrid feature compensation mechanism, a sparse attention mechanism and a gated feedforward mechanism, the dark light image enhancement method of the present invention can effectively improve the image quality in a low-light environment.

[0163] The dark light enhancement method based on hybrid feature compensation and sparse attention of the present invention includes a hybrid feature compensation mechanism, an encoder, a raw decoder and an sRGB decoder.

[0164] like Figure 3 As shown in the figure, the hybrid feature compensation mechanism consists of a feature extraction module and a feature compensation module. Based on the feature extraction module, the feature compensation module extracts and fuses multi-scale features by integrating multiple sub-networks, and dynamically adjusts the contribution of each feature representation using the self-attention mechanism, thereby enhancing the image enhancement algorithm's ability to extract and restore details, ultimately improving the overall image quality and significantly enhancing the algorithm's adaptability and robustness.

[0165] like Figure 4 As shown, assuming that the input of the feature extraction module is , the shape is [C1,H,W], C1 is the number of channels of the input feature, H and W are the height and width of the input feature map respectively.

[0166] First, input features It will go through the convolution module and the pooling module to get the features , to extract spatial features and compression features, and Perform reshaping operations to obtain features , which is convenient for subsequent processing and the reshaped features The shape is [C2],

[0167]

[0168]

[0169] in, represents the convolution module, represents the pooling module, Represents a feature reshaping operation.

[0170] Then, the features It will pass through the fully connected layer 1, the nonlinear activation layer and the fully connected layer 2 in turn to obtain the output features , the output feature shape becomes [n],

[0171]

[0172] in, , represents the fully connected layer, is a nonlinear activation function, Indicates the number of subnetworks in the feature compensation mechanism (default is 7).

[0173] like Figure 5 As shown, the feature compensation module has two inputs, one is , which is the same input as the feature extraction module, with a shape of [C1,H,W]; the other is the output of the feature extraction module , shape is [n], where n defaults to 7.

[0174] First, for the first input feature , will pass through the convolution module and nonlinear activation function in turn to obtain the feature , the shape remains unchanged,

[0175]

[0176] in, represents the convolutional layer, is a non-linear activation function.

[0177] Then, Will be input into 7 sub-networks respectively, and get 7 outputs , , , , , , Subnetwork 1 consists of 1×1 convolution modules, subnetwork 2 consists of 3×3 convolution modules, subnetwork 3 consists of 5×5 convolution modules, subnetwork 4 consists of 3×3 dilated convolution modules, subnetwork 5 consists of 5×5 dilated convolution modules, subnetwork 6 consists of 3×3 average pooling modules, and subnetwork 7 consists of 5×5 average pooling modules.

[0178]

[0179]

[0180]

[0181]

[0182]

[0183]

[0184]

[0185] in, Indicates subnetwork 1~7.

[0186] Next, for the second input (Output of the feature extraction module), the features are separated first and 7 weights are obtained . Set the 7 weights And the corresponding 7 sub-network outputs , , , , , , Multiply them together to get 7 weighted features , , , , , , After the 7 weighted features are concatenated, they are passed through the convolution module and the nonlinear activation function in turn to obtain the output of the feature compensation module. , the shape is [C1,H,W],

[0187]

[0188]

[0189]

[0190]

[0191]

[0192]

[0193]

[0194]

[0195]

[0196] in, It represents feature separation. Indicates feature splicing processing, Indicates concatenation of features along the channel dimension, represents the convolution module, is a non-linear activation function.

[0197] The principle of the encoder is to capture and distinguish feature signals (such as texture, edge, and color information) and noise information based on a hybrid feature compensation mechanism to ensure that the denoised image does not lose key details and provide rich feature representation for subsequent image denoising (raw decoder) and color rendering restoration (sRGB decoder).

[0198] like Figure 6 As shown, assuming that the input of the encoder is , with a shape of [C,H,W]. The encoder has 2 outputs, the first output is the final output of the encoder, the second output Used to save intermediate features of the encoder inner loop.

[0199] First, input features It will go through the depthwise separable convolution module, convolution module, nonlinear activation, convolution module and downsampling module in sequence (this process is one cycle), and needs to be cycled three times. The output of each cycle is the input of the next cycle. , , The input feature is the output of the convolution module, the convolution module, the nonlinearly activated convolution module, and then downsampled. The output of the first cycle is , the output of the second loop is , the output of the third loop is ,

[0200]

[0201]

[0202]

[0203]

[0204]

[0205]

[0206]

[0207] in, represents the deep convolution module, represents the convolution module, is a nonlinear activation function, Represents a downsampling module. The second output of the encoder is used to save , and .

[0208] Next, after satisfying the number of loops required, the feature with the shape [8C, H / 8, W / 8] (Downsampled 3 times) will pass through the deep convolution module, convolution module, nonlinear activation and convolution module in turn to get the final output (that is, the first output of the encoder mentioned above).

[0209] .

[0210] The principle of the raw decoder is to use the deep features and noise information provided by the encoder to accurately remove random noise in the image while retaining the texture details and structural information of the image to generate a denoised raw image.

[0211] like Figure 7 As shown, the raw decoder has two inputs, which are assumed to be , , that is, the 2 outputs of the encoder. The output of the raw decoder is 2, which are assumed to be , .

[0212] First, invert the second input The feature sequence order is obtained The first input It will go through the deep convolution module, convolution module, nonlinear activation, convolution module, upsampling module and feature fusion module in sequence. This process is a cycle, and the output of each cycle is the input of the next cycle. , , It is the feature obtained after the input feature passes through the deep convolution module, convolution module, nonlinear activation, and convolution module. , , For the output of each loop,

[0213]

[0214]

[0215]

[0216]

[0217]

[0218]

[0219]

[0220] in, represents the deep convolution module, represents the convolution module, is a nonlinear activation function, represents the upsampling module, represents the feature fusion module, Represents the xth element in the feature sequence. The second output of the raw decoder is used to save , and .

[0221] Then, the output after the cycle is passed through the depth convolution module, convolution module, nonlinear activation function and convolution module in turn to obtain the first output of the raw decoder. , the output For the predicted clear raw image,

[0222] .

[0223] The sRGB decoder has sparse attention and gated feedforward mechanism. The sRGB decoder is divided into a feedback feature fusion module and a decoding module. The feedback feature fusion module converts the second output of the raw decoder into The second output of the encoder The features in the encoder are fused to enrich the features of the encoder and enhance the ability to capture image details, while improving the separation of signals and noise and reducing the propagation of erroneous information. The decoding module uses a sparse self-attention mechanism to retain the most useful feature information and reduce the interference of noise or irrelevant information, so that the network can focus more on important information. In addition, the decoding module uses a gated feedforward mechanism to perform color correction and color space conversion on the denoised and enhanced features of the encoder and raw decoder, and maps the raw domain image to the sRGB domain to generate an enhanced image with accurate color and high quality.

[0224] like Figure 8 As shown, the feedback fusion module has two inputs, namely the second output of the encoder and the second output of the raw decoder , respectively defined as and .

[0225] First, input two , The features in the feature sequence are separated, and the first input feature After feature separation, we get , the second input feature After feature separation, we get , the separated features are passed through different convolution modules to obtain features , , , , , , then , , Add the features, Then, the added features are concatenated to obtain the feature ,

[0226]

[0227]

[0228]

[0229]

[0230]

[0231]

[0232]

[0233]

[0234]

[0235] in, It represents feature separation. represents the convolution module, represents feature addition, Represents feature concatenation.

[0236] Then, the features After the convolution module, deep convolution module and feature separation, the feature and ,

[0237]

[0238] in, It represents feature separation. represents the deep convolution module, Represents a convolutional module.

[0239] Finally, nonlinear activation functions and convolution modules are used to adaptively select and fuse useful detail information and denoising priors.

[0240] .

[0241] Overall, the feedback feature fusion module combines the features of the encoder and the raw decoder, and uses a gating mechanism to dynamically adjust the fusion of features in the spatial and channel dimensions, ensuring that important image details are preserved and noise interference is suppressed during denoising and color restoration.

[0242] like Fig. 9 As shown in , the input of the decoding module is the output of the feedback feature fusion module. Assume that the input of the decoding module is First, input feature It will go through the sparse self-attention mechanism, gated feedforward mechanism and upsampling module in sequence. This process is a cycle. The output of the previous cycle is used as the input of the next cycle. The default number of cycles is 3. The outputs of the three cycles are , , .

[0243]

[0244]

[0245]

[0246] in, represents the sparse self-attention mechanism, represents the gated feedforward mechanism, Represents an upsampling module.

[0247] After completing 3 cycles, the characteristics The features will first be obtained through the sparse self-attention mechanism and the gated feedforward mechanism , and then pass through the convolution module, nonlinear activation function and convolution module to obtain the feature , and finally through the pixel rearrangement module, the restored sRGB image is obtained ,

[0248]

[0249]

[0250]

[0251] in, represents the sparse self-attention mechanism, represents the gated feedforward mechanism, represents the convolution module, represents a nonlinear activation function, Indicates pixel rearrangement processing.

[0252] The sparse self-attention mechanism selects the largest k similarity scores between the query and the key for self-attention calculation, retaining the most useful information and reducing the interference of irrelevant features, so that feature aggregation can focus more on important information.

[0253] like Fig.10 As shown, assuming that the input of the sparse self-attention mechanism is First, input feature After layer normalization, the features are obtained by passing through the point convolution module and the depth convolution module in turn. ,feature After feature separation by feature separation module, we get , , ,

[0254]

[0255]

[0256] in, Representation layer normalization processing, represents the point convolution module, represents the deep convolution module, Represents feature separation processing.

[0257] Then, the three features , , Reshape the features to get , , ,feature , , The shapes of are changed from [C, H, W] to [N, C / N, H×W],

[0258]

[0259]

[0260]

[0261] in, is the number of feature map channels, is the feature map height, is the feature map width, N is the number of heads in the multi-head self-attention module, Indicates feature reshaping.

[0262] Next, the features Perform L2 normalization and get , for the features Perform L2 normalization and transposition to obtain ,

[0263]

[0264]

[0265] in, represents L2 normalization processing, Indicates transposing the elements of the last two dimensions of the feature.

[0266] Then, the features and Perform matrix multiplication to obtain similarity scores, and select the first 1 / 2, 1 / 3, and 1 / 4 similarity scores (respectively expressed as , , ),right , , After softmax processing, they are respectively compared with the features Perform matrix multiplication to obtain , , ,

[0267]

[0268]

[0269]

[0270]

[0271]

[0272]

[0273] in, represents matrix multiplication processing, It means taking the first k similarity scores, Indicates that softmax processing is performed on the features along the last dimension.

[0274] Next, , , Perform feature weighting, the weights are , , , these three weights are learnable, and we get the features ,

[0275]

[0276] Finally, reshape the features The shape is passed through the convolution module to obtain the output of the sparse self-attention mechanism.

[0277]

[0278] in, Represents feature reshaping, Represents a convolutional module.

[0279] The function of the gated feedforward mechanism is to achieve adaptive selection and adjustment of features. By introducing nonlinear gating units to control the flow of information, the network is allowed to more flexibly emphasize or suppress certain features when processing data, thereby enhancing the model's ability to capture key information, improving processing effects, and enabling the network to better learn and simulate complex data conversion processes. In image processing, the gated feedforward mechanism helps to distinguish and retain important visual features while suppressing noise or irrelevant information, optimizing image quality and enhancing detail performance.

[0280] like Fig.11 As shown, the input of the gated feedforward mechanism is the output of the sparse self-attention mechanism. Assume that the input of the gated feedforward mechanism is expressed as First, input feature It will be processed by layer normalization and convolution module to obtain features ,

[0281]

[0282] in, Representation layer normalization processing, Represents a convolutional module.

[0283] Next, the features Will be processed by two dilated convolution modules to obtain features and .feature It will go through a nonlinear activation function, introduce nonlinearity, and get the feature ,

[0284]

[0285]

[0286]

[0287] in, represents the dilated convolution module, is a non-linear activation function.

[0288] Finally, and After feature multiplication, it passes through the convolution module to obtain the output of the gated feedforward mechanism. ,

[0289]

[0290] in, Represents a convolutional module.

[0291] During model training, the model input is the raw image after random flipping and normalization. The model output has two, namely, the output 1 of the raw decoder and the output of the sRGB decoder. The present invention uses the L1 loss function to supervise the raw domain and the sRGB domain. The calculation formula of the loss function is as follows:

[0292]

[0293] in, represents the loss function, Represents the output 1 of the raw decoder, Indicates a clear raw image. represents the output of the sRGB decoder, represents a true sRGB image, represents the L1 loss function.

[0294] The training cycle of the model is 300 epochs (when the model is trained to 300 epochs, the model loss has converged), the batch size is 8, the initial learning rate is 0.0002, the optimizer is Adamw, the betas is [0.9, 0.999], and the optimizer adjustment strategy is Cosine annealing.

[0295] The present invention provides additional compensation for the features extracted by the encoder and decoder through a hybrid feature compensation mechanism, thereby effectively improving the denoising performance and color rendering effect of the image. The hybrid feature compensation mechanism can also dynamically select the importance of different feature representations according to the input, and more accurately restore the color details in the image, which is not available in previous methods.

[0296] The present invention proposes a novel network structure, a feature compensation module. Each sub-network in the feature compensation module is responsible for extracting features of different scales in the image. This structure allows the model to adaptively select and integrate features.

[0297] The present invention develops an adaptive feature selection mechanism, which uses self-attention technology to dynamically adjust the weights of different sub-networks to adapt to different image features and image denoising requirements.

[0298] The sparse attention mechanism proposed in the invention can effectively filter and retain the most useful feature information while reducing the interference of noise or irrelevant information. This mechanism not only improves the network's ability to focus on important information, but also significantly reduces the amount of calculation, thereby improving the overall calculation efficiency of the model.

[0299] The present invention also proposes a gated feedforward mechanism, which can effectively capture long-distance dependencies by introducing nonlinear processing, further enhancing the color rendering effect of the image. This mechanism provides a new way to enhance the functionality and efficiency of image processing networks.

[0300] The present invention proposes a novel low-light image enhancement technology, which can replace the traditional image signal processing (ISP) process. This method is particularly suitable for images taken under extreme lighting or severe weather conditions, and can significantly improve the processing effect and quality of images in these situations.

[0301] In a dark environment, images are often accompanied by a high noise level. The present invention adopts deep learning-based technology to identify and suppress image noise, so as to effectively reduce noise and improve image quality without losing image details. This method not only improves the overall aesthetics of the image, but also maintains the naturalness and detail level of the image.

[0302] The above embodiments and drawings do not limit the product form and style of the present invention. Any appropriate changes or modifications made by ordinary technicians in the relevant technical field should be deemed to be within the patent scope of the present invention.

Claims

1. A dark light image enhancement method based on a deep learning model, which uses a deep learning algorithm to enhance image data, characterized in that: The model includes a hybrid feature compensation mechanism, an encoder, a raw decoder, and an sRGB decoder; The hybrid feature compensation mechanism includes a feature extraction module and a feature compensation module. The feature extraction module extracts features from the input raw image, and the feature compensation module compensates features from the input raw image and the raw image after feature extraction. It extracts and fuses multi-scale features by integrating multiple sub-networks, and uses the self-attention mechanism to dynamically adjust the contribution of each feature representation to enhance the processing of image details and texture features. The encoder is used to capture and distinguish the key feature information and noise information of the image based on the hybrid feature compensation mechanism; The raw decoder is used to process the features extracted by the encoder to achieve image denoising, and to use the noise processability of the raw image to remove noise from the image and retain important details and textures of the image; The sRGB decoder includes a feedback feature fusion module and a decoding module. The feedback feature fusion module fuses the features in the output of the raw decoder with the output of the encoder; The decoding module uses a sparse self-attention mechanism to retain the most useful feature information and reduce the interference of noise or irrelevant information. The decoding module also uses a gated feedforward mechanism to perform color correction and color space conversion on the features that have been denoised and enhanced by the encoder and raw decoder, and maps the raw domain image to the sRGB domain to generate an enhanced image. in: The input of the feature extraction module is , the shape is [C1,H,W], C1 is the number of channels of the input feature, H and W are the height and width of the input feature map, first, the input feature After the convolution module and the pooling module, the features are obtained , to extract spatial features and compression features, and Perform feature reshaping operation to obtain features , the reshaped features The shape is [C2], in, represents the convolution module, represents the pooling module, Represents a feature reshaping operation; Then, the features After passing through the fully connected layer 1, the nonlinear activation layer and the fully connected layer 2 in sequence, the output features are obtained , the output feature shape becomes [n], in, , represents the fully connected layer, is a nonlinear activation function, Indicates the number of sub-networks in the hybrid feature compensation mechanism; The feature compensation module has two inputs, one is , which is the same input as the feature extraction module, with a shape of [C1,H,W]; the other is the output of the feature extraction module , the shape is [n], n is 7, First, for the first input , passes through the convolution module and nonlinear activation layer in turn, and obtains the feature , the shape remains unchanged, in, represents the convolution module, is a nonlinear activation function; Then, the features Input into 7 sub-networks respectively and get 7 outputs , , , , , , ; Subnetwork 1 is composed of 1×1 convolution modules, subnetwork 2 is composed of 3×3 convolution modules, subnetwork 3 is composed of 5×5 convolution modules, subnetwork 4 is composed of 3×3 dilated convolution modules, subnetwork 5 is composed of 5×5 dilated convolution modules, subnetwork 6 is composed of 3×3 average pooling modules, and subnetwork 7 is composed of 5×5 average pooling modules. in, Indicates subnetwork 1~7; Next, for the second input That is, the output of the feature extraction module, first separate its features and get 7 weights , the 7 weights And the corresponding 7 sub-network outputs , , , , , , Multiply them together to get 7 weighted features , , , , , , After the 7 weighted features are concatenated, they are passed through the convolution module and the nonlinear activation layer in turn to obtain the output of the feature compensation module. , the shape is [C1,H,W], in, It represents feature separation. Indicates feature splicing processing, Indicates concatenation of features along the channel dimension, represents the convolution module, is a nonlinear activation function; The feedback fusion module of the sRGB decoder has two inputs, namely the second output of the encoder and the second output of the raw decoder , respectively defined as and , First, input two , The features in the feature sequence are separated, and the first input feature After feature separation, we get , the second input feature After feature separation, we get , the separated features are passed through different convolution modules to obtain features , , , , , , then , , Add the features, Add the features, and then concatenate the added features to get the features , in, It represents feature separation. represents the convolution module, represents feature addition, Indicates feature concatenation processing, dim=1 means concatenating features along the channel dimension; Then, the features After passing through the convolution module, the depth convolution module and the feature separation module in turn, the feature and , in, It represents feature separation. represents the deep convolution module, Represents a convolutional module; Then, the nonlinear activation function and convolution module are used to adaptively select and fuse useful detail information and denoising priors to obtain the feature , ; in, is a nonlinear activation function, Represents a convolutional module; The input of the decoding module is the output of the feedback feature fusion module. First, input features It goes through the sparse self-attention mechanism, gated feedforward mechanism and upsampling module in sequence. This process is a cycle. The output of the previous cycle is used as the input of the next cycle. The default number of cycles is 3. The outputs of the three cycles are , , , in, represents the sparse self-attention mechanism, represents the gated feedforward mechanism, represents the upsampling module; After completing 3 cycles, the characteristics First, the features are obtained through the sparse self-attention mechanism and the gated feedforward mechanism. , and then pass through the convolution module, nonlinear activation layer and convolution module to obtain the feature , and finally through the pixel rearrangement module, the restored sRGB image is obtained , in, represents the sparse self-attention mechanism, represents the gated feedforward mechanism, represents the convolution module, represents a nonlinear activation function, Indicates pixel rearrangement processing; The sparse self-attention mechanism selects the largest k similarity scores between the query and the key for self-attention calculation, retaining the most useful information and reducing the interference of irrelevant features. The input of the sparse self-attention mechanism is , First, input features After layer normalization, the features are obtained by passing through the point convolution module and the depth convolution module in turn. ,feature After feature separation by feature separation module, we get , , , in, Representation layer normalization processing, represents the point convolution module, represents the deep convolution module, Indicates feature separation processing; Then, the three features , , Reshape the features to get , , ,feature , , The shapes of are changed from [C, H, W] to [N, C / N, H×W], in, is the number of feature map channels, is the feature map height, is the feature map width, N is the number of heads in the multi-head self-attention module, Indicates feature reshaping; Next, the features Perform L2 normalization and get , for the features Perform L2 normalization and transposition to obtain , in, represents L2 normalization processing, Indicates transposing the elements of the last two dimensions of the feature; Then, the features and Perform matrix multiplication to obtain the similarity score, and select the first 1 / 2, 1 / 3, and 1 / 4 similarity scores, respectively, and express them as , , ,right , , After softmax processing, they are respectively compared with the features Perform matrix multiplication to obtain , , , in, represents matrix multiplication processing, It means taking the first k similarity scores, Indicates that softmax processing is performed on the features along the last dimension; Next, , , Perform feature weighting, the weights are , , , these three weights are learnable, and the features are obtained , Finally, reshape the features The shape is passed through the convolution module to obtain the output of the sparse self-attention mechanism. in, Represents feature reshaping, Represents a convolutional module; The input of the gated feedforward mechanism is the output of the sparse self-attention mechanism. The input of the gated feedforward mechanism is expressed as First, input features After layer normalization and convolution module processing, the features are obtained , in, Representation layer normalization processing, Represents a convolutional module; Next, the features After being processed by two dilated convolution modules respectively, the features are obtained and ,feature After the nonlinear activation layer, nonlinearity is introduced to obtain the feature , in, represents the dilated convolution module, is a nonlinear activation function; Finally, and After feature multiplication, it passes through the convolution module to obtain the output of the gated feedforward mechanism. , in, Represents a convolutional module.

2. The dark light image enhancement method of the deep learning model according to claim 1, characterized in that: The input of the encoder is , the shape is [C,H,W], H and W are the height and width of the input feature map, respectively. The encoder has 2 outputs, the first output is the final output of the encoder, the second output Used to save the intermediate features of the encoder inner loop; First, input features It goes through the deep convolution module, convolution module, nonlinear activation layer, convolution module and downsampling module in sequence, and cycles 3 times. The output of each cycle is the input of the next cycle. , , The input feature passes through the deep convolution module, convolution module, nonlinear activation layer and the output of the convolution module, and then downsamples. The output of the first cycle is , the output of the second loop is , the output of the third loop is , in, represents the deep convolution module, represents the convolution module, is a nonlinear activation function, represents the downsampling module, The second output of the encoder is used to save , and ; Then, after the number of cycles is met, the feature with the shape [8C, H / 8, W / 8] Then it passes through the deep convolution module, convolution module, nonlinear activation layer and convolution module in turn to get the final output, which is the first output of the encoder. 。 3. The dark light image enhancement method of the deep learning model as claimed in claim 2, characterized in that: The raw decoder has two inputs, namely , , the two inputs of the raw decoder correspond to the two outputs of the encoder, and the outputs of the raw decoder are 2, respectively , ; First, invert the second input The feature sequence order is obtained , the first input It passes through the deep convolution module, convolution module, nonlinear activation layer, convolution module, upsampling module and feature fusion module in sequence. This process is a cycle, and the output of each cycle is the input of the next cycle. , , It is the feature obtained after the input feature passes through the deep convolution module, convolution module, nonlinear activation layer, and convolution module. , , For the output of each loop, in, represents the deep convolution module, represents the convolution module, is a nonlinear activation layer, represents the upsampling module, represents the feature fusion module, represents the xth element in the feature sequence, The second output of the raw decoder is used to save , and ; Then, the output after the cycle is passed through the depth convolution module, convolution module, nonlinear activation layer and convolution module in turn to obtain the first output of the raw decoder. , the output For the predicted clear raw image, 。 4. The dark light image enhancement method of the deep learning model as claimed in claim 1, characterized in that: During model training, the model input is the raw image after random flipping and normalization. The model output has two, which are the first output of the raw decoder and the output of the sRGB decoder , the L1 loss function is used to supervise the raw domain and sRGB domain. The calculation formula of the loss function is as follows: in, represents the loss function, Represents the output 1 of the raw decoder, Indicates a clear raw image. represents the output of the sRGB decoder, represents a true sRGB image, represents the L1 loss function.

5. The dark light image enhancement method of the deep learning model as claimed in claim 1, characterized in that: The training cycle of the model is 300 epochs, the batch size is 8, the initial learning rate is 0.0002, the optimizer is Adamw, the betas are [0.9, 0.999], and the optimizer adjustment strategy is Cosine annealing.

Citation Information

Cited By

  • A sparse expert network-based tugboat low-illumination image enhancement method

    CN122905907A