Low-light image enhancement method based on multi-scale global and local combination

Abstract

The invention provides a low-light image enhancement method based on multi-scale global and local combination, and the method comprises the steps: building a multi-scale prior extraction module, taking a low-light image as input, carrying out the pre-training, and extracting the prior features of the image; by introducing multi-scale prior extraction and combining with local enhancement, complex global and local dependency relationships are effectively captured, and the image enhancement quality is further improved; flexible adaptation to different inputs is realized by introducing an adaptive kernel selection module and operating dynamic selection features by using spatial variation; according to the method, a dynamic pseudo-label generation framework is also introduced, and the generalization ability of the network is improved through pseudo-label generation, dynamic confidence evaluation and knowledge distillation. The method also shows excellent performance in practical application scenes such as low-light object detection, and has a wide application prospect.

Description

Technical Field

[0001] The present invention relates to the field of low-light image enhancement, and in particular to a low-light image enhancement method based on multi-scale global and local integration. Background Art

[0002] Low-light image enhancement is a key technology in the field of computer vision, which aims to improve the quality of visual data captured under low-light conditions. Due to the dual influence of sensor physical limitations and the complexity of ambient lighting conditions, low-light images generally suffer from problems such as reduced global contrast, loss of local details, noise interference, and color distortion, which directly affect human visual perception and the performance of downstream tasks. Traditional enhancement methods are mainly based on global mapping strategies such as histogram equalization and gamma correction, but it is difficult to effectively deal with local degradation phenomena, resulting in problems such as blurred details and artifacts in the enhanced results.

[0003] In recent years, low-light enhancement methods based on deep learning have made significant progress. Among them, the algorithm guided by Retinex theory shows advantages in improving global brightness by decomposing the image into reflective and illumination components for independent optimization. However, the traditional Retinex model lacks explicit modeling capabilities for noise and local degradation, and relies on complex multi-branch network structures and physical constraints, resulting in a significant increase in model complexity and computational cost. Convolutional neural networks (CNNs) excel in detail recovery through their ability to aggregate local features, but their fixed receptive field limits their ability to model global degradation. Although the Transformer architecture achieves long-range dependency modeling through a self-attention mechanism, its quadratic computational complexity limits its application in resource-constrained scenarios.

[0004] As an emerging long sequence modeling tool, the state space model (SSM) has emerged in visual tasks with its linear computational complexity and global information capture capabilities. However, existing visual SSM models mostly follow the unidirectional scanning strategy and fail to effectively maintain the spatial locality of two-dimensional images, resulting in insufficient extraction of local degradation features. For example, although the typical visual SSM model can establish global dependencies, it lacks targeted design for local optimization tasks such as noise suppression and edge preservation. In addition, existing enhancement methods generally face the problem of rigid physical prior guidance mechanisms, making it difficult to achieve dynamic adaptive fusion of illumination components and reflection components.

[0005] In summary, although existing low-light image enhancement methods have made some progress, they still have shortcomings in dealing with global and local degradation problems at the same time. Summary of the invention

[0006] In view of the limitations of existing convolutional neural networks and Transformers in processing low-light images and the network overfitting problem often encountered in small sample data sets, the present invention proposes a low-light image enhancement method based on multi-scale global and local, which makes full use of global and local information to enhance the understanding of the model. At the same time, a student-teacher network is introduced for the second stage of fine-tuning to enhance the generalization ability of the network in the face of real scenes and improve the effect of image enhancement, so as to solve the problems raised in the above background technology. The technical solutions provided by the present invention are as follows: A low-light image enhancement method based on multi-scale global and local integration includes the following steps: Step 1, constructing a multi-scene low-light image training dataset consisting of multiple low-light images, wherein the low-light images are used to train an image enhancement network; Step 2: The image enhancement network is a U-Net-like network, which consists of a multi-scale prior extraction module, multiple local enhancement modules, and an adaptive kernel selection module. The low-light image is input into the multi-scale prior extraction module to extract the prior features of the image. F p ; The low-light image is convolved to obtain the low-light image features F l ; Step 3: Low-light image features F l and prior features F p Input local enhancement module, output local enhancement features F out1 ; Step 4: Enhance the local features F out1 and prior features F p Input adaptive kernel selection module, output enhanced features F out2 ; Step 5: Enhance the features F out2 and prior features F p Repeating steps 3 and 4 in the image enhancement network through downsampling and upsampling to train the image enhancement network; In step 6, real low-light images in different scenes are taken as input and fine-tuned using the second-stage student-teacher network to obtain an image enhancement network with the ability to generalize to real scenes.

[0007] Preferably, in step 1, the specific process of constructing a multi-scene low-light image training dataset is: selecting low-light images of multiple scenes from existing public low-light datasets, including a LOL dataset consisting of pairs of low-light images and their corresponding normal-light images, a DPED dataset containing real photos taken from three different mobile phones and a high-end SLR camera, and a Dark Face dataset focusing on face detection under low-light conditions; collecting a real low-light dataset and eliminating some blurred images, and merging the public low-light dataset images and the real low-light images to obtain a dataset for network training.

[0008] Preferably, the multi-scale prior extraction module consists of two parts. The first part is a multi-scale convolution part, which is mainly composed of three branch networks and a feature fusion module. The input low-light image is extracted with different receptive fields through the three branch networks, and the extracted features are integrated in the feature fusion module to obtain the processing result of the first part; the second part inputs the processing result from the first part, processes it in parallel in the channel and space branches, learns the importance weights on the channel and space branches, and obtains the important feature information on the channel and space branches through weighted operations; integrates the important feature information on the channel and space branches to obtain the prior feature F p .

[0009] Preferably, in the multi-scale convolution part, the first branch uses a 1x1 convolution kernel, does not change the spatial scale, and directly extracts features; the second branch uses a 3x3 convolution kernel with a void ratio of 12 to expand the receptive field to capture a wider range of contextual information; the third branch uses global average pooling to extract global contextual features to enhance the model's ability to understand the overall layout.

[0010] Preferably, on the channel branch, a global pooling operation is first performed to obtain the global information of each channel, and then two fully connected layers are passed, and the two fully connected layers use ReLU and Sigmoid as activation functions respectively to learn the importance weights on the channel branch; on the spatial branch, the input features are subjected to 1×1 convolution and Sigmoid activation function operations to learn the importance weights on the spatial branch.

[0011] Preferably, the local enhancement module consists of two branches, the low-light image features in the first branch F l The features are obtained by sequentially passing through the Layer Norm layer, Linear layer, DWConv layer, 2DSSM module and Layer Norm layer. F 1 ; Prior features F pThe features are obtained by processing the Conv layer, Linear layer and SiLU layer in sequence. F 2 ; The feature F 1 and Features F 2 Perform point multiplication and obtain local enhanced features through Linear layer processing F out1 .

[0012] Preferably, the 2DSSM module enhances the original state space model SSM by enhancing the local deviation, and the module is expressed as:

[0013] The one-dimensional sequence x t Through the hidden state h t Projected into a new one-dimensional sequence y t , A, B, C, and D represent the state, input projection, output projection, and feedthrough parameters, respectively, and E is the feature enhancement information.

[0014] Preferably, the adaptive kernel selection module consists of two branches, wherein the first branch includes Conv, SK-1, SK-2 and the kernel selection module, and the second branch includes DWConv, Sigmoid and Chunk layers; the module is based on the prior features. F p As input, it is input into the kernel selection module after being processed by the second branch, together with the local enhanced features F out1 After being processed by the first branch, the final output features are obtained through the operations of DWConv, GELU and Conv. F out2 .

[0015] Preferably, the student-teacher network consists of a student network, a teacher network, a pseudo-label quality score and a high-quality pseudo-label pool. The student network and the teacher network have the same network structure. The teacher network uses non-enhanced real low-light images as input, and the student network uses strongly enhanced images as input. The output of the teacher network is a label that is discriminated by the pseudo-label quality score to form a high-quality pseudo-label pool for high-quality training of the student network.

[0016] Preferably, the pseudo-label quality score is a hybrid scoring mechanism that integrates the physical model and the domain adaptation discriminator, as follows: Assuming that the low-light noise obeys the Poisson-Gaussian mixture model, its observation noise variance is decomposed into: ; in y For an ideal noise-free signal, αis the Poisson noise intensity, β is the standard deviation of Gaussian noise; For each pixel block p, calculate its actual noise variance: ; The residual confidence score is defined by minimizing the difference between the actual variance and the model predicted variance:

[0017] in is the mean of the pseudo-labels within the block, λ To control sensitivity; Train the binary discriminator D to distinguish between real normal light images and low light images, and use the pseudo labels output by the teacher network I enhanced Input the discriminator D and calculate its naturalness score:

[0018] The final confidence score of pixel (i, j) is weighted fused:

[0019] in, γ is a balance factor with a value of 0-1; Setting Thresholds τ , generate a binary mask M , filter high confidence regions for student network training:

[0020] The pseudo-label pool is dynamically updated according to the pseudo-label scores to ensure that the labels used for student network training are high-quality labels.

[0021] Compared with the prior art, the beneficial effects achieved by the present invention are: The present invention adopts a multi-scale prior module. By introducing multi-scale dilated convolution, the receptive field can be significantly expanded without losing resolution, so that the network can extract features at different spatial scales; by introducing channel attention, the importance of each channel is evaluated and adjusted accordingly, useful feature channels can be strengthened and unimportant channels can be suppressed, so that the network can focus on information of interest, thereby improving the quality of feature representation; by introducing spatial attention, the importance of specific areas in the image is paid attention to, which helps the model to focus on those local areas that have the greatest impact on the final task.

[0022] The present invention adopts a local enhanced spatial module, which retains local 2D dependencies during two-dimensional selective scanning through an enhanced local bias mechanism, improves the original state-space model, and enables the network to better capture non-causal relationships when processing visual data, avoiding the problem of increased neighborhood data distance caused by fixed scanning methods.

[0023] The present invention adopts an adaptive kernel selection module, which adaptively adjusts the illumination intensity according to the specific conditions of the input image, avoiding complex structural design and constraints to estimate physical priors. At the same time, through residual connections, deep convolutions and GELU functions, the module can effectively restore texture and color details in images, especially under complex degradation conditions such as low light illumination.

[0024] The present invention also adopts a dynamic pseudo-label generation mechanism, which evaluates the performance of the current model, stores the best results in the label pool, and dynamically updates the pseudo-labels to ensure that the optimal results are used, reducing noise and inconsistency. By generating high-quality pseudo-labels, unlabeled data is efficiently used, and the dependence on manually labeled data is reduced, the training efficiency of the model is improved. At the same time, the generalization ability of the model is improved, so that the model can better adapt to real-world data and perform well on unseen data. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings: Figure 1 It is a schematic diagram of the overall network structure of the method of the present invention; Figure 2 Schematic diagram of the multi-scale prior extraction network proposed in the present invention; Figure 3 is a schematic diagram of a student-teacher network proposed by the present invention; DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] See also Figure 1-Figure 3 , the present invention provides a technical solution: Embodiment 1: A low-light image enhancement method based on multi-scale global combined with local, such as Figure 1 As shown, the following steps are included: Step 1: construct a multi-scene low-light image training dataset consisting of multiple low-light images, where the low-light images are used to train an image enhancement network.

[0028] Specifically, low-light images of various scenes are selected from existing public low-light datasets, including the LOL dataset consisting of pairs of low-light images and their corresponding normal-light images, the DPED dataset containing real photos taken from three different mobile phones and a high-end SLR camera, and the Dark Face dataset focusing on face detection in low-light conditions; real low-light datasets are collected and some blurred images are eliminated, and the public low-light dataset images and real low-light images are merged to obtain a dataset for network training.

[0029] This embodiment uses 485 pairs of low-light / normal-light images in the LOL dataset for training and 15 images for testing; 500 pairs of low-light / normal-light images in the DPED dataset are used for training and 20 images are used for testing; 600 pairs of low-light / normal-light images in the Dark Face dataset are used for training and 25 images are used for testing; the real low-light dataset includes 50 low-light images.

[0030] Step 2: The image enhancement network is a U-Net-like network, which consists of a multi-scale prior extraction module, multiple local enhancement modules, and an adaptive kernel selection module. The low-light image is input into the multi-scale prior extraction module to extract the prior features of the image. F p ; The low-light image is convolved to obtain the low-light image features F l .

[0031] Multi-scale prior extraction module such as Figure 2 As shown in the figure, it consists of two parts. The first part is the multi-scale convolution part, which is mainly composed of three branch networks and a feature fusion module. The input low-light image is extracted through the three branch networks with different receptive fields. The extracted features are integrated in the feature fusion module to obtain the processing result of the first part. The second part inputs the processing result from the first part, processes it in parallel in the channel and space branches, learns the importance weights on the channel and space branches, and obtains the important feature information on the channel and space branches through weighted operations. The important feature information on the channel and space branches is integrated to obtain the prior feature. F p .

[0032] Specifically, in the multi-scale convolution part, the first branch uses a 1x1 convolution kernel to directly extract features without changing the spatial scale; the second branch uses a 3x3 convolution kernel with a void ratio of 12 to expand the receptive field to capture a wider range of contextual information; the third branch uses global average pooling to extract global contextual features and enhance the model's ability to understand the overall layout.

[0033] In the second part, on the channel branch, a global pooling operation is first performed to obtain the global information of each channel, and then two fully connected layers are passed. The two fully connected layers use ReLU and Sigmoid as activation functions respectively to learn the importance weights on the channel branch; on the spatial branch, the input features are subjected to 1×1 convolution and Sigmoid activation function operations to learn the importance weights on the spatial branch.

[0034] The above process is described as:

[0035]

[0036]

[0037] .

[0038] Step 3: Low-light image features F l and prior features F p Input local enhancement module, output local enhancement features F out1 .

[0039] like Figure 1 As shown in the LEM part, the local enhancement module consists of two branches. The first branch is the low-light image feature F l The features are obtained by sequentially passing through the Layer Norm layer, Linear layer, DWConv layer, 2DSSM module and Layer Norm layer. F 1 ; Prior features F p The features are obtained by processing the Conv layer, Linear layer and SiLU layer in sequence. F 2 ; The feature F 1 and Features F 2 Perform point multiplication and obtain local enhanced features through Linear layer processing F out1 The above process is described as:

[0040]

[0041] .

[0042] Furthermore, the local enhancement module of this embodiment enhances the original SSM by maintaining local 2D dependencies. The structured state space sequence model and SSM such as Mamba can be regarded as a continuous linear time-invariant system. Given a one-dimensional sequence x(t), it is projected into a new one-dimensional sequence y(t) through the hidden state h(t). The entire system can be defined as a linear ordinary differential equation:

[0043]

[0044] Where A, B, C, and D represent the state, input projection, output projection, and feedthrough parameters, respectively. The LEM module enhances the original SSM by introducing enhanced local deviations. The entire system is described as:

[0045] Where E is the feature enhancement information, so the calculation of the model is simple. Given the low-light image features F l and prior features F p , Layer norm and LEM modules are used to integrate spatial long-term dependencies.

[0046] Step 4: Enhance the local features F out1 and prior features F p Input adaptive kernel selection module, output enhanced features F out2 .

[0047] like Figure 1 As shown in the AKSM section, the adaptive kernel selection module consists of two branches, the first branch contains Conv, SK-1, SK-2 and kernel selection modules, and the second branch contains DWConv, Sigmoid and Chunk layers; this module uses prior features F p As input, it is input into the kernel selection module after being processed by the second branch, together with the local enhanced features F out1 After being processed by the first branch, the final output features are obtained through the operations of DWConv, GELU and Conv. F out2 This process enhances the ability to integrate information. The above process is described as:

[0048]

[0049] .

[0050] Step 5: Enhance the features F out2 and prior features F p The operations of step 3 and step 4 are repeated in the image enhancement network after downsampling and upsampling to train the image enhancement network.

[0051] Example 2: Furthermore, a student-teacher network is used to fine-tune the model trained in step 5, and real-world low-light images are introduced. The model is further optimized by combining strong geometric data enhancement, and high-quality pseudo labels are generated through a label generation framework to improve the generalization ability of the model and the actual application effect.

[0052] Student-teacher networks such as Figure 3 As shown in Figure 1, the network consists of a student network, a teacher network, a pseudo-label quality score, and a high-quality pseudo-label pool. The student network and the teacher network have the same network structure and share weights through exponential moving average. Specifically, the teacher network uses non-enhanced real low-light images as input, and the student network uses strongly enhanced images as input. After network reasoning, the following results are generated:

[0053] in, I L For real scene low-light images, I enhanced Output results for the teacher network, I pre Output results for the student network.

[0054] This embodiment uses a hybrid scoring mechanism that integrates a physical model and a domain adaptation discriminator, as follows: Assuming that the low-light noise obeys the Poisson-Gaussian mixture model, its observation noise variance is decomposed into: ; in y For an ideal noise-free signal, α is the Poisson noise intensity, β is the standard deviation of Gaussian noise; For each pixel block p, calculate its actual noise variance: ; The residual confidence score is defined by minimizing the difference between the actual variance and the model predicted variance:

[0055] in is the mean of the pseudo-labels within the block, λ To control sensitivity; Train the binary discriminator D to distinguish between real normal light images and low light images, and use the pseudo labels output by the teacher network I enhanced Input the discriminator D and calculate its naturalness score close to normal light:

[0056] The final confidence score of pixel (i, j) is weighted fused:

[0057] in, γ is a balance factor with a value of 0-1; Setting Thresholds τ , generate a binary mask M , filter high confidence regions for student network training:

[0058] The pseudo-label pool is dynamically updated according to the pseudo-label scores to ensure that the labels used for student network training are high-quality labels.

[0059] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Although the present invention is described in detail with reference to the aforementioned embodiments, those skilled in the art can still modify the technical solutions described in the aforementioned embodiments or replace some of the technical features therein by equivalents. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A low-light image enhancement method based on multi-scale global and local joint, characterized in that: The steps include: Step 1, constructing a multi-scene low-light image training dataset consisting of multiple low-light images, wherein the low-light images are used to train an image enhancement network; Step 2: The image enhancement network is a U-Net-like network, which consists of a multi-scale prior extraction module, multiple local enhancement modules, and an adaptive kernel selection module. The low-light image is input into the multi-scale prior extraction module to extract the prior features of the image. F p ; The low-light image is convolved to obtain the low-light image features F l ; Step 3: Low-light image features F l and prior features F p Input local enhancement module, output local enhancement features F out1 ; Step 4: Enhance the local features F out1 and prior features F p Input adaptive kernel selection module, output enhanced features F out2 ; Step 5: Enhance the features F out2 and prior features F p Repeating steps 3 and 4 in the image enhancement network after downsampling and upsampling to train the image enhancement network; In step 6, real low-light images in different scenes are taken as input and fine-tuned using the second-stage student-teacher network to obtain an image enhancement network with the ability to generalize to real scenes.

2. The low-light image enhancement method based on multi-scale global and local integration according to claim 1, characterized in that: In step 1, the specific process of constructing a multi-scene low-light image training dataset is as follows: select low-light images of various scenes from existing public low-light datasets, including the LOL dataset consisting of pairs of low-light images and their corresponding normal-light images, the DPED dataset containing real photos taken from three different mobile phones and a high-end SLR camera, and the Dark Face dataset that focuses on face detection under low-light conditions; collect a real low-light dataset and remove some blurred images, and merge the public low-light dataset images and the real low-light images to obtain a dataset for network training.

3. The low-light image enhancement method based on multi-scale global and local integration according to claim 2, characterized in that: The multi-scale prior extraction module is composed of two parts. The first part is the multi-scale convolution part, which is mainly composed of three branch networks and a feature fusion module. The input low-light image is extracted with different receptive fields through the three branch networks, and the extracted features are integrated in the feature fusion module to obtain the processing result of the first part; the second part inputs the processing result from the first part, processes it in parallel in the channel and space branches, learns the importance weights on the channel and space branches, and obtains the important feature information on the channel and space branches through weighted operations; integrates the important feature information on the channel and space branches to obtain the prior feature F p .

4. The low-light image enhancement method based on multi-scale global and local integration according to claim 3, characterized in that: In the multi-scale convolution part, the first branch uses a 1x1 convolution kernel, does not change the spatial scale, and directly extracts features; The second branch uses a 3x3 convolution kernel with a void ratio of 12 to expand the receptive field to capture a wider range of contextual information. The third branch uses global average pooling to extract global contextual features and enhance the model's ability to understand the overall layout.

5. The low-light image enhancement method based on multi-scale global combined with local according to claim 3, characterized in that: On the channel branch, a global pooling operation is first performed to obtain the global information of each channel, and then two fully connected layers are passed. The two fully connected layers use ReLU and Sigmoid as activation functions respectively to learn the importance weights on the channel branch; On the spatial branch, the input features are subjected to 1×1 convolution and Sigmoid activation function operations to learn the importance weights on the spatial branch.

6. The low-light image enhancement method based on multi-scale global and local integration according to claim 2, characterized in that: The local enhancement module consists of two branches. The low-light image features in the first branch F l The features are obtained by sequentially passing through the LayerNorm layer, Linear layer, DWConv layer, 2DSSM module and Layer Norm layer. F 1. Prior features F p The features are obtained by processing the Conv layer, Linear layer and SiLU layer in sequence. F 2. The features F 1 and Features F 2 Perform point multiplication and obtain local enhanced features through Linear layer processing F out1 .

7. The low-light image enhancement method based on multi-scale global and local integration according to claim 6, characterized in that: The 2DSSM module enhances the original state space model SSM by enhancing the local deviation, and the module is expressed as: ； One-dimensional sequence x t Through hidden state h t Projection into a new one-dimensional sequence y t , A, B, C, and D represent the state, input projection, output projection, and feedthrough parameters, respectively, and E is the feature enhancement information.

8. The low-light image enhancement method based on multi-scale global and local integration according to claim 2, characterized in that: The adaptive kernel selection module consists of two branches, wherein the first branch includes Conv, SK-1, SK-2 and the kernel selection module, and the second branch includes DWConv, Sigmoid and Chunk layers; F p As input, it is input into the kernel selection module after being processed by the second branch, together with the local enhanced features F out1 After being processed by the first branch, the final output features are obtained through the operations of DWConv, GELU and Conv. F out2 .

9. A low-light image enhancement method based on multi-scale global combined with local according to any one of claims 1 to 8, characterized in that: The student-teacher network consists of a student network, a teacher network, a pseudo-label quality score and a high-quality pseudo-label pool. The student network and the teacher network have the same network structure. The teacher network uses non-enhanced real low-light images as input, and the student network uses strongly enhanced images as input. The output of the teacher network is a label that is discriminated by the pseudo-label quality score to form a high-quality pseudo-label pool for high-quality training of the student network.

10. The low-light image enhancement method based on multi-scale global combined with local according to claim 9, characterized in that: The pseudo-label quality score is a hybrid scoring mechanism that integrates the physical model and the domain adaptation discriminator, as follows: Assuming that the low-light noise obeys the Poisson-Gaussian mixture model, its observation noise variance is decomposed into: ; in y For an ideal noise-free signal, α is the Poisson noise intensity, β is the standard deviation of Gaussian noise; For each pixel block p, calculate its actual noise variance: ; The residual confidence score is defined by minimizing the difference between the actual variance and the model predicted variance: ； in is the mean of the pseudo-labels within the block, λ To control sensitivity; Train the binary discriminator D to distinguish between real normal light images and low light images, and use the pseudo labels output by the teacher network I enhanced Input the discriminator D and calculate its naturalness score: ; The final confidence score of pixel (i, j) is weighted fused: ; in, γ is a balance factor with a value of 0-1; Setting Thresholds τ , generate a binary mask M , filter high confidence regions for student network training: ； The pseudo-label pool is dynamically updated according to the pseudo-label scores to ensure that the labels used for student network training are high-quality labels.

Images