Multi-exposure image enhancement method for dynamic scene

Abstract

The invention relates to the related technical field of computer vision, in particular to a multi-exposure image enhancement method for a dynamic scene, which comprises the following steps of: 1, inputting a multi-exposure image into a global brightness correction network (GBCNet), and carrying out iterative correction on the multi-exposure image by learning parameters of a nonlinear mapping curve LE by the global brightness correction network (GBCNet), therefore, global brightness correction is realized. According to the multi-exposure image enhancement method oriented to the dynamic scene, a global brightness correction network (GBCNet) is provided, the global brightness of an image is adjusted by using a learnable image brightness mapping curve, the discreteness of global brightness distribution is reduced, a local content enhancement network (LCENet) is provided, a multi-scale convolution block (Rep-MSB) capable of being reparameterized is designed, and the multi-exposure image enhancement method oriented to the dynamic scene is realized. By combining semi-instance normalization and multi-scale feature extraction, local contrast and detail representation are enhanced, and the complexity of the model is reduced by using a structure re-parameterization technology in an inference stage.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-exposure image enhancement method for dynamic scenes. Background Technology

[0002] The rapid development of intelligent transportation has driven the widespread application of image sensors in this field. However, due to limitations in sensor hardware parameters and the influence of lighting conditions, images acquired in high-dynamic scenes often suffer from underexposure or overexposure, resulting in severe image quality degradation, including brightness shift, decreased contrast, and loss of detail. This degradation not only affects the perceptual quality of human vision but also interferes with the stable operation of perception tasks (object detection, image segmentation, etc.) in autonomous driving. Multi-exposure image enhancement for high-dynamic scenes aims to adjust image brightness through algorithms, making the image brightness more consistent with that of normally exposed images, restoring local contrast, color, and detail, improving image quality, and providing reliable support for advanced computer vision tasks.

[0003] To improve the visual quality of images with different exposures in high dynamic range scenes, researchers have developed both traditional multi-exposure image enhancement methods and deep learning-based multi-exposure image enhancement methods. According to the technical approach adopted, traditional multi-exposure image enhancement methods can be divided into curve mapping-based methods, histogram equalization-based methods, and Retinex theory-based methods. While curve mapping-based methods have low computational complexity, they require careful design of specific variation parameters; otherwise, they can lead to amplification of image noise and distortion. Histogram equalization-based methods directly adjust the grayscale values ​​of the entire image, amplifying noise in dark areas of underexposed images and losing details in bright areas of overexposed images. Retinex theory-based methods are usually based on preset lighting models or heuristic rules, resulting in limited generalization and difficulty in meeting the needs of multi-exposure image enhancement. Summary of the Invention

[0004] The purpose of this invention is to provide a deblurring method for event-based spatiotemporal motion perception in real dynamic scenes, so as to solve the problem that existing single-stage networks in dynamic scenes cannot simultaneously take into account global brightness correction and local contrast relationship adjustment.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multi-exposure image enhancement method for dynamic scenes, comprising the following steps:

[0006] Step 1: Input the multi-exposure image into the Global Brightness Correction Network (GBCNet). The Global Brightness Correction Network (GBCNet) iteratively corrects the multi-exposure image by learning the parameters of the nonlinear mapping curve LE to achieve global brightness correction.

[0007] Step 2: Input the globally luminance-corrected image into the Local Content Enhancement Network (LCENet). The LCENet uses a deep residual network to calculate the residual map between the globally calibrated image and the reference image, restoring the local contrast and details of the image to optimize image quality.

[0008] Preferably, the Global Luminance Correction Network (GBCNet) includes the following sub-steps:

[0009] 1) Use 1x1 convolution to fuse the inter-channel information of the image to avoid introducing gradient or texture interference from neighboring pixels, thus ensuring the accuracy of brightness statistics.

[0010] 2) The extracted image features are subjected to global average pooling (GAP) to extract global brightness information from multi-exposure images;

[0011] 3) Use a multilayer perceptron (MLP) to estimate the parameters of the luminance mapping curve LE, and use an iterative mechanism to achieve gradual luminance correction. Specifically, the number of iterations of the LE curve is fixed at 4, and the parameters of the RGB channels of the image are estimated separately.

[0012] 4) Apply the estimated LE global brightness mapping curve to the multi-exposure image, adjust the global brightness level of the image, and complete 4 iterations in sequence until the parameter set is completely traversed.

[0013] Preferably, when the multilayer perceptron (MLP) estimates the parameters of the luminance mapping curve LE, the Tanh activation function is used to constrain the parameter values ​​to the range of (-1, 1).

[0014] Preferably, the Local Content Enhancement Network (LCENet) includes the following sub-steps:

[0015] 1) Use 3x3 convolution to aggregate local information of the image;

[0016] 2) Local contrast optimization and detail enhancement of images are efficiently achieved through 6 reparameterizable multi-scale convolutional blocks (Rep-MSB), wherein the Rep-MSB includes semi-instance normalization and multi-scale convolution;

[0017] 3) Enhance the expressive power of the network by using spatial attention and channel attention;

[0018] 4) Use two 3x3 convolutions to decode the fused features, estimate the residual of local image brightness correction, and perform a pixel-by-pixel summation operation with the result of global brightness correction to obtain the final multi-exposure image enhancement result.

[0019] Preferably, the reparameterizable multiscale convolutional block (Rep-MSB) includes the following steps:

[0020] 1) Perform semi-instance normalization on the input features to eliminate the influence of global brightness;

[0021] 2) Input the processed features into multi-scale convolutional blocks to obtain multi-scale feature information and improve the network's ability to capture details at different scales.

[0022] 3) The fused features are non-linearly activated using the Leaky function and then dimensionality is reduced using 1x1 convolution;

[0023] 4) Use spatial attention blocks (SA) to enhance the feature response of key regions;

[0024] 5) During the inference phase, the reparameterization technique is used to merge three independent convolutional branches into one branch, reducing the number of network parameters and improving the computational efficiency of the model.

[0025] Compared with the prior art, the beneficial effects of the present invention are:

[0026] 1. A progressive regression multi-exposure image enhancement framework is proposed, which decomposes the enhancement of multi-exposure images into global brightness correction and local content enhancement to address the problems of global brightness shift, local contrast shift, and detail degradation.

[0027] 2. A Global Brightness Correction Network (GBCNet) is proposed, which uses a learnable image brightness mapping curve to adjust the global brightness of the image and reduce the dispersion of the global brightness distribution.

[0028] 3. A Local Content Enhancement Network (LCENet) is proposed, which designs a reparameterizable multi-scale convolutional block (Rep-MSB). By combining semi-instance normalization and multi-scale feature extraction, local contrast and detail are enhanced, and structural reparameterization is used in the inference stage to reduce the complexity of the model. Attached Figure Description

[0029] Figure 1 This is a diagram illustrating the overall architecture of the multi-exposure image enhancement method for dynamic scenes according to the present invention.

[0030] Figure 2 This is a structural diagram of the reparameterizable multiscale convolutional block (Rep-MSB) of the present invention;

[0031] Figure 3 This is a graph showing the qualitative comparison results on the ME dataset of this invention;

[0032] Figure 4 This is a graph showing the qualitative comparison results on the SICEv2 dataset of this invention. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0034] Please see Figure 1-4 This invention provides a technical solution: a multi-exposure image enhancement method for dynamic scenes, comprising the following steps:

[0035] Step 1: To avoid the problem that Convolutional Neural Networks (CNNs) are easily affected by subtle textures when aggregating neighborhood information to extract brightness-related features, a brightness correction mapping curve is directly learned through global average brightness estimation. Images with similar global brightness levels typically share similar brightness mapping curves, which are independent of the image content. Specifically, the mapping curves for different images at the same brightness level to the statistical global brightness level tend to be consistent. Based on this prior, this invention proposes a Global Brightness Correction Network (GBCNet), as follows... Figure 1 As shown, the network iteratively corrects multi-exposure images by learning the parameters of the nonlinear mapping curve LE;

[0036] Step 2: After adjusting the overall brightness level of the image, the local areas related to the image content still have problems such as unclear contrast and missing details. This invention proposes a Local Content Enhancement Network (LCENet) based on residual learning to further optimize image quality. In this network, by learning the residual between the global correction image and the real image, it focuses on restoring essential and detailed information in local areas and improving the overall visual quality.

[0037] Furthermore, the Global Luminance Correction Network (GBCNet) includes the following sub-steps:

[0038] 1) For the input multi-exposure image, first use 1×1 convolution to fuse the inter-channel information of the image, while avoiding the introduction of gradient or texture information from neighboring pixels to ensure the accuracy of brightness statistics.

[0039] 2) Global average pooling (GAP) is performed on the extracted image features to extract global brightness information from the multi-exposure images;

[0040] 3) Use a multilayer perceptron (MLP) to estimate the parameters of the luminance mapping curve LE. Considering that multi-exposure images have a large dynamic range, a single mapping mechanism cannot correct the luminance to a consistent level. An iterative mechanism is used to achieve gradual luminance correction. In this process, the number of iterations of the LE curve is fixed at 4, and the parameters of the RGB channels of the image are estimated separately. Therefore, the MLP needs to estimate 4 sets of parameters, each set of parameters contains 3 different α values, for a total of 12 parameter values.

[0041] 4) Apply the estimated LE global brightness mapping curve to the multi-exposure image to adjust the global brightness level of the image. For each input image, select a set of α parameter values ​​for the RGB channels each time, and correct the global brightness of the multi-exposure image according to the mapping relationship shown in the formula. Complete 4 iterations in sequence until the parameter set is completely traversed. Through this step-by-step iterative method, the LE curve can achieve efficient correction of underexposed and overexposed images without sacrificing training stability, while ensuring the consistency of brightness distribution.

[0042] LE(I i ) = LE(I i-1 )+α i LE(I i-1 )(1-LE(I i-1 ))

[0043] Among them, LE(I i ) represents the image after the i-th mapping, α i The parameter α represents the permission parameter during the i-th mapping. When the value of parameter α is in the range of (-1,0), the curve mapping result will reduce the brightness level of the input image, corresponding to the brightness correction process of the overexposed image. When the value of parameter α is in the range of (0,1), the curve mapping result will increase the brightness level of the input image, corresponding to the brightness enhancement process of the overexposed image.

[0044] Furthermore, when estimating the parameters of the luminance mapping curve LE using a multilayer perceptron (MLP), the Tanh activation function is used to constrain the parameter values ​​to the range of (-1, 1).

[0045] Furthermore, the Local Content Enhancement Network (LCENet) includes the following sub-steps:

[0046] 1) Use 3x3 convolution to aggregate local information of the image;

[0047] 2) Local contrast optimization and detail enhancement of images are efficiently achieved through six reparameterizable multi-scale convolutional blocks (Rep-MSB), such as... Figure 2 As shown, the reparameterizable multiscale convolutional block (Rep-MSB) mainly consists of semi-instance normalization and multiscale convolution.

[0048] 3) Enhance the network's expressive power by using spatial attention and channel attention. In multi-exposure image processing tasks, images in different regions usually have different lighting and color characteristics. This requires the network to be able to use the differences between different regions and channels of the image. Therefore, after the multi-scale convolutional block, this invention introduces spatial attention and channel attention to fuse global spatial information and channel information respectively.

[0049] 4) Use two 3x3 convolutions to decode the fused features, estimate the residual of local image brightness correction, and perform a pixel-by-pixel summation operation with the result of global brightness correction to obtain the final multi-exposure image enhancement result.

[0050] Furthermore, reparameterizable multiscale convolutional blocks (Rep-MSB) include the following steps:

[0051] 1) Perform semi-instance normalization on the input features to eliminate the influence of global brightness. In this operation, feature slicing is performed first, and the original features are divided into two parts according to the channel dimension. One part is normalized (feature value minus mean divided by standard deviation), and the other part remains unchanged. The two parts are then concatenated in the original order.

[0052] 2) Input the processed features into the multi-scale convolutional block to obtain multi-scale feature information and improve the network's ability to capture details of different scales. In this convolutional block, three different convolutional branches first obtain features of specific scales respectively.

[0053] 3) The features from the three branches are merged, and the Leaky function is used for non-linear activation. The merged features are first reduced in dimensionality using 1×1 convolutions to ensure the compactness and effectiveness of the feature representation.

[0054] 4) Enhance the feature response of key regions by using spatial attention blocks (SA) to improve the capture of local details and regional differences;

[0055] 5) The output features of multi-scale convolutions and the input features of convolutional blocks are aggregated using residual connections to maintain feature consistency and improve network stability. During the inference phase, reparameterization is used to merge three independent convolutional branches into one branch, reducing the number of network parameters and improving the computational efficiency of the model.

[0056] Furthermore, Figure 1The framework of this invention is illustrated. First, a multi-exposure image is input into a Global Brightness Correction Network (GBCNet). This network adaptively generates curve parameters for a global nonlinear mapping by accurately sensing the global brightness statistical characteristics of the multi-exposure image. Then, the parameter estimation results are applied to the input multi-exposure image to achieve global brightness correction. Subsequently, the corrected image is input into a Local Content Enhancement Network (LCENet) for local contrast optimization and detail enhancement. In this network, a deep residual network calculates the residual map between the two corrected images and the reference image to restore the local contrast relationship and details of the image.

[0057] Example 1: To adapt to application scenarios with different levels of complexity, this invention constructs two models of different sizes, each containing a different number of multi-scale convolutional blocks. The model containing 6 multi-scale convolutional blocks is called the Small model (method-S of this invention), and the model containing 8 multi-scale convolutional blocks is called the Large model (method-L of this invention).

[0058] This invention conducts comparative experiments on two multi-exposure image datasets, ME and SICEv2. During the experiments, PSNR (Peak Signal-to-Noise Ratio) and SSIM (Structural Similarity) are used to quantitatively evaluate the quality of multi-exposure image enhancement, and the total number of parameters is used to measure the size of the model.

[0059] As shown in Tables 1 and 2, the method of this invention is quantitatively and qualitatively compared with 17 advanced multi-exposure image enhancement methods on the ME and SICEv2 datasets. In addition, the performance differences of the two versions with different parameter scales constructed in this invention (method-S and method-L) and other methods under different exposure types are evaluated. The enhancement effect of underexposed images with exposure values ​​of -1.5 and -1.0 is statistically analyzed, and the enhancement effect of overexposed images with exposure values ​​of +1.0 and +1.5 is statistically analyzed. The average index is the average enhancement effect across all exposure levels.

[0060] Table 1. Quantitative comparison experiments of multi-exposure image enhancement methods on the ME dataset.

[0061]

[0062]

[0063] The comparison results in Tables 1 and 2 show that the present invention exhibits a continuous and significant performance improvement on both datasets. On the ME dataset, compared with the second-ranked method, the present invention's method-L achieves a 0.10 dB improvement in PSNR in underexposed areas and a 0.01 dB and 0.004 improvement in PSNR and SSIM in overexposed areas, respectively, with fewer parameters. On the SICEv2 dataset, the present invention's method achieves a 0.36 dB and 0.002 improvement in PSNR and SSIM in underexposed areas, respectively, and a 0.70 dB and 0.004 improvement in PSNR and SSIM in overexposed areas, respectively, with fewer parameters. The experimental results on both datasets demonstrate that the multi-exposure image enhancement strategy, from global to local, can adapt to changes in different exposure scenarios. While correcting image brightness, it can also preserve richer details in the image. Therefore, the two-stage multi-exposure image enhancement method proposed in this invention improves image quality while being lightweight and high-performance, meeting the dual requirements of resource efficiency and enhancement effect in practical applications.

[0064] Table 2. Quantitative comparison experiments of multi-exposure image enhancement methods on the SICEv2 dataset.

[0065]

[0066]

[0067] Example 2 Figure 3 and Figure 4 The qualitative results of the proposed method for enhancing multi-exposure images on the ME and SICEv2 datasets are presented. On the ME dataset, RetinexNet and Zero-DCE both exhibit excessive brightness enhancement and severe color cast in both underexposed and overexposed images. Compared to the reference image, CMEC, MSEC, ECLNet, NEC, and FECNet still have insufficient enhancement results for underexposed images and still overexpose images with a brighter enhancement. In contrast, the enhancement results of the proposed method are closer to the brightness level of the reference image while retaining more detail information, further demonstrating the effectiveness and generalization of the proposed method in processing multi-exposure images.

[0068] On the SICEv2 dataset, scene exposure levels are more extreme, placing higher demands on multi-exposure image enhancement methods, such as... Figure 4As shown, RetinexNet relies on manually specified decomposition priors, resulting in numerous artifacts and severe color casts in the enhancement results. While Zero-DCE provides relatively reasonable brightness correction, it also exhibits color shift. MSEC still demonstrates the limitations of enhancement steps, with underexposed images showing lower brightness and overexposed images showing blurred highlight readout edges. Other methods offer relatively accurate brightness correction, but their enhancement of edge textures and highlight areas remains insufficient. The method of this invention accurately enhances brightness while maintaining color and detail consistency under extreme exposure conditions, further proving the effectiveness of the two-stage enhancement method.

[0069] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-exposure image enhancement method for dynamic scenes, characterized in that, Includes the following steps: Step 1: Input the multi-exposure image into the Global Brightness Correction Network (GBCNet). The Global Brightness Correction Network (GBCNet) iteratively corrects the multi-exposure image by learning the parameters of the nonlinear mapping curve LE to achieve global brightness correction. Step 2: Input the globally luminance-corrected image into the Local Content Enhancement Network (LCENet). The LCENet uses a deep residual network to calculate the residual map between the globally calibrated image and the reference image, restoring the local contrast and details of the image to optimize image quality.

2. The multi-exposure image enhancement method for dynamic scenes according to claim 1, characterized in that, The Global Luminance Correction Network (GBCNet) includes the following sub-steps: 1) Use 1x1 convolution to fuse the inter-channel information of the image to avoid introducing gradient or texture interference from neighboring pixels, thus ensuring the accuracy of brightness statistics. 2) The extracted image features are subjected to global average pooling (GAP) to extract global brightness information from multi-exposure images; 3) Use a multilayer perceptron (MLP) to estimate the parameters of the luminance mapping curve LE, and use an iterative mechanism to achieve gradual luminance correction. Specifically, the number of iterations of the LE curve is fixed at 4, and the parameters of the RGB channels of the image are estimated separately. 4) Apply the estimated LE global brightness mapping curve to the multi-exposure image, adjust the global brightness level of the image, and complete 4 iterations in sequence until the parameter set is completely traversed.

3. The multi-exposure image enhancement method for dynamic scenes according to claim 2, characterized in that, When estimating the parameters of the luminance mapping curve LE using the multilayer perceptron (MLP), the Tanh activation function is used to constrain the parameter values ​​to the range of (-1, 1).

4. The multi-exposure image enhancement method for dynamic scenes according to claim 1, characterized in that, The Local Content Enhancement Network (LCENet) includes the following sub-steps: 1) Use 3x3 convolution to aggregate local information of the image; 2) Local contrast optimization and detail enhancement of images are efficiently achieved through 6 reparameterizable multi-scale convolutional blocks (Rep-MSB), wherein the Rep-MSB includes semi-instance normalization and multi-scale convolution; 3) Enhance the expressive power of the network by using spatial attention and channel attention; 4) Use two 3x3 convolutions to decode the fused features, estimate the residual of local image brightness correction, and perform a pixel-by-pixel summation operation with the result of global brightness correction to obtain the final multi-exposure image enhancement result.

5. The multi-exposure image enhancement method for dynamic scenes according to claim 4, characterized in that, The reparameterizable multiscale convolutional block (Rep-MSB) includes the following steps: 1) Perform semi-instance normalization on the input features to eliminate the influence of global brightness; 2) Input the processed features into multi-scale convolutional blocks to obtain multi-scale feature information and improve the network's ability to capture details at different scales. 3) The fused features are non-linearly activated using the Leaky function and then dimensionality is reduced using 1x1 convolution; 4) Use spatial attention blocks (SA) to enhance the feature response of key regions; 5) During the inference phase, the reparameterization technique is used to merge three independent convolutional branches into one branch, reducing the number of network parameters and improving the computational efficiency of the model.

Images