A High Dynamic Range Imaging Method Based on Feature Separation and Attention Guidance
By employing feature separation and attention-guided methods, the effects of illumination and motion in image fusion under dynamic scenes are addressed, thereby improving image quality and generating high-quality, high dynamic range images.
Patent Information
- Application Number
- CN202410232613.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-03-01
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-03-01
AI Technical Summary
Existing multi-exposure image fusion methods are difficult to effectively reduce ghosting artifacts and exposure distortion in dynamic scenes, and they ignore the interaction between illumination and motion changes, resulting in poor image quality.
We employ a feature separation and attention-guided approach. By constructing a feature separation preprocessing network, we decompose images into illumination-related and motion-related features, and process them using a local entropy attention module and a feature alignment module. We then combine the Swin Transformer and channel attention to perform image fusion, thereby enhancing the extraction of image feature information.
It effectively reduces motion artifacts and exposure distortion, improves the quality of high dynamic range images, and generates high-quality HDR results with more detail and texture.
Smart Images

Figure CN118037566B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of digital image processing technology, specifically relating to a high dynamic range imaging method based on feature separation and attention guidance. Background Technology
[0002] High dynamic range (HDR) imaging aims to produce high-quality, well-exposed images even under varying lighting conditions. Due to inherent hardware limitations, most modern imaging systems (such as digital cameras) typically cannot capture the full range of natural light. Therefore, HDR imaging is an indispensable technology in modern imaging systems and has significant application value.
[0003] In the early days, specialized hardware was designed to directly capture HDR images, but these were often too expensive for widespread commercial use. Therefore, many researchers adopted multi-exposure image fusion techniques, which use a sequence of low dynamic range (LDR) images with different exposures and fuse them in a manner based on a reference LDR image to generate the corresponding HDR image. However, most multi-exposure image fusion methods fail to produce high-quality results from LDR images captured in dynamic scenes. Unlike static scenes, dynamic scenes introduce two challenging problems for HDR imaging: first, scene inconsistencies between LDR images can easily produce ghosting artifacts; second, missing content in underexposed and overexposed areas can cause distortion if not adequately compensated during the fusion stage.
[0004] With the advancement of computing power in current devices, deep learning has made significant progress, and many learning-based methods have emerged. Most multi-exposure HDR image reconstruction methods align the input LDR images before image fusion. Kalantari et al. proposed the first deep convolutional neural network (CNN) for HDR imaging of dynamic scenes. They first used optical flow to align the LDR images, and then fused the aligned images through a CNN. However, the lighting in LDR images is variable, which disrupts the constant brightness condition required for optical flow alignment, leading to considerable misalignment errors and severely impacting the quality of the final result. Inspired by learnable deformable convolutions, Tan et al. used a deformable alignment module to hierarchically align LDR images for fusion. Some effective approaches employ attention mechanisms before fusion to guide the HDR reconstruction network to focus on the LDR image's most detailed, light-optimal areas and motion-free regions, thereby reducing artifacts and distortion. Despite these significant advancements, some issues of detail blurring and motion artifacts still exist. It should be noted that these methods all directly process the source image sequence, while the source image sequence in dynamic scenes has both illumination and motion changes. They ignore the interaction between illumination and motion features when performing alignment or detail processing. Therefore, deep learning-based methods need further improvement. Summary of the Invention
[0005] To address the aforementioned technical problems, this invention proposes a high dynamic range imaging method based on feature separation and attention guidance, comprising:
[0006] S1: Obtain a low dynamic range image sequence of multiple consecutive frames, and perform block segmentation and preprocessing to obtain the high dynamic range image block corresponding to the low dynamic range image sequence block. Connect the low dynamic range image block with the corresponding high dynamic range image block channel to obtain a 6-channel low dynamic range and high dynamic range image block pair.
[0007] The acquired low dynamic range image sequence includes low-exposure, normal-exposure, and high-exposure low dynamic range images, wherein the normal-exposure low dynamic range image is used as a reference image.
[0008] S2: Construct a feature separation preprocessing network, which includes a feature separation-reconstruction module, a local entropy attention module, and a feature alignment module;
[0009] S3: Construct a fusion network, which includes a global-local feature extraction module;
[0010] S4: Train the entire network using groups of low dynamic range and high dynamic range image patches, and obtain and save the trained network model parameters.
[0011] S5: Input the images to be fused into the trained network to obtain high dynamic range images.
[0012] The beneficial effects of this invention are:
[0013] This invention decomposes the source image into two components: illumination-dependent and motion-dependent. This aims to reduce the influence of illumination on motion processing and vice versa, allowing the network to process both components more effectively. To address the lack of detail caused by oversaturated / undersaturated regions, a relational strength matrix is established using the image's local entropy characteristics. This guides the network to ignore information in these regions, promoting the extraction of detailed features. To improve the quality of HDR image fusion, a feature extractor is constructed in the image fusion module by combining the Swing Transformer and channel attention. This extractor captures global and local details of image features, enhancing image feature information during the fusion process and effectively improving the detail and texture information of the fused image. Experimental results show that, compared with existing HDR imaging methods, this invention can generate high-quality HDR results with less exposure distortion and motion artifacts. Attached Figure Description
[0014] Figure 1 This is a flowchart of the method of the present invention;
[0015] Figure 2 This is a structural diagram of the network model of the present invention;
[0016] Figure 3 This is a visualization of the local entropy attention weights of the present invention;
[0017] Figure 4 This is a diagram showing the result of the feature alignment module of the present invention;
[0018] Figure 5 This is a high dynamic range image effect obtained by the present invention;
[0019] Figure 6 This is a diagram showing the effect of independent component analysis in this invention. Detailed Implementation
[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0021] This invention provides a high dynamic range imaging method based on feature separation and attention guidance, such as... Figure 1 As shown, it includes:
[0022] S1: Acquire a sequence of low dynamic range (LVR) images from multiple consecutive frames, including low-exposure, normal-exposure, and high-exposure LVR images. Then, divide the LVR image sequence into blocks and preprocess them to obtain high dynamic range (HVR) image blocks corresponding to the LVR image sequence blocks. Connect the LVR image blocks with the corresponding HVR image block channels to obtain a 6-channel LVR and HVR image block pair. The normally exposed LVR image is used as the reference image.
[0023] Specifically, step S1 involves segmenting and preprocessing the low dynamic range images (low exposure, normal exposure, and high exposure) in each group, including:
[0024] S11: Divide each group of multi-frame low dynamic range image sequences into blocks of 128×128 pixels with a stride of 64, and then randomly rotate and flip them to obtain multiple groups of multi-frame low dynamic range image sequence blocks. i=1,2,3 The purpose of random rotation and flipping is to increase the diversity of data and facilitate network training;
[0025] S12: Use the gamma-corrected camera response function to separate low-exposure, medium-exposure, and high-exposure image blocks. i=1,2,3 Image patch h mapped to the corresponding high dynamic range domain i=1,2,3 And connected together according to channels to form low dynamic range image sequence block pairs [l i ,h i ], i = 1, 2, 3, where i = 1, 2, 3 represent the exposed, normally exposed, and overexposed images, respectively. The gamma-corrected camera response function is expressed as follows:
[0026] h i =(l i **GAMMA)÷t i i = 1, 2, 3
[0027] Where GAMMA is set to 2.2, 2.2 is an approximation of the camera response function, which can approximate the true HDR image. i It refers to the exposure time;
[0028] S13: Normalize the low dynamic range image sequence block pairs to obtain the 6-channel low dynamic range and high dynamic range image sequence block pair tensor X. i i = 1, 2, 3.
[0029] S2: Construct a feature separation preprocessing network, which includes a feature separation-reconstruction module, a local entropy attention module, and a feature alignment module, such as... Figure 2 As shown;
[0030] Specifically, the basic process of the feature separation preprocessing network is as follows: Discrete wavelet transform is used to decompose the low dynamic range image into illumination-related features and motion-related features; for illumination-related features, a local entropy attention module is used to refine feature extraction, and for motion-related features, a feature alignment module is used to perform structural consistency alignment and eliminate motion artifacts with obvious edges; then, inverse discrete wavelet transform is used to merge and reconstruct the processed illumination-related features and motion-related features to obtain the preprocessed and enhanced features to be fused.
[0031] The specific process of feature separation-reconstruction module in step S2 includes:
[0032] Patch the 6-channel low dynamic range and high dynamic range image sequence blocks onto the tensor X. i Perform 3×3 convolution operations to obtain the feature map Z with 60 channels. i i = 1, 2, 3; for the feature map Z i The first feature mapping component L is obtained by using discrete wavelet transform for feature separation. i i = 1, 2, 3 and the second eigenmap component H i ,i=1,2,3; the first feature mapping components L after processing are respectively i i = 1, 2, 3 and the second eigenmap component L i The inverse discrete wavelet transform is performed on i = 1, 2, 3 to reconstruct the feature map Z′ to be fused. i ,i=1,2,3, where i=1,2,3 represent the exposed, normal exposed and high exposed images;
[0033] The discrete wavelet transform separation and reconstruction process is represented as follows:
[0034] Discrete wavelet transform:
[0035] L i H i =DWT(Z) i ), i = 1, 2, 3
[0036] Inverse Discrete Wavelet Transform:
[0037] Z′ i =IDWT(L i H i ), i = 1, 2, 3
[0038] Among them, Z′ i L represents the feature map to be fused. i H represents the low-frequency components of the i-th exposed image feature in the horizontal, vertical, and diagonal directions, i.e., the first feature mapping component. iThe high-frequency components in the horizontal, vertical, and diagonal directions of the i-th exposed image feature are the second feature mapping components. IDWT() represents the inverse discrete wavelet transform, and i = 1, 2, 3 represent the exposed, normal exposed, and high-exposed images, respectively.
[0039] The specific processing steps of the local entropy attention module in step S2 include:
[0040] Calculate the first feature mapping component L i The local entropy of i = 1, 2, 3 is used to obtain the relation strength matrix Att through the sigmoid function. i For i = 1, 2, 3, the relation strength matrix contains the intensity distribution of each region. For over / undersaturated regions with fewer details, the intensity is smaller, and the effect is as follows: Figure 3 As shown; the relation strength matrix is mapped to the first eigenmap component L. i The attention feature L′ is obtained by performing element-wise multiplication on i = 1, 2, 3. i i = 1, 2, 3;
[0041] The local entropy is calculated as follows:
[0042]
[0043]
[0044] Where, p ab L represents the probability at (a,b) in the neighborhood area. i This represents the low-frequency components (i.e., the first feature mapping components) of the i-th exposed image feature in the horizontal, vertical, and diagonal directions, where i = 1, 2, 3 represent exposed, normally exposed, and overexposed images, respectively. L i=1,2,3 (x,y) represents the pixel value at position (x,y), L i=1,2,3 (a,b) represents the pixel value at (s,b) within the area, where area represents a 7×7 neighborhood centered at (x,y). M i=1,2,3 Let represent the local entropy of the i-th exposed image feature;
[0045] To obtain the relationship strength matrix, normalization is used, and the attention features are calculated as follows:
[0046] Att i =σ(M i ), i = 1, 2, 3
[0047]
[0048] Among them, L′ iThe attention feature is represented by i = 1, 2, 3, which represent normally exposed, normally exposed, and overexposed images, respectively, and σ() represents the sigmoid activation function. M represents element-wise multiplication. i Att represents the local entropy of the i-th exposed image feature. i L represents the relationship strength matrix. i The first feature mapping component represents the low-frequency components of the i-th exposed image feature in the horizontal, vertical, and diagonal directions.
[0049] The specific processing steps of the feature alignment module in step S2 include:
[0050] The second feature map component H i Using i = 1, 2, 3 as reference features, perform two parallel 3×3 convolution operations to obtain the first and second affine transformation matrices β1, γ1 respectively; repeat the previous step to obtain the third and fourth affine transformation matrices β2, γ2; map the second feature component H... i Multiplying i = 1, 2, 3 by the first affine transformation matrix β1 and then adding it to the second affine transformation matrix γ1, we obtain the feature H′1 aligned with the reference feature; the second feature mapping component H... i Multiplying i = 1, 2, 3 by the third affine transformation matrix β2 and then adding it to the fourth affine transformation matrix γ2 yields the feature H′2 aligned with the reference feature; the alignment effect is as follows. Figure 4 As shown.
[0051] S3: Construct a fusion network, which includes a global-local feature extraction module;
[0052] Specifically, the purpose of the fusion network is to compress and fuse the features of three different exposure image sequences into one image. During the compression process, the network integrates as many beneficial features as possible from each image. The global-local feature extraction module consists of a Swing Transformer block and channel attention. The Swing Transformer block can establish the dependencies between global information of image features, allowing the network to capture global complementary information between different exposure images, enhancing beneficial features and suppressing harmful features. Channel attention uses convolutional layers to refine the extraction of local image information, and further captures local complementary features between different exposure images by judging the importance of different channels. By integrating the global and local feature extractors, the image generation quality is further improved. The processing of the global-local feature extraction module in step S3 includes:
[0053] S31: Map the features to be fused to Z′ i After connecting i = 1, 2, 3 according to the channels, perform a 3×3 channel compressed convolution operation to obtain the preliminary fusion feature map F1;
[0054] S32: The initial fused feature map F1 is input into the global feature extractor of the Swin Transformer layer to obtain the global feature map F2, and then passed through the channel attention local feature extractor to obtain the local refined feature F3; the calculation process is expressed as follows:
[0055] LF=C 3×3 (C 3×3 (F2))
[0056] ω=σ(C 1×1 (C 1×1 (P(LF))))
[0057]
[0058] Where F3 represents local refinement features, LF represents features extracted after two 3×3 convolutions, ω represents the channel weight matrix after pooling, two 1×1 convolutions, and a sigmoid normalized activation function, F2 represents the global feature map, and C 3×3 Represents a 3×3 convolution, C 1×1 This represents a 1×1 convolution, where P represents pooling and σ() represents the sigmoid activation function. This represents element-wise multiplication;
[0059] S33: Take the local refinement feature F3 as the input of S32, repeat step S32 6 times, and then perform a 3×3 convolution operation to obtain the feature map F4. Add F1 and F4 through residual connection to obtain the feature map F5.
[0060] S34: Use feature map F5 as input to S33, repeat step S33 3 times to obtain the refined feature map F6, input F6 into a convolutional layer with a kernel of 3×3, and then pass through the ReLU activation function to obtain feature map F7;
[0061] S35: The feature map F7 is processed through tone mapping to obtain the final high dynamic range image. The tone mapping satisfies the following relationship:
[0062] T(I)=[log(1+μI)]÷[log(1+μ)]
[0063] Where T(I) represents the final high dynamic range image, I = F7, and μ represents the dynamic range compression parameter, with μ achieving the best effect when it is 5000.
[0064] S4: Train the entire network using groups of low dynamic range and high dynamic range image patches, and obtain and save the trained network model parameters.
[0065] S5: Input the images to be fused into the trained network to obtain high dynamic range images.
[0066] like Figure 5 As shown, the method of the present invention achieves better high dynamic range image quality in motion scenes than most other methods currently available, and can obtain relatively higher quality images.
[0067] This invention addresses the coupling problem between illumination and motion in dynamic scenes by establishing a preprocessing network based on the separation of illumination and motion features. This network reduces interference between illumination and motion, resulting in better illumination feature extraction and motion artifact removal.
[0068] In this embodiment, to achieve the separation of illumination and motion features, in addition to the discrete wavelet transform proposed in this invention, another implementation scheme is to use Independent Component Analysis (ICA). The core idea is that ICA can detect regions that differ between two images, and the differences between images with different exposures are the illumination regions and motion regions. By performing feature separation on images with different exposures using ICA, multiple feature maps highlighting motion regions and illumination change regions can be separated, thus achieving the separation of illumination and motion features. Figure 6 As shown, by using these feature maps as attention-guided maps to weight input images with different exposures, enhancement of illuminated areas and removal of artifacts in moving areas are achieved.
[0069] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A high dynamic range imaging method based on feature separation and attention guidance, characterized in that, include: S1: Obtain a low dynamic range image sequence of multiple consecutive frames, and perform block segmentation and preprocessing to obtain the high dynamic range image block corresponding to the low dynamic range image sequence block. Connect the low dynamic range image block with the corresponding high dynamic range image block channel to obtain a 6-channel low dynamic range and high dynamic range image block pair. The acquired low dynamic range image sequence includes low-exposure, normal-exposure, and high-exposure low dynamic range images, wherein the normal-exposure low dynamic range image is used as a reference image. S2: Construct a feature separation preprocessing network, which includes a feature separation-reconstruction module, a local entropy attention module, and a feature alignment module; The data processing procedure of the feature separation preprocessing network includes: Feature Separation-Reconstruction Module: Block tensor of 6-channel low dynamic and high dynamic range image sequence respectively, to obtain a feature map with a channel number of 60 ; a 3x3 convolution operation is performed on the feature map , to obtain a first feature map component and a second feature map component ; the first feature map component and the second feature map component processed respectively are subjected to inverse discrete wavelet transform reconstruction to obtain a feature map to be fused , wherein, indicate an exposure, normal exposure and high exposure image; Local entropy attention module: Calculate the first feature map component The local entropy is then used to obtain the relation strength matrix via the sigmoid function. ; map the relationship strength matrix to the first eigenvalue component Attention features are obtained by performing element-wise multiplication. ; Feature alignment module: The second feature mapping component As a reference feature, two parallel 3×3 convolution operations are performed to obtain the first and second affine transformation matrices, respectively. Repeat the previous step to obtain the third and fourth affine transformation matrices. Map the second feature component With the first affine transformation matrix After multiplication, it is combined with the second affine transformation matrix. Add them together to obtain the features aligned with the reference features. Map the second feature component With the third affine transformation matrix After multiplication, it is combined with the fourth affine transformation matrix. The sum is used to obtain the feature aligned with the reference feature. ; S3: Construct a fusion network, which includes a global-local feature extraction module; S4: Train the entire network using groups of low dynamic range and high dynamic range image patches, and obtain and save the trained network model parameters. S5: Input the image to be fused into the trained network to obtain a high dynamic range image.
2. The high dynamic range imaging method based on feature separation and attention guidance according to claim 1, characterized in that, Step S1 specifically includes: S11: Divide each group of multi-frame low dynamic range image sequences into blocks according to an n×n size and an n / 2 step size to obtain multiple groups of multi-frame low dynamic range image sequence blocks. S12: The low dynamic range image block, medium exposure image block, and high exposure image block in the low dynamic range image sequence block are mapped to the corresponding high dynamic range domain using the gamma-positive camera response function to obtain the high dynamic range image block. The low dynamic range image block and the corresponding high dynamic range image block are connected together according to the channel to form a low dynamic range and high dynamic range image sequence block pair. S13: Normalize the low dynamic range and high dynamic range image sequence block pairs to obtain 6-channel low dynamic range and high dynamic range image sequence block pair tensors. ,in, This indicates the exposure level, normal exposure level, and high exposure level of the image.
3. The high dynamic range imaging method based on feature separation and attention guidance according to claim 1, characterized in that, The discrete wavelet transform is used for feature separation and inverse discrete wavelet transform reconstruction, including: Discrete wavelet transform: Inverse Discrete Wavelet Transform: in, Indicates the feature mapping to be fused. This represents the low-frequency components of the nth exposed image feature in the horizontal, vertical, and diagonal directions, i.e., the first feature mapping component. Indicates the first The high-frequency components of the exposed image features in the horizontal, vertical, and diagonal directions are the second feature mapping components. Represents the inverse discrete wavelet transform. This indicates the exposure level, normal exposure level, and high exposure level of the image.
4. The high dynamic range imaging method based on feature separation and attention guidance according to claim 1, characterized in that, Calculate the first eigenmap component The local entropy includes: in, Representation of domain middle The probability at that location. Indicates the first The low-frequency components of the exposed image features in the horizontal, vertical, and diagonal directions are the first feature mapping components. Indicates the exposure, normal exposure, and high exposure images. Indicates in Pixel value at location Indicates the area pixel value, Indicates Center point field, Indicates the first Local entropy of a single exposed image feature.
5. The high dynamic range imaging method based on feature separation and attention guidance according to claim 1, characterized in that, The first Local entropy of each exposed image feature The relation strength matrix obtained through the sigmoid activation function With the first feature mapping component Attention features are obtained by performing element-wise multiplication. ,include: in, Indicating attentional characteristics, Indicates the exposure, normal exposure, and high exposure images. This represents the sigmoid activation function. This represents element-wise multiplication. Indicates the first Local entropy of a feature of an exposed image. Represents the relationship strength matrix. This represents the low-frequency components of the nth exposed image feature in the horizontal, vertical, and diagonal directions, i.e., the first feature mapping component.
6. The high dynamic range imaging method based on feature separation and attention guidance according to claim 1, characterized in that, The data processing procedure for building the converged network includes: Global-Local Feature Extraction Module: S31: Map the features to be fused After concatenating the channels, a 3×3 channel compressed convolution operation is performed to obtain the preliminary fused feature map. ; S32: Initial fusion of feature maps The input is fed into the global feature extractor of the Swin Transformer layer to obtain the global feature map. Then, the local refined features are obtained through the channel attention local feature extractor. ; S33: Refine local features Using this as input to S32, repeat step S32 six times, then perform a 3×3 convolution operation to obtain the feature map. By residual connection and Adding them together yields the feature map. ; S34: Map the features As input to S33, step S33 is repeated three times to obtain the refined extracted feature map. ,Will The input is fed into a 3×3 convolutional layer, and then passes through a ReLU activation function to obtain the feature map. ; S35: Map the features After tone mapping, the final high dynamic range image is obtained.
7. The high dynamic range imaging method based on feature separation and attention guidance according to claim 6, characterized in that, Local refinement features are obtained after channel attention local feature extractor. ,include: in, Indicates local refinement features. This represents the features extracted after two 3×3 convolutions. This represents the channel weight matrix after pooling, two 1×1 convolutions, and a sigmoid-normalized activation function. Represents global feature mapping. express convolution, express convolution, Indicates pooling, This represents the sigmoid activation function. This indicates element-wise multiplication.
8. A high dynamic range imaging method based on feature separation and attention guidance according to claim 6, characterized in that, Feature mapping After tone mapping, the final high dynamic range image output is obtained, including: in, This represents the final high dynamic range image. Indicates the dynamic range compression parameter. The effect is best when the value is 5000.
