Cross-color-gamut phase-guided double-attention fusion HDR imaging method
By employing a cross-color gamut phase-guided dual-attention fusion method, the ghosting artifacts and computational efficiency issues in HDR imaging technology under complex dynamic scenes and extreme exposures are resolved, achieving efficient and accurate image reconstruction and detail restoration, suitable for mobile devices.
Patent Information
- Application Number
- CN202610067666.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-19
- Publication Date
- 2026-05-01
AI Technical Summary
Existing HDR imaging technologies suffer from problems such as ghosting artifacts, high computational overhead, and insufficient accuracy when dealing with complex dynamic scenes and extreme exposure differences. In particular, methods based on convolutional neural networks have limitations in long-range dependency modeling and detail recovery.
We employ a cross-color gamut phase-guided dual-attention fusion method. By designing a context-aware Transformer module that combines global and local information, we introduce adaptive motion perception and suppression mechanisms and frequency domain phase fusion to optimize the training process and improve ghosting removal accuracy and computational efficiency.
It achieves stable image reconstruction under dynamic scenes and extreme exposure conditions, improves detail recovery capability and computational efficiency, reduces resource requirements, and is suitable for deployment on mobile devices.
Smart Images
Figure CN121961958A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of dual attention fusion HDR imaging technology, specifically relating to a cross-color gamut phase-guided dual attention fusion HDR imaging method. Background Technology
[0002] Multi-frame high dynamic range (HDR) imaging aims to fuse low dynamic range (LDR) images with different exposures to simultaneously preserve details in both highlights and shadows. It is widely used in mobile imaging, autonomous driving, AR / VR, and medical and remote sensing applications. However, existing technologies mainly advance along two paths: one is preprocessing schemes based on explicit registration or pixel culling using optical flow / homophony, which are prone to failure and loss of useful information in scenes with significant motion, occlusion, and saturation; the other is end-to-end implicit alignment and fusion based on convolutional neural networks. While this has made progress, it is limited by the local receptive field and content independence of convolution, making it difficult to model long-range displacements and cross-regional intensity changes, resulting in ghosting artifacts and structural distortion. The recent VisionTransformer possesses global dependency modeling capabilities, but it lacks convolutional inductive bias, suffers from insufficient generalization due to the scarcity of HDR deghosting annotation data, and has relatively weak capabilities in capturing the local neighborhood relationships and texture details necessary for multi-frame reconstruction. A more fundamental challenge lies in the inconsistency of cross-exposure features: the same object appears saturated white in overexposed frames and dark and noisy in underexposed frames, resulting in extremely low RGB feature similarity across frames. Even if spatially corresponding, reliable texture transfer is difficult to achieve, creating a semantic gap in the feature domain. Furthermore, directly measuring error in the linear HDR domain leads to an imbalance in highlight / shadow penalties, and relying solely on pixel-level loss is insufficient to guarantee perceptual quality. The core challenge of multi-frame HDR imaging lies in obtaining a ghost-free fusion substrate in dynamic scenes while maintaining structural and semantic consistency across exposures. This process requires not only motion suppression and the introduction of robust priors in the feature domain but also cross-exposure feature calibration in the frequency domain using the stability of chroma and phase. Simultaneously, it necessitates managing computational complexity by relying on hierarchical, windowed Transformers to account for local, global, and multi-scale dependencies, and ensuring training stability through perceptually consistent joint loss under data constraints. Therefore, developing an HDR fusion framework that collaboratively integrates feature domain motion suppression, chroma and phase-driven frequency domain calibration, and efficient Transformer modeling becomes a key path to achieving efficient training and high-fidelity reconstruction.
[0003] Existing HDR ghosting removal algorithms mainly include motion suppression methods, image registration methods, and deep learning-based methods. Although these methods have improved the ghosting removal effect to some extent, they still face many challenges, especially in handling complex dynamic scenes and extreme exposure differences, where they suffer from low efficiency, high computational cost, and insufficient accuracy.
[0004] Motion suppression methods reduce ghosting artifacts by globally aligning multiple frames and removing misaligned pixels. However, these methods often fail in cases of large-scale motion and dynamic foreground objects, and pixel removal leads to the loss of useful information, thus affecting the quality of the final HDR image. Furthermore, these methods are computationally complex, especially in high dynamic range scenes, resulting in low computational efficiency and slow processing speed.
[0005] Image registration methods align LDR images to reference images using optical flow or other image transformation techniques before fusion. This approach performs well in static scenes, but its registration accuracy is often insufficient under conditions of large displacement, occlusion, or strong saturation, easily producing noticeable ghosting artifacts. More importantly, image registration relies on accurate motion estimation, resulting in high computational costs and sensitivity to noise and dynamic changes.
[0006] Deep learning-based methods, especially convolutional neural networks (CNNs), have shown great potential in image feature extraction and fusion. While these methods can remove ghosting artifacts by automatically learning features, their limited local receptive fields restrict their ability to model long-range dependencies, resulting in an inability to effectively recover details when faced with large displacements, complex motions, and strong exposure differences. Furthermore, CNN methods typically rely on large amounts of labeled data for training, while existing HDR ghosting datasets are relatively limited, making models susceptible to insufficient data and overfitting during training.
[0007] Among these traditional methods, Transformer-based methods, especially the VisionTransformer (ViT) variant, have been proposed as a novel solution to capture long-range dependencies using its powerful global modeling capabilities. However, the standard ViT model suffers from high computational overhead because the computational complexity of its self-attention mechanism is quadratic with the image size; as the image size increases, the computational and storage requirements rise sharply. Although SwinTransformer introduces a windowed multi-head self-attention mechanism to alleviate this problem, this method still faces some challenges: the lack of information interaction between windows leads to blocky artifacts during image reconstruction; at the same time, the fixed-size image partitioning sacrifices accurate capture of details, especially when dealing with strong exposure differences, and still cannot guarantee high-quality image restoration. Summary of the Invention
[0008] To address the limitations of existing methods in terms of computational overhead, detail recovery, and long-range dependency modeling, this invention aims to provide a cross-color gamut phase-guided dual-attention fusion HDR imaging method. By designing a context-aware Transformer (CA-ViT) module and combining global and local information fusion, the method improves ghosting removal accuracy and computational efficiency. Simultaneously, the introduction of an adaptive motion sensing and suppression (MPM) mechanism and frequency-domain phase fusion with chromaticity stability effectively enhances image detail recovery. By designing a joint loss function, the training process is optimized, enabling the model to train stably on small datasets and generalize effectively. By providing an efficient, accurate, and resource-efficient HDR ghosting removal algorithm, this invention can operate stably in dynamic scenes and under extreme exposure conditions, promoting the practical application of HDR imaging technology.
[0009] To achieve the above objectives, the technical solution adopted by the present invention is as follows: A cross-color gamut phase-guided dual-attention fusion HDR imaging method includes the following steps; Step (1): Data preprocessing for denoising and cleaning the original multiple exposure image sequence; Step (2): Load and transform the preprocessed data to obtain standardized image data; Step (3): The standardized image data is mapped from nonlinear pixel values to linear physical illumination values by applying the camera response curve in reverse, and a linearized multi-frame image is output. Step (4): The linearized multi-frame images are stitched together along the channel dimension to construct a unified input tensor containing complete dynamic range information; Step (5): Perform a convolution operation on the input tensor to extract shallow features of local texture and edges of the image; Step (6): Based on the shallow features, a dual attention mechanism is introduced to perform interaction and aggregation of global features, and output the fused high dynamic range feature map.
[0010] The specific steps (1) are as follows: (1.1) Image cropping Multiple exposure LDR image sequences, covering a brightness range from underexposed to overexposed, are acquired through fixed-position shooting or simulation and used as input data for the network. Simultaneously, the camera response function (CRF) is used to correct the backlight response of the multiple exposure LDR image sequences, mapping them to the linear brightness domain and performing weighted fusion to generate HDR images with physically accurate lighting information as ground truth labels. The original scene's LDR image and the corresponding HDR ground truth are cropped as follows: The image blocks are divided into pixel blocks to obtain the original LDR image blocks and HDR image blocks; to ensure the continuity of image content, overlapping cropping is performed using a step size of 64 pixels; (1.2) Data Augmentation For each cropped image patch, data augmentation techniques, such as rotation and flipping, are applied to increase the diversity of training samples and enhance the generalization ability of the model; enhanced versions of LDR and HDR image patches are obtained. After the above processing, the original image patch and its enhanced image patch are saved as new independent samples, which together constitute the training set of the network. Step (2) specifically involves: Before loading the training set into the model for training, read three LDR image patches: underexposed. Normal exposure is also called a reference frame. and overexposure Normalized to scope: Simultaneously read the corresponding exposure time , , .
[0011] Step (3) specifically involves: the pixel values of an LDR image are typically nonlinear values that have undergone camera response function and Gamma correction; in order to recover the true irradiance information, the pixel values of the LDR image are linearized by applying inverse Gamma correction and dividing the result by the exposure time. To approximate the implementation: in This is the linearized HDR irradiance map. It is the input LDR image. =2.2, It is a minimum value 1e -8 To prevent division by zero errors.
[0012] Step (4) specifically involves: enabling the network to simultaneously acquire the input LDR image. Non-linear image details and linearized HDR irradiance map information The two are then spliced together along the channel dimension; This constitutes the actual three LDR images input to the network model. : At this point, each input consists of three frames of LDR images. Both are 6-channel tensors, that is .
[0013] The specific steps (5) are as follows: (5.1) Feature domain motion suppression: To suppress inconsistencies caused by camera / object motion or occlusion, this invention designs a motion-aware pre-modulation module (MPM) to provide motion artifact-free features for subsequent cross-exposure consistency calibration and depth fusion. Compared to pixel-level registration, this module performs attention calculations only in the feature domain, requires no optical flow, is more robust to large displacements, occlusions, and saturation, and has controllable computational overhead.
[0014] Three LDR images Convolution with 3x3 kernel After shallow feature extraction, the following was obtained: in This represents the shallow RGB feature map obtained after shallow convolution of the i-th frame of the input image. H and W represent the spatial height and width of the feature map, respectively, and their dimensions are usually consistent with the spatial dimensions of the input image. C represents the channel dimension of the feature map, i.e., the number of convolution kernels, and i represents the frame index of the multi-exposure image sequence, with a value ranging from 1 to 2. These correspond to short exposure, medium exposure (as reference frames), and long exposure images, respectively. Using the reference frame, the MPM performs motion modeling and suppression on the RGB path; In windowed self-attention, Structural self-consistency enhancement is performed to obtain refined features. The calculation process is as follows: in These respectively represent the features of the reference frame. The query matrix, key matrix, and value matrix generated by linear projection. These represent the learnable linear projection weight matrices corresponding to the generated query, key, and value matrices, respectively. This represents the calculated attention map, used to measure long-distance dependencies between pixels within a feature. The feature dimension of the key vector. These are learnable weights, which are assigned values according to the initialization strategy at the beginning of training and then learned end-to-end through backpropagation. against ,by and Perform cross-frame correlation calculations to generate a spatial attention map. And accordingly, suppress movement and occlusion areas: in This represents the spatial attention map generated for the i-th non-reference frame. This represents the function for calculating cross-frame correlation. This represents the refined features of the i-th frame after soft suppression processing. To perform element-wise multiplication, soft suppression of inconsistencies in non-reference frames is achieved, avoiding information loss caused by pixel-level hard culling and preserving reusable static details. (5.2) Cross-exposure feature consistency calibration: To address the inconsistencies in cross-frame and cross-channel features and structural mismatches caused by overexposure saturation and underexposure noise during multiple exposures, a Chromatic Anchored Structure Calibration (CASC) module is proposed. This module uses the chromaticity stability of YCbCr as an anchor and performs cross-color gamut phase-guided structure injection (PSI) in the frequency domain. After injecting robust structural phase into both the RGB and YCbCr amplitude channels, the results are returned to the spatial domain, producing consistent and structurally robust features across exposures, providing a reliable characterization for subsequent fusion and reconstruction. Reference Frame Structural consistency enhancement has already been implemented in the MPM module, which acts as a structural anchor. Further frequency-domain phase rewriting of this anchor would disrupt the high-confidence structure and increase computational overhead. Therefore, the CASC module only handles underexposed inputs. With overexposure input Execution in two frames.
[0015] and The operating steps are the same, for Transformation from RGB domain to YCbCr domain: in For brightness, it carries the vast majority of structural and edge information. , The chromaticity difference component describes color changes and is less sensitive to structure. Since structural consistency primarily depends on the luminance domain, to prevent the introduction of color shift and false color that weaken the stability of chromaticity anchoring, luminance extension is applied to Y: Obtain the YCbCr tensor .
[0016] The overall structure of the CASC module involves first extracting shallow features from the YCbCr branch, and then performing average pooling to suppress noise while preserving chromaticity stability. in This represents the shallow feature map of the YCbCr domain obtained after channel concatenation and convolution operations on the input data. Indicates the kernel size as Convolution operation, This indicates a concatenation operation along the channel dimension. This represents the high dynamic range luminance component after linearization. This indicates the average pooling operation. This represents the smoothed YCbCr feature map obtained after average pooling. Next, perform max pooling on the output of the MPM module mentioned earlier in the RGB branch to preserve structural cues near extreme or saturation values: The two-branch aggregated features are mapped to the frequency domain using a fast Fourier transform, decomposing them into amplitude and phase spectra: in These represent the spatial domain aggregated features of the RGB branch and the YCbCr branch after pre-pooling processing, respectively. This represents the complex frequency domain feature representation obtained after the above features have undergone a Fast Fourier Transform (FFT). and For the amplitude spectrum of the corresponding domain, and This represents the phase spectrum of the corresponding domain. Represents the Fast Fourier Transform; Lightweight, learnable point convolutions are applied to both the amplitude and phase: in This represents the refined amplitude spectrum after processing by the mapping network. This represents the refined phase spectrum after processing by the mapping network. Two-story Convolution + LeakyReLU+ Convolution is used to suppress spectral noise and adjust the scale to improve the stability of the subsequent link phase fusion. A unified phase structure is obtained by performing circular phase addition on the complex plane. : in This represents the unified phase structure after cross-branch fusion. For phase normalization operators, Argument calculation; Next, the two phase paths are replaced with a unified phase, and the spatial domain features are reconstructed: in Indicates the use of unified phase The spatial domain enhancement features are reconstructed by replacing the original phase and combining them with their respective refined amplitude spectra, and then subjected to inverse fast Fourier transform (IFFT). The stable structure of the YCbCr domain is introduced across the color gamut into the RGB domain representation, while preserving their respective amplitude energy distributions, resulting in two features consistent across the exposure structure. Finally, to further align fine-grained semantics in the spatial domain, a symmetrical cross-attention mechanism is implemented, followed by a feedforward network (FFN) and residual connections. From the RGB branch to the YCbCr branch, where For Query, For Key and Value: in These represent the spatial domain input features of the RGB branch and the YCbCr branch, respectively, reconstructed by the IFFT module. These represent the query vector, key vector, and value vector generated in the attention calculation path from RGB to YCbCr, respectively. This represents the cross-gamut cross-attention map calculated by querying the YCbCr branch from the RGB branch. This represents the cross-gamut complementary feature residuals after linear projection. This represents the intermediate RGB characteristics after cross-color gamut interaction. This represents the refined features of the first frame (short exposure) after bidirectional cross-attention interaction and feedforward network processing. DWConv is a depthwise separable convolution. For learnable temperature parameters, Proj is Linear mapping; FFN employs a gated depthwise convolutional feedforward network. From the YCbCr branch to the RGB branch, where For Query, For Key and Value: The refined features from the two branches are concatenated along the channel dimension to obtain the CASC module output of the non-reference frame: Overexposed frames Obtained by the same steps Subsequently , Anchoring features with reference frame Multi-frame aggregated feature maps are obtained by concatenating the channels. : Spatial features are divided into non-overlapping patches and linearly mapped to tokens using PatchEmbedding: Where L represents the length of the token sequence. This represents the PatchEmbedding mapping function. This represents the initial Transformer input sequence generated after embedding processing, where P is the patch size and D is the embedding dimension.
[0017] The specific steps (6) are as follows: A dual attention fusion network (DAFN) is used as the backend fusion and reconstruction module; The backend fusion and reconstruction module consists of multiple dual-attention refinement modules (DARB) connected in series, and uses residual connections: The DARB module consists of two parts. First, the left-hand Global Context Aggregation (GCA) module establishes a consistent scene prior, and then the right-hand Context-Guided Semantic Fusion (CGSF) module completes the final fine-grained fusion. The specific steps are as follows: The GCA module adopts a similar structure to the standard SwinTransformer block and retains a window-based self-attention mechanism to obtain a globally consistent context, and then uses MLP to improve representation capabilities: Where Z represents the sequence of visual tokens input to the current GCA module. The representation is normalized at the layer level, and U represents the intermediate transition feature after attention mechanism processing and residual connection. This represents a multilayer perceptron. This represents the global context features obtained after processing by the basic Transformer block of the GCA module. W-MSA computes attention on a sliding window, which has linear complexity and indirectly establishes cross-window dependencies. In global context Based on this, the bidirectional complementarity of the three-exposure semantics is explicitly modeled, first by segmenting according to channels to obtain... Corresponding to underexposure, reference frame, and overexposure, and linear projection: in For learnable projection, d represents independent projection for each head in the case of multiple heads. Let represent the query vector, key vector, and value vector generated by the linear projection of the features of the i-th frame, respectively. Perform cross attention on bidirectional paths: Query vector of reference branch As a condition, for non-reference branches key-value pairs Perform conditional information aggregation: in For learnable temperature parameters, This represents the information aggregation feature from non-reference frames to reference frames. The query vector representing the reference frame. Represents the key vector of a non-reference frame. Represents the value vector of a non-reference frame; The non-reference channel is calibrated by the reference channel to align its semantic coordinates with the intensity response: in For learnable temperature parameters, This represents the feature calibration representation from the reference frame to the non-reference frame. The query vector represents a non-reference frame. The key vector representing the reference frame. Represents the value vector of the reference frame; The non-reference channel is calibrated by the reference channel to align its semantic coordinates with the intensity response: in This is another set of learnable temperature parameters. Its function is to anchor the feature representations of underexposed and overexposed channels to the stable structure of the reference channel, explicitly eliminating semantic mismatches and response drift across exposures; Finally, the four outputs are concatenated and output via MLP: in This represents the cross-group spatial fusion feature output after deep fusion via a multilayer perceptron (MLP). This represents the information aggregation feature from non-reference frames to reference frames. This represents the feature calibration representation from the reference frame to the non-reference frame; After passing through multiple DARB modules, the image is returned to the spatial domain via Patchun Embedding and then reconstructed by the reconstruction head to obtain the HDR image. in For PatchunEmbedding, For channel-aligned global residual connections, Constrain the result to Sigmoid , This represents the final deep visual token sequence output after processing by N cascaded DARB modules. This represents a pre-reconstructed feature map that integrates deep semantics and shallow details. This represents the final generated high dynamic range (HDR) predicted image.
[0018] The beneficial effects of this invention are: To address the issues of lost effective information and easy failure of optical flow under large displacement / occlusion / saturation in traditional "pre-registration + pixel culling", the MPM feature domain motion suppression of this invention uses the reference frame as an anchor and generates a suppression map through differential attention to perform soft suppression on non-reference frames, completing alignment and purification in the feature domain, avoiding the loss of detail caused by hard culling and significantly reducing ghosting.
[0019] Compared to existing CNN / Transformer methods that only align within the same color gamut, the CASC (Chronic Anchored Structure Calibration) proposed in this invention performs frequency domain processing on both RGB and YCbCr branches. By introducing a cross-color gamut phase structure, the robust structural phase of the YCbCr path is recombined with the amplitude of the two paths and returned to the spatial domain. This unifies the structural coordinate system, maintains energy fidelity, effectively solves the semantic differences caused by overexposure saturation and underexposure noise, and significantly improves the reliability of cross-frame texture transfer.
[0020] Existing simple splicing + convolution or single-path attention methods are prone to undirected mixing. The DAFN module of this invention first establishes a globally consistent prior using GCA within the shift window, and then uses CGSF to achieve semantic complementarity and alignment between reference and non-reference frames through reference-domain conditional aggregation and bidirectional cross-attention orientation with non-reference calibration. This concatenation significantly reduces residual artifacts and structural drift in extreme exposure areas, resulting in more stable detail reconstruction.
[0021] Compared to global self-attention or frequency domain processing on all three frames: windowed attention reduces attention computation from quadratic complexity to near linear complexity; CASC only applies to two non-reference frames, reducing frequency domain operators and memory usage by approximately 30%; shared KV branch + depthwise separable convolution (DWConv) + 1×1 projection reduces parameters and computation in cross-path refinement and DAFN; channel-level single-entry stitching combined with global and block-level residuals ensures stable training and fast convergence. Therefore, it maintains excellent speed, memory, and parameter efficiency even with multi-module stacking, facilitating deployment on mobile devices.
[0022] Compared to schemes based on optical flow, homography, or only local convolution, MPM's feature domain soft suppression combined with DAFN's global-semantic concatenation alignment enables the model to obtain a ghost-free and structurally consistent fusion substrate even under long-range displacement, missing regions caused by occlusion, strong saturation, and deep darkness. Attached Figure Description
[0023] Figure 1 This is a schematic diagram of the overall structure of a three-stage HDR imaging network.
[0024] Figure 2 This is a schematic diagram of a motion sensing premodulation module (MPM).
[0025] Figure 3 This is a schematic diagram of the Color Anchoring Structure Calibration Module (CASC).
[0026] Figure 4 This is a schematic diagram of cross-gamut phase injection (PSI).
[0027] Figure 5 This is a schematic diagram of a Dual Attention Fusion Network (DAFN). Detailed Implementation
[0028] The present invention will now be described in further detail with reference to the accompanying drawings.
[0029] A cross-color gamut phase-guided dual-attention fusion HDR imaging method includes the following steps; The HDR imaging network proposed in this invention adopts an end-to-end encoder-decoder architecture, combining an attention mechanism and a multi-scale feature fusion strategy. Its core idea is to utilize a multi-branch structure to process the input low dynamic range image sequence in parallel, extracting motion, structural, and color information separately. Finally, a powerful dual-attention fusion network is used for deep fusion. The network framework is as follows: Figure 1 The overall network architecture includes the following steps: (1) Data preprocessing (1.1) Image cropping The original LDR image and the corresponding HDR ground truth image of the scene were cropped to Image blocks of pixels. To ensure the continuity of image content, overlapping cropping is performed using a stride of 64 pixels.
[0030] (1.2) Data Augmentation For each cropped image patch, data augmentation techniques, such as rotation and flipping, are used to increase the diversity of training samples and enhance the model's generalization ability.
[0031] After the above processing, the original image patch and its enhanced version are saved as new independent samples, which together constitute the training set of the network.
[0032] (2) Data loading and transformation Before loading the dataset into the model for training, read three LDR image patches: underexposed. Normal exposure is also called a reference frame. and overexposure Normalized to scope: Simultaneously read the corresponding exposure time , , .
[0033] (3) Linearization Pixel values in LDR images are typically non-linear values adjusted for camera response function and gamma correction. To recover true irradiance information, these values are linearized. This is done by applying inverse gamma correction and dividing the result by the exposure time. To approximate the implementation: in This is the linearized HDR irradiance map. It is the input LDR image. =2.2, It is a minimum value 1e -8 To prevent division by zero errors.
[0034] (4) Channel splicing In order for the network to simultaneously capture the original nonlinear image details and linearized irradiance information The two are then spliced together along the channel dimension.
[0035] This constitutes the actual input to the network model. : At this point, each input Both are 6-channel tensors, that is .
[0036] (5) Shallow feature extraction (5.1) Feature Domain Motion Inhibition To suppress inconsistencies caused by camera / object motion or occlusion, this invention designs a motion-aware pre-modulation module (MPM) to provide motion artifact-free features for subsequent cross-exposure consistency calibration and depth fusion. Compared to pixel-level registration, this module performs attention calculations only in the feature domain, requires no optical flow, is more robust to large displacements, occlusions, and saturation, and has controllable computational overhead.
[0037] like Figure 2 As shown, three LDR images Convolution with 3x3 kernel After shallow feature extraction, the following was obtained: in Using the reference frame, MPM performs motion modeling and suppression on the RGB path.
[0038] In windowed self-attention, Structural self-consistency enhancement is performed to obtain refined features. The calculation process is as follows: in These are learnable weights, assigned values according to the initialization strategy at the start of training, and then learned end-to-end through backpropagation.
[0039] against ,by and Perform cross-frame correlation calculations to generate a spatial attention map. And accordingly, suppress movement and occlusion areas: in To perform element-wise multiplication, inconsistencies in non-reference frames are softly suppressed, avoiding information loss caused by pixel-level hard culling and preserving reusable static details.
[0040] (5.2) Cross-exposure feature consistency calibration To address the inconsistencies in cross-frame and cross-channel features and structural mismatches caused by overexposure saturation and underexposure noise during multiple exposures, a Chromatic Anchored Structure Calibration (CASC) module is proposed. This module uses the chromaticity stability of YCbCr as an anchor and performs cross-color gamut phase-guided structure injection (PSI) in the frequency domain. After injecting robust structural phase into both the RGB and YCbCr amplitude channels, the results are returned to the spatial domain, producing consistent and structurally robust features across exposures, providing a reliable characterization for subsequent fusion and reconstruction. Reference Frame Structural consistency enhancement has already been implemented in the MPM module, which acts as a structural anchor. Further frequency-domain phase rewriting of this anchor would disrupt the high-confidence structure and increase computational overhead. Therefore, the CASC module only handles underexposed inputs. With overexposure input Execution in two frames.
[0041] because and The operation steps are the same, so we will use the following example: For example, Figure 3 right Transformation from RGB domain to YCbCr domain: in For brightness, it carries the vast majority of structural and edge information. , The chromaticity difference component describes color changes and is less sensitive to structure.
[0042] Since structural consistency primarily depends on the luminance domain, to prevent the introduction of color shifts and false colors that weaken the stability of chromaticity anchoring, luminance extension is applied to Y: Obtain the YCbCr tensor .
[0043] like Figure 3 The overall structure of the CASC module is shown. First, shallow features are extracted from the YCbCr branch. Then, to suppress noise while preserving chromaticity stability, average pooling is performed. Next, perform max pooling on the output of the MPM module mentioned earlier in the RGB branch to preserve structural cues near extreme or saturation values: Next, as follows Figure 4 The PSI structure diagram maps the two-branch convergence features to the frequency domain using a fast Fourier transform, decomposing them into amplitude and phase spectra: in and For the amplitude spectrum of the corresponding domain, and This represents the phase spectrum of the corresponding domain.
[0044] To suppress frequency domain noise and improve numerical stability, lightweight, learnable point convolutions are applied to both the amplitude and phase: in Two-story Convolution + LeakyReLU+ Convolution is mainly used to suppress spectral noise and adjust the scale to improve the stability of post-link phase fusion.
[0045] To avoid linear addition For periodic transitions, circular phase addition is performed on the complex plane to obtain a unified phase structure. : in For phase normalization operators, This is for argument calculation.
[0046] Next, the two phase paths are replaced with a unified phase, and the spatial domain features are reconstructed: The stable structure of the YCbCr domain is introduced across the color gamut into the RGB domain representation, while preserving their respective amplitude energy distributions, resulting in two features consistent across the exposure structure.
[0047] Finally, to further align fine-grained semantics in the spatial domain, a symmetrical cross-attention mechanism is implemented, followed by a feedforward network (FFN) and residual connections.
[0048] From the RGB branch to the YCbCr branch, where For Query, For Key and Value: Where DWConv is a depthwise separable convolution. For learnable temperature parameters, Proj is For linear mapping, FFN employs a gated depthwise convolutional feedforward network.
[0049] From the YCbCr branch to the RGB branch, where For Query, For Key and Value: The refined features from the two branches are concatenated along the channel dimension to obtain the CASC module output of the non-reference frame: Overexposed frames Obtained by the same steps Subsequently , Anchoring features with reference frame In channel splicing: Spatial features are divided into non-overlapping patches and linearly mapped to tokens using PatchEmbedding: Where P is the patch size and D is the embedding dimension.
[0050] (6) Global feature fusion After completing motion suppression (MPM) and cross-exposure consistency calibration (CASC), the three features There are still problems such as insufficient global structural consistency, semantic loss in extreme exposure areas, and oversmoothing of detail reconstruction; simple cascading or single-path attention fusion is prone to residual artifacts and texture weakening, and the computational cost is relatively high. In order to solve the above problems, this invention designs a dual attention fusion network DAFN as a back-end fusion and reconstruction module.
[0051] This module consists of multiple dual-attention refinement modules (DARB) connected in series, and uses residual connections: like Figure 5 The DARB module shown consists of two parts. First, the Global Context Aggregation (GCA) module on the left establishes a consistent scene prior. Then, the Context-Guided Semantic Fusion (CGSF) module on the right completes the final fine-grained fusion. The specific steps are as follows: The GCA module adopts a similar structure to the standard SwinTransformer block and retains a window-based self-attention mechanism to obtain a globally consistent context, and then uses MLP to improve representation capabilities: W-MSA computes attention on a sliding window, achieving linear complexity and indirectly establishing cross-window dependencies.
[0052] In global context Based on this, we explicitly model the bidirectional complementarity of the three-exposure semantics. First, we obtain the results by channel segmentation. Corresponding to underexposure, reference frame, and overexposure, and linear projection: in For learnable projection, d represents independent projection for each head when there are multiple heads.
[0053] Perform cross attention on bidirectional paths: Query vector of reference branch As a condition, for non-reference branches key-value pairs Perform conditional information aggregation: in This is a learnable temperature parameter. Its function is to allow a stable reference frame to selectively aggregate reliable details and edge textures in underexposed and overexposed channels under attention guidance, in order to fill in missing information in extreme areas of the reference frame.
[0054] The non-reference channel is calibrated by the reference channel to align its semantic coordinates with the intensity response: in This is a learnable temperature parameter. Its function is to allow a stable reference frame to selectively aggregate reliable details and edge textures in underexposed and overexposed channels under attention guidance, in order to fill in missing information in extreme areas of the reference frame.
[0055] The non-reference channel is calibrated by the reference channel to align its semantic coordinates with the intensity response: in This is another set of learnable temperature parameters. Its function is to anchor the feature representations of underexposed and overexposed channels to the stable structure of the reference channel, explicitly eliminating semantic mismatches and response drift across exposures.
[0056] Finally, the four outputs are concatenated and output via MLP: After passing through multiple DARB modules, the image is returned to the spatial domain via Patchun Embedding and then reconstructed by the reconstruction head to obtain the HDR image. in For PatchunEmbedding, For channel-aligned global residual connections, Constrain the result to Sigmoid .
[0057] This invention innovatively proposes a three-stage end-to-end network solution framework for multiple exposures in the field of HDR imaging: first, motion suppression and structure anchoring (MPM) are completed in the feature domain; then, structural consistency calibration (CASC) of non-reference frames is completed by combining cross-color and frequency domains; and finally, fine-grained semantic reconstruction and pixel-level reconstruction under global consistency constraints are completed in the dual attention fusion network (DAFN).
[0058] Structural self-consistency enhancement is performed on the reference frame, and difference attention is computed on non-reference frames conditioned on the reference features to generate spatial suppression maps. It provides soft suppression (element-by-element modulation) of motion and occlusion areas, thereby robustly handling large displacements, occlusions and saturation without relying on optical flow or pixel registration.
[0059] To address the inconsistencies and structural mismatches in cross-frame and cross-channel features caused by overexposure saturation and underexposure noise, a novel approach is proposed: after color gamut conversion, phase unification and injection are performed in the frequency domain. Non-reference frames are decomposed into amplitude / phase in both RGB and YCbCr branches using FFT. A unified structural phase is obtained by circumferential phase fusion and then recombined with the two amplitude channels using IFFT back to the spatial domain, achieving cross-exposure calibration with structural alignment and energy fidelity.
[0060] This invention proposes a sequential dual-attention fusion approach, prioritizing global context over semantic context: a stable, globally consistent context is first constructed during the Global Consistency (GCA) stage, and then fine-grained semantic alignment and completion are achieved during the Global Cross-Signal (CGSF) stage using directional cross-attention between reference and non-reference frames. This sequential approach not only constrains the search space and improves stability but also enables cross-window global modeling with near-linear complexity.
[0061] Under the premise of multi-module serialization, this invention maintains high efficiency and deployability through systematic design: windowed attention is used to reduce self-attention from quadratic complexity to near linear, significantly suppressing the growth of computing power and memory; CASC only operates on two non-reference frames, avoiding repeated frequency domain processing of reference frames, reducing overall frequency domain operator calls and memory usage by about 30%; in cross-path refinement, a shared KV branch is introduced in combination with depthwise separable convolution (DWConv) and 1×1 projection, which greatly compresses the number of parameters while ensuring information transmission; the entire network adopts a stable training path of channel-level single-entry splicing, global residual and block-level residual, which accelerates convergence and facilitates deployment on edge devices.
[0062] To verify the effectiveness of the cross-color gamut phase-guided dual-attention fusion HDR imaging method proposed in this invention, a set of typical scenes with large dynamic range and local motion were selected for simulation experiments: Input configuration: The input is a sequence of three LDR images with different exposures captured in the same scene. The resolutions are all 1500(H)×1000(W)×3, corresponding to underexposure, normal exposure (reference frame), and overexposure, respectively, and the corresponding exposure times are read. , , .
[0063] Processing steps: Step (1): The input 1500×1000 original LDR image is divided into blocks. To adapt to the network input and ensure the continuity of image content, a sliding window mechanism is used to crop the high-resolution image. The cropping window size is set to... With a sliding window step size of 64 pixels (i.e., an overlap rate of 50%), 345 LDR image patch sequences were obtained, each containing three images of size [missing information]. Image blocks ( ), serving as the benchmark unit for subsequent single-step reasoning; Step (2): Take the three LDR image blocks output in step (1), with pixel values ranging from... The pixel values are loaded as input into the computation unit and normalized to... The floating-point range. Simultaneously read the exposure times corresponding to three images. , , Output the normalized LDR image patch tensor ; Step (3): Input the normalized LDR image block and their corresponding exposure time According to the formula Perform reverse photometric response correction, where the parameters are strictly set as follows: =2.2, Outputs a linear HDR irradiance patch that restores physical illumination information. ; Step (4): The nonlinear LDR image block from step (2) With the linear HDR image patch of step (3) For each frame of image, its LDR image block and HDR image block are concatenated along the channel dimension. , , The completed network input tensor , , The size of each tensor becomes 6; Step (5): Input the three steps from step (4) 6 , , ,go through Convolutional layers extract shallow features For non-reference features, a color space conversion is performed using a color gamut transformation matrix: According to the formula The luminance component is processed, where The feature map is subjected to FFT transformation. After phase unification and injection operations are completed in the frequency domain, it is reconstructed by IFFT. Finally, the processed features are concatenated with the features of the reference frame in the channel dimension to obtain a multi-channel aggregated feature map containing information from multiple frames. The space size remains as ; Step (6): Input the aggregated feature map output from step (5) ,Will The spatial features are divided into non-overlapping patches (patch size P) and mapped to a token sequence. The token sequence is sequentially processed by the Global Context Aggregation (GCA) module and the Context-Guided Semantic Fusion (CGSF) module to complete global dependency modeling and bidirectional semantic alignment. The Patchun Embedding operation restores the token sequence to spatial features, which are then processed by the terminal convolutional layer and the Sigmoid activation function. Perform pixel-level reconstruction, outputting a single image with a resolution of [resolution value missing]. HDR image patches; Finally, output all of the steps in (6). Following the sliding window positions recorded in step (1), all HDR image blocks are stitched back to their original resolution. For overlapping areas, a weighted average is used to eliminate stitching artifacts, resulting in a complete image with a resolution of [resolution value missing]. The high dynamic range (HDR) image was processed with ghosting removal, lighting reconstruction, and tone mapping.
Claims
1. A cross-color gamut phase-guided dual-attention fusion HDR imaging method, characterized in that, Includes the following steps; Step (1): Data preprocessing for denoising and cleaning the original multiple exposure LDR image sequence; Step (2): Load and transform the preprocessed data to obtain standardized image data; Step (3): The standardized image data is mapped from nonlinear pixel values to linear physical illumination values by applying the camera response curve in reverse, and a linearized multi-frame image is output. Step (4): The linearized multi-frame images are stitched together along the channel dimension to construct a unified input tensor containing complete dynamic range information; Step (5): Perform convolution operation on the input tensor to extract shallow feature representations of local texture and edges of the image; Step (6): Based on the shallow features, a dual attention mechanism is introduced to perform interaction and aggregation of global features, and output the fused high dynamic range feature map.
2. The cross-color gamut phase-guided dual-attention fusion HDR imaging method according to claim 1, characterized in that, The specific steps (1) are as follows: (1.1) A multi-exposure LDR image sequence is obtained by shooting from a fixed position or by simulation, covering the brightness range from underexposure to overexposure, and used as the input data of the network; at the same time, the camera response function CRF is used to perform backlight response correction on the multi-exposure LDR image sequence, map it to the linear brightness domain and perform weighted fusion to generate an HDR image with physical real illumination information as the ground truth label. The original LDR image and the corresponding HDR ground truth are cropped into image patches to obtain the original LDR image patch and HDR image patch; (1.2) For each cropped image patch, data augmentation techniques are used to increase the diversity of training samples and obtain enhanced versions of LDR and HDR image patches; The original image patches and their augmented image patches together constitute the training set of the network.
3. The cross-color gamut phase-guided dual-attention fusion HDR imaging method according to claim 2, characterized in that, Step (2) specifically involves: Before loading the training set into the model for training, read three LDR image patches: underexposed. Normal exposure and overexposure and normalized to scope: Simultaneously read the corresponding exposure time , , Normal exposure It is used as a reference frame.
4. The cross-color gamut phase-guided dual-attention fusion HDR imaging method according to claim 3, characterized in that, The specific steps (3) are as follows: The pixel values of an LDR image are non-linear values that have undergone camera response function and Gamma correction; linearizing the pixel values of an LDR image is achieved by applying inverse Gamma correction and dividing the result by the exposure time. To approximate the implementation: in This is the linearized HDR irradiance map. It is the input LDR image. It is a minimum value 1e -8 This is to prevent division by zero errors.
5. The cross-color gamut phase-guided dual-attention fusion HDR imaging method according to claim 4, characterized in that, Step (4) specifically involves: LDR image Non-linear image detail and linearized HDR irradiance map The components are stitched together along the channel dimension. The three LDR images that constitute the actual input to the network model : At this point, each input consists of three LDR images. Both are 6-channel tensors, that is .
6. The cross-color gamut phase-guided dual-attention fusion HDR imaging method according to claim 5, characterized in that, Step (5) specifically involves: (5.1) Three LDR images Convolution with 3x3 kernel After shallow feature extraction, the following is obtained: in Let represent the shallow RGB feature map obtained after shallow convolution of the input image of frame i. H and W represent the spatial height and width of the feature map, respectively. C represents the channel dimension of the feature map, i.e., the number of convolution kernels. i represents the frame index of the multiple exposure image sequence, with a value ranging from 1 to 2. These correspond to short exposure, medium exposure, and long exposure images, respectively. MPM completes motion modeling and suppression in the RGB path. In windowed self-attention, Structural self-consistency enhancement is performed to obtain refined features. ; (5.2) The CASC module only handles underexposed inputs. With overexposure input Execution in two frames; and The operating steps are the same, for Transformation from RGB domain to YCbCr domain: in For brightness, it carries the vast majority of structural and edge information. , The chromaticity difference component describes color changes and is less sensitive to structure. Luminance expansion of Y: Obtain the YCbCr tensor ; The overall structure of the CASC module involves first extracting shallow features from the YCbCr branch, followed by average pooling. in This represents the shallow feature map of the YCbCr domain obtained after channel concatenation and convolution of the input data. Indicates the kernel size as Convolution operation, This indicates a concatenation operation along the channel dimension. This represents the high dynamic range luminance component after linearization. This indicates the average pooling operation. This represents the smoothed YCbCr feature map obtained after average pooling. Next, perform max pooling on the output of the MPM module mentioned earlier in the RGB branch to preserve structural cues near extreme or saturation values: The two-branch aggregated features are mapped to the frequency domain using a fast Fourier transform, decomposing them into amplitude and phase spectra: in These represent the spatial domain aggregated features of the RGB branch and the YCbCr branch after pre-pooling processing, respectively. This represents the complex frequency domain feature representation obtained after the above features have undergone a Fast Fourier Transform. and For the amplitude spectrum of the corresponding domain, and This represents the phase spectrum of the corresponding domain. Represents the Fast Fourier Transform; Lightweight, learnable point convolutions are applied to both the amplitude and phase: in This represents the refined amplitude spectrum after processing by the mapping network. This represents the refined phase spectrum after processing by the mapping network. Two-story Convolution + LeakyReLU+ convolution; A unified phase structure is obtained by performing circular phase addition on the complex plane. : in This represents the unified phase structure after cross-branch fusion. For phase normalization operators, Argument calculation; Replace the two phase paths with a unified phase and reconstruct the spatial domain features: in Indicates the use of unified phase The spatial domain enhancement features are reconstructed by replacing the original phase and combining them with their respective refined amplitude spectra, and then undergoing inverse fast Fourier transform. The stable structure of the YCbCr domain is introduced across the color gamut into the RGB domain representation, while preserving their respective amplitude energy distributions, resulting in two features consistent across the exposure structure. Finally, to further align fine-grained semantics in the spatial domain, a symmetrical cross-attention mechanism is implemented, followed by a feedforward network (FFN) and residual connections. From the RGB branch to the YCbCr branch, where For Query, For Key and Value: in These represent the spatial domain input features of the RGB branch and the YCbCr branch, respectively, reconstructed by the IFFT module. These represent the query vector, key vector, and value vector generated in the attention calculation path from RGB to YCbCr, respectively. This represents the cross-gamut cross-attention map calculated by querying the YCbCr branch from the RGB branch. This represents the cross-gamut complementary feature residuals after linear projection. This represents the intermediate RGB characteristics after cross-color gamut interaction. This represents the refined features of the first frame after bidirectional cross-attention interaction and feedforward network processing. DWConv is a depthwise separable convolution. For learnable temperature parameters, Proj is For linear mapping, FFN employs a gated depthwise convolutional feedforward network.
7. The cross-color gamut phase-guided dual-attention fusion HDR imaging method according to claim 6, characterized in that, From the YCbCr branch to the RGB branch, where For Query, For Key and Value: The refined features from the two branches are concatenated along the channel dimension to obtain the CASC module output of the non-reference frame: Overexposed frames Obtained by the same steps Subsequently , Anchoring features with reference frame Multi-frame aggregated feature maps are obtained by concatenating the channel dimensions. : Spatial features are divided into non-overlapping patches and linearly mapped to tokens using PatchEmbedding: Where L represents the length of the token sequence. This represents the PatchEmbedding mapping function. This represents the initial Transformer input sequence generated after embedding processing, where P is the patch size and D is the embedding dimension.
8. The cross-color gamut phase-guided dual-attention fusion HDR imaging method according to claim 7, characterized in that, In step (5.1), the detailed feature calculation process is as follows: in These respectively represent the features of the reference frame. The query matrix, key matrix, and value matrix generated by linear projection. These represent the learnable linear projection weight matrices corresponding to the generated query, key, and value matrices, respectively. This represents the calculated self-attention map, used to measure long-distance dependencies between pixels within a feature. The feature dimension of the key vector. These are learnable weights, which are assigned values according to the initialization strategy at the beginning of training and then learned end-to-end through backpropagation. against ,by and Perform cross-frame correlation calculations to generate a spatial attention map. And accordingly, suppress movement and occlusion areas: in This represents the spatial attention map generated for the i-th non-reference frame. This represents the function for calculating cross-frame correlation. This represents the refined features of the i-th frame after soft suppression processing. This is for element-wise multiplication.
9. The cross-color gamut phase-guided dual-attention fusion HDR imaging method according to claim 8, characterized in that, The specific steps (6) are as follows: A dual attention fusion network (DAFN) is used as the backend fusion and reconstruction module; The backend fusion and reconstruction module consists of multiple dual-attention refinement modules (DARB) connected in series, and uses residual connections: The DARB module consists of two parts: first, the left-side Global Context Aggregation (GCA) module establishes a consistent scene prior, and then the right-side Context-Guided Semantic Fusion (CGSF) module completes the final fine-grained fusion.
10. The cross-color gamut phase-guided dual-attention fusion HDR imaging method according to claim 9, characterized in that, The GCA module adopts a similar structure to the standard SwinTransformer block and retains a window-based self-attention mechanism to obtain a globally consistent context, and then uses MLP to improve representation capabilities: Where Z represents the sequence of visual tokens input to the current GCA module. The representation is normalized at the layer level, and U represents the intermediate transition feature after attention mechanism processing and residual connection. This represents a multilayer perceptron. This represents the global context features obtained after processing by the basic Transformer block of the GCA module. W-MSA computes attention on a sliding window. In global context Based on this, the bidirectional complementarity of the three-exposure semantics is explicitly modeled, first by segmenting according to channels to obtain... Corresponding to underexposure, reference frame, and overexposure, and linear projection: in For learnable projection, d represents independent projection for each head in the case of multiple heads. Let represent the query vector, key vector, and value vector generated by the linear projection of the features of the i-th frame, respectively; Perform cross attention on bidirectional paths: Query vector of reference branch As a condition, for non-reference branches key-value pairs Perform conditional information aggregation: in For learnable temperature parameters, This represents the information aggregation feature from non-reference frames to reference frames. The query vector representing the reference frame. Represents the key vector of a non-reference frame. Represents the value vector of a non-reference frame; The non-reference channel is calibrated by the reference channel to align its semantic coordinates with the intensity response: in For learnable temperature parameters, This represents the feature calibration representation from the reference frame to the non-reference frame. The query vector represents a non-reference frame. The key vector representing the reference frame. Represents the value vector of the reference frame; Finally, the four outputs are concatenated and output via MLP: in This represents the cross-group spatial fusion feature output after deep fusion via a multilayer perceptron (MLP). This represents the information aggregation feature from non-reference frames to reference frames. This represents the feature calibration representation from the reference frame to the non-reference frame; After passing through multiple DARB modules, the image is returned to the spatial domain via Patchun Embedding and then reconstructed by the reconstruction head to obtain the HDR image. in For PatchunEmbedding, For channel-aligned global residual connections, Constrain the result to Sigmoid , This represents the final deep visual token sequence output after processing by N cascaded DARB modules. This represents a pre-reconstructed feature map that integrates deep semantics and shallow details. This represents the final generated high dynamic range (HDR) predicted image.