Single-exposure spectral-depth joint reconstruction method in high dynamic range scene

Abstract

The present application provides a kind of single exposure spectral-depth joint reconstruction method under high dynamic range scene, provides a single shooting end-to-end framework, realizes the joint optimization of spectral reconstruction and depth estimation under high dynamic range scene, can complete spectral-depth joint reconstruction based on single shooting HDR scene, avoids the motion artifact problem of multi-exposure method;HDR high dynamic range preprocessing network learns task perception feature representation through the back propagation of downstream spectrum and depth loss, can complete task perception based HDR feature learning without HDR ground truth image, ensure that feature representation matches the demand of reconstruction task.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and computational imaging technology, and particularly relates to a single-exposure spectral-depth joint reconstruction method in high dynamic range scenes. Background Technology

[0002] Spectral reconstruction and depth estimation are core tasks in computer vision. Spectral reconstruction infers high-dimensional surface reflectance from a limited set of RGB channels, capturing material properties; depth estimation recovers 3D structure from cues such as parallax, texture gradients, or defocus in monocular images. These two tasks are complementary, and their joint acquisition enables a unified representation of object composition, spatial layout, and physical properties, which is beneficial for applications such as remote sensing, industrial inspection, robot perception, and augmented reality. The joint acquisition of spectral and depth information produces a synergistic effect of "1+1>2". In intelligent manufacturing, simultaneously acquiring the material composition (through spectroscopy) and geometric dimensions (through depth) of parts enables comprehensive quality control. In precision agriculture, combining the three-dimensional structural information (plant height, canopy density) and spectral characteristics (chlorophyll content, water content) of crops allows for more accurate yield prediction and disease diagnosis. In medical imaging, fusing the spectral characteristics (blood oxygen saturation, hemoglobin concentration) and three-dimensional morphology (tumor volume, surface texture) of tissues provides more comprehensive diagnostic information. In remote sensing archaeology, spectral information reveals the material composition of underground remains, while depth information displays micro-topographical undulations; combining the two can discover ancient sites that are difficult to detect with the naked eye. In autonomous driving, spectral information helps identify road surface materials (asphalt, cement, water accumulation, ice), while depth information provides obstacle distances; combining the two significantly improves perception capabilities in adverse weather conditions. In robot grasping, spectral features determine the material of objects (metal, plastic, glass), while depth information plans the grasping posture, enabling adaptive grasping of objects of different materials.

[0003] Existing depth-spectral joint imaging methods typically assume a standard dynamic range and rely on conventional 8-bit representations, which severely limits their performance under high dynamic range conditions. High dynamic range (HDR) scenes exhibit brightness variations spanning several orders of magnitude, which traditional sensors cannot effectively respond to. This leads to saturation and loss of detail in bright areas, while dark areas suffer from severe noise and quantization errors, making it difficult to accurately capture spectral and geometric information. This degradation significantly disrupts spectral reconstruction and depth estimation because fundamental geometric boundaries and material features are distorted or lost in poorly exposed areas.

[0004] Deep learning has revolutionized these two tasks: Convolutional Neural Networks (CNN)-based models achieve single-shot RGB-to-hyperspectral mapping, Transformers capture long-range dependencies, and self-supervised / unsupervised monocular deep frameworks reduce reliance on annotations. Pre-trained visual Transformers further enhance geometric inference capabilities for dense predictions. Recent joint or multi-task methods leverage shared representations and cross-task constraints to improve robustness to sparsely textured or geometrically blurred regions.

[0005] 1. Traditional spectral-depth joint reconstruction method

[0006] In recent years, researchers have begun to explore the joint optimization of spectral reconstruction and depth estimation: (1) Dual-camera stereo system: Fu et al. (2019) proposed using a low spatial resolution spectral camera and a high spatial resolution RGB camera to simultaneously capture measurements from two different perspectives, and jointly reconstruct a high-resolution spectral cube and depth map through a two-stage scheme. First, the low-resolution spectral measurements are super-resolution, then synthesized into an RGB image and used in conjunction with the high-resolution RGB measurements to estimate the disparity map. Finally, the depth is reconstructed using a stereo configuration and the spectral reconstruction is improved.

[0007] (2) Co-imaging based on coded aperture: Baek et al. (2021) proposed a time-multiplexed phase coded aperture system, which uses a color coded aperture combined with a time-varying phase coded aperture and a spatially varying pixel shutter to generate a spatially varying point spread function to facilitate the differentiation of depth information. At the same time, spectral information is encoded by selectively filtering specific spectral bands.

[0008] (3) End-to-end optical-digital joint optimization: Chang and Wetzstein (2019) proposed an end-to-end learning method for depth estimation by diffraction rotation, which jointly optimizes the phase mask of the rotation point spread function (RPSF) and the weights of the depth estimation neural network, and introduces a differentiable aperture mask physical model and accurate camera imaging pipeline simulation.

[0009] (4) Multi-task learning framework: Wang et al. (2019) optimized spectral and depth targets within a multi-task learning framework. Experiments showed that the jointly learned features contained richer semantic and structural cues than task-specific features and exhibited stronger robustness in regions with sparse textures or geometric ambiguity.

[0010] However, these methods are all designed for standard dynamic range images, rely on conventional 8-bit representation, and assume that all pixels are exposed in the same way, which severely limits their performance under high dynamic range conditions.

[0011] 2. HDR High Dynamic Range Imaging Technology Based on Multi-Exposure Fusion Multi-exposure HDR imaging is the most commonly used method for acquiring high dynamic range images. It reconstructs the complete dynamic range by capturing multiple LDR images of the same scene with different exposure times and fusing them. The core of this technology lies in: (1) Registration and Alignment Methods: Optical flow algorithms or feature matching methods are used to spatially align images with different exposures. For example, Kalantari et al. (2017) used optical flow algorithms to align low-exposure and high-exposure images to a medium-exposure reference image, and then input them into a deep neural network for fusion. Global or local geometric transformations are used to handle camera shake and background movement.

[0012] (2) Ghosting techniques: To address ghosting artifacts caused by moving objects in dynamic scenes, researchers have proposed various detection and suppression methods. Khan et al. (2006) and Sen et al. (2012) identified moving regions through motion detection, reduced the weight of these regions, or used only information from the reference frame. Recent deep learning methods, such as Wu et al. (2018) and Yan et al. (2019), employ attention mechanisms and deformable convolutions to implicitly handle alignment problems, eliminating misaligned regions by calculating the spatial correlation between the reference image and the non-reference image.

[0013] (3) Fusion Strategy: The weighted graph-based adaptive fusion method performs weighted synthesis based on the quality of each exposure image (such as saturation, good exposure, and contrast). The classic method proposed by Debevec and Malik (1997) recovers scene irradiance through the camera response function, which is the theoretical basis of multi-exposure HDR imaging.

[0014] 3. HDR High Dynamic Range Image Reconstruction Technology Based on Single-Frame Deep Learning To address the limitations of multi-exposure methods, researchers have proposed a deep learning method for reconstructing HDR images from a single low dynamic range (LDR) image: (1) End-to-end convolutional neural network methods: Eilertsen et al. (2017) proposed HDRCNN, which uses a fully convolutional network with an autoencoder structure to directly reconstruct HDR images from a single 8-bit LDR image, focusing on recovering lost information in saturated regions (such as highlights). The network is trained on a large-scale HDR dataset and utilizes the MIT Places database for transfer learning to enhance robustness. Marnerides et al. (2018) proposed ExpandNet, which uses dilated convolutions to increase the receptive field and perform high dynamic range expansion from LDR content.

[0015] (2) Inverse tone mapping method: Liu et al. (2020) proposed to reconstruct HDR images by learning the inverse camera imaging pipeline, modeling the camera's truncation process and post-processing operations. This method uses a dual learning framework to learn both the forward mapping from LDR to HDR and the inverse mapping from HDR to LDR. It guides the learning of the main module by constraining the output and introduces an attention mechanism to adjust the contrast and saturation problems caused by dynamic range changes.

[0016] (3) HDR Imaging Based on Depth Optics: Metzler et al. (2020) proposed Deep Optics for HDR, interpreting the problem as the joint training of an optical encoder and an electronic decoder. The encoder is parameterized by the point spread function (PSF) of the lens, the bottleneck is the sensor with limited dynamic range, and the decoder is a convolutional neural network. By optimizing the lens surface and CNN end-to-end, optimized diffractive optical elements are fabricated as hardware add-ons, and a single sensor image containing optically encoded HDR information is recorded during inference.

[0017] (4) Adaptive illumination modulation methods: Some methods introduce adaptive illumination modulation based on local brightness, applying spatially varying tone mapping to selectively enhance bright and dark areas according to local brightness. However, these methods mainly optimize perceived brightness and do not consider the specific needs of downstream visual tasks.

[0018] Therefore, it can be seen that the existing technology has the following technical defects: 1. The inadequacy of the spectral-depth joint method for HDR high dynamic range scenes. (1) Limitations of the standard dynamic range assumption: Existing joint methods assume a standard dynamic range and consistent exposure. In HDR scenes, saturation in overexposed areas and severe noise in underexposed areas make it difficult to accurately capture spectral and geometric information. Experiments show that when the scene dynamic range exceeds 8-bit representation capability, the reconstruction accuracy decreases significantly.

[0019] (2) Performance degradation under extreme lighting: In overexposed and underexposed areas, basic geometric boundaries and material features are distorted or lost, and the transition bands between different exposure areas are particularly critical but difficult to reconstruct accurately. The loss of information in these areas seriously affects the accuracy of scene understanding.

[0020] (3) Hardware complexity and cost: High-complexity hardware such as CASSI (Coded Aperture Snapshot Spectroscopic Imaging) or active illumination methods such as structured light greatly limit the application scope, especially for outdoor use. The dual-camera system of Fu et al. (2019) requires precise calibration and synchronization, which increases the system complexity.

[0021] (4) Lack of unified end-to-end optimization: Existing methods usually design optical systems and post-processing algorithms separately, or adopt a two-stage optimization strategy, resulting in suboptimal performance. There is a lack of a joint optimization framework for HDR preprocessing, spectral reconstruction and depth estimation, and it is difficult to achieve global optimum by training each module independently.

[0022] 2. Inherent limitations of multi-exposure HDR high dynamic range methods (1) Motion artifacts and ghosting problems: Even with advanced optical flow alignment or deformable convolution methods, registration errors are still easily generated when there are large-scale motions, occlusions and appearance changes caused by different exposures, resulting in ghosting artifacts in the fusion results. Studies by Gallo et al. (2009) and Zimmer et al. (2011) show that non-rigid motion is particularly difficult to handle.

[0023] (2) Time cost and hardware requirements: The need for multiple exposures limits the imaging speed and is not suitable for real-time processing of dynamic scenes. In practical applications, the time delay of multi-frame capture may reach tens of milliseconds or even longer, resulting in the inability to accurately record scene changes.

[0024] (3) Information loss problem: In order to avoid ghosting, the ghosting algorithm often adopts a conservative strategy, reducing the weight of motion area or using only reference frame information. This will discard useful information in non-reference frames, especially when the reference frame contains overexposed or underexposed areas, affecting the final HDR quality.

[0025] 3. Limitations of Single-Frame Deep Learning HDR High Dynamic Range Methods (1) Saturated details are difficult to recover accurately: Although CNNs can synthesize visually believable HDR content, saturated scene details often cannot be faithfully recovered. Metzler et al. (2020) pointed out that purely CNN-based methods are essentially "hallucinating" missing content and lack real physical constraints.

[0026] (2) Lack of task-oriented optimization: Existing HDR methods are mainly designed to enhance visual quality or image-based lighting (IBL), with the optimization goal being the perceived brightness or visual effect of HDR display, without considering the specific needs of downstream tasks such as spectral reconstruction and depth estimation. Uniform tone mapping cannot adapt to the different requirements of spatially varied lighting for different reconstruction targets.

[0027] (3) Blurred geometric and spectral details: Local inference methods may blur geometric discontinuities and spectral details, and the lack of end-to-end joint optimization leads to a mismatch between feature representation and subsequent reconstruction tasks. Studies by Eilertsen et al. (2017) have shown that features without targeted optimization often contain redundant information, affecting reconstruction accuracy. Summary of the Invention

[0028] To address the aforementioned issues, this invention provides a single-exposure spectral-depth joint reconstruction method for high dynamic range (HDR) scenes, which enables spectral-depth joint imaging in HDR scenes.

[0029] A single-exposure spectral-depth joint reconstruction method for high dynamic range scenes includes the following steps: Input a given low dynamic range image Input HDR high dynamic range preprocessing network Dynamic range extension is performed to obtain a high dynamic range image. ; High dynamic range images Multi-scale semantic features are extracted from the pre-trained visual base model DINO v3. Then, a lightweight refinement network was adopted. Multi-scale semantic features Upsampling and detail restoration are performed to output a pixel-level depth estimation map. ; High dynamic range images Feed into Transformer spectral reconstruction network Transformer spectral reconstruction network High dynamic range images through self-attention mechanism Perform correlation modeling and output the reconstructed hyperspectral image. .

[0030] Furthermore, the visual baseline model DINO v3 and the Transformer spectral reconstruction network were trained. Joint loss function at time as follows:

[0031] in, For spectral branching loss function, for The corresponding balance coefficient, For the deep branch loss function, for The corresponding balance coefficient; in, The calculation method is as follows:

[0032] in, The bias coefficient, This is the loss term used to constrain spatial smoothness. This is the loss term used to constrain the spectral continuity between adjacent bands. The loss term is used to suppress high-frequency artifacts by applying second-order Laplace regularization.

[0033] Furthermore, the HDR high dynamic range preprocessing network This includes the mask estimation branch, the state network branch, and the backbone network; The mask estimation branch is used to acquire low dynamic range input images. The corresponding mask; The state network branch is used to input images from low dynamic range. Four different scales of illumination features were extracted. The backbone network is used to synthesize local illumination features based on illumination features at four different scales; Apply the mask to the low dynamic range input image The result of element-wise multiplication is added to the local illumination features to obtain a high dynamic range image. .

[0034] Furthermore, the mask estimation branch is obtained by cascading a first 3×3 convolutional layer, a first Mish layer, a second 3×3 convolutional layer, a second Mish layer, and a 1×1 convolutional layer in sequence; Low dynamic range input image After being processed sequentially through the first 3×3 convolutional layer, the first Mish layer, the second 3×3 convolutional layer, the second Mish layer, and the 1×1 convolutional layer, the mask is output by the 1×1 convolutional layer.

[0035] 5. A single-exposure spectral-depth joint reconstruction method for high dynamic range scenes as described in claim 3, characterized in that the state network branch includes a cascaded branch and four 1×1 convolutional layers; wherein the cascaded branch is obtained by cascading the first 1×1 convolutional layer, the first Mish layer, the first downsampling module, the second Mish layer, the second downsampling module, the third Mish layer, the third downsampling module, and the fourth Mish layer in sequence; Low dynamic range input image After processing via a concatenated branch, the output of the first Mish layer is fused using a second 1×1 convolutional layer to obtain the illumination features at the first scale. The output of the second Mish layer is fused using a third 1×1 convolutional layer to obtain the illumination features at the second scale. The output of the third Mish layer is fused using a fourth 1×1 convolutional layer to obtain the illumination features at the third scale. The output of the fourth Mish layer is fused using the fifth 1×1 convolutional layer to obtain the illumination features at the fourth scale. .

[0036] Furthermore, the backbone network includes an encoder, a decoder, and an external ResBlock module; wherein, the encoder is obtained by cascading a first built-in ResBlock module, a first downsampling module, a second built-in ResBlock module, a second downsampling module, a third built-in ResBlock module, and a third downsampling module in sequence; the decoder is obtained by cascading a first upsampling module, a fourth built-in ResBlock module, a second upsampling module, a fifth built-in ResBlock module, a third upsampling module, and a sixth built-in ResBlock module in sequence. Among them, the illumination features of the first scale Input the first built-in ResBlock module; second scale lighting features. Input the second built-in ResBlock module; the third scale of lighting features. Input the third built-in ResBlock module; fourth scale lighting features. The output of the third downsampling module is superimposed on the output of the external ResBlock module and then input into the external ResBlock module; the output of the external ResBlock module is used as the input of the first upsampling module; and the output of the sixth built-in ResBlock module is used as the final local illumination feature.

[0037] Furthermore, each ResBlock module is obtained by cascading a LIAM layer, a first 3×3 convolutional layer, a Mish layer, and a second 3×3 convolutional layer in sequence; wherein, the LIAM layer includes an intensity compensation module and an adaptive modulation module; the final output of the LIAM layer... Output of the strength compensation module and the output of the adaptive modulation module The results are obtained by superposition. At the same time, the adaptive modulation module includes normal exposure branch, overexposure branch and underexposure branch. in, ,in, For the first Illumination characteristics at various scales , This is a 1×1 convolution operation. For brightness projection operation, For a multilayer perceptron function; if illumination features are not used as input to the external ResBlock module and the fourth to sixth built-in ResBlock modules, then the external ResBlock module and the fourth to sixth built-in ResBlock modules correspond to... =0; ,in, The output of the normal exposure branch. For the output of the overexposure branch, The output of the underexposed branch, and has , These are the spatial attention gates corresponding to the normal exposure branch, overexposure branch, and underexposure branch. For the Sigmoid function, These are independent convolutional mappings corresponding to the normal exposure branch, overexposure branch, and underexposure branch. For the output of the previous module cascaded in the current LIAM layer, the first built-in ResBlock module corresponds to... The value is 0, corresponding to the second and third built-in ResBlock modules. All are outputs of the downsampling module, corresponding to the fourth to sixth built-in ResBlock modules. This is the output of the upsampling module; the corresponding output of the external ResBlock module. Illumination characteristics at the fourth scale The result is superimposed with the output of the third downsampling module; At the same time, the input to the LIAM layer will be... The output of the ResBlock module is superimposed on the output of the second 3×3 convolutional layer.

[0038] Furthermore, the Transformer spectral reconstruction network It includes the CIM branch and the SRB branch; the SRB branch is obtained by cascading the first 3×3 convolutional layer, multiple SRB modules, and the second 3×3 convolutional layer in sequence. Transformer Spectral Reconstruction Network The final output Output from CIM branch The output of the first 3×3 convolutional layer The output of the second 3×3 convolutional layer The result of fusion is:

[0039] in, For convolution operations, This is the activation function.

[0040] Furthermore, the output of the CIM branch The method for obtaining it is as follows:

[0041] in, This is a 1×1 convolution operation. For batch normalization operations, For GELU activation function, The feature space attention operation consists of a 5×5 depthwise separable convolution operation, a 1×1 point convolution operation, and a Sigmoid activation function operation.

[0042] Furthermore, each SRB module includes an encoder, a decoder, a skip connection section, and an external channel attention CAB module; wherein, the encoder is obtained by cascading an embedded convolutional layer, a first built-in channel attention CAB module, a first downsampling module, a second built-in channel attention CAB module, and a second downsampling module in sequence; the decoder includes a first upsampling module, a third built-in channel attention CAB module, a second upsampling module, a fourth built-in channel attention CAB module, and a mapping module; the skip connection section includes a first 1×1 convolutional layer and a second 1×1 convolutional layer; Input features of the current SRB module The input features are processed by the encoder via embedded convolutional layers. The output of the second downsampling module is then fed into the external channel attention CAB module for feature extraction. The output features of the external channel attention CAB module then enter the first upsampling module. The output of the second built-in channel attention CAB module is concatenated with the output of the first upsampling module and then fed into the first 1×1 convolutional layer. The output of the first 1×1 convolutional layer serves as the input to the third built-in channel attention CAB module, and its output enters the second upsampling module. The output of the first built-in channel attention CAB module is concatenated with the output of the second upsampling module and then fed into the second 1×1 convolutional layer. The output of the second 1×1 convolutional layer serves as the input to the fourth built-in channel attention CAB module, and its output enters the mapping module. The output and input features of the mapping module are then processed together. The sum of the two is used as the final output of the current SRB module; When the current SRB module is the first SRB module, the input features are... The output of the first 3×3 convolutional layer in the SRB branch, when the current SRB module is not the first SRB module, is the input feature. This is the final output of the SRB module preceding the current SRB module.

[0043] Beneficial effects: 1. This invention provides a single-exposure spectral-depth joint reconstruction method for high dynamic range (HDR) scenes, and provides a single-shot end-to-end framework to achieve joint optimization of spectral reconstruction and depth estimation in HDR scenes. Spectral-depth joint reconstruction can be completed based on a single-shot HDR scene, avoiding motion artifacts in multi-exposure methods. The HDR high dynamic range preprocessing network learns task-aware feature representations through backpropagation of downstream spectral and depth losses, completing task-aware HDR feature learning without HDR ground truth images, ensuring that the feature representation matches the reconstruction task requirements.

[0044] 2. This invention provides a single-exposure spectral-depth joint reconstruction method for high dynamic range scenes, and proposes a brightness-aware illumination adaptive modulation (LIAM) mechanism, which dynamically adjusts the feature response based on local exposure conditions to achieve effective processing of spatially non-uniform illumination; that is, the LIAM mechanism recalibrates local brightness and enhances task-related representations under HDR conditions, while maintaining scene realism.

[0045] 3. This invention provides a single-exposure spectral-depth joint reconstruction method for high dynamic range scenes. The spectral reconstruction network SRT utilizes channel interpolation and adaptive total variational regularization to maintain a fine-grained spectral structure while suppressing high brightness noise. The depth estimation branch based on DINOv3, combined with adaptive edge-aware regularization, can achieve accurate, consistent, and robust depth map estimation even under complex extreme lighting and occlusion conditions. Attached Figure Description

[0046] Figure 1 This invention provides an end-to-end single-shot spectral-depth joint reconstruction framework. Figure 2 The HDR high dynamic range preprocessing network provided by this invention Architecture diagram; Figure 3 The HDR high dynamic range preprocessing network provided by this invention Model structure diagram; Figure 4 A model structure diagram of the brightness-sensing illumination adaptive modulation mechanism (LIAM) provided by the present invention; Figure 5 The multilayer perceptron (MLP) diagram provided for this invention; Figure 6 The Transformer spectral reconstruction network provided by this invention Model structure diagram; Figure 7 The channel attention structure diagram provided by this invention. Detailed Implementation

[0047] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0048] I. System Overall Architecture This invention proposes an end-to-end single-shot spectral-depth joint reconstruction framework for spectral-depth joint imaging in high dynamic range (HDR) scenes.

[0049] like Figure 1 As shown, the spectral-depth joint reconstruction framework of the present invention consists of three main components: 1. HDR High Dynamic Range Preprocessing Network It adopts a lightweight structure and integrates a Luminance-aware Illumination Adaptive Modulation (LIAM) mechanism to adaptively enhance the brightness and contrast distribution of input features.

[0050] 2. Transformer Spectral Reconstruction Network Based on the Transformer architecture, a channel interpolation scheme is used to achieve efficient RGB to hyperspectral mapping, and adaptive total variational regularization is combined to balance detail preservation and noise suppression.

[0051] 3. Depth Estimation Network Branch: Using a DINOv3 encoder combined with a DPT decoder, detailed geometry is recovered, maintaining the accuracy of depth prediction under extreme lighting conditions.

[0052] A key feature of this framework is that it can perform end-to-end joint optimization without requiring HDR ground truth; HDR preprocessing is entirely guided by the backpropagation algorithm of the downstream task loss.

[0053] The key innovation lies in end-to-end joint optimization, achieved through a unified loss function. A coupled modeling of spectral reflectance and depth is achieved, where depth provides geometrically constrained and stable material estimation, and spectral cues provide fine texture information to enhance structure perception.

[0054] Forward propagation process Given a low dynamic range input image First, it passes through an HDR high dynamic range preprocessing network. Dynamic range extension is performed to obtain a high dynamic range image:

[0055] by As a shared input, the network is divided into two parallel processing branches.

[0056] Depth estimation branch: Input the pre-trained visual base model DINO v3 to extract multi-scale semantic features Subsequently, the lightweight refinement network was used. The features are upsampled and details are restored to output a pixel-level depth estimation map:

[0057] Hyperspectral reconstruction branch: Feed into the spectral Transformer network By modeling spatial-spectral correlation through a self-attention mechanism, a reconstructed hyperspectral image is output:

[0058] The two branches are computed in parallel without interfering with each other, together forming the complete forward reasoning path of the system.

[0059] The total loss is obtained by weighted summation of the losses from the two branches:

[0060] in, For spectral branching loss function, for The corresponding balance coefficient, For the deep branch loss function, for The corresponding balance coefficients; it should be noted that the HDR high dynamic range preprocessing network By optimizing the learning task's perceptual representation through end-to-end optimization, no HDR ground truth is required. HDR preprocessing is entirely guided by backpropagation of the downstream task's loss, enabling the system to adaptively learn the feature representations most beneficial for spectral reconstruction and depth estimation.

[0061] in, The calculation method is as follows:

[0062] in, The bias coefficient, This is the loss term used to constrain spatial smoothness. This is the loss term used to constrain the spectral continuity between adjacent bands. The loss term is used to suppress high-frequency artifacts by applying second-order Laplace regularization.

[0063] Backpropagation and parameter optimization process According to the chain rule, the total loss The gradients of the parameters in each module are propagated backward layer by layer. The gradient of the depth estimation branch is propagated along... → Path backhaul; gradient along the hyperspectral reconstruction branch → Path backhaul. Two paths in... The points converge, and their parameter gradients are the sum of the gradients propagated back from the two branches:

[0064] All learnable parameters The Adam optimizer is used to uniformly update gradients based on the aforementioned gradients, achieving end-to-end joint optimization. It should be noted that the DINOv2 backbone can be frozen or fine-tuned during the training phase depending on task requirements; if frozen, the gradients will not pass through... Continue passing it on.

[0065] Based on this, the present invention provides a single-exposure spectral-depth joint reconstruction method for high dynamic range scenes, comprising the following steps: Input a given low dynamic range image Input HDR high dynamic range preprocessing network Dynamic range extension is performed to obtain a high dynamic range image. ; High dynamic range images Multi-scale semantic features are extracted from the pre-trained visual base model DINO v3. Then, a lightweight refinement network was adopted. Multi-scale semantic features Upsampling and detail restoration are performed to output a pixel-level depth estimation map. ; High dynamic range images Feed into Transformer spectral reconstruction network Transformer spectral reconstruction network High dynamic range images through self-attention mechanism Perform correlation modeling and output the reconstructed hyperspectral image. .

[0066] II. HDR High Dynamic Range Preprocessing Network

[0067] like Figure 2 As shown, the HDR high dynamic range preprocessing network It adopts a U-Net-style encoder-decoder architecture. The entire network consists of a mask estimation branch, a state network branch, and a backbone network branch, and finally fuses the mask and output through residual connections.

[0068] Specifically, the mask estimation branch is used to acquire low dynamic range input images. The corresponding mask; the state network branch is used to input images from low dynamic range. Four different scales of illumination features are extracted; the backbone network is used to synthesize local illumination features based on the four different scales of illumination features; the mask is then compared with the low dynamic range input image. The result of element-wise multiplication is added to the local illumination features to obtain a high dynamic range image. .

[0069] It should be noted that the mask estimation branch consists of simple stacked convolutional layers: first, a 3×3 convolution is used to process the input image. The number of channels is reduced to half the dimension of the basic features, then reduced to one-quarter after Mish activation and 3×3 convolution, and finally mapped to the number of output channels through 1×1 convolution to generate a mask with the same spatial size as the input image. This mask is used for subsequent element-wise multiplication with the backbone network output to achieve local weighting.

[0070] The state network branch is responsible for extracting multi-scale illumination features from the image. The input low-dynamic image first passes through a sequence of 3×3 convolutions, Mish activations, and 1×1 convolutions to obtain initial conditional features. These features are then sequentially passed through state network conditions (Cond1×1) to generate state network conditional features C at different scales. ), that is, state network condition C.

[0071] The backbone network consists of encoders and decoders. The encoder extracts multi-scale features through hierarchical convolutions: At each feature level, conditional features are computed using 1×1 and 3×3 convolutions. This captures local illumination features. These features are then processed by a reconstruction network with a LIAM layer.

[0072] like Figure 3 As shown, the mask estimation branch is obtained by cascading a first 3×3 convolutional layer, a first Mish layer, a second 3×3 convolutional layer, a second Mish layer, and a 1×1 convolutional layer in sequence; Low dynamic range input image After being processed sequentially through the first 3×3 convolutional layer, the first Mish layer, the second 3×3 convolutional layer, the second Mish layer, and the 1×1 convolutional layer, the mask is output by the 1×1 convolutional layer.

[0073] The state network branch includes a cascaded branch and four 1×1 convolutional layers; wherein, the cascaded branch is obtained by cascading the first 1×1 convolutional layer, the first Mish layer, the first downsampling module, the second Mish layer, the second downsampling module, the third Mish layer, the third downsampling module, and the fourth Mish layer in sequence; Low dynamic range input image After processing via a concatenated branch, the output of the first Mish layer is fused using a second 1×1 convolutional layer to obtain the illumination features at the first scale. The output of the second Mish layer is fused using a third 1×1 convolutional layer to obtain the illumination features at the second scale. The output of the third Mish layer is fused using a fourth 1×1 convolutional layer to obtain the illumination features at the third scale. The output of the fourth Mish layer is fused using the fifth 1×1 convolutional layer to obtain the illumination features at the fourth scale. .

[0074] The backbone network includes an encoder, a decoder, and an external ResBlock module; wherein, the encoder is obtained by cascading a first built-in ResBlock module, a first downsampling module, a second built-in ResBlock module, a second downsampling module, a third built-in ResBlock module, and a third downsampling module in sequence; the decoder is obtained by cascading a first upsampling module, a fourth built-in ResBlock module, a second upsampling module, a fifth built-in ResBlock module, a third upsampling module, and a sixth built-in ResBlock module in sequence; Among them, the illumination features of the first scale Input the first built-in ResBlock module; second scale lighting features. Input the second built-in ResBlock module; the third scale of lighting features. Input the third built-in ResBlock module; fourth scale lighting features. The output of the third downsampling module is superimposed on the output of the external ResBlock module and then input into the external ResBlock module; the output of the external ResBlock module is used as the input of the first upsampling module; and the output of the sixth built-in ResBlock module is used as the final local illumination feature.

[0075] Furthermore, each ResBlock module is obtained by cascading a LIAM layer, a first 3×3 convolutional layer, a Mish layer, and a second 3×3 convolutional layer in sequence; where, for example... Figure 4 As shown, the LIAM layer includes an intensity compensation module and an adaptive modulation module; the final output of the LIAM layer... Output of the strength compensation module and the output of the adaptive modulation module The results are obtained by superposition. At the same time, the adaptive modulation module includes normal exposure branch, overexposure branch and underexposure branch. in, ,in, For the first Illumination characteristics at various scales , This is a 1×1 convolution operation. For brightness projection operation, For a multilayer perceptron function; if illumination features are not used as input to the external ResBlock module and the fourth to sixth built-in ResBlock modules, then the external ResBlock module and the fourth to sixth built-in ResBlock modules correspond to... =0; ,in, The output of the normal exposure branch. For the output of the overexposure branch, The output of the underexposed branch, and has , These are the spatial attention gates corresponding to the normal exposure branch, overexposure branch, and underexposure branch. For the Sigmoid function, These are independent convolutional mappings corresponding to the normal exposure branch, overexposure branch, and underexposure branch. For the output of the previous module cascaded in the current LIAM layer, the first built-in ResBlock module corresponds to... The value is 0, corresponding to the second and third built-in ResBlock modules. All are outputs of the downsampling module, corresponding to the fourth to sixth built-in ResBlock modules. This is the output of the upsampling module; the corresponding output of the external ResBlock module. Illumination characteristics at the fourth scale The result is superimposed with the output of the third downsampling module; At the same time, the input to the LIAM layer will be... The output of the ResBlock module is superimposed on the output of the second 3×3 convolutional layer.

[0076] In other words, the LIAM layer in the ResBlock module consists of an adaptive modulation module and an intensity compensation module. In the adaptive modulation module, for the input features... F The adaptive modulation module first performs channel compression using 1×1 convolution to obtain low-dimensional feature representations. Subsequently, a three-branch adaptive gating mechanism is introduced. Each branch takes the Fr feature as input and generates spatial attention gates for normally exposed, overexposed, and underexposed areas respectively through independent convolutional mapping and sigmoid activation. The output of each branch is... , Finally, the three branch outputs are fused element-wise to obtain the comprehensive modulation characteristics. .

[0077] In the strength compensation module, the strength compensation module also uses... Convolution is used to reduce the dimensionality of input features. Then, through a luminance projection operation, the spatial features are mapped to the luminance perception space to extract the global intensity distribution representation. The result is expressed as follows: Figure 5 The two-layer multi-layer perceptron (MLP) regression shown obtains the exposure compensation vector, which is then used to apply affine modulation to the features, outputting the compensation features. Finally, the output features of the two modules are fused by residual superposition to obtain the output features of LIAM.

[0078] It should be noted that the three-branch modulation mechanism of LIAM can be simplified to a two-branch mechanism (distinguishing only between overexposure and underexposure), reducing computational complexity while maintaining the main functionality. Specifically, pixels are divided into bright and dark areas using a single threshold. A suppression strategy is employed for bright areas, and an enhancement strategy for dark areas, thereby reducing the number of parameters in the gating network and improving inference speed. This will not be elaborated upon further in this invention.

[0079] III. Transformer Spectral Reconstruction Network

[0080] like Figure 6 As shown, the Transformer spectral reconstruction network It includes the CIM branch (Channel Interpolation Module) and the SRB branch; the SRB branch is obtained by cascading the first 3×3 convolutional layer, multiple SRB modules, and the second 3×3 convolutional layer in sequence. Transformer Spectral Reconstruction Network The final output Output from CIM branch The output of the first 3×3 convolutional layer The output of the second 3×3 convolutional layer The result of fusion is:

[0081] in, For convolution operations, This is the activation function.

[0082] In other words, and The data is concatenated along the channel dimension and then fused using a 1×1 convolution to compress the number of channels back to the basic feature dimension. The fused features are then mapped to the target spectral channel number via a 3×3 convolution, and simultaneously added to the result of the same convolution with the shallow features, forming a global residual connection. Finally, the output feature map is cropped to the original input size, yielding the hyperspectral reconstruction result. The entire model, through multi-level skip connections and a bi-branch design, fully integrates spectral priors and spatial details, achieving accurate spectral reconstruction.

[0083] It should be noted that spectral reconstruction from RGB images is inherently an ill-posed problem. The CIM branch cascades channel projection units and spatial refinement units sequentially, projecting the RGB input onto a high-dimensional spectral sensing representation in the following manner. of:

[0084] in, This is a 1×1 convolution operation. For batch normalization operations, For GELU activation function, The feature space attention operation consists of a 5×5 depthwise separable convolution operation, a 1×1 point convolution operation, and a Sigmoid activation function operation.

[0085] Therefore, the CIM branch consists of, in sequence, a 1×1 convolution (mapping the c channel to half the dimension of the channel to be processed), batch normalization, GELU activation, and another 1×1 convolution (mapping to the dimension of the channel to be processed). The spatial refinement unit is the Feature Spatial Attention (FSA) module, which consists of a 5×5 depthwise separable convolution (the number of groups equals the number of input channels), a 1×1 pointwise convolution, and a Sigmoid activation. The output is multiplied element-wise by the input to achieve spatial attention weighting. The depthwise convolution and pointwise convolution emphasize spatial information regions such as material boundaries and texture variations, providing complementary spectral priors for reconstruction. Finally, the spectral features are output.

[0086] Furthermore, the SRB branches are progressively refined through cascading SRBs on the main branch. Each SRB implements a U-Net-style encoder-decoder structure, including channel attention blocks (CABs). The encoder captures multi-scale spectral-spatial patterns through hierarchical downsampling, and the decoder recovers spatial resolution through transposed convolutions and skip connections.

[0087] like Figure 6As shown, any SRB module includes an encoder, a decoder, a skip connection part, and an external channel attention CAB module; wherein, the encoder is obtained by cascading an embedded convolutional layer, a first built-in channel attention CAB module, a first downsampling module, a second built-in channel attention CAB module, and a second downsampling module in sequence; the decoder includes a first upsampling module, a third built-in channel attention CAB module, a second upsampling module, a fourth built-in channel attention CAB module, and a mapping module; the skip connection part includes a first 1×1 convolutional layer and a second 1×1 convolutional layer; Current input features of the SRB module The input features are processed by the encoder via embedded convolutional layers. The output of the second downsampling module is then fed into the external channel attention CAB module for feature extraction. The output features of the external channel attention CAB module then enter the first upsampling module. The output of the second built-in channel attention CAB module is concatenated with the output of the first upsampling module and then fed into the first 1×1 convolutional layer. The output of the first 1×1 convolutional layer serves as the input to the third built-in channel attention CAB module, and its output enters the second upsampling module. The output of the first built-in channel attention CAB module is concatenated with the output of the second upsampling module and then fed into the second 1×1 convolutional layer. The output of the second 1×1 convolutional layer serves as the input to the fourth built-in channel attention CAB module, and its output enters the mapping module. The output and input features of the mapping module are then processed together. The sum of the two is used as the final output of the current SRB module; When the current SRB module is the first SRB module, the input features are... The output of the first 3×3 convolutional layer in the SRB branch, when the current SRB module is not the first SRB module, is the input feature. This is the final output of the SRB module preceding the current SRB module.

[0088] In other words, each SRB module employs a symmetrical encoder-decoder structure supplemented by skip connections to extract and reconstruct spectral information in a multi-scale manner. The initial part of the SRB is an embedding convolution, which transforms the input feature dimension into the internal processing dimension. Next, the encoder stage begins, consisting of multiple layers. Each layer first uses a CAB module to enhance spatial-channel joint attention, then downsamples the features using a 4×4 convolution with a stride of 2, doubling the number of channels. The features output from each layer are saved as skip connections. The deepest layer of the encoder is further processed by the bottleneck CAB module. The decoder stage then upsamples progressively: each level first restores the spatial resolution and halves the number of channels using a 2×2 transposed convolution, then concatenates the current features with the skip connection features of the corresponding encoder layer along the channel dimension, fuses them using a 1×1 convolution, and then feeds them into another CAB module for feature refinement. Finally, a 3×3 convolution maps the internal dimensions back to the input dimension, and adds it to the original input of the SRB to form the residual output.

[0089] It should be noted that the core attention unit within the entire SRB module is the CAB module, which in turn consists of multiple repeating blocks, such as... Figure 7 As shown, each block contains an SCA module and a feedforward network with normalized pre-layers. Spectral Channel Self-Attention (SCA) models inter-band correlations through normalized channel attention. The feedforward network (FFN) adaptively recalibrates the response with the channel attention to address the varying reconstruction difficulties of different spectral bands. The SCA module implements spatial cross-attention: it reshapes the input features into a sequence, generates queries, keys, and values ​​through linear transformation, and performs multi-head dot product attention. The attention score is scaled by a learnable scaling factor and normalized with Softmax. The attention output is multiplied by the value and then linearly projected back, while simultaneously superimposed with a positional encoding generated by depthwise convolution, ultimately connected to the input residual. The feedforward network consists of two 1×1 convolutional layers with a GELU activation inserted in between, followed by a channel attention mechanism. This mechanism collects channel descriptions through global average pooling, generates channel weights through two fully connected layers and a Sigmoid activation, recalibrates the feedforward output, and the result is also connected to the input residual. By stacking them alternately, the CAB module can effectively model long-distance dependencies and cross-channel interactions.

[0090] IV. Depth Estimation Branch 1. DINOv3 encoder We employ the newly proposed DINOv3 encoder, a state-of-the-art self-supervised vision foundation model. Small, foundational, and large models are created using ViT-S, ViT-B, and ViT-L variants, respectively. HDR-enhanced images I_HDR are input into the pre-trained DINOv3 encoder to extract multi-scale semantic features.

[0091] 2. DPT decoder The DPT (Dense Prediction Transformer) decoder architecture is adopted, and the multi-scale fusion strategy inspired by RefineNet is used to gradually refine the features and generate a depth map D ∈ R^(H×W).

[0092] 3. Adaptive edge-aware regularization The depth branch uses adaptive edge-aware TV regularization optimization to smooth flat areas while preserving edges, ensuring that the depth map has good smoothness while maintaining clear object boundaries.

[0093] In summary, this invention provides an end-to-end HDR-spectrum-depth joint optimization framework, in which the HDR module learns task-aware features through backpropagation of downstream task loss, without requiring ground-value HDR images. It employs a Luminescence-Aware Adaptive Modulation (LIAM) mechanism and its three-branch modulation strategy; a spectral reconstruction network (SRT) based on channel interpolation and adaptive total variational regularization to achieve a balance between preserving fine spectral structure and suppressing noise; an adaptive gating mechanism to dynamically allocate processing weights for different exposure regions through softmax normalization; a learnable attention mask fusion strategy to adaptively weight the contributions of the original input and HDR enhancement features; a spectral-depth cross-modal coupling optimization method to achieve synergy between geometric constraints and spectral cues through a unified loss function; a multi-scale feature spatial attention (FSA) module to emphasize spatial information regions such as material boundaries and texture variations; and a self-supervised depth estimation method based on DINOv3, combined with adaptive edge-aware regularization to maintain robustness of depth prediction under extreme lighting conditions.

[0094] Compared with the prior art, the present invention has the following advantages: 1. Significantly improved reconstruction accuracy: Experiments show that the spectral reconstruction achieves a PSNR of 34.70 dB (an improvement of 2.05 dB), an RMSE of 0.0200, and a spatial mean error of 0.0499; the depth estimation achieves an AbsRel of 0.165 (a reduction of 31.5%), and the threshold accuracy is 85.9%, 93.9%, and 96.5%.

[0095] 2. Real-time processing of a single shot: Abandoning the traditional multi-exposure fusion method, avoiding motion artifacts and ghosting problems, it is suitable for real-time processing of dynamic scenes.

[0096] 3. Robustness under extreme lighting: The adaptive modulation strategy of the LIAM mechanism successfully handles spatially non-uniform lighting, maintaining high-quality reconstruction in both overexposed and underexposed areas.

[0097] 4. End-to-end task-aware optimization: The HDR module learns indirectly through the loss of downstream tasks, without the need for HDR ground truth, ensuring that the feature representation perfectly matches the requirements of spectral and depth reconstruction.

[0098] 5. Maintain scene realism: Adaptive lighting modulation enhances feature representation while maintaining scene realism, avoiding distortion caused by overprocessing.

[0099] 6. Cross-modal collaborative enhancement: Joint optimization of spectral and depth information enables cross-modal collaboration, with depth providing geometric constraints to stabilize material estimation and spectral cues enhancing structure perception.

[0100] 7. Balance between detail preservation and noise suppression: Adaptive total variational regularization preserves fine spectral structure while suppressing high brightness noise, preventing over-smoothing and texture loss.

[0101] Of course, the present invention may have other various embodiments. Without departing from the spirit and essence of the present invention, those skilled in the art can make various corresponding changes and modifications according to the present invention, but these corresponding changes and modifications should all fall within the protection scope of the appended claims.

Claims

1. A single-exposure spectral-depth joint reconstruction method for high dynamic range scenes, characterized in that, Includes the following steps: Input a given low dynamic range image Input HDR high dynamic range preprocessing network Dynamic range extension is performed to obtain a high dynamic range image. ; High dynamic range images Multi-scale semantic features are extracted from the pre-trained visual base model DINO v3. Then, a lightweight refinement network was adopted. Multi-scale semantic features Upsampling and detail restoration are performed to output a pixel-level depth estimation map. ; High dynamic range images Feed into Transformer spectral reconstruction network Transformer spectral reconstruction network High dynamic range images through self-attention mechanism Perform correlation modeling and output the reconstructed hyperspectral image. .

2. The single-exposure spectral-depth joint reconstruction method for high dynamic range scenes as described in claim 1, characterized in that, Training the visual foundation model DINO v3 with the Transformer spectral reconstruction network Joint loss function at time as follows: in, For spectral branching loss function, for The corresponding balance coefficient, For the deep branch loss function, for The corresponding balance coefficient; in, The calculation method is as follows: in, The bias coefficient, This is the loss term used to constrain spatial smoothness. This is the loss term used to constrain the spectral continuity between adjacent bands. The loss term is used to suppress high-frequency artifacts by applying second-order Laplace regularization.

3. The single-exposure spectral-depth joint reconstruction method for high dynamic range scenes as described in claim 1, characterized in that, The HDR high dynamic range preprocessing network This includes the mask estimation branch, the state network branch, and the backbone network; The mask estimation branch is used to acquire low dynamic range input images. The corresponding mask; The state network branch is used to input images from low dynamic range. Four different scales of illumination features were extracted. The backbone network is used to synthesize local illumination features based on illumination features at four different scales; Apply the mask to the low dynamic range input image The result of element-wise multiplication is added to the local illumination features to obtain a high dynamic range image. .

4. The single-exposure spectral-depth joint reconstruction method for high dynamic range scenes as described in claim 3, characterized in that, The mask estimation branch is obtained by cascading a first 3×3 convolutional layer, a first Mish layer, a second 3×3 convolutional layer, a second Mish layer, and a 1×1 convolutional layer in sequence; Low dynamic range input image After being processed sequentially through the first 3×3 convolutional layer, the first Mish layer, the second 3×3 convolutional layer, the second Mish layer, and the 1×1 convolutional layer, the mask is output by the 1×1 convolutional layer.

5. The single-exposure spectral-depth joint reconstruction method for high dynamic range scenes as described in claim 3, characterized in that, The state network branch includes a cascaded branch and four 1×1 convolutional layers; wherein, the cascaded branch is obtained by cascading the first 1×1 convolutional layer, the first Mish layer, the first downsampling module, the second Mish layer, the second downsampling module, the third Mish layer, the third downsampling module, and the fourth Mish layer in sequence; Low dynamic range input image After processing via a concatenated branch, the output of the first Mish layer is fused using a second 1×1 convolutional layer to obtain the illumination features at the first scale. The output of the second Mish layer is fused using a third 1×1 convolutional layer to obtain the illumination features at the second scale. The output of the third Mish layer is fused using a fourth 1×1 convolutional layer to obtain the illumination features at the third scale. The output of the fourth Mish layer is fused using the fifth 1×1 convolutional layer to obtain the illumination features at the fourth scale. .

6. The single-exposure spectral-depth joint reconstruction method for high dynamic range scenes as described in claim 3, characterized in that, The backbone network includes an encoder, a decoder, and an external ResBlock module; wherein, the encoder is obtained by cascading a first built-in ResBlock module, a first downsampling module, a second built-in ResBlock module, a second downsampling module, a third built-in ResBlock module, and a third downsampling module in sequence; the decoder is obtained by cascading a first upsampling module, a fourth built-in ResBlock module, a second upsampling module, a fifth built-in ResBlock module, a third upsampling module, and a sixth built-in ResBlock module in sequence; Among them, the illumination features of the first scale Input the first built-in ResBlock module; second scale lighting features. Input the second built-in ResBlock module; the third scale of lighting features. Input the third built-in ResBlock module; fourth scale lighting features. The output of the third downsampling module is superimposed on the output of the external ResBlock module and then input into the external ResBlock module; the output of the external ResBlock module is used as the input of the first upsampling module; and the output of the sixth built-in ResBlock module is used as the final local illumination feature.

7. The single-exposure spectral-depth joint reconstruction method for high dynamic range scenes as described in claim 6, characterized in that, Each ResBlock module is obtained by cascading a LIAM layer, a first 3×3 convolutional layer, a Mish layer, and a second 3×3 convolutional layer in sequence; the LIAM layer includes an intensity compensation module and an adaptive modulation module; the final output of the LIAM layer... Output of the strength compensation module and the output of the adaptive modulation module The results are obtained by superposition. At the same time, the adaptive modulation module includes normal exposure branch, overexposure branch and underexposure branch. in, ,in, For the first Illumination characteristics at various scales , This is a 1×1 convolution operation. For brightness projection operation, For a multilayer perceptron function; if illumination features are not used as input to the external ResBlock module and the fourth to sixth built-in ResBlock modules, then the external ResBlock module and the fourth to sixth built-in ResBlock modules correspond to... =0; ,in, The output of the normal exposure branch, For the output of the overexposure branch, The output of the underexposed branch, and has , These are the spatial attention gates corresponding to the normal exposure branch, overexposure branch, and underexposure branch. For the Sigmoid function, These are independent convolutional mappings corresponding to the normal exposure branch, overexposure branch, and underexposure branch. For the output of the previous module cascaded in the current LIAM layer, the first built-in ResBlock module corresponds to... The value is 0, corresponding to the second and third built-in ResBlock modules. All are outputs of the downsampling module, corresponding to the fourth to sixth built-in ResBlock modules. This is the output of the upsampling module; the corresponding output of the external ResBlock module. Illumination characteristics at the fourth scale The result is superimposed with the output of the third downsampling module; At the same time, the input to the LIAM layer will be... The output of the ResBlock module is superimposed on the output of the second 3×3 convolutional layer.

8. The single-exposure spectral-depth joint reconstruction method for high dynamic range scenes as described in claim 1, characterized in that, The Transformer spectral reconstruction network It includes the CIM branch and the SRB branch; the SRB branch is obtained by cascading the first 3×3 convolutional layer, multiple SRB modules, and the second 3×3 convolutional layer in sequence. Transformer Spectral Reconstruction Network The final output Output from CIM branch The output of the first 3×3 convolutional layer The output of the second 3×3 convolutional layer The result of fusion is: in, For convolution operations, This is the activation function.

9. The single-exposure spectral-depth joint reconstruction method for high dynamic range scenes as described in claim 8, characterized in that, CIM branch output The method for obtaining it is as follows: in, This is a 1×1 convolution operation. For batch normalization operations, For GELU activation function, The feature space attention operation consists of a 5×5 depthwise separable convolution operation, a 1×1 point convolution operation, and a Sigmoid activation function operation.

10. The single-exposure spectral-depth joint reconstruction method for high dynamic range scenes as described in claim 8, characterized in that, Each SRB module includes an encoder, a decoder, a skip connection section, and an external channel attention CAB module. The encoder is obtained by cascading an embedded convolutional layer, a first built-in channel attention CAB module, a first downsampling module, a second built-in channel attention CAB module, and a second downsampling module. The decoder includes a first upsampling module, a third built-in channel attention CAB module, a second upsampling module, a fourth built-in channel attention CAB module, and a mapping module. The skip connection section includes a first 1×1 convolutional layer and a second 1×1 convolutional layer. Current input features of the SRB module The input features are processed by the encoder via embedded convolutional layers. The output of the second downsampling module is then fed into the external channel attention CAB module for feature extraction. The output features of the external channel attention CAB module then enter the first upsampling module. The output of the second built-in channel attention CAB module is concatenated with the output of the first upsampling module and then fed into the first 1×1 convolutional layer. The output of the first 1×1 convolutional layer serves as the input to the third built-in channel attention CAB module, and its output enters the second upsampling module. The output of the first built-in channel attention CAB module is concatenated with the output of the second upsampling module and then fed into the second 1×1 convolutional layer. The output of the second 1×1 convolutional layer serves as the input to the fourth built-in channel attention CAB module, and its output enters the mapping module. The output and input features of the mapping module are then processed together. The sum of the two is used as the final output of the current SRB module; When the current SRB module is the first SRB module, the input features are... The output of the first 3×3 convolutional layer in the SRB branch, when the current SRB module is not the first SRB module, is the input feature. This is the final output of the SRB module preceding the current SRB module.

Images