Semantic knowledge migration-based high dynamic range imaging method
By introducing domain transfer and knowledge distillation techniques into high dynamic range image reconstruction, and utilizing semantic segmentation networks and prior fusion modules, high-quality reconstruction of high dynamic range images is achieved, solving the cross-domain semantic matching problem and improving image quality and efficiency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- NORTHWESTERN POLYTECHNICAL UNIV
- Filing Date
- 2025-12-11
- Publication Date
- 2026-04-28
AI Technical Summary
Existing technologies struggle to effectively utilize semantic priors from the standard dynamic range domain for high dynamic range image reconstruction, especially under multi-exposure conditions where image detail is lost and contrast is insufficient. Furthermore, existing methods exhibit errors in cross-domain semantic matching.
By employing domain transfer and knowledge distillation techniques, the semantic priors of the standard dynamic range domain are implicitly embedded into the high dynamic range domain. Using a semantic segmentation network and a prior fusion module, cross-domain semantic knowledge transfer is achieved, thereby optimizing the high dynamic range image reconstruction model.
It improves the quality and consistency of high dynamic range image reconstruction, solves the cross-domain semantic matching problem, enhances image contrast and detail integrity, and maintains efficient inference performance.
Smart Images
Figure CN121937337A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of imaging technology, and more specifically to a high dynamic range imaging method based on semantic knowledge transfer. Background Technology
[0002] High dynamic range (HDR) imaging is a fundamental problem in computer vision, aiming to capture images with a wide range of light in the real world more accurately. Traditional imaging methods often suffer from underexposure or overexposure in strong or weak light, resulting in loss of image detail or insufficient contrast. HDR imaging combines multiple low dynamic range (LMR) images with different exposure levels and finely controls the dynamic range of each pixel to capture details in both bright and dark areas within a single image. However, the difficulty of recovering HDR images from multiple LMR images increases dramatically when the LMR images exhibit significant degradation and content loss. An effective method for recovering extremely degraded regions is to utilize semantic priors of the scene; however, to date, no work has specifically explored this approach for multi-exposure HDR imaging. First, existing prior extraction models are primarily designed for standard dynamic range (DMR) images, and the gap between the HDR and standard DMR image domains makes it difficult to effectively obtain high-quality priors for HDR imaging, especially considering that many HDR image datasets are stored in raw image formats, further exacerbating this challenge. Furthermore, even with appropriate semantic priors, using sRGB-based priors for high dynamic range (HDR) images in the original image format can introduce incorrect matches due to differences in data representation and dimensionality. In addition, HDR imaging involves degraded HDR images of multiple dynamic scenes, and the semantic information between different frames may differ significantly, resulting in incomplete and potentially conflicting prior information for individual HDR image frames.
[0003] It should be noted that the information disclosed in the background section above is only used to enhance the understanding of the background of the present invention, and therefore may include information that does not constitute prior art known to those skilled in the art. Summary of the Invention
[0004] To overcome the shortcomings of the prior art, the present invention aims to propose a semantic-based high dynamic range imaging method. This method bridges the gap between the semantic knowledge of the standard dynamic range domain and the high dynamic range domain by implicitly embedding semantic knowledge into the original image format domain through self-distillation of the standard dynamic range domain, thereby effectively utilizing the advanced capabilities of semantic priors for high dynamic range image reconstruction.
[0005] Other features and advantages of the invention will become apparent from the following detailed description, or may be learned in part by practice of the invention.
[0006] According to a first aspect of the present invention, a high dynamic range imaging method based on semantic knowledge transfer is provided, the method comprising: Acquire a series of low dynamic range images captured in dynamic scenes with different exposures, generate an initial high dynamic range image using an initial high dynamic range reconstruction model, and convert the initial high dynamic range image into a reconstructed image in the standard dynamic range domain through a domain transfer operation. The semantic segmentation network SAM is used to process the reconstructed image in the standard dynamic range domain to obtain semantic priors; the semantic priors are combined with the semantic priors to guide the reconstruction model to process the reconstructed image in the standard dynamic range domain to obtain a refined reconstructed image; Based on the knowledge distillation method, cross-domain semantic prior transfer is achieved at the three levels of content, color and features. The initial high dynamic range reconstruction model is optimized by combining loss functions. In the inference stage, only the optimized initial high dynamic range reconstruction model is used to output the final high dynamic range image.
[0007] In some exemplary embodiments, the conversion of the initial high dynamic range image into a reconstructed image in the standard dynamic range domain via domain migration operation specifically involves:
[0008] In the formula, This indicates a tone mapping operation, also known as a domain transfer operation. This represents the original reconstruction model, also known as the student model. Indicates model parameters, This represents the initial reconstructed image, i.e., the output of the student model. This represents the reconstructed image after tone mapping transformation to the standard dynamic range domain, i.e., the input to the teacher model. This indicates a low dynamic range image.
[0009] In some exemplary embodiments, the semantic prior-guided reconstruction model is composed of a multi-stage encoder-decoder structure, with each stage containing a source from... of .
[0010] In some exemplary embodiments, the step of using semantic priors combined with a semantic prior-guided reconstruction model to process the reconstructed image in the standard dynamic range domain to obtain a refined reconstructed image includes: The encoder encodes the reconstructed map from the standard dynamic range domain, and the decoder outputs the feature map; The semantic prior is interacted with the decoder output feature map at the pixel level through the prior fusion module FPN to obtain a refined feature map, thereby obtaining a refined reconstructed image.
[0011] In some exemplary embodiments, the prior fusion module projects the encoder output features into a query vector, converts the semantic prior into a key vector and a value vector, calculates an attention map through a transposed attention mechanism, and combines the feedforward network to output a refined feature map.
[0012] In some exemplary implementations, at the content level, by minimizing the pixel space The loss and the perceptual loss of the feature representation space are used to extract the semantic prior from the semantic prior to guide the reconstruction model to the initial reconstruction model:
[0013] in, It is the output of the semantic prior that guides the reconstruction model. This is the output of the initial reconstruction model. express loss, express The network passes through the first i Extracted after the second max pooling operation j Each convolutional feature It is a balanced hyperparameter.
[0014] In some exemplary implementations, at the color level, a semantically guided histogram loss function is introduced to ensure color consistency between the semantically prior-guided reconstruction model and the initial reconstruction model, while utilizing local geometric information obtained from scene semantics and global color information obtained from content:
[0015] in, Indicates element-wise multiplication. express use The first segment in the obtained segmentation graph i One channel, The histogram loss function is calculated over these regions. This represents the number of channels.
[0016] In some exemplary embodiments, at the feature level, a semantic knowledge alignment module is introduced, which includes two encoders, a mask feature mixer and a decoder, to minimize the mean square error between the enhanced features and the teacher features through encoding mapping, complementary random mask mixing and decoding reconstruction.
[0017] According to a second aspect of the present invention, a storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the high dynamic range imaging method based on semantic knowledge transfer described in the first aspect.
[0018] According to a third aspect of the present invention, a computer program product is provided, on which a computer program is stored, wherein when the computer program is executed by a processor, it implements the high dynamic range imaging method based on semantic knowledge transfer described in the first aspect above.
[0019] According to a fourth aspect of the present invention, an electronic device is provided, comprising: Processor; and Memory for storing the executable instructions of the processor; The processor is configured to implement the high dynamic range imaging method based on semantic knowledge transfer described in the first aspect above by executing the executable instructions.
[0020] The high dynamic range imaging method based on semantic knowledge transfer provided by the embodiments of the present invention has the following advantages compared with the prior art: 1. Breakthrough in cross-domain semantic transfer bottleneck: For the first time, the semantic prior of the standard dynamic range domain is transferred to the high dynamic range imaging scene. Through domain transfer alignment and self-distillation technology, the problem of poor adaptability of existing semantic models and mismatch caused by data format differences is solved, filling the gap in this technology direction.
[0021] 2. Improve the reconstruction quality of extremely degraded scenes: By using the SAM model to extract multi-scale semantic priors and combining the pixel-level interaction and transposed attention mechanism of the prior fusion module, the degraded areas of low dynamic range images are accurately repaired, while preserving details in both bright and dark areas, which greatly improves image contrast and detail integrity.
[0022] 3. Achieve multi-dimensional consistency constraints: Construct a distillation system from three levels: content, color, and features to ensure accurate alignment of the reconstructed image in terms of content, color distribution, and semantic features, and avoid cross-domain feature degradation.
[0023] 4. Balancing performance and inference efficiency: During the training phase, the initial reconstruction model is optimized through dual-link collaborative optimization. During the inference phase, only a single model needs to run, without relying on external semantic networks. This significantly improves the real-time performance and engineering practicality of high dynamic range image reconstruction without increasing inference complexity.
[0024] 5. Strong versatility and adaptability: No need to retrain a dedicated high dynamic range semantic model, it is compatible with various existing high dynamic range reconstruction frameworks, and is suitable for various data scenarios such as original image formats and sRGB formats, reducing the threshold for technology implementation.
[0025] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit the invention. Attached Figure Description
[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention. It is obvious that the drawings described below are merely some embodiments of the invention, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort.
[0027] Figure 1 This is a flowchart of the method of the present invention.
[0028] Figure 2 These are low dynamic range images with multiple frames and different exposures.
[0029] Figure 3 This is a high dynamic range image reconstructed using the method of this invention. Detailed Implementation
[0030] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, they are provided so that the invention will be more comprehensive and complete, and will fully convey the concept of the exemplary embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.
[0031] Furthermore, the accompanying drawings are merely illustrative of the invention and are not necessarily drawn to scale. The same reference numerals in the drawings denote the same or similar parts, and therefore repeated descriptions of them will be omitted. Some block diagrams shown in the drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0032] Using semantic priors to address ill-posed problems in low-level vision tasks holds great potential; however, no work has yet specifically explored this approach for multi-exposure high dynamic range (HMR) imaging. To address this shortcoming, this invention proposes a general framework to transfer semantic knowledge from pre-trained segmentation models in the standard dynamic range domain to enhance existing HMR imaging methods without retraining semantic prior models specifically for HMR. This invention proposes a semantic prior transfer scheme that utilizes knowledge distillation to efficiently achieve cross-domain semantic prior transfer at three levels: color, content, and features, thereby enhancing the content of reconstructed images. The method of this invention is more aligned with real-world needs, fully leveraging semantic priors for a deeper understanding of image content in HMR imaging.
[0033] This example implementation provides a high dynamic range imaging method based on semantic knowledge transfer, which may specifically include the following steps: Step 1: Use any existing high dynamic range imaging method to generate semantically rich high dynamic range images, and perform domain transformation on the obtained preliminary reconstructed images to convert them to the standard dynamic range domain; Step 2: Use a pre-trained semantic segmentation network to process the reconstructed image converted to the standard dynamic range domain to obtain semantic knowledge. Use the semantic knowledge and semantic priors to guide the reconstruction model to process the reconstructed image converted to the standard dynamic range domain to obtain a refined reconstructed image.
[0034] The steps in this exemplary embodiment will now be described in more detail with reference to the accompanying drawings and embodiments.
[0035] like Figure 1 As shown, the method of this invention includes two training paths: a semantic enhancement stage (second-stage model) and a basic reconstruction stage (first-stage model). During training, the first stage first provides an initial high dynamic range image reconstruction result, which is then unified to the display domain / sRGB domain by a domain transfer operator. The second stage, guided by the prior knowledge of a pre-trained segmentation backbone, generates a semantically enhanced reconstructed image. This enhanced result is then used as a teacher signal, and semantic information is transferred to the first stage through cross-domain self-distillation, enabling it to possess semantic perception capabilities without introducing an external semantic network. The inference stage retains only the first-stage model. The specific process is as follows: Step 1: As Figure 2 As shown, the results obtained from a series of dynamic scene shots with different exposures Low dynamic range (LVR) images are used to generate initial high dynamic range (HVR) images using existing HVR reconstruction models. These initial HVR images are then converted to sRGB domain results using a domain transfer operation.
[0036] in, This indicates a tone mapping operation, also known as a domain transfer operation. For sRGB format data, it includes a... The law function, for data in the original image format, also includes a differentiable demosaic operation; Represents the original reconstruction model. Indicates model parameters, This represents the initial reconstructed image, i.e., the output of the student model. This represents the reconstructed image after tone mapping transformation to the standard dynamic range domain, which is the input to the teacher model.
[0037] Step 2: Use a pre-trained semantic segmentation network (SegmentAnything Model) processes reconstructed images converted to the standard dynamic range domain to obtain semantic knowledge. It uses semantic knowledge combined with semantic priors to guide the reconstruction model to process reconstructed images converted to the standard dynamic range domain to obtain refined reconstructed images.
[0038] The semantic prior-guided reconstruction model consists of a multi-stage encoder-decoder structure, with each stage containing a sequence of semantic priors from the priors. of .
[0039] Sub-step 1: Encode the tone-mapped image obtained from the base reconstruction stage in step 1 using an encoder. Decoder output feature map The shape of the output feature map at each stage is as follows: ,in , and These represent the number of channels, height, and width of the input image, respectively.
[0040] Sub-step 2: Select The model acts as a semantic prior extractor, using its output features before representing the head to generate three multi-scale semantic priors. Having the corresponding The same , The result Through Feature Pyramid Network (FPN) and Perform multi-level feature interaction processing and adjust it to be compatible with... They have the same number of channels.
[0041] Sub-step 3: To integrate these semantic priors, a prior fusion module is introduced. This module is in and Pixel-level interactions are performed between them to produce a refined feature map. Specifically, the prior fusion module takes two inputs: an intermediate feature. and a prior feature Using pointwise 1×1 convolution and depthwise 3×3 convolution to... Projected onto query vector The weight is and Through similar operations, Convert to key vector Sum value vector Then, a transposed attention mechanism is used to compute the attention map. Therefore, the process can be described as follows:
[0042] In the formula, For trainable parameters, for feedforward network This is the normalization operator.
[0043] Sub-step 4: Using cross-format / cross-domain semantic consistency as a constraint framework, a knowledge distillation strategy is adopted to simultaneously enhance the alignment between the initial high dynamic range image and the semantic refinement result through two paths: color consistency and content consistency.
[0044] In the early stages of training, semantic priors guide the model reconstruction, enabling the generation of high dynamic range images with tone maps that incorporate semantic priors. The quality of the reconstructed image obtained from the initial reconstruction model is superior to that obtained from the initial reconstruction model, both subjectively and objectively. To narrow the performance gap between the two and endow the initial reconstruction model with reconstruction capabilities similar to those of the semantic prior-guided reconstruction model, a semantic prior transfer scheme is proposed. In the semantic prior transfer scheme, the high-quality reconstructed image obtained from step two is used as a teacher signal to guide the training of the high dynamic range reconstruction model from step one. The main contents of this scheme are as follows.
[0045] First, a key challenge when using the results of a semantically prior-guided reconstruction model as a teacher signal is the difference between the domains (high dynamic range / standard dynamic range) or representations (RAW / sRGB) operated by the initial reconstruction model and the semantically prior-guided reconstruction model. To address this, a simple yet effective solution is proposed: before performing prior distillation, a domain transfer operator is used to align the outputs of different models in terms of domain and representation. Previous work has only focused on content-level distillation, but color and content are also important in high dynamic range image reconstruction tasks; therefore, knowledge transfer is performed at both the content and color levels. At the content level, knowledge transfer is performed by minimizing the pixel space... The loss and the perceptual loss of the feature representation space are used to extract the semantic prior from the semantic prior to guide the reconstruction model to the initial reconstruction model:
[0046] in, Represents the region transfer operator; It is the output of the semantic prior that guides the reconstruction model. This is the output of the initial reconstruction model. express loss. express The network passes through the first The first extraction after the second max pooling operation Each convolutional feature It is a balanced hyperparameter.
[0047] At the color layer, a semantically guided histogram loss function is introduced to ensure color consistency between the semantically prior-guided reconstruction model and the initial reconstruction model. Simultaneously, local geometric information obtained from scene semantics and global color information obtained from content are utilized.
[0048] in, Indicates element-wise multiplication. express use The first segment in the obtained segmentation graph i One channel, It is the histogram loss function calculated over these regions.
[0049] During the training phase, semantic priors guide the reconstruction model using domain transfer labels for supervision, and through... The loss is optimized and denoted as :
[0050] In the formula: It is an image tag. This is the output of step two.
[0051] In the framework of improving reconstruction quality through semantic knowledge transfer, output-level distillation only aligns at the color and content levels, but the problem of semantic differences at the feature layer still exists. Since the teacher model operates in the sRGB / SDR domain while the student model operates in the HDR / RAW domain, their intermediate feature spaces exhibit inconsistent distributions. Direct alignment in the original feature space can lead to unstable gradient propagation or feature degradation. Therefore, this invention introduces a semantic knowledge alignment module to achieve progressive alignment of cross-domain semantic features within a potentially unified space, thereby enhancing distillation performance and semantic consistency.
[0052] The semantic knowledge alignment module mainly includes two encoders, which process the intermediate features of the teacher model and the student model respectively. The system performs feature embedding, a mask feature mixer, and a decoder to reconstruct the aligned enhanced features. .
[0053] The module generates corresponding representations in the latent space after encoding and mapping the features:
[0054] in, The encoder consists of two 3×3 convolutions and a ReLU activation function. and These represent the encoded intermediate features of the teacher model and the student model, respectively.
[0055] Then through a pair of complementary random masks By combining features from two domains, semantic partial replacement can be achieved.
[0056] in, This represents element-wise multiplication. Indicates decoder, This indicates the intermediate characteristics after mixing.
[0057] To minimize the feature differences between the teacher model and the student model in the latent semantic space, the semantic feature alignment loss is defined as:
[0058] in, The mean squared error is the factor that minimizes the enhanced features. With teacher characteristics The mean squared error is calculated. This loss implements a consistency constraint at the semantic level, enabling the student model to progressively learn the teacher's semantic representation in the latent space.
[0059] Meanwhile, the initial reconstruction model is supervised using its original reconstruction loss function, denoted as . All objective functions of the model in step one are denoted as follows: These losses are combined as follows:
[0060] in, and It's a hyperparameter, set to... and In the inference phase of this invention, only the initial reconstructed model participates in the inference process, without requiring semantic prior guidance for the reconstructed model to participate.
[0061] It should be noted that, as another aspect, this application also provides a storage medium, which may be included in an electronic device or may exist independently without being assembled into the electronic device. The storage medium carries one or more programs, which, when executed by an electronic device, cause the electronic device to perform the methods described in the following embodiments.
[0062] In one embodiment, this application provides a computer program product including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.
[0063] Furthermore, the above figures are merely illustrative of the processes included in the method according to exemplary embodiments of the present invention, and are not intended to be limiting. It is readily understood that the processes shown in the above figures do not indicate or limit the temporal order of these processes. Additionally, it is readily understood that these processes may be executed synchronously or asynchronously, for example, in multiple modules.
[0064] Other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention herein. This application is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of the invention are indicated by the claims.
[0065] It should be understood that the present invention is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is defined only by the appended claims.
Claims
1. A high dynamic range imaging method based on semantic knowledge transfer, characterized in that, The method includes: A series of low dynamic range images captured in dynamic scenes with different exposures are acquired, and an initial high dynamic range image is generated using an initial high dynamic range reconstruction model. The initial high dynamic range image is then converted into a reconstructed image in the standard dynamic range domain through a domain transfer operation. The semantic segmentation network SAM is used to process the reconstructed image in the standard dynamic range domain to obtain semantic priors; the semantic priors are combined with the semantic priors to guide the reconstruction model to process the reconstructed image in the standard dynamic range domain to obtain a refined reconstructed image; Based on the knowledge distillation method, cross-domain semantic prior transfer is achieved at the three levels of content, color and features. The initial high dynamic range reconstruction model is optimized by combining loss functions. In the inference stage, only the optimized initial high dynamic range reconstruction model is used to output the final high dynamic range image.
2. The high dynamic range imaging method based on semantic knowledge transfer according to claim 1, characterized in that, The process of converting the initial high dynamic range image into a reconstructed image in the standard dynamic range domain through domain migration operations specifically involves: In the formula, This indicates a tone mapping operation, also known as a domain transfer operation. This represents the original reconstruction model, also known as the student model. Indicates model parameters, This represents the initial reconstructed image, i.e., the output of the student model. This represents the reconstructed image after tone mapping transformation to the standard dynamic range domain, i.e., the input to the teacher model. This indicates a low dynamic range image.
3. The high dynamic range imaging method based on semantic knowledge transfer according to claim 1, characterized in that, The semantic prior-guided reconstruction model is composed of a multi-stage encoder-decoder structure, with each stage containing a source from... of .
4. The high dynamic range imaging method based on semantic knowledge transfer according to claim 3, characterized in that, The process of using semantic priors combined with a semantic prior-guided reconstruction model to process the reconstructed image in the standard dynamic range domain to obtain a refined reconstructed image includes: The encoder encodes the reconstructed map from the standard dynamic range domain, and the decoder outputs the feature map; The semantic prior is interacted with the decoder output feature map at the pixel level through the prior fusion module FPN to obtain a refined feature map, thereby obtaining a refined reconstructed image.
5. The high dynamic range imaging method based on semantic knowledge transfer according to claim 4, characterized in that, The prior fusion module projects the encoder output features into a query vector, converts the semantic prior into key vectors and value vectors, calculates an attention map through a transposed attention mechanism, and combines it with a feedforward network to output a refined feature map.
6. The high dynamic range imaging method based on semantic knowledge transfer according to claim 1, characterized in that, At the content level, by minimizing pixel space The loss and the perceptual loss of the feature representation space are used to extract the semantic prior from the semantic prior to guide the reconstruction model to the initial reconstruction model: in, It is the output of the semantic prior that guides the reconstruction model. This is the output of the initial reconstruction model. express loss, express The network passes through the first i The first extraction after the second max pooling operation j Each convolutional feature It is a balanced hyperparameter.
7. The high dynamic range imaging method based on semantic knowledge transfer according to claim 1, characterized in that, At the color level, a semantically guided histogram loss function is introduced to ensure color consistency between the semantically prior-guided reconstruction model and the initial reconstruction model. Simultaneously, it utilizes local geometric information obtained from scene semantics and global color information obtained from content. in, Indicates element-wise multiplication. express use The first segment in the obtained segmentation graph i One channel, The histogram loss function is calculated over these regions. This represents the number of channels.
8. The high dynamic range imaging method based on semantic knowledge transfer according to claim 1, characterized in that, At the feature level, a semantic knowledge alignment module is introduced, which includes two encoders, a mask feature mixer and a decoder. Through encoding mapping, complementary random mask mixing and decoding reconstruction, the mean square error between the enhanced features and the teacher features is minimized.
9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the high dynamic range imaging method based on semantic knowledge transfer as described in any one of claims 1 to 8.
10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the high dynamic range imaging method based on semantic knowledge transfer as described in any one of claims 1 to 8.