A high dynamic range data generation and generalization method

Abstract

The present application relates to a kind of high dynamic range data generation and generalization method, comprising: collecting and obtaining motion material and high dynamic range background data, and by rendering tool processing, obtain rendering dataset, that is, obtain high dynamic range motion dataset;Domain migration method is used, and the model trained on rendering dataset is generalized to real dataset.Compared with prior art, the present application can generate high-quality, large-scale dynamic scene high dynamic range data in batches on one hand, greatly reduce the cost of manpower and equipment, on the other hand, it can solve the distribution difference between rendering data and real data, support domain migration on labeled and unlabeled real data, so as to improve the robustness and generalization ability of high dynamic range reconstruction algorithm in real scene.

Description

Technical Field

[0001] This invention relates to the field of image data processing technology, and in particular to a method for generating and generalizing high dynamic range data. Background Technology

[0002] Multiple exposure synthesis is a common method for capturing High-Dynamic Range (HDR) images, particularly suitable for static scenes. When shooting dynamic scenes, strict control of moving subjects is crucial. This method relies on capturing multiple photos at different exposures and combining them into a single image with a wider dynamic range. Typically, at least three images are captured: an overexposed image, a normally exposed image, and an underexposed image. Then, specialized image processing algorithms are used to synthesize these images, capturing details within different brightness ranges of the scene and ultimately generating a high dynamic range image. In existing research, Kalantari et al. [1] The first high dynamic range (HMR) dataset was collected to support HMR reconstruction algorithm research, including 74 training pairs and 15 test pairs, providing a standardized platform for training deep learning-based HMR reconstruction models. Subsequently, these researchers used similar acquisition methods to collect multiple HMR datasets from different perspectives to support HMR reconstruction algorithm research. Tel et al. [2] A dataset focusing on foreground objects, environmental factors, and significant motion changes was collected, including 108 training samples and 36 test samples, each capturing a dynamic scene with significant foreground or camera motion. However, the number of motion elements and scene types remains relatively limited. Shu et al. [3] A similar ghosting removal dataset was constructed, containing 450 training samples and 50 test samples, further improving adaptability to dynamic scenes. With the development of mobile imaging, Kong et al. [4] A dataset with an extended range of motion and saturation regions was captured using a smartphone, containing 96 training samples and 27 test samples. However, this data acquisition method requires multiple shooting frames, thus placing high demands on the camera's stability and performance. Furthermore, when shooting scenes with moving objects, strict control of the motion material is necessary, as image alignment issues may occur between multiple exposures, leading to artifacts or motion blur. In addition, this data is still only in the hundreds, and the amount of data remains limited.

[0003] Another common method is to use a beam splitter to acquire high dynamic range (HDR) data. By splitting the light beam into two, and then photographing each beam separately with different exposure times, images with varying exposures are generated. These images can then be combined into a single HDR image. (Froehlich et al.) [5]Two beam splitters are used to acquire a batch of high dynamic range data. However, beam splitters cause some photon loss, which affects image quality, and for scenes with a very large dynamic range, the beam splitter method may not be able to capture a sufficient range of brightness.

[0004] In summary, existing methods for acquiring high dynamic range (HDR) data in dynamic scenes rely on controlling moving materials and setting up high dynamic range scenes. HDR data is synthesized by simultaneously capturing multiple images with different exposures. This process involves controlling the trajectory of moving objects and synthesizing multiple exposures, making it complex and costly, and difficult to acquire diverse HDR dynamic data on a large scale; the data volume is typically only around 100. Another method uses a beam splitter for HDR data acquisition, but this method not only causes some photon loss but also cannot acquire dynamic scenes with large dynamic ranges. Furthermore, because current HDR datasets have limited motion types and insufficient dynamic range, HDR reconstruction models trained on these datasets perform poorly in aligning the motion of dynamic objects, especially in scenes with large motion and high dynamic range, easily exhibiting visual defects such as ghosting, blurring, and artifacts, severely affecting the effectiveness and robustness of HDR reconstruction. In conclusion, while these methods have advanced HDR imaging, their limitations, particularly in dynamic scenes or scenes with extremely large dynamic ranges, still make it difficult to meet the research needs for high-quality and efficient HDR imaging algorithms.

[0005] The existing technical documents are as follows:

[0006] [1]Kalantari NK,Ramamoorthi R.Deep high dynamic range imaging ofdynamic scenes[J].ACM Transactions on Graphics(TOG),2017,36(4):1-12.

[0007] [2]Tel S, Wu Z, Zhang Y, et al.Alignment-free HDR Deghosting withSemantics Consistent Transformer[C] / / Proceedings of the IEEE / CVFInternational Conference on Computer Vision.2023:12836-12845.

[0008] [3]Shu Y,Shen L,Hu X,et al.Towards Real-World HDR VideoReconstruction:A Large-Scale Benchmark Dataset and A Two-Stage AlignmentNetwork[C] / / Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition.2024:2879-2888.

[0009] [4]Kong L, Li B, Xiong Y, et al.SAFNet: Selective Alignment FusionNetwork for Efficient HDR Imaging[C] / / European Conference on ComputerVision.Springer,Cham,2025:256-273.

[0010] [5]Froehlich J, Grandinetti S, Eberhardt B, et al.Creating cinematicwide gamut HDR-video for the evaluation of tone mapping operators and HDR-displays[C] / / Digital photography X.SPIE,2014,9023:279-288. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of the existing technology by providing a high dynamic range data generation and generalization method that can generate high-quality, large-scale dynamic scene HDR data in batches and solve the distribution difference between rendered data and real data.

[0012] The objective of this invention can be achieved through the following technical solution: a method for generating and generalizing high dynamic range data, comprising the following steps:

[0013] S1. Collect and acquire motion data and high dynamic range background data, and process them through rendering tools to obtain a rendering dataset, i.e., a high dynamic range motion dataset.

[0014] S2. Using the domain transfer method, the model trained on the rendered dataset is generalized to the real dataset.

[0015] Furthermore, in step S1, the motion material includes local motion, global motion, and mixed motion; the high dynamic range background includes indoor background and outdoor background; and the high dynamic range motion dataset includes indoor local motion, indoor global motion, indoor mixed motion, outdoor local motion, outdoor global motion, and outdoor mixed motion data.

[0016] Furthermore, step S2 specifically involves constructing a plug-and-play domain migration network structure to achieve generalization from rendered data to real data.

[0017] Furthermore, the domain transfer network structure includes a shared branch, a transfer branch, and a pre-trained network. The shared branch is used to control the sharing of the same knowledge between the rendered data and the real data, and the transfer branch is used to control the knowledge transfer from the rendered data to the real data.

[0018] Furthermore, the shared branch consists of two 1x1 convolutional layers. The first convolutional layer has the same number of input channels as the pre-trained network structure and has 1 output channel. The second convolutional layer has 1 input channel and the same number of output channels as the pre-trained network structure.

[0019] Furthermore, the transfer branch consists of two 1x1 convolutional layers. The first convolutional layer has the same number of input channels as the pre-trained network structure and 128 output channels. The second convolutional layer has 128 input channels and the same number of output channels as the pre-trained network structure.

[0020] Furthermore, the pre-trained network maintains the same structure as the original network. The working process of the domain transfer network structure includes: inputting the input features into the pre-trained network, the shared branch, and the transfer branch respectively; then multiplying the output of the shared branch by the scaling parameter α and the output of the transfer branch by the scaling parameter β; finally, adding the scaled outputs of the shared branch, the scaled outputs of the transfer branch, and the pre-trained network together as a new output feature to replace the output features of the original pre-trained network structure.

[0021] Furthermore, in step S2, if the real dataset has labels, the domain transfer network structure is trained using a gradient-optimized training method.

[0022] If the real dataset has no labels, the domain transfer network structure is trained using the domain transfer framework during testing.

[0023] Furthermore, the domain transfer framework includes a data augmentation module, a student model employing a domain transfer network structure, a teacher model employing a domain transfer network structure, and a moving average weight update module. The data augmentation module is used to augment the input low dynamic range image.

[0024] The student model receives low dynamic range images and obtains the student model output;

[0025] The teacher model receives the enhanced low dynamic range image and obtains multiple teacher model outputs;

[0026] The moving average weight update module uses the moving average weight update method to update the network weights of the teacher model.

[0027] Furthermore, the working process of the domain transfer framework includes: inputting the data-enhanced low dynamic range image into the teacher model to obtain the output results of N teacher models; calculating the mean μ and variance σ of these N output results, where the variance σ is used to update the scaling parameters α = 1 - σ and β = 1 + σ; the mean μ is used as a pseudo-label to calculate the loss function and perform gradient updates with the output results of the student network to optimize the parameters of the student model;

[0028] Furthermore, the parameters of the teacher model are updated using a moving average method to update the weights, and this teacher model is used as the final inference model.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] This invention collects motion footage and high dynamic range (HDL) background data, processes it using rendering tools to obtain a rendered dataset (i.e., a HDL motion dataset), and employs a domain migration method to migrate the rendered dataset to a real dataset. On one hand, it can generate high-quality, large-scale dynamic scene HDL data in batches, significantly reducing manpower and equipment costs. On the other hand, it can resolve the distribution differences between rendered data and real data, supporting domain migration on both labeled and unlabeled real data, thereby improving the robustness and generalization ability of the HDL reconstruction algorithm in real-world scenes.

[0031] This invention utilizes motion data and high dynamic range (HDR) backgrounds to construct HDR scenes. Through rendering tools, a large amount of motion data is combined with HDR backgrounds to efficiently render large-scale, high-quality HDR motion datasets, including indoor, outdoor, and various other motion types. This method significantly improves the diversity and scale of the dataset, providing richer training data for HDR reconstruction tasks.

[0032] This invention designs a plug-and-play domain transfer network structure, including a shared branch, a transfer branch, and a pre-trained network. The shared branch controls the sharing of the same knowledge between rendered data and real data, while the transfer branch controls the knowledge transfer from rendered data to real data. Through the synergistic effect of these two branches, generalization from rendered data to real data can be effectively achieved. Furthermore, when labeled real data is available, domain transfer can be directly completed through gradient optimization training. When the real data is unlabeled, a test-time domain transfer method is used, employing data augmentation, a student model and a teacher model based on the domain transfer network structure, and a moving average weight update method to ensure successful transfer even without labeled data. This solves the problem of being unable to build large-scale training data pairs or only being able to build a small amount of training data, effectively enabling domain transfer of data regardless of whether the real data is labeled. Attached Figure Description

[0033] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0034] Figure 2 This is a schematic diagram illustrating the construction of a high dynamic range scene in the embodiment;

[0035] Figure 3 This is a schematic diagram of high dynamic range motion data in the embodiment;

[0036] Figure 4 This is a schematic diagram of the domain migration network structure in this invention;

[0037] Figure 5 This is a schematic diagram illustrating the working process of the domain migration framework in this invention;

[0038] Figure 6 This is a schematic diagram comparing the reconstruction effect of the method of the present invention with other existing methods in the motion region;

[0039] Figure 7 This is a schematic diagram comparing the reconstruction effects of the method of the present invention with other existing methods in a high dynamic range scenario under direct sunlight. Detailed Implementation

[0040] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0041] Example

[0042] like Figure 1 As shown, a high dynamic range data generation and generalization method includes the following steps:

[0043] S1. Collect and acquire motion data and high dynamic range background data, and process them through rendering tools to obtain a rendering dataset, i.e., a high dynamic range motion dataset.

[0044] S2. Using the domain transfer method, the model trained on the rendered dataset is generalized to the real dataset.

[0045] In step S1, a high dynamic range scene is constructed using motion data and a high dynamic range background (e.g., Figure 2 (As shown). By using rendering tools, a large amount of motion footage is combined with a high dynamic range background, thereby efficiently rendering large-scale, high-quality high dynamic range motion datasets. For example... Figure 3 As shown, the rendered dataset includes high-quality, high-dynamic-range motion data for indoor and outdoor scenes, as well as various other motion types. This approach significantly enhances the diversity and scale of the dataset, providing richer training data for high-dynamic-range reconstruction tasks. By generating high-quality, large-scale dynamic scene high-dynamic-range data in batches, it drastically reduces manpower and equipment costs.

[0046] In step S2, to enable the model trained on rendered data to better generalize to real data, this scheme proposes a domain transfer method. This method effectively transfers the model trained on the rendered dataset to the real dataset, thereby significantly improving the model's performance. This domain transfer method adopts a plug-and-play design, and the specific domain transfer network structure is as follows: Figure 4 As shown, it includes a shared branch and a migration branch. The shared branch is used to control knowledge sharing, while the migration branch focuses on domain migration. Through the synergy of these two branches, it is possible to effectively generalize from rendered data to real data.

[0047] Specifically, the domain transfer network structure includes: a shared branch, a transfer branch, and a pre-trained network structure. The shared branch is used to control the sharing of the same knowledge between the rendered data and the real data. It consists of two 1x1 convolutional layers. The first convolutional layer has the same number of input channels as the pre-trained network structure and has 1 output channel. The second convolutional layer has 1 input channel and the same number of output channels as the pre-trained network structure.

[0048] The transfer branch is used to control the knowledge transfer from rendered data to real data. It consists of two 1x1 convolutional layers. The first convolutional layer has the same number of input channels as the pre-trained network structure and 128 output channels. The second convolutional layer has 128 input channels and the same number of output channels as the pre-trained network structure.

[0049] The pre-trained network maintains the same structure as the original network.

[0050] Finally, the output of the shared branch network is multiplied by the scaling parameter α, and the output of the transfer branch network is multiplied by the scaling parameter β, where the initial values ​​of scaling parameters α and β are 1. Then, the outputs of the scaled shared branch, the scaled transfer branch, and the pre-trained network structure are summed to form a new output feature, replacing the output feature of the original pre-trained network structure.

[0051] When labeled real data is available, domain transfer can be completed directly through simple training methods (such as gradient optimization).

[0052] When the real data has no labels, then use as follows: Figure 5 The domain transfer framework shown implements domain transfer, ensuring successful model transfer and improved performance even without labeled data. Specifically, this framework includes a data augmentation module, a student model with a domain transfer structure, a teacher model with a domain transfer structure, and a moving average weight update module. The data augmentation module enhances the input low dynamic range image using N different enhancement methods, including exposure enhancement and noise enhancement. The student model with the domain transfer structure receives the low dynamic range image as input and outputs the student model. The teacher model with the domain transfer structure receives the enhanced low dynamic range image as input and outputs N teacher models. The moving average weight update module updates the network weights of the teacher models using a moving average weight update method.

[0053] The domain transfer testing process is as follows: The data-augmented low dynamic range image is processed through the teacher model to obtain N outputs. The mean μ and variance σ of these N outputs are calculated. The variance is used to update the scaling parameters α = 1 - σ and β = 1 + σ. The mean is used as a pseudo-label to calculate the loss function and perform gradient updates with the student network's output, optimizing the student model's parameters. Then, the teacher model's parameters are updated using a moving average method to update the weights, and this teacher model is used as the final inference model. This entire process only needs to be executed once.

[0054] To verify the effectiveness of this solution, this embodiment compares the reconstruction results of the proposed method with other existing methods. Among these methods, SCTNet is a representative high dynamic range (HMR) reconstruction method based on an attention mechanism, while SAFNet is a representative method for HMR reconstruction using convolutional neural networks. In the SCTNet method, the dataset used is called the SCT dataset, while the SAFNet method uses the Challenge123 dataset. This embodiment conducts experimental verification based on these two state-of-the-art HMR reconstruction methods and their corresponding datasets. The results are as follows... Figure 6 and Figure 7 As shown, by Figure 6It can be seen that this scheme exhibits the best performance in the motion region, while other methods still show obvious ghosting phenomena when trained on different datasets. Figure 7 It can be seen that this scheme can achieve excellent reconstruction results in extremely high dynamic range scenarios such as direct sunlight, while other methods cannot achieve the same reconstruction quality when trained in the same scenario.

[0055] Existing high dynamic range (HDR) data acquisition methods are limited by data scale, resulting in insufficient generalization ability of models trained on small datasets. Using the large-scale, diverse datasets generated by this approach, the HDR reconstruction method based on these datasets shows significantly improved generalization and robustness across different scenes and datasets, especially in motion scenes and HDR scenes, achieving superior reconstruction results.

[0056] In summary, this solution addresses the performance bottleneck in high dynamic range (HMR) reconstruction caused by the scarcity of dynamic scene data. It proposes an efficient method for generating HMR training data, producing high-quality HMR data that provides diverse dynamic scenes and rich lighting variations, significantly increasing the quantity and diversity of HMR data. Utilizing the generated rendering data, richer training samples can be provided in deep learning models, thereby significantly improving the generalization ability and robustness of HMR reconstruction models. This effectively reduces ghosting, blurring, and artifacts in fast-moving and HMR scenes, enhancing the robustness and stability of HMR reconstruction models in dynamic scenes.

[0057] This solution leverages Unreal Engine's advanced graphics rendering technology to generate high-quality dynamic scene HDR data in batches, significantly reducing manpower and equipment costs. Through rendering tools, it constructs rich motion materials and dynamic range scenes, generating tens of thousands of high-quality, high dynamic range images, significantly improving data diversity and scale. To address the distributional discrepancy between rendered and real-world data, this solution proposes a plug-and-play domain transfer method, supporting domain transfer on both labeled and unlabeled real-world data. This enhances the robustness and generalization ability of high dynamic range reconstruction algorithms in real-world scenes, achieving generalization from rendered to real-world data regardless of label status. It solves the challenge of building large-scale training data pairs or only a small amount of training data, effectively performing domain transfer regardless of whether the real-world data is labeled, especially in data-constrained tasks such as high dynamic range reconstruction. This solution significantly improves the high dynamic range imaging effect of high dynamic range and dynamic motion scenes, demonstrating high innovation and broad application prospects, applicable to fields such as computer vision, image enhancement, autonomous driving, and virtual reality.

Claims

1. A method for generating and generalizing high dynamic range data, characterized in that, Includes the following steps: S1. Collect and acquire motion data and high dynamic range background data, and process them through rendering tools to obtain a rendering dataset, i.e., a high dynamic range motion dataset. S2. Using the domain transfer method, the model trained on the rendered dataset is generalized to the real dataset; In S2, if the real dataset has labels, the domain transfer network structure is trained using gradient optimization. If the real dataset has no labels, the domain transfer network structure is trained using the domain transfer framework during testing. The domain transfer framework includes a data augmentation module, a student model with a domain transfer network structure, a teacher model with a domain transfer network structure, and a moving average weight update module. The data augmentation module is used to augment the input low dynamic range image. The student model receives low dynamic range images and obtains the student model output; The teacher model receives the enhanced low dynamic range image and obtains multiple teacher model outputs; The moving average weight update module uses the moving average weight update method to update the network weights of the teacher model; The domain transfer framework operates by: inputting a data-enhanced low dynamic range image into a teacher model to obtain N outputs from the teacher model, and then averaging these N outputs. and variance Calculate, where variance Used to update scaling parameters =1- , =1+ Mean As pseudo-labels, they are used to calculate the loss function and perform gradient updates with the output of the student network, thereby optimizing the parameters of the student model. Furthermore, the parameters of the teacher model are updated using a moving average method to update the weights, and this teacher model is used as the final inference model.

2. The high dynamic range data generation and generalization method according to claim 1, characterized in that, The motion data in step S1 includes local motion, global motion, and mixed motion; the high dynamic range background includes indoor background and outdoor background; and the high dynamic range motion dataset includes indoor local motion, indoor global motion, indoor mixed motion, outdoor local motion, outdoor global motion, and outdoor mixed motion data.

3. The high dynamic range data generation and generalization method according to claim 1, characterized in that, Step S2 specifically involves constructing a plug-and-play domain migration network structure to achieve generalization from rendered data to real data.

4. The high dynamic range data generation and generalization method according to claim 3, characterized in that, The domain transfer network structure includes a shared branch, a transfer branch, and a pre-trained network. The shared branch is used to control the sharing of the same knowledge between the rendered data and the real data, and the transfer branch is used to control the knowledge transfer from the rendered data to the real data.

5. The high dynamic range data generation and generalization method according to claim 4, characterized in that, The shared branch consists of two 1x1 convolutional layers. The first convolutional layer has the same number of input channels as the pre-trained network structure and 1 output channel. The second convolutional layer has 1 input channel and the same number of output channels as the pre-trained network structure.

6. The high dynamic range data generation and generalization method according to claim 4, characterized in that, The transfer branch consists of two 1x1 convolutional layers. The first convolutional layer has the same number of input channels as the pre-trained network structure and 128 output channels. The second convolutional layer has 128 input channels and the same number of output channels as the pre-trained network structure.

7. The high dynamic range data generation and generalization method according to claim 4, characterized in that, The pre-trained network maintains the same structure as the original network. The operation of the domain transfer network structure includes: inputting the input features into the pre-trained network, the shared branch, and the transfer branch respectively, and then multiplying the output of the shared branch by a scaling parameter. Multiply the output of the migration branch by the scaling parameter. Finally, the scaled shared branch, the scaled transfer branch, and the output of the pre-trained network are added together to form a new output feature, which replaces the output feature of the original pre-trained network structure.

Images