High dynamic range imaging exposure interpolation method, device, equipment and medium

Abstract

The invention relates to the technical field of image processing, and provides a high dynamic range imaging exposure interpolation method and device, equipment and a medium. The method comprises the following steps: acquiring a high-exposure image and a low-exposure image for the same scene, wherein the exposure time of the high-exposure image is longer than that of the low-exposure image; respectively mapping the high-exposure image and the low-exposure image to a target exposure time to obtain a first intermediate image and a second intermediate image; performing weighted fusion processing according to the first intermediate image and the second intermediate image to generate an initial intermediate exposure image; and inputting the initial intermediate exposure image into a pre-trained residual diffusion model, and performing residual detail recovery on the initial intermediate exposure image through the pre-trained residual diffusion model so as to generate a high dynamic range image based on the recovered residual details. According to the method, through a technical path of combining physical exposure interpolation and residual diffusion optimization, high-quality and high-dynamic range image reconstruction with natural vision and rich details is realized.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a high dynamic range imaging exposure interpolation method, apparatus, device, and medium. Background Technology

[0002] High Dynamic Range (HDR) imaging, with its ability to render rich brightness levels and scene details, has wide applications in computational photography, virtual reality, consumer electronics, and image enhancement. However, traditional HDR generation methods mostly rely on capturing multiple images with different exposures and then fusing them. This method is prone to misalignment and ghosting in dynamic scenes or when shooting handheld, resulting in noticeable artifacts in the final image and severely affecting the quality and stability of the final image.

[0003] With the continuous development of imaging hardware, new HDR devices can capture two aligned images with a large exposure ratio at once, effectively eliminating ghosting. However, this method can only provide two extreme exposure images, and generating high-quality HDR images directly under these conditions remains quite challenging. To avoid the limitations of shooting multiple images, some methods attempt to achieve exposure interpolation through single images or physical modeling. However, due to the nonlinearity of the actual camera response, model-based physical interpolation methods often underperform in bright or complex texture areas, easily resulting in "Brightness Order Reversal (BOR)"—where highlights in a dark image may be darker than shadows in a bright image, leading to unnatural visual effects. To address this issue, the concept of exposure interpolation has been proposed, which involves first synthesizing an intermediate exposure image as a transition, and then optimizing the details of this intermediate exposure image. While model-based physical interpolation methods can maintain brightness consistency, their performance under extreme conditions is very limited due to their reliance on simplified camera response functions, and they often suffer from detail loss or color deviation.

[0004] In recent years, deep learning technology has been widely used in HDR imaging tasks. Among them, convolutional neural networks can learn end-to-end mapping relationships, but due to limitations in network structure, they still have shortcomings in capturing complex illumination distributions and nonlinear response characteristics. In addition, diffusion models have shown strong detail recovery capabilities in image generation and restoration, but when directly applied to HDR image restoration, the computational cost is high and the convergence is unstable. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a high dynamic range imaging exposure interpolation method, apparatus, device and medium to solve the above-mentioned technical problem.

[0006] The technical solution of this invention to solve the above-mentioned technical problems is as follows: a high dynamic range imaging exposure interpolation method, comprising: acquiring a high-exposure image and a low-exposure image for the same scene, wherein the exposure time of the high-exposure image is greater than the exposure time of the low-exposure image; mapping the low-exposure image and the high-exposure image to a target exposure time respectively to obtain a first intermediate image and a second intermediate image; performing weighted fusion processing on the first intermediate image and the second intermediate image to generate an initial intermediate exposure image; inputting the initial intermediate exposure image into a pre-trained residual diffusion model, performing residual detail restoration on the initial intermediate exposure image through the pre-trained residual diffusion model, and generating a high dynamic range image based on the restored residual details.

[0007] The beneficial effects of this invention are as follows: This invention uses a technical approach that combines physical exposure interpolation with residual diffusion optimization. It generates an initial intermediate exposure image with a reasonable structure and no ghosting artifacts using a physical model, and restores high-frequency details such as texture and edges using a residual diffusion model. While effectively suppressing the brightness order reversal phenomenon, it significantly improves the ability to restore details, and finally achieves high-quality, high dynamic range image reconstruction with natural visuals and rich details.

[0008] Based on the above technical solution, the present invention can be further improved as follows.

[0009] Further, the step of mapping the low-exposure image and the high-exposure image to a target exposure time to obtain a first intermediate image and a second intermediate image includes: obtaining the exposure time of the high-exposure image and the exposure time of the low-exposure image; calculating the irradiance corresponding to each pixel in the high-exposure image and the irradiance corresponding to each pixel in the low-exposure image according to the camera response model, the exposure time of the high-exposure image, and the exposure time of the low-exposure image; calculating the pixel value of each pixel in the high-exposure image and the pixel value of each pixel in the low-exposure image at the target exposure time according to the camera response model, the irradiance corresponding to each pixel in the high-exposure image, the irradiance corresponding to each pixel in the low-exposure image, and the target exposure time; obtaining the first intermediate image based on the pixel values ​​of each pixel in the low-exposure image at the target exposure time; and obtaining the second intermediate image based on the pixel values ​​of each pixel in the high-exposure image at the target exposure time.

[0010] Further, the step of generating an initial intermediate exposure image by performing weighted fusion processing based on the first intermediate image and the second intermediate image includes: calculating a first weight corresponding to each pixel in the first intermediate image based on the pixel value of each pixel in the first intermediate image and a preset minimum pixel value; calculating a second weight corresponding to each pixel in the second intermediate image based on the pixel value of each pixel in the second intermediate image and a preset maximum pixel value; and calculating the pixel value of each pixel in the initial intermediate exposure image based on the first weight of each pixel in the first intermediate image, the second weight of each pixel in the second intermediate image, the pixel value of each pixel in the first intermediate image, and the pixel value of each pixel in the second intermediate image, thereby obtaining the initial intermediate exposure image.

[0011] Furthermore, the pre-trained residual diffusion model is trained as follows: multiple sets of training data are acquired, each set of training data including a target high dynamic range image and an initial intermediate exposure image corresponding to the target high dynamic range image; for each set of training data, random noise conforming to a Gaussian distribution is gradually injected into the target high dynamic range image, and residual information is gradually injected into the target high dynamic range image according to a preset ratio, until it transitions to the initial intermediate exposure image corresponding to the target high dynamic range image, resulting in multiple noisy images; wherein, the residual information represents the residual information between the target high dynamic range image and the initial intermediate exposure image corresponding to the target high dynamic range image; for each set of training data, the initial intermediate exposure image corresponding to the target high dynamic range image is used as a conditional input to train the neural network to learn how to gradually recover residual information from different noisy images.

[0012] Furthermore, the step of restoring residual details in the initial intermediate exposure image using the pre-trained residual diffusion model to generate a high dynamic range image based on the restored residual details includes: injecting initial noise conforming to a Gaussian distribution into the initial intermediate exposure image using the pre-trained residual diffusion model to construct an initial noise state; and based on the initial noise state and the initial intermediate exposure image, progressively predicting residual information using the pre-trained residual diffusion model to generate the high dynamic range image.

[0013] Furthermore, the step of generating the high dynamic range image by progressively predicting residual information using the pre-trained residual diffusion model based on the initial noise state and the initial intermediate exposure image includes: starting from the initial noise state, progressively performing the following operations until a zero noise level is reached: predicting the residual information of the current step using a trained neural network based on the current noise state and the initial intermediate exposure image; calculating the noise state of the next step using the pre-trained residual diffusion model based on the predicted residual information; and when the iteration process is completed, the final noise state is the high dynamic range image.

[0014] Furthermore, the step of mapping the low-exposure image and the high-exposure image to the target exposure time to obtain the first intermediate image and the second intermediate image further includes: for a first pixel point in the high-exposure image or the low-exposure image whose corresponding pixel value is located between two adjacent integer pixel values, calculating the irradiance corresponding to the first pixel point by linear interpolation based on the two adjacent integer pixel values ​​and the irradiance corresponding to the two adjacent integer pixel values.

[0015] To address the aforementioned technical problems, the present invention also provides a high dynamic range imaging exposure interpolation device, comprising: An image acquisition module is used to acquire a high-exposure image and a low-exposure image of the same scene, wherein the exposure time of the high-exposure image is greater than the exposure time of the low-exposure image; An image mapping module is used to map the low-exposure image and the high-exposure image to a target exposure time, respectively, to obtain a first intermediate image and a second intermediate image; The weighted fusion module is used to perform weighted fusion processing on the first intermediate image and the second intermediate image to generate an initial intermediate exposure image; The image generation module is used to input the initial intermediate exposure image into a pre-trained residual diffusion model, and to perform residual detail restoration on the initial intermediate exposure image through the pre-trained residual diffusion model, so as to generate a high dynamic range image based on the restored residual details.

[0016] To address the aforementioned technical problems, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the high dynamic range imaging exposure interpolation method as described above.

[0017] To address the aforementioned technical problems, the present invention also provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute a high dynamic range imaging exposure interpolation method as described above. Attached Figure Description

[0018] Figure 1 This is a flowchart of a high dynamic range imaging exposure interpolation method according to the present invention; Figure 2 This is a schematic diagram of the initial intermediate exposure image generation process in Embodiment 1 of the present invention; Figure 3 This is a schematic diagram of a high dynamic range imaging exposure interpolation method according to Embodiment 1 of the present invention; Figure 4 This is a schematic diagram of a high dynamic range imaging exposure interpolation device according to the present invention; Figure 5 This is a schematic diagram of an electronic device according to the present invention. Detailed Implementation

[0019] The principles and features of the present invention are described below. The examples given are only for explaining the present invention and are not intended to limit the scope of the present invention.

[0020] The purpose of this invention is to address the shortcomings of existing HDR exposure interpolation methods by proposing an HDR exposure interpolation method that combines physical modeling and residual diffusion correction. This method can improve detail recovery while ensuring global exposure accuracy and significantly reduce the probability of BOR artifacts, achieving natural and clear HDR image reconstruction.

[0021] Example 1 Based on this, such as Figure 1 As shown, this embodiment provides a high dynamic range imaging exposure interpolation method, including: S101. Obtain a high-exposure image and a low-exposure image for the same scene, wherein the exposure time of the high-exposure image is longer than the exposure time of the low-exposure image.

[0022] S102. Map the low-exposure image and the high-exposure image to the target exposure time respectively to obtain the first intermediate image and the second intermediate image.

[0023] S103. Perform weighted fusion processing on the first intermediate image and the second intermediate image to generate an initial intermediate exposure image.

[0024] S104. Input the initial intermediate exposure image into the pre-trained residual diffusion model, and restore the residual details of the initial intermediate exposure image through the pre-trained residual diffusion model to generate a high dynamic range image based on the restored residual details.

[0025] This method combines physical exposure interpolation with residual diffusion optimization. It uses a physical model to generate an initial intermediate exposure image with a reasonable structure and no ghosting artifacts, and uses a residual diffusion model to restore high-frequency details such as texture and edges. While effectively suppressing the brightness order reversal phenomenon, it significantly improves the ability to restore details, and finally achieves high-quality, high dynamic range image reconstruction with natural visuals and rich details.

[0026] The two input images are a low-exposure image and a high-exposure image captured in the same scene, with different exposure times, denoted as t1 and t2 respectively. Low-exposure images retain details in bright areas, while high-exposure images retain details in dark areas. For acquiring both low-exposure and high-exposure images, new HDR cameras can output a pair of aligned LDR images simultaneously in a single shutter release, or they can be acquired sequentially by a conventional camera over short time intervals.

[0027] Optionally, in an embodiment, mapping the low-exposure image and the high-exposure image to a target exposure time to obtain a first intermediate image and a second intermediate image includes: obtaining the exposure time of the high-exposure image and the exposure time of the low-exposure image; calculating the irradiance corresponding to each pixel in the high-exposure image and the irradiance corresponding to each pixel in the low-exposure image based on the camera response model, the exposure time of the high-exposure image, and the exposure time of the low-exposure image; calculating the pixel value of each pixel in the high-exposure image and the pixel value of each pixel in the low-exposure image at the target exposure time based on the camera response model, the irradiance corresponding to each pixel in the high-exposure image, the irradiance corresponding to each pixel in the low-exposure image, and the target exposure time; obtaining the first intermediate image based on the pixel values ​​of each pixel in the low-exposure image at the target exposure time; and obtaining the second intermediate image based on the pixel values ​​of each pixel in the high-exposure image at the target exposure time.

[0028] Specifically, in the imaging principle of digital cameras, pixel value... It is the result obtained after the incident light undergoes a series of nonlinear processing. Specifically, for each color channel... Pixel values ​​can be represented as: ; in, Represents irradiance per unit area per unit time. It's the exposure time. And... The Camera Response Function (CRF) includes non-linear mappings such as analog-to-digital conversion and gamma compression. Its purpose is to convert intensity-level values ​​into pixel-level values, mapping image values ​​to pixel values. This formula can be derived as follows: .

[0029] To extract from low-exposure images With high-exposure images Intermediate exposure image of the composite target Assuming the required exposure times for the three images are respectively , , It is necessary to construct intensity mapping functions (IMFs) to map pixel values ​​in the reference image to response values ​​under the target exposure conditions.

[0030] Specifically, let each color channel of the i-th image (i=1,2) be... Its mapping function is expressed as It can be calculated based on different time ratios: ; This formula combines the above. The calculation formula can then be derived as follows: ; in, Indicates pixel value The inverse mapping is the inverse response function of irradiance. This is the exposure time ratio between the target and the reference image. This formula can be viewed as a process of scaling the reference image in the irradiance domain and then mapping it back to the pixel domain.

[0031] Based on the above formula, the low-exposure image and the high-exposure image are processed separately, mapping the two images with different exposure ratios to two corresponding intermediate images, thus generating the first intermediate image. Second intermediate image , This represents the pixel value at pixel p in the low-exposure image. This represents the pixel value at pixel p in the high-exposure image. Overall, as shown... Figure 2 The image shown is after the IMF module.

[0032] Optionally, in an embodiment, mapping the low-exposure image and the high-exposure image to the target exposure time respectively to obtain a first intermediate image and a second intermediate image further includes: for a first pixel point in the high-exposure image or the low-exposure image whose corresponding pixel value is located between two adjacent integer pixel values, calculating the irradiance corresponding to the first pixel point by linear interpolation based on the two adjacent integer pixel values ​​and the irradiance corresponding to the two adjacent integer pixel values.

[0033] In practice, the inverse function of the camera response function The values ​​often lie between the irradiance values ​​corresponding to two adjacent integer pixel values, making it impossible to directly obtain the precise value of that pixel location. Existing methods typically use an exponential function to fit the scatter plot for estimation, but while the fitted curve is smooth, it cannot guarantee that all the pixels to be determined fall exactly on the curve, which can easily affect the quality of the final interpolated image.

[0034] Therefore, this embodiment employs a linear interpolation method to calculate the interpolation point for the intermediate pixel value between adjacent known points. An "interpolation point" refers to the intermediate pixel position located between two known pixel values ​​but not directly corresponding to it in the original response function.

[0035] Specifically, if the pixel value is known , The response function and inverse function correspond to the irradiance respectively. , Then for those located in the interval any pixel value within Its irradiance can be obtained using the following formula. : ; in, That is, the interpolation point. This is the corresponding interpolation result.

[0036] Since pixel values ​​are densely distributed in real images, linear interpolation can maintain the smoothness of the curve and ensure that all interpolation points are located on the approximate trajectory of the true response function, thereby improving the accuracy of the response function fitting and generating intermediate exposure images that are closer to the real scene.

[0037] Optionally, in an embodiment, generating an initial intermediate exposure image by performing weighted fusion processing on the first intermediate image and the second intermediate image includes: calculating a first weight corresponding to each pixel in the first intermediate image based on the pixel value of each pixel in the first intermediate image and a preset minimum pixel value; calculating a second weight corresponding to each pixel in the second intermediate image based on the pixel value of each pixel in the second intermediate image and a preset maximum pixel value; and calculating the pixel value of each pixel in the initial intermediate exposure image based on the first weight of each pixel in the first intermediate image, the second weight of each pixel in the second intermediate image, the pixel value of each pixel in the first intermediate image, and the pixel value of each pixel in the second intermediate image, thereby obtaining the initial intermediate exposure image.

[0038] As previously noted, the IMF has reliability issues: when mapping from underexposed areas to brighter images, or from overexposed areas to darker images, IMF estimates are often inaccurate, easily causing artifacts or loss of detail. Therefore, injection is necessary. , This ensures that different areas are treated differently for different exposures. For example, pixels in bright areas of high exposure are weakened, while pixels in dark areas of low exposure are weakened, thus ensuring that low-confidence areas are not trusted.

[0039] Specifically, in order to combine two corresponding intermediate images , merging to generate an initial intermediate image For different pixel points p, , The pixel value corresponding to pixel p in the middle The pixel value at pixel p in the initial intermediate image generated by weighted fusion The formula is as follows: ; in, and These are the pixel values ​​at pixel p in the low-exposure and high-exposure images, respectively. , The fusion weights are based on pixel saturation and local brightness, and are defined as follows: ; .

[0040] and , Defined as: ; ; in, Represents pixel value, and These represent the lower and upper bounds of the pixel value, respectively. (Function) and This is used to normalize pixel values, allowing them to be linearly scaled in both low and high exposure regions to ensure the stability and controllability of the mapping process. Images with different exposures undergo... , The generated image is Figure 2 The result is shown after processing by the WeightFunction module.

[0041] This operation effectively alleviates the brightness flip problem and provides a more stable and natural intermediate exposure image.

[0042] Although this physical modeling method weights different parts of the image with different exposures—for example, discarding information from the highlight areas (sunlight areas) of a high-exposure image by setting their weight to 0—the same applies to the shadow areas of a low-exposure image. This method selectively discards areas with poor detail, providing a structurally sound estimate of intermediate exposure images. However, due to its inability to effectively model complex regions such as nonlinear materials and lighting boundaries, it may still suffer from a lack of detail. Therefore, this method further introduces a diffusion model to improve the performance of the initial image. Perform residual refinement.

[0043] Diffusion models based on residual learning are better suited for processing structured and sparse residual information. They can effectively compensate for high-frequency details while preserving physical consistency, making them suitable for residual optimization processing of intermediate exposure images.

[0044] Optionally, in an embodiment, the pre-trained residual diffusion model is trained as follows: multiple sets of training data are acquired, each set of training data including a target high dynamic range image and an initial intermediate exposure image corresponding to the target high dynamic range image; for each set of training data, random noise conforming to a Gaussian distribution is gradually injected into the target high dynamic range image, and residual information is gradually injected into the target high dynamic range image according to a preset ratio, until it transitions to the initial intermediate exposure image corresponding to the target high dynamic range image, resulting in multiple noisy images; wherein, the residual information represents the residual information between the target high dynamic range image and the initial intermediate exposure image corresponding to the target high dynamic range image; for each set of training data, the initial intermediate exposure image corresponding to the target high dynamic range image is used as a conditional input to train the neural network to learn how to gradually recover residual information from different noisy images.

[0045] For each set of training data, three corresponding images are obtained: a high-exposure image, a low-exposure image, and a target high dynamic range image for the same scene. For each pair of high-exposure and low-exposure images, mapping and weighted fusion processing are performed sequentially using the physical modeling method described above to obtain an initial intermediate exposure image. The target high dynamic range image and its corresponding initial intermediate exposure image are used to train the diffusion model.

[0046] Optionally, in an embodiment, residual detail recovery is performed on the initial intermediate exposure image using a pre-trained residual diffusion model to generate a high dynamic range image based on the recovered residual details. This includes: injecting initial noise conforming to a Gaussian distribution into the initial intermediate exposure image using the pre-trained residual diffusion model to construct an initial noise state; and based on the initial noise state and the initial intermediate exposure image, progressively predicting residual information using the pre-trained residual diffusion model to generate a high dynamic range image.

[0047] Optionally, in an embodiment, based on the initial noise state and the initial intermediate exposure image, residual information is predicted step by step using a pre-trained residual diffusion model to generate a high dynamic range image, including: starting from the initial noise state, performing the following operations step by step until a zero noise level is reached: predicting the residual information of the current step using a trained neural network based on the current noise state and the initial intermediate exposure image; calculating the noise state of the next step using a pre-trained residual diffusion model based on the predicted residual information; and when the iteration process is completed, the final noise state is the high dynamic range image.

[0048] Assuming that The corresponding HDR image is It is worth noting that, unlike generating the target image directly from scratch, the neural enhancement method proposed in this application only requires the diffusion model to learn the residuals.

[0049] for The residual diffusion optimization is denoted as the intermediate exposure image generated by the model-based physical exposure interpolation method. The target HDR image is denoted as The residual between the two is defined as: .

[0050] The residual contains detailed areas that are difficult to model, such as textures, edges, and highlights. To effectively recover these details, the residual repair process is embedded in the diffusion chain, allowing the model to focus on compensating for non-modeled areas.

[0051] In the forward process: from the target HDR image Initially, residuals are gradually injected to construct intermediate states. Finally, it transitions to the initial image obtained from physical modeling. The image contains noise information added by the diffusion model. This step is done during the training of the diffusion model to allow it to fit and learn the forward process of diffusion, facilitating the establishment and learning of the forward propagation process. Residual modeling and denoising.

[0052] Define the initial residual information as follows: And introduce a custom monotonically increasing sequence of residual weights. This is used to control the amount of residual information injected during each diffusion step. The magnitude of the residual injection in each diffusion step... Defined as: ;in , Then the specific residual injection amount for each diffusion step can be determined by... Let represent this. Therefore, the transfer distribution of the forward diffusion process can be expressed as: ; in, This represents the intermediate image state from the previous time step. The current time step state (including residual injection and noise); Here is the noise covariance matrix; The mean is Covariance is The Gaussian distribution.

[0053] This design ensures that the model can progressively learn the residual distribution and noise perturbations during forward diffusion, providing effective priors for inverse denoising and residual recovery. Further parameterization techniques can be used to refine the model. Perform efficient sampling. Meanwhile, Sampling can be performed using the following reparameterization techniques: .

[0054] In the reverse process: The goal of the reverse diffusion process is to enable the diffusion model to learn based on physical interpolation results. (Intermediate exposure image), gradually restoring its relationship with the target HDR image. The residual process compensates for high-frequency details and nonlinear region errors that physical interpolation cannot model, while enabling the model to directly provide... The ability to reason in real time.

[0055] The residual recovery process is modeled as a conditional generation problem, that is, given... In the case of estimating the posterior distribution: ; in, , From Restore to The learnable inverse transfer kernel. During the inference phase, the parameters of this distribution are determined by the conditional neural network structured as U-Net. Prediction. It is based on the current state. and condition information As input, output the mean and variance parameters of the inverse transition kernel. Assume the inverse transition kernel at each step follows a Gaussian distribution: .

[0056] It is worth noting that this U-Net structure is a conditional neural network. Training is required, and the training goal is to enable the backsampling process to accurately recover the residual, thereby achieving detail compensation for areas not covered by the physically modeled image.

[0057] During the training phase, the network optimizes by minimizing the distance (variational lower bound) between the predicted distribution and the forward-diverged true distribution: ; in, This represents the Kullback-Leibler (KL) divergence. and Satisfies the condition when most similar Minimum, at the same time Represents the true posterior distribution process. It refers to the posterior process that the model needs to predict and fit, the difference being the true posterior distribution process. It is known However, prediction simulation It is unknown The latter requires a conditional neural network with a U-Net structure for prediction. .

[0058] join in This is to ensure that the posterior process of the prediction simulation is as close as possible to the actual posterior distribution process, and to simulate the theoretical effect as closely as possible.

[0059] Combining the aforementioned formula, the true posterior distribution It can be represented by the following explicit form: .

[0060] Considering the variance parameter and and It is irrelevant, therefore the setting is: .

[0061] for Average parameters Reparameterization is as follows: ; in, It is a conditional neural network with a U-Net structure, used in inference to predict HDR images. Comparison with the true posterior Constructed based on the mean, the difference being that the true posterior is known. The predicted posterior is predicted by a conditional neural network with a U-Net structure. Right now This allows the use of physical interpolation maps in backward reasoning. The provided structural priors allow the diffusion model to focus on learning detailed differences.

[0062] To train the model, minimize the following objective function: .

[0063] This optimization method directly drives the diffusion model to approximate the true residual at each step, thereby improving the detail recovery effect. To reduce the computational cost of training, diffusion modeling is performed in the VQGAN latent space, compressing the image spatial resolution by a factor of 4, requiring only noise injection and denoising of the latent code. This not only preserves global structural information but also significantly reduces GPU memory usage.

[0064] In the inference phase of high dynamic range imaging, the entire process first proceeds as follows: Figure 2 As shown: Two images with different exposure ratios are acquired, and a medium-exposure image is generated using a physically based interpolation method. Subsequently, as Figure 3 As shown in the inference, the diffusion model in the input During this process, a certain amount of noise is injected directly into the model based on the distribution learned during training, to simulate the state during the reverse denoising process. Then, the model gradually generates high-quality HDR exposure interpolated images through the reverse process of residual diffusion.

[0065] It's important to note that only back-diffusion is performed during the inference phase; the forward diffusion or reverse denoising processes used in training are not involved. The data used in the training phase consists of intermediate states generated by forward diffusion and their corresponding residuals. The neural network learns the inverse transfer kernel for each step by fitting this data. The parameters are used to effectively recover details not modeled in the physical model during the inference phase.

[0066] Unlike traditional diffusion methods that directly predict HDR images, this method reverses the process by gradually restoring the residuals based on the physical interpolation results. This decouples structure preservation from detail enhancement, making the diffusion model more likely to converge and providing stronger recovery capabilities in the high-frequency region.

[0067] Example 2 like Figure 4 As shown, this embodiment provides a high dynamic range imaging exposure interpolation device 200, including: The image acquisition module 201 is used to acquire high-exposure images and low-exposure images of the same scene, wherein the exposure time of the high-exposure image is longer than that of the low-exposure image. Image mapping module 202 is used to map low-exposure image and high-exposure image to target exposure time respectively to obtain first intermediate image and second intermediate image; The weighted fusion module 203 is used to perform weighted fusion processing on the first intermediate image and the second intermediate image to generate an initial intermediate exposure image; The image generation module 204 is used to input the initial intermediate exposure image into a pre-trained residual diffusion model, and to restore the residual details of the initial intermediate exposure image through the pre-trained residual diffusion model, so as to generate a high dynamic range image based on the restored residual details.

[0068] Optionally, in an embodiment, the image mapping module 202 includes: The parameter acquisition unit is used to acquire the exposure time of the high-exposure image and the exposure time of the low-exposure image. The irradiance calculation unit is used to calculate the irradiance corresponding to each pixel in the high-exposure image and the irradiance corresponding to each pixel in the low-exposure image, respectively, based on the camera response model, the exposure time of the high-exposure image, and the exposure time of the low-exposure image. The pixel value calculation unit is used to calculate the pixel value of each pixel in the high-exposure image and the pixel value of each pixel in the low-exposure image at the target exposure time, based on the camera response model, the irradiance corresponding to each pixel in the high-exposure image, the irradiance corresponding to each pixel in the low-exposure image, and the target exposure time. The first image generation unit is used to obtain a first intermediate image based on the pixel values ​​of each pixel in the low-exposure image at the target exposure time; The second image generation unit is used to obtain a second intermediate image based on the pixel values ​​of each pixel in the high-exposure image at the target exposure time.

[0069] Optionally, in an embodiment, the weighted fusion module 203 includes: The first weight calculation unit is used to calculate the first weight corresponding to each pixel in the first intermediate image based on the pixel value of each pixel in the first intermediate image and the preset minimum pixel value. The second weight calculation unit is used to calculate the second weight corresponding to each pixel in the second intermediate image based on the pixel value of each pixel in the second intermediate image and the preset maximum pixel value. The weighted fusion unit is used to calculate the pixel value of each pixel in the initial intermediate exposure image based on the first weight corresponding to each pixel in the first intermediate image, the second weight corresponding to each pixel in the second intermediate image, the pixel value of each pixel in the first intermediate image, and the pixel value of each pixel in the second intermediate image, so as to obtain the initial intermediate exposure image.

[0070] Optionally, in an embodiment, the pre-trained residual diffusion model is trained as follows: multiple sets of training data are acquired, each set of training data including a target high dynamic range image and an initial intermediate exposure image corresponding to the target high dynamic range image; for each set of training data, random noise conforming to a Gaussian distribution is gradually injected into the target high dynamic range image, and residual information is gradually injected into the target high dynamic range image according to a preset ratio, until it transitions to the initial intermediate exposure image corresponding to the target high dynamic range image, resulting in multiple noisy images; wherein, the residual information represents the residual information between the target high dynamic range image and the initial intermediate exposure image corresponding to the target high dynamic range image; for each set of training data, the initial intermediate exposure image corresponding to the target high dynamic range image is used as a conditional input to train the neural network to learn how to gradually recover residual information from different noisy images.

[0071] Optionally, in an embodiment, the image generation module 204 includes: The noise injection unit is used to inject initial noise conforming to a Gaussian distribution into the initial intermediate exposure image through a pre-trained residual diffusion model, thereby constructing the initial noise state. The third image generation unit is used to generate a high dynamic range image by progressively predicting residual information based on the initial noise state and the initial intermediate exposure image through a pre-trained residual diffusion model.

[0072] Optionally, in an embodiment, the third image generation unit includes: The image generation subunit is used to perform the following operations step by step from the initial noise state until a zero noise level is reached: based on the current noise state and the initial intermediate exposure image, predict the residual information of the current step through a trained neural network; calculate the noise state of the next step through a pre-trained residual diffusion model based on the predicted residual information; when the iteration process is completed, the final noise state is the high dynamic range image.

[0073] Optionally, in an embodiment, the image mapping module 202 further includes: The linear interpolation unit is used to calculate the irradiance corresponding to the first pixel point in a high-exposure or low-exposure image, whose pixel value is located between two adjacent integer pixel values, based on the two adjacent integer pixel values ​​and the irradiance corresponding to the two adjacent integer pixel values, using the linear interpolation method.

[0074] In some embodiments, the high dynamic range imaging exposure interpolation device 200 of the present invention can be implemented in a combination of hardware and software. As an example, the high dynamic range imaging exposure interpolation device 200 of the present invention can be a processor in the form of a hardware decoding processor, which is programmed to execute the high dynamic range imaging exposure interpolation method of the present invention. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0075] The modules described in the embodiments of this invention can be implemented in software or hardware. The names of the modules are not, in some cases, limiting the scope of the module itself.

[0076] Example 3 like Figure 5 As shown, this embodiment provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements a high dynamic range imaging exposure interpolation method as described in Embodiment 1.

[0077] In other words, an electronic device according to an embodiment of the present invention may include, but is not limited to, a processor and a memory; the memory is used to store a computer program; the processor is used to execute a high dynamic range imaging exposure interpolation method shown in any embodiment of the present invention by calling the computer program.

[0078] In one alternative embodiment, an electronic device is provided, such as Figure 5 As shown, Figure 5 The illustrated electronic device 300 includes a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, for example, via a bus 302. Optionally, the electronic device 300 may further include a transceiver 304, which can be used for data interaction between the electronic device and other electronic devices, such as sending and / or receiving data. It should be noted that in practical applications, the transceiver 304 is not limited to one type, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of the present invention.

[0079] Processor 301 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this invention. Processor 301 may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0080] Bus 302 may include a path for transmitting information between the aforementioned components. Bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. Bus 302 can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 5 The bus 302 is represented by only one thick line, but this does not mean that there is only one bus or one type of bus.

[0081] The memory 303 may be a ROM (Read Only Memory) or other type of static storage device capable of storing static information and instructions, RAM (Random Access Memory) or other type of dynamic storage device capable of storing information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), CD-ROM (Compact Disc Read Only Memory) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), magnetic disk storage media or other magnetic storage devices, or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto.

[0082] The memory 303 is used to store application code (computer program) for executing the present invention, and its execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the content shown in the foregoing method embodiments.

[0083] Among them, electronic devices can also be terminal devices, which can be any device that can install applications, including at least one of smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, smart TVs, and smart in-vehicle devices.

[0084] It should be noted that, Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.

[0085] Example 4 This embodiment provides a non-transitory computer-readable storage medium that stores computer instructions for causing a computer to execute a high dynamic range imaging exposure interpolation method as described in Embodiment 1.

[0086] Alternatively, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, a floppy disk, and an optical data storage device, etc.

[0087] In an exemplary embodiment, a computer program product or computer program is also provided, which includes computer instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the electronic device to perform the high dynamic range imaging exposure interpolation method described above.

[0088] Computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0089] It should be understood that the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of methods and computer program products according to various embodiments of the present invention. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0090] The computer-readable storage medium provided in this invention can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EEPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this invention, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0091] The aforementioned computer-readable storage medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the method shown in the above embodiments.

[0092] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

[0093] It should be noted that the terms "first," "second," etc., used in the specification and claims of this application are used to distinguish similar objects and represent a limitation on a specific order or sequence. Where appropriate, the order of use for similar objects can be interchanged so that the embodiments of this application described herein can be implemented in an order other than that shown or described.

[0094] Those skilled in the art will recognize that this invention can be implemented as a system, method, or computer program product. Therefore, this invention can be specifically implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, this invention can also be implemented as a computer program product contained in one or more computer-readable media, which includes computer-readable program code.

[0095] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A high dynamic range imaging exposure interpolation method, characterized in that, include: Acquire high-exposure and low-exposure images of the same scene, wherein the exposure time of the high-exposure image is greater than the exposure time of the low-exposure image; The low-exposure image and the high-exposure image are respectively mapped to the target exposure time to obtain the first intermediate image and the second intermediate image; A weighted fusion process is performed on the first intermediate image and the second intermediate image to generate an initial intermediate exposure image; The initial intermediate exposure image is input into a pre-trained residual diffusion model, and residual details are restored on the initial intermediate exposure image through the pre-trained residual diffusion model, so as to generate a high dynamic range image based on the restored residual details.

2. The high dynamic range imaging exposure interpolation method according to claim 1, characterized in that, The step of mapping the low-exposure image and the high-exposure image to a target exposure time to obtain a first intermediate image and a second intermediate image includes: Obtain the exposure time of the high-exposure image and the exposure time of the low-exposure image; Based on the camera response model, the exposure time of the high-exposure image, and the exposure time of the low-exposure image, the irradiance corresponding to each pixel in the high-exposure image and the irradiance corresponding to each pixel in the low-exposure image are calculated respectively. Based on the camera response model, the irradiance corresponding to each pixel in the high-exposure image, the irradiance corresponding to each pixel in the low-exposure image, and the target exposure time, the pixel value of each pixel in the high-exposure image and the pixel value of each pixel in the low-exposure image are calculated respectively at the target exposure time. The first intermediate image is obtained based on the pixel values ​​of each pixel in the low-exposure image at the target exposure time; The second intermediate image is obtained based on the pixel values ​​of each pixel in the high-exposure image at the target exposure time.

3. The high dynamic range imaging exposure interpolation method according to claim 1, characterized in that, The step of generating an initial intermediate exposure image by performing weighted fusion processing based on the first intermediate image and the second intermediate image includes: The first weight corresponding to each pixel in the first intermediate image is calculated based on the pixel value of each pixel in the first intermediate image and the preset minimum pixel value. The second weight corresponding to each pixel in the second intermediate image is calculated based on the pixel value of each pixel in the second intermediate image and the preset maximum pixel value. The pixel value of each pixel in the initial intermediate exposure image is calculated based on the first weight corresponding to each pixel in the first intermediate image, the second weight corresponding to each pixel in the second intermediate image, the pixel value of each pixel in the first intermediate image, and the pixel value of each pixel in the second intermediate image, so as to obtain the initial intermediate exposure image.

4. The high dynamic range imaging exposure interpolation method according to claim 1, characterized in that, The pre-trained residual diffusion model was trained in the following way: Acquire multiple sets of training data, each set of training data including a target high dynamic range image and an initial intermediate exposure image corresponding to the target high dynamic range image; For each set of training data, random noise conforming to a Gaussian distribution is gradually injected into the target high dynamic range image, and residual information is gradually injected into the target high dynamic range image according to a preset ratio until it transitions to the initial intermediate exposure image corresponding to the target high dynamic range image, thus obtaining multiple noisy images. The residual information represents the residual information between the target high dynamic range image and the initial intermediate exposure image corresponding to the target high dynamic range image; For each set of training data, the initial intermediate exposure image corresponding to the target high dynamic range image is used as the conditional input to train the neural network to learn how to gradually recover residual information from different noisy images.

5. The high dynamic range imaging exposure interpolation method according to claim 1, characterized in that, The step of restoring residual details from the initial intermediate exposure image using the pre-trained residual diffusion model to generate a high dynamic range image based on the restored residual details includes: The initial noise state is constructed by injecting Gaussian-distributed initial noise into the initial intermediate exposure image through the pre-trained residual diffusion model. Based on the initial noise state and the initial intermediate exposure image, the high dynamic range image is generated by progressively predicting residual information using the pre-trained residual diffusion model.

6. The high dynamic range imaging exposure interpolation method according to claim 5, characterized in that, The process of generating the high dynamic range image by progressively predicting residual information using the pre-trained residual diffusion model based on the initial noise state and the initial intermediate exposure image includes: Starting from the initial noise state, perform the following operations step by step until a zero noise level is reached: Based on the current noise state and the initial intermediate exposure image, the residual information for the current step is predicted using a trained neural network. Based on the predicted residual information, the noise state for the next step is calculated using the pre-trained residual diffusion model; Once the iteration process is complete, the final noise state is the high dynamic range image.

7. The high dynamic range imaging exposure interpolation method according to claim 2, characterized in that, The step of mapping the low-exposure image and the high-exposure image to a target exposure time to obtain a first intermediate image and a second intermediate image also includes: For a first pixel in the high-exposure image or the low-exposure image whose pixel value is located between two adjacent integer pixel values, the irradiance corresponding to the first pixel is calculated by linear interpolation based on the two adjacent integer pixel values ​​and the irradiance corresponding to the two adjacent integer pixel values.

8. A high dynamic range imaging exposure interpolation device, characterized in that, include: An image acquisition module is used to acquire a high-exposure image and a low-exposure image of the same scene, wherein the exposure time of the high-exposure image is greater than the exposure time of the low-exposure image; The image mapping module is used to map the low-exposure image and the high-exposure image to the target exposure time respectively to obtain the first intermediate image and the second intermediate image; The weighted fusion module is used to perform weighted fusion processing on the first intermediate image and the second intermediate image to generate an initial intermediate exposure image; The image generation module is used to input the initial intermediate exposure image into a pre-trained residual diffusion model, and to perform residual detail restoration on the initial intermediate exposure image through the pre-trained residual diffusion model, so as to generate a high dynamic range image based on the restored residual details.

9. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement a high dynamic range imaging exposure interpolation method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to execute the high dynamic range imaging exposure interpolation method according to any one of claims 1 to 7.

Images