A high dynamic range image reconstruction method based on a progressive alignment fusion depth expansion model

Abstract

The present application relates to the field of high dynamic range image reconstruction, and proposes a high dynamic range image reconstruction method based on a progressive alignment fusion depth unfolding model, comprising the following steps: step S1, modeling of multi-exposure high dynamic range image reconstruction; step S2, iterative solution of the multi-exposure high dynamic range image reconstruction problem; step S3, building of a depth unfolding model; and step S4, model training; the present application models the multi-exposure high dynamic range image reconstruction from the perspective of maximum posterior problem, introduces a prior regularization term for constraining the spatial correspondence relationship between low dynamic range images, decomposes the task into alignment and fusion sub-problems, and solves them in an alternating and progressive manner to obtain high-quality results; the present application develops an interpretable and robust progressive alignment and fusion depth unfolding network, which has more advanced performance.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of high dynamic range image reconstruction, in particular to a high dynamic range image depth unfolding model reconstruction method using multiple low dynamic range images of different exposures. BACKGROUND

[0002] With the development of computer vision and graphics, high dynamic range image reconstruction technology has attracted more and more attention. Multi-exposure high dynamic range (HDR) image reconstruction aims to effectively utilize the information of multiple low dynamic range (LDR) images captured under different exposures to reconstruct a high dynamic range image with a larger dynamic range. High dynamic range images have a wider dynamic range, providing a more realistic visual effect and bringing an attractive visual experience, making them an indispensable important tool for satellite remote sensing, medical image processing, virtual reality, augmented reality, autonomous driving and other applications.

[0003] Existing high dynamic range image reconstruction methods usually use deep learning models to train and test, obtaining a network model that can reconstruct high dynamic range images using multiple exposure images. Although existing models can already obtain high dynamic range images with good reconstruction results based on multiple low dynamic range images of different exposures, using multiple non-aligned low dynamic range images to reconstruct high dynamic range images, due to the presence of large foreground character movement or camera movement, or poor exposure conditions, resulting in the reconstructed high dynamic range image has obvious artifacts. Traditional methods mainly include the following categories: one method rejects non-aligned pixels, one method merges and fuses images at the image block level, and another method explicitly aligns low dynamic range images. The above three categories of methods are referred to as rejection-based methods, image block-based methods, and alignment-based methods. For rejection-based methods, since it rejects pixels in various motion areas, it will result in a large amount of information loss; for image block-based methods, the computational complexity is large; for alignment-based methods, based on a relatively accurate pre-alignment, first align the multiple input images of different exposures, and then perform image fusion, so such a way has a strong dependence on pre-alignment, and has challenges in motion areas or poor exposure conditions, and poor alignment results will further affect the reconstruction results.

[0004] Although existing high dynamic range image reconstruction methods can produce accurate reconstruction results when low dynamic range images are well aligned, the misalignment problem caused by dynamic scenes or camera jitter easily produces artifacts. To solve the artifact problem, traditional methods usually rely on discarding the role of misaligned pixels for reconstruction, or merging images at the image block level, or explicitly aligning low dynamic range images and then fusing the aligned images to reconstruct high dynamic range images. However, these methods are still far from being practical. The pixel rejection-based method can miss important details in moving areas, the image block-based method is computationally intensive, and the alignment-based method heavily relies on accurate alignment, which is challenging when there is large motion or poor exposure in the image content.

[0005] With the rise of deep learning-based methods, utilizing data priors and flexible modeling can provide more effective solutions to these challenges. They usually follow two main paradigms:

[0006] Most methods adopt a two-stage approach: first align low dynamic range images using an alignment module, then refine, fuse features and reconstruct high dynamic range images through a complex network, referred to as the "alignment-fusion" paradigm.

[0007] In contrast, other methods bypass alignment and directly fuse features through a stacking module, summarized as the "fusion-only" paradigm. Although both paradigms have been proven to be effective, they often face the risk of not explicitly aligning and potential information loss after alignment, and both paradigms have the problem of limited explainability, with the model built only as a "black box", which is the main drawback that needs to be solved by the present invention. SUMMARY

[0008] To solve the above problems, the present invention proposes a progressive alignment and fusion method based on a deep unfolding network, which has stronger explainability. The traditional optimization problem solving method is combined with the deep learning model, so as to use the powerful learning ability of the deep learning model on the basis of mathematical theory, realize the explainability and excellent high dynamic range image reconstruction performance.

[0009] Technical solutions

[0010] The application provides a high dynamic range image reconstruction method based on deep unfolding progressive alignment fusion, which models a multi-exposure high dynamic range reconstruction problem from a maximum posteriori angle, introduces a key image prior constraint term, uses prior information between overexposed images and intermediate exposure reference images, another same prior constraint term uses prior information between underexposed images and intermediate exposure reference images, and then can decouple the problem into two sub-problems, i.e., an alignment problem and a fusion problem, iteratively solves the two problems, and realizes high-quality multi-exposure high dynamic range image reconstruction capability.

[0011] A high dynamic range image reconstruction method based on a progressive alignment fusion deep unfolding model, comprising the following steps:

[0012] Step S1 modeling of a multi-exposure high dynamic range image reconstruction problem

[0013] The application models a multi-exposure high dynamic range image reconstruction problem from a maximum posteriori problem, introduces a prior regularization term for constraining spatial correspondence between low dynamic range images, and obtains an optimization objective function for reconstruction, as shown below:

[0014]

[0015] Wherein, A normal exposure reference image, a high dynamic range image to be reconstructed and Need to be aligned in content. , , The three different exposure images are respectively , , The degradation matrix of The reconstruction target, i.e., a high dynamic range image, , Is an auxiliary variable introduced. , Is a weight coefficient corresponding to each regularization term. And Is a prior regularization term used to describe spatial correspondence.

[0016] Step S2 iterative solution of the multi-exposure high dynamic range image reconstruction problem

[0017] The solution of the optimization objective function is firstly loose into two sub-problems, i.e., an alignment problem and a fusion problem, then an iterative method is used to solve the two sub-problems, so that the solution of the whole problem is solved in an iterative manner of alternately solving between the two sub-problems until the problem converges to find an optimal solution.

[0018] Step S3 building of a deep unfolding model

[0019] The iterative formula derived in step S2 is expanded, each stage corresponds to a depth expansion module, the module structure corresponding to each stage is the same, the parameters are not shared, the depth expansion modules of each stage are stacked to build a depth expansion model.

[0020] Step S4 model training

[0021] The progressive alignment fusion depth expansion model of step S3 is trained.

[0022] Compared with the prior art, the present application has the following beneficial effects:

[0023] (1) Unified maximum a posteriori problem angle for high dynamic range image reconstruction modeling: the present application models multi-exposure high dynamic range image reconstruction from the maximum a posteriori problem angle, introduces a prior regularization term that constrains the spatial correspondence relationship between low dynamic range images, decomposes the task into alignment and fusion sub-problems, and solves them in an alternating progressive manner to obtain high-quality results.

[0024] (2) End-to-end interpretable network: the present application develops an interpretable and robust progressive alignment and fusion depth expansion network that allows end-to-end training of high dynamic range image reconstruction, which is composed of a plurality of stacked alignment and fusion modules associated with iterative stages with physical meaning.

[0025] (3) More advanced performance: extensive qualitative and quantitative experiments show that the progressive alignment and fusion depth expansion network proposed by the present application achieves more advanced performance in high dynamic range image reconstruction, providing visually appealing results that conform to human perception aesthetics. BRIEF DESCRIPTION OF DRAWINGS

[0026] Figure 1 is a flow chart of the overall framework of the method of the present application;

[0027] Figure 2 is a flow chart of the depth expansion model framework of an embodiment of the present application;

[0028] Figure 3 is a detailed diagram of the depth expansion model module of an embodiment of the present application;

[0029] Figure 4 is a visualization comparison result diagram of an embodiment of the present application on Kalantari's dataset;

[0030] Figure 5 is a visualization comparison result diagram of an embodiment of the present application on Hu's dataset;

[0031] Figure 6 is a visualization comparison result diagram of an embodiment of the present application on Tel's dataset;

[0032] Figure 7 is a visualization comparison result diagram of an embodiment of the present application on Tursen's dataset. Detailed Implementation

[0033] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.

[0034] As shown in Figure 1, a high dynamic range image reconstruction method based on a progressive alignment fusion depth unfolding model includes the following steps:

[0035] Step S1: Modeling the problem of multi-exposure high dynamic range image reconstruction

[0036] From the perspective of the maximum a posteriori problem, we model the problem of reconstructing multi-exposure high dynamic range images. The following is a detailed description of the modeling process.

[0037] The degradation process from a high dynamic range image to a low dynamic range image can be expressed mathematically as follows:

[0038] ,

[0039] in Represents a high dynamic range image. Indicates the degradation process. It is a low dynamic range image. This refers to noise introduced during the degradation process. This invention reformulates the ill-posed problem of the aforementioned degradation process as an optimization problem within the framework of a maximum a posteriori problem, including a data fidelity term and a regularization term. Data fidelity is typically defined as... Norm, expressed by the following energy function:

[0040]

[0041] in For reconstructing high dynamic range images, This is a weighted hyperparameter for the regularization term.

[0042] The objective of this invention is to merge three low dynamic range images with different exposures (i.e., underexposed images). Normally exposed image and overexposed images This results in a single, high-quality, artifact-free high dynamic range image. Specifically, a low dynamic range image... As a reference image, the predicted high dynamic range image must be compared with... Content alignment. Therefore, this invention achieves this by introducing a non-referenced low dynamic range image. and To extend Equation 2, they provide complementary information from scenes of different exposure levels, thus enhancing the effect of the reconstruction from However, directly applying priors from non-reference low dynamic range images can limit their effective utilization, because of the misalignment of content between low dynamic range images. To solve this problem, the present application further introduces two spatial correspondence prior regularization terms which explicitly model the rich priors between multiple non-reference LDR images and the reference low dynamic range image for high dynamic range image reconstruction. denote the spatial alignment auxiliary variables of which gradually align the image structure and content with during the iterative optimization process.

[0043] In summary, the present application reformulates the optimization problem of HDR reconstruction as:

[0044]

[0045] where are the degradation matrices of three different exposure images are the weight coefficients of each regularization term. Equation 3 is the optimization objective of the high dynamic range image reconstruction finally modeled by the present application. Step S2 Iterative solution of the multi-exposure high dynamic range image reconstruction problem

[0046] To effectively solve Equation 3, the present application first decomposes it into two sub-problems of alignment and fusion, and solves them alternately.

[0047] Alignment sub-problem:

[0048]

[0049]

[0050]

[0051] Fusion sub-problem:

[0052]

[0053] ​​​​​​​​​​For the alignment sub-problem in equation 4a and equation 4b, the gradient descent operator is further defined

[0054] ,

[0055] where, is the step size, is the gradient of the spatial correspondence prior term with respect to the alignment auxiliary variable , where =1,3. The step optimization can be expressed as:

[0056]

[0057]

[0058] For the fusion sub-problem in equation 4c, the present application uses the Half Quantic Splitting (HQS) method to decouple the data fidelity term and the regularization term.

[0059] First, the present application introduces two auxiliary variables and , which correspond to two prior regularization terms that constrain the spatial correspondence between different low dynamic range images, and further constrain and to be as close as possible to the target image .

[0060]

[0061] , is the weight hyper-parameter.

[0062] The fusion problem can then be decomposed into three sub-problems, each of which iteratively updates:

[0063]

[0064]

[0065]

[0066] Given the reconstructed image and the alignment auxiliary variables and , the present application defines the proximal operators and for the optimization update of and . The specific definitions of the two proximal operators are as follows:

[0067] ,

[0068] .

[0069] Equations 7a and 7b can be solved by the following iterative equations:

[0070]

[0071]

[0072] Equation 7c represents a quadratic regularized least square problem, and there is a closed-form solution as follows:

[0073]

[0074] where, is an identity matrix, is the transpose matrix of the degenerate matrix . The matrix inversion is computationally expensive, so is regarded as a whole, denoted as .

[0075] The present application further designs a neural network to learn the complex degenerate matrix and respectively.

[0076] Step S3 deep unfolding model building

[0077] The iterative formula derived in step S2 is unfolded, and each stage corresponds to a deep unfolding module. The module structure corresponding to each stage is the same, and the parameters are not shared. The deep unfolding modules of each stage are stacked to build a deep unfolding model.

[0078] It is worth noting that, as described in step S1 and step S2, the solution of the unfolding paradigm is performed in the image space, where the low dynamic range image , , directly participates in the optimization process. In contrast, in step S3 of actually building the deep unfolding model, the present application performs iterative optimization in the feature space rather than the image space, and uses the unfolded deep network to propose a learnable end-to-end solution.

[0079] In detail, the deep unfolding model specifically includes three stages: feature extraction and initialization, high dynamic range feature reconstruction, and high dynamic range image reconstruction. The three stages are unfolded in turn as follows:

[0080] Stage 1 Feature extraction and initialization

[0081] First, initialization is performed, that is, features of multiple different exposure low dynamic range images are extracted using a feature extractor, and the extracted features are used to initialize the iterative process of the next stage;

[0082] The input picture of the model is wherein is a low dynamic range image, is the corresponding image after gamma correction, which is used to provide more information for reconstruction for the model. The above input picture is extracted by three convolution layers with convolution kernels, and the image domain is projected to the feature domain wherein, batch size, channel number, height and width respectively. The feature extraction process is as follows:

[0083]

[0084] The extracted features , , are used to initialize the feature variable in the 0th iteration process in the high dynamic range feature reconstruction stage , , .

[0085] Stage 2 high dynamic range feature reconstruction

[0086] Then, high dynamic range feature reconstruction is performed, that is, iteration is performed in the feature domain, and alignment and fusion operations are gradually realized, so that the features of three different exposure range images are aligned and fused side by side, and finally the high dynamic range image target reconstruction feature is obtained;

[0087] The present application proposes a deep unfolding model of the stage, uses a stacked alignment fusion module AFM, and solves the optimization target of the reconstructed high dynamic range image proposed in step S1 from an iterative optimization unfolding algorithm. It should be noted that all stages of the deep unfolding model of the present application maintain the same structure, but the parameters are not shared between stages. The iterative process is represented as:

[0088]

[0089] wherein, is the output of the previous iteration stage, as the input of the current iteration stage, is the output of the current iteration stage, and so on.

[0090] Each iteration stage can be divided into two sub-stages of feature alignment and feature fusion,Figure 3 The module details of the proposed modules in the deep unfolding model are shown, wherein: (a) the spatial alignment module uses a window-based cross-attention mechanism to align and respectively. (b) the channel fusion module fuses and respectively with to obtain and (c) the data consistency module uses and to update to obtain the preliminary updated (d) the matrix related to degradation is learned using the module.

[0091] Stage 2.1 Feature Alignment

[0092] Corresponding to the step S2 alignment sub-problem formula 5a and formula 5b, the present application proposes two simple and effective spatial alignment modules for aligning and respectively to as two gradient descent operators and .

[0093] As shown in Figure 3 (a), the spatial alignment module is constructed as a window-based cross-attention block, denoted as:

[0094]

[0095] wherein, is a window-based cross-attention alignment module (WCAA, Window-based Cross-Attention Alignment module), is a feed-forward network (FFN, Feed-Forward Network). Through this design, the present application can query the information in the reference image of normal exposure, while using to maintain the spatial structure of the reference image.

[0096] The detailed implementation of is as follows:

[0097]

[0098] wherein,

[0099]

[0100]

[0101]

[0102] is a learnable transformation, is and the feature channel dimension, denotes a multi-layer perceptron (MLP) for learning the degradation matrix , where The details of this module are illustrated in Fig. (d) and the learning of the model for the degradation matrix Figure 3 is achieved through end-to-end training.

[0103] It is worth noting that the present application uses a window-based cross-attention mechanism to focus on the local spatial information between features, since the alignment is mainly for high frequencies and structural details. In contrast, some global transformation methods result in suboptimal alignment results due to the application of uniform transformation on the entire image.

[0104] Stage 2.2 Feature Fusion

[0105] The feature fusion process includes the update of three feature variables, i.e. .

[0106] Update and : Since the update and are symmetric, they are stated together. According to Equations 8a and 8b, the present application proposes a two-step fusion module for learning the proximal operator between features. First, it fuses the features in space, and then performs channel-level fusion. This process can be represented as:

[0107]

[0108]

[0109]

[0110] where denotes a spatial fusion module (SFM) with the input being the concatenated result of the three, and the output being ;

[0111] and are respectively used to align​ The updated channel fusion module (CFM) adopts a cross-attention mechanism based on channels to obtain a further refined and fused result and .

[0112] The specific details of the channel fusion module are shown in Figure 3 (b).

[0113] The update : For the optimization of the variable of the previous stage, the updated result obtained from formula 14(b) and formula 14(c) is used and , the present application carries out preliminary update through the data consistency module (DCM), and obtains , as shown in the following formula:

[0114]

[0115] The specific implementation details of the data consistency module are shown in Figure 3 (c).

[0116] After the preliminary update of , the present application uses the multi-layer perception fusion optimized variable , , to further update , and obtains a further refined result .

[0117]

[0118] Finally, the final result of the present iteration process update , , is passed to the next iteration process until the completion of all iteration processes of the high dynamic range feature reconstruction stage.

[0119] Stage 3 High Dynamic Range Image Reconstruction

[0120] Finally, high dynamic range image reconstruction is performed, that is, the obtained target reconstruction feature is projected from the feature domain to the image domain using the deep learning module to obtain the final high dynamic range image reconstruction result.

[0121] After completing all iteration reconstruction processes in feature domains, the high dynamic range feature is finally reconstructed To ensure the stability of the image reconstruction process, this invention employs a residual strategy, and then projects the features onto the reconstructed high dynamic range image. The high dynamic range image reconstruction stage is defined as follows:

[0122]

[0123] in, and These represent the Sigmoid activation function and the convolution operation (Conv, Convolutional layer), respectively.

[0124] Details of the progressive alignment and fusion depth unfolding model of this invention are as follows: Figure 2 As shown, it includes three stages: feature extraction and initialization, high dynamic range feature reconstruction, and high dynamic range image reconstruction. The alignment and fusion module reconstructs the target's high dynamic range features through alignment and fusion iterations in each stage. In the feature alignment stage, it is achieved through... -Spatial alignment module and - The spatial alignment module will feature and Align to feature In the feature fusion stage, the spatial fusion module first performs preliminary fusion to obtain intermediate results. and Next, the channel fusion module, namely the U-space fusion module and the V-space fusion module, respectively obtains further fusion results. and Finally, the data consistency module was optimized to achieve the following: Features are further refined using a multilayer perceptron with residual connections before being passed to the next stage. .

[0125] Step S4: Model Training

[0126] The progressive alignment and fusion depth unrolling model in training step S3 is used. Loss and perceived loss The model is trained end-to-end using a linear combination of these methods. Loss and perceived loss As shown below:

[0127]

[0128]

[0129] in, For the first from VGG-19 Features extracted from layers, Indicates the process -law function processed image.

[0130] Therefore, the total loss is:

[0131]

[0132] wherein, is a weight coefficient, which is set to 0.005.

[0133] Considering that calculating the loss in the high dynamic range image domain will result in reduced efficiency of training, the present application adopts a -law function to calculate the loss after tone mapping.

[0134]

[0135] wherein, is set to 5000.

[0136] The main innovation points of the present application are:

[0137] 1. Optimization problem modeling from the perspective of maximum posterior problem

[0138] In order to break through the excessive constraints and relatively complex limitations of other methods in problem modeling, the present application proposes to model the problem from the perspective of maximum posterior, and proposes an ill-posed optimization target based on the degradation process. The key point of the present application is to reasonably utilize the prior information between the three different exposure images in the modeling process, that is, to take the normally exposed image as the reference image, because the high dynamic range image finally reconstructed needs to be aligned with the normally exposed image in content, and the two spatial alignment prior regular terms are introduced to cleverly introduce the overexposed image and the underexposed image into the modeling, so as to ensure the full use of the information of the three low dynamic range images and provide complete constraints. If the overexposed image and the underexposed image are directly introduced, due to the inevitable misalignment phenomenon between the three low dynamic range images, it will inevitably lead to poor utilization of the prior, and further lead to poor reconstruction results. Therefore, the present application further improves the spatial alignment prior regular term, introduces two auxiliary variables for alignment corresponding to the overexposed image and the underexposed image respectively, and constantly optimizes with the optimization of the reconstruction target in the iteration process, and aligns with the reconstruction target, so that the reconstruction process can utilize the most matched prior information in each iteration process.

[0139] The iterative solving process aiming at the optimization target further proposes an effective traditional iterative solving process aiming at the optimization target modeled by the application. Firstly, the solving of the optimization problem is relaxed into the alternative solving of two optimization problems, i.e. alignment sub-problems and fusion sub-problems, and the variables in the two sub-problems are alternately transmitted to each other, so that the joint optimization of the two sub-problems is better realized by means of the complementary properties of alignment and fusion. In the alignment sub-problem, the application proposes a gradient descent operator, which is solved by means of iterative solving. In the fusion sub-problem, the application uses the solving skill of semi-quadratic splitting, realizes the decoupling of the data fidelity term and each regularization term by introducing two auxiliary variables, and further splits the fusion sub-problem into the alternative iterative solving of three sub-problems. The effective traditional iterative solving mode proposed by the application guarantees the effective decoupling between complex problems, solves them in turn, realizes the joint optimization, utilizes the complementary properties between the alignment sub-problem and the fusion sub-problem, realizes the progressive alternative solving with explicit physical meaning, and guarantees the high quality and optimality of the final optimization result.

[0140] The effective solution based on the deep unfolding model The traditional iterative optimization solving mode has the disadvantages of complex parameter adjustment and poor optimization performance, and various deep learning models (DNNs, Deep Neural Networks) commonly used in the deep learning field have the black box problem of insufficient interpretability. In order to fully utilize the advantages of traditional iterative optimization solving and deep learning models, the application proposes an effective high dynamic range image reconstruction solution based on deep unfolding models (DUNs, Deep Unfolding Networks), and builds a progressive alignment and fusion deep unfolding model. The model is divided into three stages: feature extraction and initialization, high dynamic range feature reconstruction, and high dynamic range image reconstruction. The high dynamic range feature reconstruction is the key in the three stages, and the three key feature variables are alternately updated, , , , realize the progressive alignment and synchronous fusion process. The completion of alignment is beneficial to the subsequent fusion, because alignment reduces the non-alignment phenomenon in content, so that the fusion result has less artifacts and more detailed information, and the completion of fusion is beneficial to further alignment, because the intermediate result has more alignment reference information than the fusion result of the previous step. Therefore, through the building of the above modules, the high-quality high dynamic range image is guaranteed.

[0141] Embodiment 1

[0142] An experimental platform is built to simulate the method of the application, as follows:

[0143] Dataset

[0144] Experiments are tested on Kalantari's dataset, Hu's dataset and Tel's dataset. To demonstrate the generalization ability of the model, the present application additionally uses Tursen's dataset for qualitative testing only. When testing on Kalantari's dataset, Hu's dataset and Tel's dataset, the training model settings remain consistent. The test on Tursen's dataset uses the model trained on Kalantari's dataset for inference testing, as this dataset does not have reference high dynamic range images, so it is only for qualitative testing.

[0145] Method application

[0146] The present application uses PyTorch implementation, and the high dynamic range feature reconstruction in the deep expansion model is set to default 4 stages, that is, T = 4. Each stage is composed of 2 spatial alignment modules, 1 spatial fusion module, and 2 channel fusion modules. During training, the present application samples 128x128 size image blocks from the dataset, and applies data augmentation techniques, including random cropping, rotation and flipping. The present application uses the Adam optimizer, the batch size is 6, the initial learning rate is 5x10 −4 , and is decayed to 5x10 −6 using the cosine decay strategy.

[0147] Implementation effect comparison

[0148] For easy comparison, the present application selects several classic methods or state-of-the-art methods that have been improved for the multi-exposure high dynamic range image reconstruction task. These methods include DHDR, AHDR, NHDRR, HDR-GAN, APNT, CA-ViT, HyHDR, DiffHDR, SCTNet, LFDiff. The following mainly reports the performance comparison of the present application and other comparative methods in the high dynamic range domain PSNR, SSIM and the image domain after tone mapping PNSR, SSIM, respectively, denoted as PNSR- , SSIM- , PNSR- , PNSR- .

[0149] Table 1 gives the quantitative results on Kalantari's dataset. In order to distinguish the obvious, the two methods with the first and second performance are indicated by bold and underlined respectively.

[0150] Table 1 Comparison of the present application and other models on Kalantari's dataset

[0151]

[0152] Table 2 gives quantitative results on Hu's dataset. For the sake of clear distinction, the first and second two methods are indicated by bold and underlined respectively.

[0153] Table 2 Comparison results of the present application and other models on Hu's dataset

[0154]

[0155] Table 3 gives quantitative results on Tel's dataset. For the sake of clear distinction, the first and second two methods are indicated by bold and underlined respectively.

[0156] Table 3 Comparison results of the present application and other models on Tel's dataset

[0157]

[0158] Figure 4 gives the visual comparison results on Kalantari's dataset. In the upper half, the three pictures marked by green boxes are multi-exposure low dynamic range images, which are the input of the model, the middle is the reconstruction result of the present application, and the right is the input of the two image blocks for comparison. The two lines in the lower half are the reconstruction results of the two image blocks by each method. The red arrows indicate the poor positions in the reconstruction results, which are far from the actual image and have a great impact on visual perception.

[0159] After comparison, it can be seen that the present application is very close to the actual image, which shows excellent reconstruction performance. Compared with other methods, the present application has better robustness to misalignment and can reduce the existence of artifacts; at the same time, the present application has a stronger advantage in generating details of the image, such as the horizontal lines on the building in the blue block.

[0160] Figure 5 gives the visual comparison results on Hu's dataset. The present application is very close to the actual image, and compared with other methods, the present application has better artifact suppression effect and no obvious artifacts, and the visual effect of the present application is more excellent.

[0161] Figure 6 gives the visual comparison results on Tel's dataset. The present application is very close to the actual image, and compared with other methods, the present application has better artifact suppression effect and no obvious artifacts, and the visual effect of the present application is more excellent.

[0162] Fig. 7 shows the visual comparison results on Tursen's dataset. The dataset does not have actual images, and can only be evaluated by subjective effects, and quantitative indicators cannot be calculated. By comparison, it can be seen that the reconstruction effect of the present application is almost free of obvious artifacts in areas prone to artifacts such as non-aligned and poorly exposed image blocks, and can better capture the details of the image, bringing a more excellent visual experience and realism.

Claims

1. A high dynamic range image reconstruction method based on a progressive alignment fusion deep unfolding model, characterized in that, Comprising the following steps: Step S1 modeling of the multi-exposure high dynamic range image reconstruction problem The multi-exposure high dynamic range image reconstruction problem is modeled from the perspective of the maximum posterior problem, a prior regularization term that constrains the spatial correspondence between low dynamic range images is introduced, and an optimization objective function for reconstruction is obtained, as follows: wherein, denotes a reference image of normal exposure, a high dynamic range image of the reconstruction target is reconstructed from and needs to be aligned in content; , , are three different exposure images respectively , , is a degradation matrix; is the reconstruction target, i.e. a high dynamic range image, , is an auxiliary variable introduced; and are prior regularization terms used to describe the spatial correspondence, , are weight coefficients corresponding to each regularization term; Step S2 iterative solution of the multi-exposure high dynamic range image reconstruction problem The solution of the optimization objective function is first relaxed into two sub-problems of alignment and fusion, and then the two sub-problems are solved by using an iterative method, so that the solution of the whole problem is solved in an iterative manner of alternating between the two sub-problems until the problem converges to find the optimal solution; Step S3 building of a deep unfolding model The iterative formula derived in step S2 is unfolded, each stage corresponds to a deep unfolding module, the module structure corresponding to each stage is the same and the parameters are not shared, the deep unfolding modules of each stage are stacked to build a deep unfolding model; Step S4 model training The progressive alignment and fusion deep unfolding model of step S3 is trained.

2. The method of claim 1, wherein the method further comprises: Step S1 is specifically: From the perspective of the maximum posterior problem, the multi-exposure high dynamic range image reconstruction problem is modeled, and the specific process and details of the modeling are as follows: The degradation process from a high dynamic range image to a low dynamic range image is mathematically expressed as: ， wherein denotes a high dynamic range image, denotes a degradation process, is a low dynamic range image, is noise introduced in the degradation process; Reformulating the ill-posed problem of the above degradation process as an optimization problem under the maximum a posteriori problem framework, including a data fidelity term and a regularization term ; The data fidelity term is usually defined as The norm, with the following energy function: wherein is the reconstructed high dynamic range image, is a regularization term weighting hyperparameter; The goal of the algorithm is to merge three different low dynamic range images of underexposed , normally exposed , and overexposed images into a single high quality high dynamic range image without artifacts; in particular, the low dynamic range images are used as reference images, while the predicted high dynamic range image must be aligned with the content; by introducing non-reference low dynamic range images and to extend equation (2), and provide complementary information from the scene at different exposure levels, thus enabling an enhanced effect of reconstruction ; Two spatial correspondence prior regularization terms are introduced. and , and Multiple non-reference LDR images were explicitly simulated. , and reference low dynamic range image Rich priors between them are used for high dynamic range image reconstruction. and It is a priori regularization term used to describe spatial correspondences; and They represent and Spatial alignment auxiliary variables are used to gradually align the image structure and content with the spatial alignment during iterative optimization. Alignment; Based on the above, the optimization problem of HDR reconstruction is re-expressed as: wherein, , are three different exposure images , , degradation matrix, , is the weight coefficient corresponding to each regularization term; formula (3) is the optimization objective of the final modeling of the high dynamic range image reconstruction. 3.The method of claim 1, wherein, Step S2 is specifically: Alignment sub-problem: Fusion sub-problem: For the alignment sub-problem in formula (4a) and formula (4b), a gradient descent operator is further defined ， wherein, is a step size, is a gradient of the spatial correspondence prior term with respect to the alignment auxiliary variable , wherein = 1, 3; the step optimization can be expressed as: For the fusion sub-problem in formula (4c), a semi-quadratic splitting method is used to decouple the data fidelity term and the regularization term: Two auxiliary variables are introduced and , and correspond to two prior regularization terms that constrain the spatial correspondence between different low dynamic range images, and further constrain and to be close to the target image ; , is a weight hyperparameter; The fusion problem is decomposed into three sub-problems, each of which is updated iteratively: Given reconstructed image and an alignment auxiliary variable and , define the proximal operator and for the optimization update of and ; the specific definitions of the two proximal operators are as follows: ， , Formula (7a) and formula (7b) can be solved by the following iterative formula: Formula (7c) represents a quadratic regularization least squares problem, which has a closed-form solution as follows: where is the identity matrix, is the degenerate matrix is the transpose matrix; the matrix inverse is computationally expensive, so is considered as a whole, denoted as . 4.The method of claim 1, wherein, In step S3, the deep unfolding model specifically includes three stages: feature extraction and initialization, high dynamic range feature reconstruction, and high dynamic range image reconstruction, as follows: Stage 1 feature extraction and initialization First, initialization is performed, that is, the features of multiple low dynamic range images with different exposures are extracted using a feature extractor, and the extracted features are used to initialize the iterative process of the next stage; The picture of the input model is wherein is a low dynamic range image, is a corresponding image after gamma correction, used to provide more information for reconstruction for the model; the picture of the input model is extracted by three convolution kernel convolution layers to extract features and project the image domain to the feature domain wherein, respectively refer to batch size, channel number, height, and width; the feature extraction process is expressed as follows: extracted features , , for initializing the feature variables in the 0th iteration process inside the high dynamic range feature reconstruction phase , , ; Stage 2 high dynamic range feature reconstruction Then, high dynamic range feature reconstruction is performed, that is, iteration is performed in the feature domain to gradually achieve alignment and fusion operations, and three different exposure range image features are aligned and fused side by side to finally obtain high dynamic range image target reconstruction features; The depth unfolding model uses stacked aligned fusion modules AFM from an iterative optimization unfolding algorithm to solve the optimization objective of reconstructing a high dynamic range image proposed in step S1 ; all stages of the depth unfolding model maintain the same structure, but the parameters are not shared between stages; the iterative process is represented as: wherein, is the output of the previous iteration stage as input to the current iteration stage, is the output of the current iteration stage, and so on. Each iteration stage is subdivided into two sub-stages of feature alignment and feature fusion: Stage 2.1 feature alignment Corresponding to the sub-problem formulas (5a) and (5b) aligned with step S2, the spatial alignment module is configured to align and respectively to as two gradient descent operators therein and ; The spatial alignment module is constructed as a window-based cross-attention block, denoted as: wherein, is a window-based cross-attention alignment module, is a feed-forward network; through the spatial alignment module, information in the reference image of normal exposure is queried, while is used to maintain the spatial structure of the reference image; The following is a detailed implementation of the above. wherein, is a learnable transformation, is and a feature channel dimension, denotes a multi-layer perceptron (MLP) for learning a degenerated matrix, wherein by end-to-end training, the learning of the model for the degenerated matrix is achieved; Stage 2.2 feature fusion The feature fusion process includes the update of three feature variables, namely ; updating and According to the formula (8a) and formula (8b), a two-step fusion module is used to learn the proximity operator between features; first, it fuses the features in space, and then performs channel-level fusion; denoted as: wherein, represents a spatial fusion module, the input is the result of the cascade of the three, the output is ; and are respectively used for fusing the updated channels The updated channel fusion module adopts a channel-based cross-attention mechanism to obtain a further refined and fused result and ; updates : for the optimization of the variables of the previous stage , the update results obtained from equation (14b) and equation (14c) and are used to make a preliminary update by the data consistency module DCM to obtain , as shown in the following equation: After the preliminary update on , the optimized variables are fused using a multi-layer perceptron , , to further update and obtain a further refined result : Finally, the final result of the present iteration is updated , , passed to the next iteration until the full number of iterations of the high dynamic range feature reconstruction phase is completed. Stage 3 high dynamic range image reconstruction Finally, high dynamic range image reconstruction is performed, i.e., the obtained target reconstruction features are projected from the feature domain to the image domain by using a deep learning module to obtain a final high dynamic range image reconstruction result. After completing all the iterative reconstruction processes in the feature domain , the final high dynamic range feature image is reconstructed ; in order to ensure the stability of the image reconstruction process, the residual strategy is adopted, and then the feature is projected to the reconstructed high dynamic range image ; the high dynamic range image reconstruction stage is defined as follows: wherein, and denote a Sigmoid activation function and a convolution operation, respectively. 5.The method of claim 1, wherein In step S4, an end-to-end training of the progressive alignment fusion depth expansion model is performed using a linear combination of the reconstruction loss and the perceptual loss reconstruction loss and perceptual loss reconstruction loss and perceptual loss reconstruction loss and perceptual loss as follows: wherein, are features extracted from the 7th layer in VGG-19, are features extracted from the 7th layer in VGG-19, represents an image processed by a sigmoid function; represents an image processed by a sigmoid function; Thus, the total loss is: wherein, is a weight coefficient, set to 0.005; In view of the fact that computing the loss in the high dynamic range image domain results in a reduction in the efficiency of training, the loss after tone mapping is computed using a -law function wherein, is set to 5000.

Images