A high dynamic range imaging and ghost removal method based on generative adversarial network

Abstract

The present invention belongs to the field of digital image processing, and specifically relates to a high dynamic range imaging and ghost removal method based on a generative adversarial network; the method extracts semantic and pixel features of continuous frames of video images by introducing a feature extraction module based on a dual attention mechanism to enhance the details and texture features of image fusion; in order to effectively improve the imaging quality and algorithm robustness, and at the same time solve the gradient vanishing and mode collapse problems in the GAN network, this paper studies a loss function based on spherical distance to solve the large-scale ghost problem generated by large displacement of moving targets in the synthesis stage.

Description

Technical Field

[0001] The present invention belongs to the field of digital image processing, and in particular relates to a high dynamic range imaging and ghost removal method based on a generative adversarial network. Background Art

[0002] High Dynamic Range (HDR) imaging is a set of technologies used to achieve a larger exposure dynamic range (i.e., a larger difference between light and dark) than ordinary digital imaging technologies. It not only provides better visual expression, but also provides a new and powerful driving force for the leapfrog development of digital media. At present, the image fusion method based on Low Dynamic Range (LDR) is still one of the main ways to obtain HDR imaging. In the image fusion process, how to effectively remove the ghosting effect caused by dynamic scenes, camera shake, inaccurate image registration and even large displacement of moving targets is a key problem that needs to be solved in HDR imaging. The technical difficulties it faces mainly include: 1) The global Multi-Exposure Fusion (MEF) algorithm is difficult to achieve the fusion goal of full light and dark details and rich colors close to natural scenes; and the use of local multi-exposure image fusion algorithm for solution will increase the complexity of the algorithm, and at the same time, the image details may be unnatural due to local over-enhancement, or halo and noise may be enhanced; 2) In real dynamic scenes, there are many complex and irregular movements, and the time complexity of existing ghost removal technical means and methods is relatively high; 3) Existing deep learning methods provide some solutions for massive video image fusion, but there is still room for improvement in enriching training data sets and improving the generalization ability of deep learning algorithms.

[0003] In the era of artificial intelligence, the deep learning and application of big data machines have developed in depth, and computer vision technology has been redefined. Relying on powerful computing hardware support and intensive training of large data sets, the robustness of deep learning (DL) networks has been continuously enhanced. Deep learning has gradually become the main method for processing complex high-dimensional data such as images, text corpora and sound signals, especially in the fields of image recognition and image processing, showing great potential. Convolutional Neural Networks (CNN), Artificial Neural Networks (ANN), Generative Adversarial Networks (GAN) and other structures have also been gradually proposed and widely used in image classification, face recognition, image enhancement, semantic segmentation, image reconstruction, image enhancement, super-resolution and other problems. At present, deep learning has been applied to HDR imaging, and deep HDR imaging methods based on convolutional neural networks (CNN) and generative adversarial networks (GAN) have been tried and achieved good results. Among them, the model of the GAN network is more suitable for ghost area recognition, so it has rapidly risen in the field of HDR imaging.

[0004] Deep learning has been used to achieve image reconstruction, especially for panoramic image reconstruction and ghost removal, and has achieved outstanding results. However, since most of the current optimizations are based on network improvements, there is still a lack of more convincing arguments for the mathematical expression of image pixels or spatial features at the network layer, and there is insufficient theoretical argumentation for the schemes that affect imaging quality. Therefore, methods based on deep learning also need to be further improved. Summary of the invention

[0005] To solve the above problems, this paper mainly studies the dynamic image fusion and ghost removal methods based on large-scale motion scenes, and proposes solutions for image reconstruction and ghost removal based on traditional multi-exposure fusion (MEF) and deep learning.

[0006] A high dynamic range imaging and ghost removal method based on a generative adversarial network, comprising:

[0007] S1. Acquire a continuous multi-frame low dynamic range image sequence, which includes a low-exposure LDR image, a normal-exposure LDR image, and a high-exposure LDR image, and perform preprocessing and block division to obtain a group of low-exposure and normal-exposure image block pairs, and normal-exposure and high-exposure image block pairs for training; wherein the normal-exposure LDR image is used as a reference image;

[0008] S2. Construct a generative adversarial network for high dynamic range imaging, which includes a generator network and a discriminator network;

[0009] S3. Use the groups of low-exposure and normal-exposure image patch pairs, as well as the normal-exposure and high-exposure image patch pairs to train the generative adversarial network, use the Wasserstein distance loss function to optimize until convergence to the Nash equilibrium, and obtain and save the trained generative adversarial network;

[0010] S4. Input the image to be fused into the trained generative adversarial network to obtain an HDR image.

[0011] Furthermore, the specific process of preprocessing and dividing each group of low-exposure LDR images, normal-exposure LDR images, and high-exposure LDR images in step S1 includes:

[0012] S11. Using the camera response function to establish a pixel mapping function between the low-exposure LDR image, the high-exposure LDR image and the normal-exposure LDR image;

[0013] S12. normalizing the normally exposed low dynamic range image, the low-exposure LDR image after pixel mapping, and the high-exposure LDR image;

[0014] S13. Divide the image normalized in step S12 into blocks according to the size of n×n to obtain image block pairs consisting of low-exposure image blocks and normal-exposure image blocks, and image block pairs consisting of normal-exposure image blocks and high-exposure image blocks.

[0015] Furthermore, the generator network includes an encoding module, a synthesis module and a decoding module, the encoding module includes a dual attention module and a feature extraction module, and the dual attention module includes a channel attention unit and a spatial attention unit.

[0016] Furthermore, the processing of the spatial attention unit includes:

[0017] The low exposure image block P s1 and the normal exposure image block P r , Normal exposure image block P r and high exposure image block P s3 The local features of the image blocks are respectively subjected to 1×1 convolution operations; the image blocks after the convolution operations are respectively transposed and scaled to generate the first feature map, the second feature map and the third feature map; the first feature map is multiplied by the second feature map and then passed through the softmax function to obtain the first relationship strength matrix, and the second feature map is multiplied by the third feature map and then passed through the softmax function to obtain the second relationship strength matrix; the first relationship strength matrix is ​​combined with the image block P s1 Multiply the first spatial attention feature and add the second relationship reinforcement matrix to the image block P s3 Multiply them to get the second spatial attention feature;

[0018] The processing of the channel attention unit includes:

[0019] For image block P s1 , P r and P s3 Transpose and scale transformation are performed respectively to generate the first conversion feature, the second conversion feature and the third conversion feature; the first conversion feature is multiplied by the second conversion feature and then passed through the softmax function to obtain the first channel attention matrix, and the second conversion feature is multiplied by the third conversion feature and then passed through the softmax function to obtain the second channel attention matrix; the first channel attention matrix is ​​combined with the image block P s1 Multiply the first channel attention feature and add the second channel attention matrix to the image block P s3 Multiply to get the second channel attention feature.

[0020] Furthermore, the specific process of step S3 training the adversarial generative network includes:

[0021] S31. First, fix the parameters of the generator network and train the discriminator network, including:

[0022] S32. Obtain fake sample image blocks, that is, input the grouped low-exposure and normal-exposure image block pairs, and the normal-exposure and high-exposure image block pairs into the generator network to obtain m fused image blocks;

[0023] S33. Obtaining true sample image blocks, that is, obtaining m low-exposure source image blocks and high-exposure source image blocks from low-exposure and high-exposure low dynamic range images respectively;

[0024] S34. Send the fake sample image blocks and the real sample image blocks together to the discriminator network training, calculate the discriminant loss of the discriminator network, back-propagate the discriminant loss to update the discriminator network parameters, and use Adam to optimize;

[0025] S35. Next, the parameters of the discriminator network are fixed and the generator network is trained, including:

[0026] S36. Obtain m fake sample image blocks from the fake sample image blocks generated by training, and obtain m reference image blocks from the reference image;

[0027] S37. Extract pixel features of fake sample image blocks and reference image blocks and update generator network parameters

[0028] S38. Repeat steps S31-37 until the training of the generative adversarial network is completed. Further, the Wasserstein distance loss function is expressed as:

[0029]

[0030] To facilitate the solution, the above Wasserstein distance loss function is transformed and expressed as:

[0031]

[0032] Among them, P r represents the reference image block, P g Indicates the generated image block, the Lipschitz constant of the continuous function f is K, represents the expected value of the reference image, represents the expected generated image, sup represents the supremum, I r represents the reference image, I g Indicates the generated image.

[0033] Beneficial effects of the present invention:

[0034] This paper introduces the Generative Adversarial Network (GAN) into the ghost removal scheme of dynamic images. For massive video images, an HDR-WGAN image fusion and ghost removal algorithm is proposed. The scheme uses the learning mechanism and technical advantages of the competition between the generator G and the discriminator D to judge the inconsistent pixels and areas between the fusion result and the real image, and effectively removes ghosts. In order to solve the problems of gradient disappearance and mode collapse that are prone to occur during the training of the GAN network, a loss function with Wasserstein distance as the generator and discriminator is established. When the distribution of the two images does not overlap or the overlapping part is negligible, Wasserstein distance can still provide meaningful gradients; in order to improve the quality of HDR image fusion, especially the fusion of continuous frames of video images, this chapter introduces a dual attention learning module in the image synthesis module to obtain contextual semantics and feature information in the spatial dimension and channel dimension of the image, enhance the image feature information in the image synthesis module, and effectively improve the details and texture information of the generated fused image. Experimental results show that compared with the existing ghost removal algorithms, the HDR-WGAN algorithm proposed in this paper is particularly suitable for image reconstruction in complex dynamic scenes, and can completely remove ghosts. The images have richer texture and detail information, which effectively improves the imaging quality and algorithm robustness. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The HDR imaging mechanism based on the GAN network in the embodiment of the present invention;

[0036] Figure 2 It is the HDR-WGAN high dynamic range imaging algorithm model of the embodiment of the present invention;

[0037] Figure 3 This is a structural diagram of the spatial attention unit PAM according to an embodiment of the present invention;

[0038] Figure 4 This is a structural diagram of a channel attention unit CAM according to an embodiment of the present invention;

[0039] Figure 5 This is a flow chart of the HDR-WGAN high dynamic range imaging algorithm according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] GAN is a network model based on the game between the generator network (Generator, G for short) and the discriminator network (Discriminator, D for short). The goal of the generator G is to generate enough real data to confuse the discriminator D as much as possible, while the discriminator D needs to identify all the images generated by the generator G. Through the game between each other, a balance is finally reached, which is called Nash equilibrium, so that the generator network can finally learn the distribution of the input data. Its basic structure is as follows: Figure 1 As shown. The training process of the GAN network generally adopts the stochastic gradient descent method for training. Its training strategy is similar to solving the "NP-hard" problem of multiple optimization objectives. The generator network and the discriminator network are fixed, cross-trained, and iterated until both networks reach Nash equilibrium. The specific execution process is described as follows:

[0042] (a) Input the noisy data into the generator G to obtain the corresponding "generated sample". The initial "generated sample" is far from the target result and is called "false data", denoted by D(z);

[0043] (b) Input the real data into the discriminator D as the basis for discrimination, denoted as x;

[0044] (c) The data generated by one of the first two steps is used as the input of the discriminator network. The discriminator D is a binary classifier that determines whether an image is "real". Let the input of the discriminator D be x = G(z), where x represents a real image. Then D(x) represents the probability of a real image. 1 represents a 100% real image, while the output 0 represents an image that is not real.

[0045] (d) then calculating the loss function based on the designed statistical probability model;

[0046] (e) Based on the loss functions of the discriminant model and the generative model, the back propagation algorithm is used to update the model parameters.

[0047] In view of the problems existing in the existing high dynamic range imaging technology based on deep learning, especially based on generative adversarial network (GAN), in addition to considering the "gradient vanishing" and "mode collapse" problems that need to be overcome in the network training itself, it is also necessary to consider the ghosting effect caused by the presence of large moving targets in the scene. For this reason, the present invention proposes a high dynamic range imaging and ghost removal method based on generative adversarial network, and the method specifically includes:

[0048] S1. Acquire a sequence of multiple continuous low dynamic range images, which includes a low-exposure LDR image I s1 , Normally exposed LDR image I s2 and the high-exposure LDR image I s3 , and preprocess and divide the low dynamic range image sequence into blocks, and obtain a group of image block pairs consisting of low-exposure and normal-exposure image blocks, and an image block pair consisting of normal-exposure and high-exposure image blocks for training; wherein the normally-exposed LDR image I s2 As a reference image, the reference image is denoted as I r ;

[0049] S2. Construct a generative adversarial network for high dynamic range imaging, which includes a generator network and a discriminator network;

[0050] S3. Use the groups of low-exposure and normal-exposure image patch pairs, as well as the normal-exposure and high-exposure image patch pairs to train the generative adversarial network, use the Wasserstein distance loss function to optimize until convergence to the Nash equilibrium, and obtain and save the trained generative adversarial network;

[0051] S4. Input the image to be fused into the trained generative adversarial network to obtain an HDR image.

[0052] Specifically, the images to be fused may be a plurality of consecutive frames of video images or a plurality of consecutive frames of low dynamic range images.

[0053] Specifically, the specific process of preprocessing the low-exposure LDR image, the normal-exposure LDR image and the high-exposure LDR image of each group in step S1 includes:

[0054] S11. pixel-map the low-exposure LDR image, the high-exposure LDR image, and the normal-exposure LDR image using a camera response function;

[0055] S12. normalizing the normally exposed low dynamic range image, the low-exposure LDR image after pixel mapping, and the high-exposure LDR image;

[0056] S13. Divide the image normalized in step S12 into blocks according to the size of n×n to obtain a plurality of image block pairs consisting of low-exposure image blocks and normal-exposure image blocks, and image block pairs consisting of normal-exposure image blocks and high-exposure image blocks.

[0057] In one embodiment, the block diagram of the HDR-WGAN imaging and ghost removal method is as follows: Figure 2 As shown in the figure, the generative adversarial network contains a generator network and a discriminator network. The generator network includes an encoding module, a synthesis module, and a decoding module. In addition to the feature extractor, the encoding module also adds a dual attention module. The input image is extracted through the dual attention channel to enhance the image details. The dual attention module includes a channel attention unit and a spatial attention unit. The two complementary attention units capture context-related semantic information in the spatial dimension and the channel dimension respectively, and then enhance the feature extraction. Based on its significant advantages in feature extraction, it is widely used in the field of medical or remote sensing image classification. HDR image video fusion, the semantics of the previous and next frame images are highly correlated, and dual attention is introduced to enhance feature extraction, thereby effectively improving the expression and generation capabilities of the GAN network.

[0058] like Figure 3 As shown in Figure 2, the processing of the spatial attention unit includes:

[0059] The low exposure image block P s1 and the normal exposure image block P r , Normal exposure image block P r and high exposure image block P s3 The local features of the image block P are respectively subjected to 1×1 convolution operation; s1 , P r and P s3 Transpose and reshape (C×H×W) are performed respectively to generate the first feature map T(Is1), the second feature map T(Is2) and the third feature map T(Is3), and T(Is1), T(Is2), T(Is3)∈R∧(C×H×W), N=H×W; the first feature map is multiplied by the second feature map and then passed through the softmax function to obtain the first relationship strength matrix, and the second feature map is multiplied by the third feature map and then passed through the softmax function to obtain the second relationship strength matrix; the first relationship strength matrix is ​​compared with the image block P s1 Multiply the first spatial attention feature and add the second relationship reinforcement matrix to the image block P s3 Multiplying together gives the second spatial attention feature, expressed as:

[0060]

[0061] Among them, E i represents the feature after spatial attention enhancement, μ represents the mean of the input source image, T j represents the newly generated convolutional feature map, I i represents the input feature vector, N represents the image block size, s ji It represents the influence of channel i on channel j.

[0062] like Figure 4 As shown, the processing of the channel attention unit includes:

[0063] For image block P s1 , P r and P s3 Transpose and scale transformation are performed respectively to generate the first conversion feature, the second conversion feature and the third conversion feature; the first conversion feature is multiplied by the second conversion feature and then passed through the softmax function to obtain the first channel attention matrix, and the second conversion feature is multiplied by the third conversion feature and then passed through the softmax function to obtain the second channel attention matrix; the first channel attention matrix is ​​combined with the image block P s1 Multiply the first channel attention feature and add the second channel attention matrix to the image block P s3 Multiply to get the second channel attention feature, expressed as:

[0064]

[0065] Among them, E j. represents the feature after the final enhancement of channel attention, λ represents a constant, I i represents the input source graph, C represents the feature matrix, x ji It represents the influence of channel i on channel j.

[0066] The feature extraction module contains three branches X1, X2, and X3 with the same structure, which are used to extract image blocks P s1 , P r and P s3 Each branch consists of three serially connected coding units, which are connected by downsampling convolutional layers. The two downsampling of the feature extraction module divides the network into three different scales, where the input of branch X1 is the low-exposure image block P. s1 and the normal exposure image block P r The output after the dual attention module and the low-exposure image I s1 The low-exposure image block P after preprocessing s1 ; The input of branch X3 is the normal exposure image I s2 The normal exposure image block P after preprocessing s2 ; The input of branch X3 is the high exposure image block P s3and the normal exposure image block P r The output after the dual attention module and the high exposure image I s3 The high exposure image block P after preprocessing s3 .

[0067] Corresponding to the three scales of the encoding module, the synthesis module consists of three scale fusion branches Each fusion branch uses the output features of the coding units of the corresponding scale on the three-way branches in the coding module as input, and adds the output features of the fusion branch to the output features of the coding units of the corresponding scale on the normal exposure branch in the coding module through element-by-element addition operation to form a residual structure based on the features of the normal exposure image.

[0068] The decoding module consists of three decoder units Composition, among which, The first scale corresponding to the encoding module, The second scale corresponding to the encoding module; The input of is composed of two parts of features concatenated in the channel dimension, namely, the output features of the fusion branch of the synthesis module of the corresponding scale and the output features of the fusion branch of the synthesis module of the next scale; The input of is composed of two parts of features concatenated in the channel dimension, namely The output features and Output the result after feature upsampling.

[0069] The discriminator network consists of five convolutional layers connected in series.

[0070] In one embodiment, during the GAN network training process, the purpose of the generator G is to eliminate the ghosting effect caused by the position inconsistency of the moving target between the source images as much as possible, improve the image fusion quality, and continuously approach the real image value in the discriminator D; the goal of the discriminator D is to try to identify the "fake image" generated by the generator G as much as possible. The ideal result of the "dynamic game" between G and D is that G generates a "fake" G(Is), and D has difficulty in distinguishing G(Is), at which time D(x) = 0.5. Therefore, the HDR-WGAN high dynamic range imaging algorithm model proposed in the present invention is expressed as:

[0071] min G max D V(D,G)=E x-Pdata [log(D(I r ))]+E z-P(z) [log(1-D(G(I s )))]

[0072] Among them, G represents the generator, D represents the discriminator, V is the maximum and minimum objective optimization function of the discriminator D and the generator G, and I r represents the reference image, I s represents the input image, E x-Pdata represents the true sample image set (GT), E z-P(z) Represents the generator result atlas, x is the true sample image, and z is the fake sample image.

[0073] HDR-WGAN high dynamic range imaging algorithm Figure 5 As shown, including:

[0074] S201. Obtaining LDR images I with different exposures s1 ,I s2 and I s3 , select the LDR image I with good exposure s2 As the reference image, denoted as I r ;

[0075] S202. LDR images I with different exposures s1 ,I s2 and I s3 Preprocessing is performed, and blocks are divided into n×n size blocks to obtain groups of different exposure image blocks for training, and the groups of different exposure image blocks are mapped to the HDR domain to obtain different exposure image blocks in the HDR domain, and the LDR domain image blocks and the corresponding HDR domain image blocks are combined into 6-channel image blocks;

[0076] S203. First, the parameters of the generator network are fixed and the discriminator network is trained, including:

[0077] S204. Obtain fake sample image blocks, that is, input the grouped low-exposure and normal-exposure image block pairs, and the normal-exposure and high-exposure image block pairs into the generator network to obtain m fused image blocks P H1 ,P H2 ,...,P Hm ;

[0078] S205. Obtain true sample image blocks, that is, obtain m low-exposure source image blocks P from low-exposure and high-exposure low dynamic range images respectively. s11 ,P s12 ,...,P s1m and high exposure source image patch P s21 ,P s22 ,...,P s2m ;

[0079] S206. The fake sample image blocks and the real sample image blocks are sent to the discriminator network for training, the discriminant loss of the discriminator network is calculated, the discriminant loss is back-propagated to update the discriminator network parameters, and Adam is used for optimization;

[0080] S207. Next, the parameters of the discriminator network are fixed and the generator network is trained, including:

[0081] S208. Obtain m fake sample image blocks from the fake sample image blocks generated by training, and obtain m reference image blocks in the HDR domain from the reference image;

[0082] S209. Extract pixel features of the fake sample image block and the reference image block and update the generator network parameters;

[0083] S210. Repeat steps S203-209 until the generative adversarial network training is completed.

[0084] Specifically, the GAN network has problems such as difficulty in training, the loss functions of the generator and the discriminator cannot indicate the training process, as well as problems such as gradient disappearance and mode collapse. Moreover, the KL divergence and JS divergence are sudden changes. In contrast, the Wasserstein spherical distance is smooth. If the distributions of the two images do not overlap or the overlapping parts are negligible, the Wasserstein distance can still provide meaningful gradients. Based on this, the GAN network generation model in this embodiment uses the Wasserstein distance (also known as Earth-Move, referred to as EM distance) as the loss function:

[0085]

[0086] To facilitate the solution, the above formula is transformed as follows:

[0087]

[0088] Among them, the Lipschitz constant of the continuous function f(x) is K, and a set of parameters w can be used to define a series of possible functions f w (x), under the condition that w does not exceed a certain range, the loss function of the discriminator and the generator is expressed as follows:

[0089] L=E x～Pr [f w (x)]-E x～Pg [f w (x)]

[0090] Among them, when the parameter w is as large as possible, the discriminator's f w(x) is used to approximate the Wasserstein distance between the real image and the generated image. If the generator wants to approximately minimize the Wasserstein distance, it can minimize the formula L. The first term in the formula is independent of the generator, so the loss function of the generator and discriminator of WGAN is,

[0091] L G =-E xpg [f w (x)]

[0092] L vs L G They are inverse functions of each other. This relationship can indicate network training. The smaller the value, the smaller the Wasserstein distance metric function, and the more optimized the GAN network training.

[0093] All algorithms proposed in this paper are uniformly run on MD Ryzen 5 3600 6-Core Process 4.0 GHz, running memory is 16.00 GB, GPU Nvidia GeForce RTX 2060Super, 64-bit Windows 10 system.

[0094] Some dynamic data sets containing three images with different exposures are used, including day and night as well as indoor and outdoor scenes. The exposure difference of the source image is large and the motion in the scene is complex. In order to better analyze the effect of ghost removal of the algorithm in this chapter, scenes with large displacement of moving targets (Large Displacement of Motion, LDM for short) and scenes with deviations in shooting angles are selected for experiments.

[0095] Due to the existence of moving targets in dynamic scenes, some evaluation indicators cannot fully represent the image quality, and the objective evaluation and subjective observation results cannot be completely consistent. Therefore, this paper considers both objective and subjective evaluations.

[0096] (1) For subjective evaluation, the algorithms in this chapter are evaluated from the following five dimensions, such as image brightness, image resolution, texture details, edge preservation, and ghost freedom.

[0097] (2) For objective evaluation, both parameterized and non-parametric image quality assessment methods are used. For image naturalness and visual quality, the visual information fidelity index (VIFP), relative polar edge coherence (RECO) and tone image quality index (TMQI) are selected for quantitative comparison. The VIFP index is based on the natural scene statistical model (NSS); for non-parametric image quality assessment methods, the naturalness image quality evaluator (NIQE), TENENGRAD gradient function, information entropy are selected, and the HDR high dynamic image quality evaluation index (HDR-vdp-2.2) is also used.

[0098] Aiming at the problem of massive mobile video image fusion, the present invention establishes the loss function of the generator and discriminator based on spherical distance (Wasserstein), which effectively improves the accuracy of image registration; introduces the attention learning mechanism in the image synthesis module to enhance the details and texture features of the generated image. The proposed HDR imaging algorithm based on deep learning (High Dynamic Range Imaging with Wasserstein GAN, HDR-WGAN for short) is particularly suitable for the field of massive video image synthesis. The synthesized image has significant effects in both quality improvement and ghost removal, and the algorithm time complexity is greatly reduced compared with the traditional MEF algorithm.

[0099] In the present invention, unless otherwise clearly stipulated and limited, the terms such as "installation", "setting", "connection", "fixation" and "rotation" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral one; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium; it can be the internal connection of two elements or the interaction relationship between two elements. Unless otherwise clearly defined, ordinary technicians in this field can understand the specific meanings of the above terms in the present invention according to the specific circumstances.

[0100] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A high dynamic range imaging and ghost removal method based on a generative adversarial network, characterized in that: include: S1. Acquire a continuous multi-frame low dynamic range image sequence, which includes low-exposure, normal-exposure and high-exposure low dynamic range images, and perform preprocessing and block division to obtain a group of low-exposure and normal-exposure image block pairs, and normal-exposure and high-exposure image block pairs for training; wherein the normal-exposure low dynamic range image is used as a reference image; The specific process of preprocessing and dividing each group of low-exposure, normal-exposure and high-exposure low dynamic range images in step S1 includes: S11. Using the camera response function to establish a pixel mapping function between low-exposure, high-exposure and normal-exposure low dynamic range images; S12. performing normalization processing on the normally exposed low dynamic range image, and the low exposure and high exposure low dynamic range images after pixel mapping; S13. Divide the image normalized in step S12 into blocks according to the size of n×n to obtain image block pairs consisting of low-exposure image blocks and normal-exposure image blocks, and image block pairs consisting of normal-exposure image blocks and high-exposure image blocks; S2. Construct a generative adversarial network for high dynamic range imaging, which includes a generator network and a discriminator network; The generator network includes an encoding module, a synthesis module and a decoding module. The encoding module includes a dual attention module and a feature extraction module. The dual attention module includes a channel attention unit and a spatial attention unit. S3. Use the groups of low-exposure and normal-exposure image patch pairs, as well as the normal-exposure and high-exposure image patch pairs to train the generative adversarial network, use the Wasserstein distance loss function to optimize until convergence to the Nash equilibrium, and obtain and save the trained generative adversarial network; The Wasserstein distance loss function is expressed as: To facilitate the solution, the above Wasserstein distance loss function is transformed and expressed as: Among them, P r represents the reference image block, P g Indicates the generated image block, the Lipschitz constant of the continuous function f is K, represents the expected value of the reference image, represents the expected value of the generated image, sup represents the supremum, I r represents the reference image, I g Indicates the generated image; S4. Input the image to be fused into the trained generative adversarial network to obtain an HDR image.

2. The high dynamic range imaging and ghost removal method based on generative adversarial network according to claim 1, characterized in that: The processing of the spatial attention unit includes: The low exposure image block P s1 and the normal exposure image block P r , Normal exposure image block P r and high exposure image block P s3 The local features of the image blocks are respectively subjected to 1×1 convolution operations; the image blocks after the convolution operations are respectively transposed and scaled to generate the first feature map, the second feature map and the third feature map; the first feature map is multiplied by the second feature map and then passed through the softmax function to obtain the first relationship strength matrix, and the second feature map is multiplied by the third feature map and then passed through the softmax function to obtain the second relationship strength matrix; the first relationship strength matrix is ​​combined with the image block P s1 Multiply the first spatial attention feature and add the second relationship reinforcement matrix to the image block P s3 Multiply them to get the second spatial attention feature; The processing of the channel attention unit includes: For the image block P s1 , P r and P s3 Transpose and scale transformation are performed respectively to generate the first conversion feature, the second conversion feature and the third conversion feature; the first conversion feature is multiplied by the second conversion feature and then passed through the softmax function to obtain the first channel attention matrix, and the second conversion feature is multiplied by the third conversion feature and then passed through the softmax function to obtain the second channel attention matrix; the first channel attention matrix is ​​combined with the image block P s1 Multiply the first channel attention feature and add the second channel attention matrix to the image block P s3 Multiply to get the second channel attention feature.

3. The high dynamic range imaging and ghost removal method based on generative adversarial network according to claim 1, characterized in that: The specific process of step S3 training the adversarial generation network includes: S31. First, fix the parameters of the generator network and train the discriminator network, including: S32. Obtain fake sample image blocks, that is, input the grouped low-exposure and normal-exposure image block pairs, and the normal-exposure and high-exposure image block pairs into the generator network to obtain m fused image blocks; S33. Obtaining true sample image blocks, that is, obtaining m low-exposure source image blocks and high-exposure source image blocks from low-exposure and high-exposure low dynamic range images respectively; S34. Send the fake sample image blocks and the real sample image blocks together to the discriminator network training, calculate the discriminant loss of the discriminator network, back-propagate the discriminant loss to update the discriminator network parameters, and use Adam to optimize; S35. Next, the parameters of the discriminator network are fixed and the generator network is trained, including: S36. Obtain m fake sample image blocks from the fake sample image blocks generated by training, and obtain m reference image blocks from the reference image; S37. Extract pixel features of the fake sample image block and the reference image block and update the generator network parameters; S38. Repeat steps S31-37 until the generative adversarial network training is completed.

Images