An image stroboscopic removal method based on a cycle generative adversarial network

Abstract

The application relates to an image stroboscopic removal method based on a cycle generative adversarial network, which trains a cycle generative adversarial network model on the basis of a synthetic stroboscopic image dataset, and the cycle generative adversarial network model comprises two generation networks and two discrimination networks, the two generation networks are respectively responsible for generating a non-stroboscopic image from an input stroboscopic image and generating a stroboscopic image from an input non-stroboscopic image, and the discrimination network utilizes a real image and a generated image of the generation network to train the generation network in combination with a generative adversarial idea. Finally, the purpose of finally outputting a corresponding non-stroboscopic image of an input stroboscopic image is achieved, and the practicability of the image stroboscopic removal method is effectively improved.

Description

Technical Field

[0001] This invention belongs to the field of computer image processing and mainly relates to image flicker removal under artificial lighting imaging environments. Specifically, it is an image flicker removal method based on recurrent generative adversarial networks. Background Technology

[0002] In artificial lighting environments, such as when fluorescent lamps and LED lights are powered by AC mains electricity, their light intensity fluctuates due to the periodic changes in the mains current. Because rolling shutter cameras use line-by-line exposure, the images captured by these cameras can capture the brightness fluctuations of the light source at different exposure times for each line, specifically manifested as striped patterns of light and dark variations, i.e., the flicker effect of the image.

[0003] Generative Adversarial Networks (GANs) are generative models that typically consist of two sub-models with opposing optimization objectives: a generator, responsible for fitting the latent distribution of real data, and a discriminator, which acts as a binary classifier to distinguish whether the input data is real or fabricated by the generator. CycleGANs are a type of GAN trained in unsupervised mode, designed to address the difficulty of obtaining paired data in supervised learning. CycleGAN's proposed cycle consistency loss allows the model to learn a one-to-one mapping between the source and target domains even in the absence of paired data. Both the generator and discriminator networks employ convolutional neural networks.

[0004] Generative adversarial networks (GANs) take real images as input and output another image with specific attributes.

[0005] Existing image flicker removal methods include: using multiple images with different exposure times for image fusion to suppress flicker; and using additional prior information such as the line scanning frequency of the imaging device and the power grid frequency to design digital filters to remove flicker components from the image.

[0006] However, existing flicker removal methods are difficult to apply to single images or situations where camera or power grid parameters are unknown, limiting their application scenarios and making them less practical. Summary of the Invention

[0007] To address the problems existing in the prior art, the present invention aims to provide an image flicker removal method based on cyclic generative adversarial networks. This method introduces the concept of cyclic generative adversarial networks, eliminating the need for additional prior information or matching flicker-free images for reference, thereby enhancing the application scenarios and practicality of the flicker removal method.

[0008] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0009] A method for removing image flicker based on recurrent generative adversarial networks includes the following steps:

[0010] Step 1: Training the image flicker removal model;

[0011] Step 1.1: Construct a flicker-free cyclic generative adversarial network framework;

[0012] The flicker-removing cyclic generative adversarial network framework includes a generator network G, a generator network R, a discriminator network D1, and a discriminator network D2. The generator network R is used to learn the mapping for eliminating image flicker; the generator network G is used to learn the generation of flickering images; the discriminator network D1 is used to distinguish between the flicker-removed image and the real flicker-free image; and the discriminator network D2 is used to distinguish between the flickering image generated by the model and the real flickering image.

[0013] Step 1.2: Input a flicker / flicker-free image training set to train the image flicker removal model. The flicker / flicker-free image training set includes a flicker image X and a flicker-free image Y. The generator network R generates a flicker-free image Yfake based on the input flicker image X, and generates an ontology-mapped flicker-free image Yiden based on the input flicker-free image Y. The generator network G generates a flicker-containing image Xfake based on the input flicker-free image Y, and generates an ontology-mapped flicker image Xiden based on the input flicker image X. The model's flicker loss loss_Flicker, gradient loss loss_Grad, and ontology-mapped loss loss_Iden are calculated.

[0014] The generated de-flicker image Yfake and flicker image Xfake are input into the generator network G and generator network R respectively to generate a cyclic flicker image Xcycle and a cyclic de-flicker image Ycycle, and the cycle consistency loss loss_Cycle is calculated.

[0015] Input the flicker-free images Y and Yfake into the discrimination network D1, and output the probability of the real image [P1]. 1 real P1 2 real P1 3 real ] and the probability of generating images [P1] 1 fake P1 2 fake P1 3 fake ], and calculate the discriminator loss loss_Disc1 corresponding to D1;

[0016] Input the stroboscopic images X and Xfake into the discrimination network D2, and output the probability of the real image [P2].1 real P2 2 real P2 3 real [P2] and the probability of generating images 1 fake P2 2 fake P2 3 fake ], and calculate the discriminator loss loss_Disc2 corresponding to D2;

[0017] Update the network parameters of D1 and D2 according to loss_Disc1 and loss_Disc2;

[0018] The de-flickering image Yfake is then input into the discrimination network D1, which outputs the probability of generating an image [P1]. 1 fake P1 2 fake P1 3 fake ] Calculate the adversarial loss loss_Gen_R of the generator network R.

[0019] Input the generated strobe image Xfake into the discrimination network D2, and output the probability of the generated image [P2]. 1 fake P2 2 fake P2 3 fake ], and calculate the adversarial loss loss_Gen_G of the generator network G;

[0020] Calculate the total losses loss_R and loss_G of the generator network R and generator network G, and update the network parameters to obtain the image flicker removal model.

[0021] Step 2: Use the image flicker removal model trained in Step 1 to remove image flicker. That is, input the flickering image into the image flicker removal model trained in Step 1 and output the flicker-free image.

[0022] The flicker loss_Flicker is calculated as follows:

[0023] loss_Flicker=|X p -Yfake p |1+|Y p -Xfake p |1

[0024] Wherein, the p-index represents the average value of the two-dimensional pixel matrix calculated in each of the R, G, and B channels.

[0025] The gradient loss, loss_Grad, is calculated as follows:

[0026]

[0027] in, This indicates that the gradient is calculated in the horizontal direction. This indicates that the gradient is calculated in the vertical direction.

[0028] The ontology mapping loss_Iden is calculated as follows:

[0029] loss_Iden=|X-Xiden|1+|Y-Yiden|1.

[0030] The cycle consistency loss_Cycle is calculated as follows:

[0031] loss_Cycle=|X-Xcycle|1+|Y-Ycycle|1.

[0032] The discriminator losses loss_Disc1 and loss_Disc2 are calculated as follows:

[0033] loss_Disc1=-Σ i (log(1-P1 i fake) + log(P1 i real))

[0034] loss_Disc2=-Σ i (log(1-P2 i fake) + log(P2) i real).

[0035] The adversarial losses of the generative network, loss_Gen_R and loss_Gen_G, are calculated as follows:

[0036] loss_Gen_R=-Σ i log(P1ifake);

[0037] loss_Gen_G=-Σ i log(P2 i (fake).

[0038] The total loss of the generator network R and the generator network G is calculated as follows:

[0039] loss_R=loss_Flicker+loss_Grad+loss_Iden+loss_Cycle+loss_Gen_R;

[0040] loss_G=loss_Flicker+loss_Iden+loss_Cycle+loss_Gen_G.

[0041] The flickering image X in the flicker / flicker-free image training set is obtained by flickering synthesis of the flicker-free image Y, as detailed below;

[0042] Based on the changing pattern of the stroboscopic signal, a sinusoidal signal matching the pattern is generated and filled in the column direction to match the image size;

[0043] The filled signal is superimposed on the flicker-free image Y to obtain the synthesized flicker image X.

[0044] The generator network G and generator network R have the same structure, both including convolutional layers, deconvolutional layers, skip connections, ReLU nonlinear activation function layers, and instance normalization layers;

[0045] The discrimination network D1 and discrimination network D2 have the same structure, both including an average pooling layer, a convolutional layer, a LeaklyReLU nonlinear activation function layer, and a spectral normalization layer.

[0046] By adopting the above scheme, this invention trains a recurrent generative adversarial network (GAN) model based on a synthetic stroboscopic image dataset. This model comprises two generator networks and two discriminator networks. The generator networks are responsible for generating flicker-free images from input stroboscopic images and generating stroboscopic images from input flicker-free images, respectively. The discriminator networks use the input real images and the generated images from the generator networks, combined with the generative adversarial approach, to train the generator networks. Ultimately, this achieves the goal of outputting a flicker-free image from an input stroboscopic image, effectively improving the practicality of the image flicker removal method.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] 1. Unsupervised training is used, which eliminates the need for images with different shutter speeds or exposure levels, thus reducing the difficulty of data acquisition.

[0049] 2. Based on the statistical laws of flicker, flicker loss and gradient loss are proposed, which improve the image quality after flicker elimination.

[0050] 3. No additional information such as the scanning frequency of the imaging equipment or the power grid frequency is required as prior knowledge.

[0051] 4. It has no restrictions on the background or content of the image, and has good generalization performance and broad application prospects. Attached Figure Description

[0052] The present invention will now be further described with reference to the accompanying drawings.

[0053] Figure 1 It is a diagram of a recurrent generative adversarial network model for image stroboscopic removal;

[0054] Figure 2 This is an example graph of the dataset;

[0055] Figure 3 This is a structural diagram of the generator network G;

[0056] Figure 4 This is a structural diagram of the identification network D;

[0057] Figure 5 This invention demonstrates the flicker removal effect of the proposed model on flickering images and compares it with the flicker removal effect of a general recurrent generative adversarial network that does not use flicker loss and gradient loss. Detailed Implementation

[0058] This invention discloses an image flicker removal method based on a recurrent generative adversarial network, which specifically includes the following steps:

[0059] Step 1: Training the image flicker removal model;

[0060] Step 1.1: Construct a flicker-free cyclic generative adversarial network framework;

[0061] like Figure 1 As shown, the flicker-free cyclic generative adversarial network framework includes a generator network G, a generator network R, a discriminator network D1, and a discriminator network D2. The generator network R is used to learn the mapping for eliminating image flicker; the generator network G is used to learn the generation of flickering images; the discriminator network D1 is used to distinguish between the flicker-free image and the real flicker-free image; and the discriminator network D2 is used to distinguish between the flickering image generated by the model and the real flickering image.

[0062] To enable the generative network to better capture the data distribution of real images, the structures of the generative networks R and G are defined, such as... Figure 2 As shown. The discriminator networks D1 and D2 employ multi-scale discrimination and spectral normalization, simultaneously discriminating at multiple scales of the input image to guide the training of the generator network. This allows the generator network to learn the flicker-free mapping, preserving the image's texture and details to the greatest extent possible. The structure of the discriminator network D is defined as follows: Figure 3 As shown. The generator network G and generator network R have the same structure, both containing convolutional layers, deconvolutional layers, skip connections, ReLU nonlinear activation function layers, and instance normalization layers. The discriminator networks D1 and discriminator networks D2 have the same structure, both containing average pooling layers, convolutional layers, LeaklyReLU nonlinear activation function layers, and spectral normalization layers.

[0063] Step 1.2: Input the flicker / flicker-free image training set to train the image flicker removal model; the flicker / flicker-free image training set includes flicker image X and flicker-free image Y.

[0064] Existing image flicker removal methods often require paired flicker-free images as references or additional imaging information as priors to estimate flicker components before using digital filters to remove the flicker, resulting in poor practicality. Unlike existing flicker removal methods, this invention introduces a recurrent generative adversarial network (GAN) for training with unpaired flicker / flicker-free images, aiming to learn the mapping from flicker images to flicker-free images. To this end, this invention, based on the existing IndoorCVPR-09 dataset, obtains a training dataset by artificially synthesizing flicker images. Example dataset diagrams are shown below. Figure 4 As shown.

[0065] The flickering image X in the flicker / flicker-free image training set is obtained by flickering synthesis of the flicker-free image Y, as detailed below;

[0066] Based on the changing pattern of the stroboscopic signal, a sinusoidal signal matching the pattern is generated and filled in the column direction to match the image size;

[0067] The filled signal is superimposed on the flicker-free image Y to obtain the synthesized flicker image X.

[0068] The generator network R generates a flicker-free image Yfake from the input flicker image X, and generates an ontology-mapped flicker-free image Yiden from the input flicker-free image Y; the generator network G generates a flicker-included image Xfake from the input flicker-free image Y, and generates an ontology-mapped flicker image Xiden from the input flicker image X, and calculates the model's flicker loss loss_Flicker, gradient loss loss_Grad, and ontology-mapped loss loss_Iden;

[0069] loss_Flicker=|X p -Yfake p |1+|Y p -Xfake p |1;

[0070]

[0071] loss_Iden=|X-Xiden|1+|Y-Yiden|1;

[0072] Wherein, the p-index represents the average value of the two-dimensional pixel matrix calculated in each of the R, G, and B channels; This indicates that the gradient is calculated in the horizontal direction. This indicates that the gradient is calculated in the vertical direction.

[0073] The generated de-flicker image Yfake and flicker image Xfake are input into the generator network G and generator network R respectively to generate a cyclic flicker image Xcycle and a cyclic de-flicker image Ycycle, and the cycle consistency loss loss_Cycle is calculated.

[0074] loss_Cycle=|X-Xcycle|1+|Y-Ycycle|1.

[0075] Input the flicker-free images Y and Yfake into the discrimination network D1, and output the probability of the real image [P1]. 1 real P1 2 real P1 3 real ] and the probability of generating images [P1] 1 fake P1 2 fake P1 3 fake ], and calculate the discriminator loss loss_Disc1 corresponding to D1, and update the parameters of the discriminator network D1 according to the following formula:

[0076] loss_Disc1=-Σ i (log(1-P1 i fake) + log(P1 i real))

[0077] Input the stroboscopic images X and Xfake into the discrimination network D2, and output the probability of the real image [P2]. 1 real P2 2 real P2 3 real [P2] and the probability of generating images 1 fake P2 2 fake P2 3 fake ], and calculate the discriminator loss loss_Disc2 corresponding to D2, and update the parameters of the discriminator network D2 according to the following formula:

[0078] loss_Disc2=-Σ i (log(1-P2 i fake) + log(P2) i real))

[0079] The de-flickering image Yfake is then input into the discrimination network D1, which outputs the probability of generating an image [P1]. 1 fake P1 2 fake P1 3 fake The adversarial loss loss_Gen_R of the generator network R is calculated according to the following formula:

[0080] loss_Gen_R=-Σ i log(P1 i fake);

[0081] Input the generated strobe image Xfake into the discrimination network D2, and output the probability of the generated image [P2]. 1 fake P2 2 fake P2 3 fake The adversarial loss loss_Gen_G of the generator network G is calculated according to the following formula:

[0082] loss_Gen_G=-Σ i log(P2 i fake);

[0083] The total losses loss_R and loss_G of the generator network R and generator network G are calculated according to the following formulas, and the network parameters are updated to obtain the image flicker removal model:

[0084] loss_R=loss_Flicker+loss_Grad+loss_Iden+loss_Cycle+loss_Gen_R;

[0085] loss_G=loss_Flicker+loss_Iden+loss_Cycle+loss_Gen_G.

[0086] Step 2: Use the image flicker removal model trained in Step 1 to remove image flicker. That is, input the flickering image into the image flicker removal model trained in Step 1 and output the flicker-free image.

[0087] This invention proposes an image flicker removal model based on a recurrent generative adversarial network (GAN). This model includes a training phase and an application phase. The training phase includes the following steps:

[0088] 1. Input the original flickering image X and the original flicker-free image Y into the corresponding data channels (R—>G—>R, G—>R—>G) to obtain the model-removed flickering image Yfake and the model-generated flickering image Xfake respectively. Map the model body to the flicker-free image Yiden, and map the model body to the flickering image Xiden. Loop through the removal of the flickering image Ycycle, and loop through the generation of the flickering image Xcycle.

[0089] 2. Input the real flicker-free image Y and the model-removed flicker image Yfake into the discrimination network D1. Input the synthetic flicker image X and the model-generated flicker image Xfake into the discrimination network D2. Output the real and fake probabilities of the two respectively.

[0090] 3. Update the network weights of the generation network and the discrimination network according to the corresponding error formula.

[0091] Actual usage process of the model: Input the stroboscopic image into the generator network R, and you can get the de-stroboscopic image Yfake.

[0092] Figure 5 The image shows the flicker removal effect of the proposed model on flickering images and a comparison with the flicker removal effect of a general recurrent generative adversarial network (CycleCAN) without flicker loss and gradient loss. Figure 5 As can be seen, the present invention has a better effect in eliminating flicker.

[0093] In summary, this invention trains a recurrent generative adversarial network (GAN) model based on a synthetic stroboscopic image dataset. This model comprises two generator networks and two discriminator networks. The generator networks are responsible for generating flicker-free images from input stroboscopic images and generating stroboscopic images from input flicker-free images, respectively. The discriminator networks train the generator networks using the input real images and the generated images from the generator networks, combining the principles of generative adversarial approaches. Ultimately, this achieves the goal of outputting a flicker-free image from an input stroboscopic image, effectively improving the practicality of image flicker removal methods.

[0094] The above description is merely an embodiment of the present invention and does not constitute any limitation on the technical scope of the present invention. Therefore, any minor modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention shall still fall within the scope of the technical solution of the present invention.

Claims

1. A cyclic generative adversarial network-based image stroboscopic removal method, characterized in that: The method comprises the following steps: Step 1, training of an image flicker removal model; Step 1.1, constructing a cycle generative adversarial network framework for flicker removal; The cycle generative adversarial network framework comprises a generation network G, a generation network R, a discrimination network D1 and a discrimination network D2, wherein the generation network R is used to learn a mapping for removing image flicker; the generation network G is used to learn generation of flicker images; the discrimination network D1 is used to discriminate differences between images removed of flicker by the model and real images without flicker; and the discrimination network D2 is used to discriminate differences between flicker images generated by the model and real flicker images; Step 1.2, inputting a flicker / no-flicker image training set to train the image flicker removal model; the flicker / no-flicker image training set comprises flicker images X and no-flicker images Y; the generation network R generates images Yfake removed of flicker according to the input flicker images X, and generates body mapping no-flicker images Yiden according to the input no-flicker images Y; the generation network G generates images Xfake containing flicker according to the input no-flicker images Y, and generates body mapping flicker images Xiden according to the input flicker images X, and calculates a flicker loss loss_Flicker, a gradient loss loss_Grad and a body mapping loss loss_Iden; the generated flicker-removed images Yfake and flicker images Xfake are inputted into the generation network G and the generation network R respectively to generate cyclic flicker images Xcycle and cyclic flicker-removed images Ycycle, and a cycle consistency loss loss_Cycle is calculated; The de-flicker image Y and the Yfake are input into the discrimination network D1 respectively, and the possibility [P1 1 real ,P1 2 real ,P1 3 real ] of the real image and the possibility [P1 1 fake ,P1 2 fake ,P1 3 fake ] of the generated image are output, and a discriminator loss loss_Disc1 corresponding to the discrimination network D1 is calculated. The stroboscopic images X, Xfake are input into the discrimination network D2 respectively, and the possibility [P2 1 real ,P2 2 real ,P2 3 real ] of the real image and the possibility [P2 1 fake ,P2 2 fake ,P2 3 fake ] of the generated image are output, and the discriminator loss loss_Disc2 corresponding to the discrimination network D2 is calculated. The network parameters of the discrimination network D1 and the discrimination network D2 are updated according to loss_Disc1 and loss_Disc2; The de-flicker image Yfake is input into the discrimination network D1, and the possibility [P1 of generating the image is output. 1 fake ,P1 2 fake ,P1 3 fake ] and the adversarial loss loss_Gen_R of the generation network R is calculated. The generated stroboscopic image Xfake is input into the discriminator network D2, and the possibility [P2 of generating the image is output 1 fake ,P2 2 fake ,P2 3 fake ] and the adversarial loss loss_Gen_G of the generation network G is calculated; The total loss loss_R and loss_G of the generation network R and the generation network G are calculated, and the network parameters are updated to obtain the image flicker removal model; Step 2, image flicker removal is performed by using the image flicker removal model trained in step 1, that is, a flicker image is inputted into the image flicker removal model trained in step 1 to output a flicker-removed image. 2.The image stroboscopic removal method based on cycle generative adversarial network according to claim 1, wherein: The flicker loss loss_Flicker is calculated as follows: loss_Flicker = |X p -Yfake p |1+|Y p -Xfake p |1 Wherein, p index represents calculation of an average value of a two-dimensional pixel matrix in R, G and B channels. 3.The image stroboscopic removal method based on cycle generative adversarial network according to claim 1, characterized in that: The gradient loss loss_Grad is calculated as follows: wherein, represents calculating a gradient in the horizontal direction, represents calculating a gradient in the vertical direction.

4. The image stroboscopic removal method based on cycle generative adversarial network according to claim 1, characterized in that: The body mapping loss loss_Iden is calculated as follows: loss_Iden = |X-Xiden|1+|Y-Yiden|1.

5. The image stroboscopic removal method based on cycle generative adversarial network according to claim 1, characterized in that: The cycle consistency loss loss_Cycle is calculated as follows: loss_Cycle = |X-Xcycle|1+|Y-Ycycle|1.

6. The image stroboscopic removal method based on cycle generative adversarial network according to claim 1, characterized in that: The discriminator loss loss_Disc1 and the discriminator loss loss_Disc2 are calculated as follows: loss_Disc1 = -∑ i (log(1 - P1 i fake) + log(P1 i real)) loss_Disc2 = -∑ i (log(l - P2 i fake) + log(P2 i real)).

7. The image stroboscopic removal method based on cycle generative adversarial network according to claim 1, characterized in that: The adversarial loss loss_Gen_R and loss_Gen_G of the generation network are calculated as follows: loss_Gen_R = -∑ i log(P1 i fake); loss_Gen_G = -∑ i log(P2 i fake).

8. The image stroboscopic removal method based on cycle generative adversarial network according to claim 1, characterized in that: The total loss of the generation network R and the generation network G is calculated as follows: loss_R = loss_Flicker + loss_Grad + loss_Iden + loss_Cycle + loss_Gen_R; loss_G = loss_Flicker + loss_Iden + loss_Cycle + loss_Gen_G.

9. The image stroboscopic removal method based on cycle generative adversarial network according to claim 1, characterized in that: The stroboscopic image X in the stroboscopic / non-stroboscopic image training set is obtained by stroboscopic synthesis from the non-stroboscopic image Y, specifically as follows: According to the change rule of the stroboscopic signal, a sinusoidal signal matching the mode is generated and filled in the column direction to be consistent with the image size; The filled signal is superimposed with the non-stroboscopic image Y to obtain the synthesized stroboscopic image X.

10. The image stroboscopic removal method based on cycle generative adversarial network according to claim 1, characterized in that: The generation network G and the generation network R are the same in structure, and both include a convolution layer, a deconvolution layer, a jump connection, a ReLU nonlinear activation function layer, and an instance normalization layer; The discriminator network D1 and the discriminator network D2 are the same in structure, and both include an average pooling layer, a convolution layer, a LeaklyReLU nonlinear activation function layer, and a spectral normalization layer.

Images