A rain scene image synthesis method and device learning from rendering and a storage medium

Abstract

The present application relates to a kind of rain scene image synthesis method, device and storage medium from rendering learning, the method steps include: rendering stage creates high-resolution paired rain scene-background image dataset, contains multiple scenes and the image pair of different time under illumination condition;Learning stage introduces guide diffusion model in implicit diffusion model, learns a implicit space equivalent to image space in perception using autoencoder model, uses encoder to carry out perceptual compression, obtains the latent variable equivalent to the pixel of image space in implicit space;Forward process and reverse process of diffusion model are completed in implicit space, wherein the reverse process uses cross attention and series two conditional mechanisms to constrain;The latent variable of output is transformed to image space by decoder, and high-resolution rain scene image is generated.Compared with prior art, the present application has the advantages of authenticity based on rendering method and efficiency based on learning method.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to a high-resolution rain scene image synthesis method learned from rendering. BACKGROUND

[0002] Single image rain removal (SIRR) is a widely concerned task. Due to the powerful fitting ability of neural networks, SIRR methods based on deep learning are the current main trend, and the effect of such data-driven methods is greatly affected by the quality of the data set. The existing SIRR data set acquisition methods can be mainly divided into three types: real data set, artificially generated data set and synthetic data set. The real data set is obtained by shooting real-world scene images in rainy days. This method is often limited by weather conditions and it is difficult to obtain paired data. The artificially generated data set simulates the rainy scene in the real world by using a water sprayer, and obtains the image by shooting with a camera, thereby obtaining a pair of clean and rainy scene images. However, this method consumes a large amount of time and manpower. The image synthesis method can synthesize rainy scene images from clear background images with little or no human intervention, which can save time and manpower and provide the possibility for large-scale paired image data set acquisition.

[0003] The existing rain scene image synthesis method can be mainly divided into two types: rendering-based method and learning-based method. The rendering-based method models the raindrop dynamics model and the appearance model of the rain streaks. By inputting the scene depth map, light source attribute and some custom rain-related attribute parameters, the rain is rendered, and the rain layer and the background image are mixed to obtain the synthesized rain scene image in a way consistent with the physical principle, which can truly reflect the color appearance of the rain under a specific lighting environment. The learning-based method uses a real rain scene image data set to train a generative model, so that the model can capture the complex distribution of rain streaks in real images, thereby automatically and efficiently generating diversified and non-repetitive rain streaks without the need for subjective human intervention and experience parameter setting.

[0004] For example, the scheme disclosed in Chinese patent application CN114332460A, although these methods can be used for synthetic data sets, there are still some limitations. The input data of the rendering-based method is complex, involving some empirically set parameters, limiting the type of generated rain, and physical simulation and rendering also increase a large amount of time overhead. The learning-based method generally takes the rain layer as a single-channel grayscale layer, and often mixes the rain layer and the background image through linear superposition, ignoring the color appearance of the rain and the optical phenomena such as refraction and transmission of the rain to the environment. In addition, the existing synthetic data sets lack diversity in lighting environments, with images under daytime lighting environments being the main ones, and images under complex lighting environments such as night are less included, in addition, the resolution of these synthetic images is low.

[0005] As Figure 1 shown, the comparison of rain scene images randomly selected from 5 synthetic data sets BDD350, COCO, Rain100H, Rain100L and RainCityscapes and real rain scene images is shown, there is a big difference between the color appearance of the rain layer in the rain scene images in the existing data set and the real rain scene images. The SIRR model trained with these data sets is difficult to generalize to complex lighting environments such as night, and the performance is greatly affected. As Figure 2 shown, the rain removal results of 4 deep learning-based SIRR models on real night rain scene images are shown, from the results, it can be seen that several SIRR models are difficult to completely remove the colored rain streaks and restore the clear background. Therefore, at present, there is a lack of effective high-resolution rain scene image synthesis method under complex lighting environments to create a high-quality paired rain scene data set to train deep learning-based SIRR models, so that they can be generalized to complex lighting environments such as night. SUMMARY

[0006] The purpose of the present application is to provide a high-resolution rain scene image synthesis method and device learned from rendering.

[0007] The purpose of the present application can be achieved by the following technical solutions:

[0008] As a first aspect of the present application, a rain scene image synthesis method learned from rendering is provided, the method steps comprising:

[0009] a rendering stage, a high-resolution paired rain scene-background image data set is created, the data set contains a plurality of scenes and lighting conditions at different times including rain layer mask images, background images and rain scene images;

[0010] In the learning stage, a guided diffusion model is introduced into the implicit diffusion model, an encoder-decoder model is used to learn an implicit space equivalent to the image space in perception, and an encoder is used to compress the image in perception to obtain an implicit variable equivalent to the pixel in the image space in the implicit space; the forward process and the reverse process of the diffusion model are completed in the implicit space, wherein the reverse process is constrained by using cross-attention and series two conditional mechanisms; the output implicit variable is transformed to the image space by a decoder to generate a high-resolution rain scene image.

[0011] Further, in the rendering stage, the specific steps are as follows:

[0012] A scene model is created using a modeling tool, and a raindrop model is created using a particle simulator; the scene model and the raindrop model are combined to obtain a rain scene model; after the environment related parameters are set, a rendering engine is used to render a rain layer mask image, a background image and a rain scene image to form an image pair.

[0013] Further, the implicit diffusion model gradually adds noise to the data on the Markov chain at time step T, and converts between the original data distribution and the Gaussian distribution, and the specific steps include:

[0014] The implicit diffusion model first trains a perceptual compression autoencoder model, including an encoder ε and a decoder The encoder is used to compress the image from a high-dimensional pixel space to a low-dimensional implicit space;

[0015] In the forward process, noise is gradually added to the real data x0~q(x0):

[0016]

[0017] Wherein, is a hyperparameter, x1,...,x T is an implicit variable with the same dimension as the original data x0;

[0018] By using the reparameterization technique, any time step is sampled Wherein

[0019] By training the implicit diffusion model, the reverse process is learned, and the variable A learnable Gaussian conversion is applied Step-by-step denoising is performed, wherein a neural network is used to predict the statistical characteristics μ θ and ∑ θ of p θ , wherein μ θ represents the mean vector of the Gaussian distribution, ∑ θ represents the covariance matrix of the Gaussian distribution; μ θ is reparameterized into a denoising networkθ (x t , t);

[0020] Based on a given high-resolution rainy day image x high , the corresponding latent variable encoded by the encoder is z high , and the objective function of the latent diffusion model LDM is represented as:

[0021]

[0022] Where ∈ represents a variable sampled from a standard Gaussian distribution, and t is uniformly sampled from {1,..., T}.

[0023] Further, the guided diffusion model is trained on the latent variables of low-resolution images, and the latent variables of low-resolution rainy scene images are roughly predicted; in the high-resolution image generation process, the latent variables predicted by the guided diffusion model are used as the conditions of the reverse process of the diffusion model to guide image generation; the specific steps include:

[0024] Downsample the input background image to obtain a low-resolution image Combine the low-resolution image with the mask image to obtain a low-resolution mask image

[0025] Encode the low-resolution image and the low-resolution mask image into the latent space to obtain the latent variable And Input into the guided diffusion model to predict the latent variable of the low-resolution rainy scene image The objective function of the guided diffusion model GDM is represented as:

[0026]

[0027] Where z low is the latent variable of the real low-resolution rainy scene image, and the backbone of the guided diffusion model is implemented using UNet.

[0028] Further, the specific steps of using the conditional mechanism to constrain in the reverse process include:

[0029] Predict the latent variable of the low-resolution rainy scene image using the guided model;

[0030] Based on the cross-attention mechanism, enhance the bottom backbone of the diffusion model with the latent variable of the low-resolution rainy scene image;

[0031] Combine the high-resolution background image and the high-resolution rain layer mask to obtain a high-resolution mask image, and compress it through the encoder to obtain the latent variable, which is used as the condition of the reverse process through the concatenation mechanism.

[0032] Further, the specific steps of constraining based on the cross-attention mechanism include:

[0033] The reverse process of the implicit diffusion model is modeled as a conditional distribution p(z|y), and a conditional denoising network ∈ θ (z t , t, y) is used to constrain the reverse process;

[0034] The low-resolution rain scene latent variable predicted by the guided diffusion model is mapped to the intermediate layer of the diffusion model bottom UNet through the cross-attention mechanism as a condition, mapped to the intermediate layer of the diffusion model bottom UNet through the cross-attention layer, denoted as:

[0035]

[0036] wherein, represents the intermediate representation of the UNet implemented ∈ θ , and is a learnable projection matrix.

[0037] Further, the specific steps of constraining based on the series mechanism include:

[0038] The high-resolution background image and the rain layer mask are synthesized into a mask image is compressed into a latent variable by an encoder The latent variable is concatenated with the input hidden variable as the condition of the reverse process, specifically, the variable input to the reverse process

[0039] Based on the series and cross-attention conditional mechanism, the conditional implicit diffusion model LDM is learned through the following objective function:

[0040]

[0041] wherein, represents the variable input to the reverse process, represents the latent variable predicted by the guided model for the low-resolution rain scene image.

[0042] Further, after generating the rain scene image by the model, post-processing is performed on the generated rain scene image:

[0043] The generated rain scene image and the background image are mixed using the Lighten mixing mode, and the rain layer mask is used to keep the pixel color unchanged outside the rain range:

[0044]

[0045] Wherein, M represents a rain mask layer, and represents element-wise multiplication, represents generating a rain scene image, represents a high-resolution background image.

[0046] As a second aspect of the present application, a rain scene image synthesis device learning from rendering is provided, comprising a memory, a processor, and a program stored in the memory, and the processor implements the rain scene image synthesis method learning from rendering as described above when executing the program.

[0047] As a third aspect of the present application, a storage medium is provided, which stores a program, and the program implements the rain scene image synthesis method learning from rendering as described above when executed.

[0048] Compared with the prior art, the present application has the following beneficial effects:

[0049] The present application provides a practical high-resolution rain scene image synthesis method learning from rendering:

[0050] In the rendering stage, the present application uses a rendering-based method to render realistic high-resolution paired rain scene-background images, and creates a paired rain scene-background image dataset containing realistic paired rain scene images at different times under multiple scenes and lighting conditions.

[0051] In the learning stage, the dataset obtained by rendering is used to train a high-resolution rain scene image generation network HRIGNet, and a guided diffusion model is introduced in the implicit diffusion model to guide the synthesis of high-resolution images with low-resolution images; and cross-attention and concatenation conditional mechanisms are used to control the synthesis of rain scene images; the lighting information can be learned from the background image to generate high-resolution rain scene images under the same lighting conditions as the background image.

[0052] The method provided by the present application has the advantages of realism based on the rendering method and efficiency based on the learning method, and avoids the disadvantages of low efficiency based on the rendering method and poor realism based on the learning method. BRIEF DESCRIPTION OF DRAWINGS

[0053] Figure 1 The rain scene images of the existing synthesis dataset are compared, (a) BDD350; (b) COCO; (c) Rain100H; (d) Rain100L; (e) RainCityscapes; (f)-(h) real rain scene images;

[0054] Figure 2Comparison chart of rain removal results of existing deep learning-based SIRR model on real night rain scene image, (a) input image; (b) Restormer; (c) Maxim; (d) DGNL-Net; (e) PReNet;

[0055] Figure 3 Flowchart of the rain scene image synthesis method of the present application for learning from rendering;

[0056] Figure 4 Rain scene image of the HRI dataset of the present application, the first row is a rain scene image of a road scene, and the second row is a rain scene image of an urban street scene;

[0057] Figure 5 HRIGNet architecture of the present application;

[0058] Figure 6 Comparison chart of generated rain scene images and output images of the present application, (a) background image; (b) reference image; (c) generated rain scene image; (d) post-processed output image;

[0059] Figure 7 Comparison chart of output images of the present application and basic models, the first row is a clear background image and a generated image, and the second row is a real rain scene image and an output image, (a) clear background image; (b) ASSET generated image; (c) LDM generated image; (d) Dit generated image; (e) HRIGNet generated image; (f) real rain scene image; (g) ASSET output image; (h) LDM output image; (i) Dit output image; (j) HRIGNet output image. DETAILED DESCRIPTION

[0060] The present application will be described in detail below in conjunction with the accompanying drawings and specific embodiments. The present embodiment is implemented on the basis of the technical solution of the present application, and gives a detailed implementation and specific operation process, but the protection scope of the present application is not limited to the following examples.

[0061] Example 1

[0062] In the present application, in order to effectively synthesize a large number of high-resolution rainy scene images under complex lighting environments, a practical learning-from-rendering pipeline is proposed. The pipeline is divided into two stages: in the rendering stage, a high-resolution paired rainy scene-background image dataset is created, and in the learning stage, a rainy scene image generation network is trained using the dataset. The pipeline combines the realism of the rendering-based method and the efficiency of the learning-based method, and avoids the low efficiency of the rendering-based method and the poor realism of the learning-based method.

[0063] In order to train a high-quality rainy scene image generation network, a realistic paired rainy scene-background image dataset is needed. Considering that it is time-consuming and laborious to collect real rainy scene images taken by a camera, and it is difficult to obtain corresponding background images, the present application proposes a method for constructing a high-resolution rainy scene image (HRI, High-Resolution Rainy Image) dataset using a rendering-based method. The dataset contains realistic paired rainy scene-background images under different lighting environments in multiple scenes and different time periods.

[0064] In order to learn the lighting information from clear background images and generate high-resolution rainy scene images under corresponding lighting environments, a high-resolution rainy scene image generation network (HRIGNet, High-Resolution Rainy Image Generation Network) based on diffusion models (DMs, Diffusion Models) is proposed in the learning stage.

[0065] Using low-resolution images to guide the synthesis of high-resolution images can provide more guidance information for the synthesis of high-resolution images, and is expected to improve the quality of the synthesized images. The present application introduces a guiding diffusion model into the latent diffusion model (LDM, Latent Diffusion Model); in order to pair the generated rainy scene images with the input background images, effective constraints need to be applied to the image generation process, and cross-attention and concatenation conditional mechanisms are used to control the synthesis of rainy scene images, and the latent codes of the low-resolution rainy scene images and the high-resolution masked images predicted by the guiding diffusion model are used as conditions, respectively.

[0066] To effectively synthesize high-resolution rain scene images under complex lighting conditions and create a high-quality synthetic rain scene dataset, a practical pipeline for learning from rendering is proposed. As shown in Figure 3 FIG. 1 is an overview of the pipeline. Specifically, the pipeline combines rendering-based methods and learning-based methods and is divided into two stages, a rendering stage and a learning stage. In the rendering stage, rendering-based methods are used to render realistic high-resolution paired rain scene-background images and create a paired rain scene-background image dataset. In the learning stage, a rain scene image generation network is trained using the dataset obtained by rendering to efficiently generate high-resolution rain scene images. The pipeline has the advantages of realism of rendering-based methods and efficiency of learning-based methods, and avoids the disadvantages of low efficiency of rendering-based methods and poor realism of learning-based methods.

[0067] 1. Rendering stage

[0068] To train a high-quality rain scene image generation network, a realistic rain scene paired image dataset is needed, which contains pairs of background images, rain layer mask images, and rain scene images under different lighting conditions. Considering that it is time-consuming and laborious to collect real rain scene images taken by cameras, and it is difficult to obtain corresponding background images and rain layer mask images, a rendering-based method is used to construct the dataset.

[0069] Offline rendering technology based on ray tracing can simulate most of the natural phenomena of object surface interaction in the real physical world and render photo-realistic images. It is currently widely used in the fields of film, animation, design, etc. Blender is an open-source 3D content creation software that can easily and freely create true three-dimensional scene models, including various common light sources. The physics engine and particle system in Blender can be used to simulate rain effects, and a GPU ray tracing renderer Cycles based on physical algorithms is provided. Realistic rain scene images can be rendered.

[0070] The rendering stage is implemented based on Blender, and the rendering stage is shown in Figure 3 Specifically, a modeling tool is used to create a scene model, a particle simulator is used to create a raindrop model, the scene model and the raindrop model are combined to obtain a rain scene model, and after setting up the lighting and other environment-related parameters, a rendering engine is used to render a rain layer mask image, a background image, and a rain scene image, thereby forming an image pair.

[0071] 1.1 High-resolution rain scene image dataset (HRI)

[0072] In the rendering stage, we construct a realistic high-resolution rain scene paired image dataset HRI. The HRI dataset contains 1300 image pairs, as shown in Table 1, including two scenes, lane and city street, with image resolutions of 720x960 and 512x512, respectively. For the lane scene, there are 1000 image pairs, including images from 4 camera viewpoints, 50 time instants for each viewpoint, and 5 intensities and directions of rain for each time instant. For the city street scene, there are 300 image pairs, including images from 6 camera viewpoints, 25 time instants from day to night for each viewpoint, and 2 intensities and directions of rain for each time instant. Some of the images are shown in Figure 4

[0073] The dataset is divided into training and testing sets according to camera viewpoints. For the lane scene, the training set includes images from 3 camera viewpoints, and the testing set includes images from 1 camera viewpoint. For the city street scene, the training set includes images from 5 camera viewpoints, and the testing set includes images from 1 camera viewpoint. Therefore, the training set includes a total of 1000 image pairs, and the testing set includes a total of 300 image pairs.

[0074] Table 1 HRI dataset

[0075]

[0076] 2. High-resolution rain scene image generation network (HRIGNet)

[0077] This invention proposes a high-resolution rain scene image generation network (HRIGNet) that can synthesize high-resolution rain scene images from clear background images and corresponding rain layer mask images. Specifically, given an RGB clear scene image and a mask image representing the position of rain streaks in the scene image, this method can generate rain streaks at the corresponding positions of the mask, and the synthesized rain streaks have the corresponding lighting conditions and color appearance of the background image. In addition, this method can generate high-resolution images with a resolution of 512x512.

[0078] The architecture of HRIGNet is shown in Figure 5 According to LDM, in order to reduce the overhead of training the diffusion model on high-resolution images, a self-encoder model is used to learn a latent space that is perceptually equivalent to the image space. The encoder is used to perceptually compress the image, and the latent variable equivalent to the pixel in the image space is obtained in the latent space. Therefore, the forward and backward processes of the diffusion model can be completed in the latent space, and finally the output latent variable is transformed to the image space through the decoder.

[0079] ​To control the image synthesis process of the diffusion model, both serial and cross attention conditioning mechanisms are used to constrain the reverse process. A guiding diffusion model is first used to predict the latent code of the low-resolution rain scene image. Based on the cross attention mechanism, the latent code is used to enhance the bottom UNet backbone of the diffusion model. The training and sampling of the diffusion model are computationally less expensive at low resolution. Therefore, the low-resolution rain scene image is first predicted, and the predicted latent code of the low-resolution rain scene image is used to guide the high-resolution synthesis, providing more guidance information for high-resolution image synthesis and improving the quality of the synthesized image. To impose stronger constraints on the image synthesis process, the background image and the rain layer masked image are synthesized through the serial mechanism as the condition of the reverse process.

[0080] 2.1 Implicit diffusion model

[0081] The diffusion model is a probabilistic model that gradually adds noise to the data on a Markov chain at time step T, converting between the original data distribution and a Gaussian distribution. The forward process of the diffusion model gradually adds noise to the real data x0~q(x0) where is a hyperparameter, and x1,...,x T is the latent variable with the same dimension as the original data x0. By reparameterization trick, any time step can be sampled where By training the diffusion model, the reverse process, i.e., the inverse conversion of the forward process, can be learned to apply a learnable Gaussian transformation to the variable Step-by-step denoising, where a neural network is used to predict the statistical characteristics μ θ and∑ θ of p θ . The reparameterization of μ θ is a denoising network ∈ t (x high , t), and the corresponding objective function can be simplified as:

[0082]

[0083] where ∈ represents a variable sampled from a standard Gaussian distribution, and t is uniformly sampled from {1,...,T}.

[0084] For image synthesis, to reduce the computational cost of training the diffusion model on high-resolution images, the LDM first trains a perceptual compression autoencoder model VQGAN, including an encoder ε and a decoder​ An encoder is used to compress the image from a high-dimensional pixel space to a low-dimensional latent space, where high-frequency, hard-to-explain details are abstracted, making the training of the diffusion model more efficient in the low-dimensional latent space. Given a high-resolution RGB rainy day image x high The corresponding latent variable encoded by the encoder is z high The objective function of the LDM can be expressed as:

[0085]

[0086] 2.2 Guiding diffusion model

[0087] The training and sampling of the diffusion model are computationally less expensive under low-resolution images, and guiding the synthesis of high-resolution images with low-resolution images can provide more guidance information for the synthesis of high-resolution images, which is expected to improve the quality of the synthesized images.

[0088] This embodiment uses a guiding diffusion model to train on the latent variables of low-resolution images. Since the dimension of the latent variable is small, the guiding diffusion model can quickly perform the training and sampling process, roughly predict the latent variable of the low-resolution rainy scene image, and then use the predicted latent variable of the model as the condition of the reverse process of the diffusion model to guide the image generation in the subsequent high-resolution image generation process. Specifically, the input RGB background image is down-sampled to obtain a low-resolution image After combining with the mask image, a low-resolution mask image is obtained Encode the two images into the latent space to obtain the latent variable and Input into the guiding diffusion model to predict the latent variable of the low-resolution rainy scene image The objective function of the guiding diffusion model can be expressed as:

[0089]

[0090] where z low is the latent variable of the real low-resolution rainy scene image, and the backbone of the guiding diffusion model GDM is implemented using UNet.

[0091] 2.3 Conditional mechanism

[0092] Modeling the reverse process of the diffusion model as a conditional distribution p(z|y) can constrain the reverse process to achieve the purpose of controlling the image synthesis process, which can be modeled as a conditional denoising network ∈ θ (z t, t, y) implementation. In the context of image synthesis, LDM enables different modalities of input to be conditioned on the DMs through cross-attention mechanisms. In this method, the low-resolution rain scene latent variable predicted by the guiding transformer As a condition, we have which is mapped to the intermediate layer of UNet through a cross-attention layer, denoted as where

[0093]

[0094]

[0095]

[0096] Here, denotes the UNet implementation of θ The intermediate representation of is a learnable projection matrix.

[0097] In addition to using cross-attention conditioning mechanisms, a concatenation conditioning mechanism is also used to impose stronger constraints on the image generation process. The high-resolution background image and the rain layer mask are combined into a mask image which is compressed into a latent variable by the VQGAN encoder which is concatenated with the input hidden variable to serve as a condition for the inverse process. Specifically, the variable input to the inverse process is

[0098] Based on the concatenation and cross-attention conditioning mechanisms, the conditional implicit diffusion model LDM is learned through the following objective function:

[0099]

[0100] Therefore, the overall objective function of HRIGNet is

[0101] L HRIG = L GDM + L LDM (5)

[0102] 2.4 Post-processing

[0103] Through HRIGNet, given a clear background image and a rain layer mask, a corresponding rain scene image can be obtained. For simplicity, the impact of weather such as fog is not considered, and it is assumed that the area without rain in the image has the same color as the clear background image, and in general, brighter rain streaks will brighten the color at that location, and darker rain streaks will not make the color at that location darker than the background color, but will remain the same color as originally.

[0104] Therefore, after obtaining the rain scene image generated by the model, further post-processing is performed on it, the rain scene image and the background image are mixed by using the mixing mode of Lighten, and the pixel color outside the rain range is kept unchanged by using the rain layer mask, that is

[0105]

[0106] wherein M represents a rain mask layer, and represents element-wise multiplication.

[0107] As Figure 6 shown, a comparison between the generated rain scene image and the post-processed output image is shown, and it can be seen that the post-processed image is better in visual effect.

[0108] 3. Effect verification

[0109] In the experimental part, the HRIGNet model is trained based on the HRI dataset, and is quantified and visually compared with multiple image generation baseline models to verify the ability of HRIGNet in high-resolution rain scene image synthesis. At the same time, ablation experiments are also conducted on the guiding model and diffusion backbone of HRIGNet to verify the effect of HRIGNet.

[0110] 3.1 Training details

[0111] First, use low-resolution images with a size of 256x256 to train the L GDM Pre-train a GuidingDiffusion model, then fix the weights of Guiding Diffusion, use high-resolution images with a size of 512x512, and train the L LDM HRIGNet based on formula 4.

[0112] During the training of Guiding Diffusion and HRIG Net, the AdamW optimizer is used. When training GuidingDiffusion, the First Stage Model and the Cond Stage Model use the same VQGAN model, that is Figure 5 E2 in the above formula, and the model weight uses the vq-f8-256 pre-trained by LDM. The initial learning rate of Diffusion is 2x10 -6 , the batchsize size is 4, the image size of UNet backbone is 32x32, and the model channels are 224.

[0113] When training HRIG Net, the First Stage Model and the Cond Stage Model use the same VQGAN model, i.e. Figure 5 E1 in the above formula, the model weight uses vq-f4 pre-trained by LDM. The initial learning rate of Diffusion is 2x10 -6 , the batch size is 1, the image size of UNet backbone is 128x128, and the model channel is 224.

[0114] 3.2 Comparison with Baselines model

[0115] In order to verify the ability of HRIG Net in high-resolution rain scene image synthesis, the method of the present application and several image generation models baselines are compared: ASSET, LDM, DiT, and the evaluation indexes are FID, LPIPS, SSIM, and PSNR. The specific model settings are in the additional materials. As shown in the experimental results of Table 2, the model proposed in the present application achieves the best results in these indexes. The rain scene image synthesis results of several methods are shown in Figure 7 From the figure, it can be seen that the model of the present application can well capture the light and color in the background environment and map it to the generated rain layer, so that it has a realistic and corresponding color appearance to the background image.

[0116] Table 2 Comparison of baselines and HRIGNet in 512x512 resolution

[0117] Method Resolution FID↓ LPIPS↓ SSIM↑ PSNR↑ ASSET 512×512 330.918 0.255 0.792 23.562 LDM 512×512 166.874 0.243 0.784 22.932 DiT 512×512 279.663 0.342 0.719 19.810 HRIGNet 512×512 130.186 0.203 0.819 24.030

[0118] 3.3 Ablation experiment

[0119] Through the ablation experiment, the effect of using Diffusion as the guiding model is evaluated. The performance of using the Transformer and Diffusion model as the guiding model of HRIGNet is compared. As shown in Table 3, in the FID, LPIPS and SSIM indexes, the model using Guiding Diffusion is better than the model using Guiding Transformer. The diffusion model is fitted with a simple Gaussian distribution at each time step, which is easy to converge, so it is easier to achieve good results. This can reasonably explain why the image synthesis result using the guiding diffusion model is better at low resolution.

[0120] Table 3. Comparison of HRIG Net using different guiding models at 512x512 resolution

[0121] Guiding Model Image Resolution FID↓ LPIPS↓ SSIM↑ PSNR↑ Transformer 512×512 133.738 0.204 0.818 24.056 Diffusion 512×512 130.186 0.203 0.819 24.030

[0122] To explore the impact of using different backbones in the Diffusion model on HRIG Net, an ablation experiment was also conducted on the backbone, using UNet and Transformer respectively. The results are shown in Table 4. The HRIG Net using UNet performed better than the one using Transformer. According to the scalable nature of the Transformer in DiT, the Transformer backbone model was also used. However, due to this characteristic, the model using the Transformer backbone performed poorly in the case of insufficient parameters. In addition, the UNet backbone used in the model of the present application converges faster than the model based on the Transformer backbone.

[0123] Table 4. Comparison of HRIG Net using different backbones at 512x512 resolution

[0124] Backbone Image Resolution FID↓ LPIPS↓ SSIM↑ PSNR↑ Transformer 512×512 217.182 0.263 0.780 22.469 UNet 512×512 130.186 0.203 0.819 24.030

[0125] Embodiment 2

[0126] As a second aspect of the present application, the present application also provides an electronic device comprising: one or more processors; memory for storing one or more programs; when the one or more programs are executed by the one or more processors, the one or more processors implement the rain scene image synthesis method learned from rendering as described above. In addition to the above-mentioned processors, memories and interfaces, any data processing-capable device in which the device in the embodiment is generally composed of other hardware according to the actual function of the data processing-capable device, which will not be described here.

[0127] Embodiment 3

[0128] As a third aspect of the present application, the present application also provides a computer readable storage medium having stored thereon computer instructions which, when executed by a processor, implement the method for synthesizing rain scene image from rendering as described above. The computer readable storage medium can be an internal storage unit of any of the devices having data processing capability as described in any of the preceding embodiments, such as a hard disk or a memory. The computer readable storage medium can also be an external storage device, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc. Further, the computer readable storage medium can include both an internal storage unit of any of the devices having data processing capability and an external storage device. The computer readable storage medium is used to store the computer program and other programs and data required by the devices having data processing capability, and can also be used to temporarily store data that has been output or will be output.

[0129] The preferred embodiments of the present application have been described in detail. It should be understood that modifications and variations can be made by those of ordinary skill in the art without departing from the spirit and scope of the present application. Therefore, it is intended that the scope of the present application be governed by the scope of the claims and their equivalents, based on the disclosure provided herein.

Claims

1. A rain scene image synthesis method of learning from renderings, characterized by, The method steps include: a rendering stage, creating a high-resolution paired rain scene-background image dataset, the dataset containing multiple image pairs including rain layer mask images, background images and rain scene images at different times under different scenes and lighting conditions; a learning stage, introducing a guided diffusion model into an implicit diffusion model, using an autoencoder model to learn an implicit space equivalent to the image space in perception, using an encoder to compress the image in perception, obtaining an implicit variable equivalent to the pixel in the image space in the implicit space; completing the forward process and the reverse process of the diffusion model in the implicit space, wherein the reverse process is constrained using cross-attention and concatenation two conditional mechanisms; transforming the output implicit variable to the image space through the decoder to generate a high-resolution rain scene image.

2. The rain scene image synthesis method learning from rendering of claim 1, wherein, The rendering stage has the following specific steps: Using a modeling tool to create a scene model and a particle simulator to create a raindrop model; merging the scene model and the raindrop model to obtain a rain scene model; after setting the environment-related parameters, using a rendering engine to render a rain layer mask image, a background image and a rain scene image to form an image pair.

3. The rain scene image synthesis method learning from rendering of claim 1, wherein, The implicit diffusion model gradually adds noise to the data on the Markov chain at time step T, converting between the original data distribution and the Gaussian distribution, and the specific steps include: The implicit diffusion model first trains a perceptual compression auto-encoder model, including an encoder and a decoder , compresses the image from a high-dimensional pixel space to a low-dimensional hidden space with the encoder; In the forward process for real data Add noise step by step: wherein, is a hyperparameter, is a latent variable of the same dimension as the real data is a latent variable of the same dimension as the real data By reparameterization trick, sample at any time step wherein ; By training an implicit diffusion model, learn the reverse process, on the variables Apply a learnable Gaussian transform Perform step-wise denoising, where a neural network is used to predict statistical features and where denotes the mean vector of the Gaussian distribution, denotes the covariance matrix of the Gaussian distribution; transform into a denoising network ; Based on a given high-resolution rainy day image The corresponding latent variable encoded by the encoder is The objective function of the implicit diffusion model LDM is represented as: wherein, denotes a variable sampled from a standard Gaussian distribution, is uniformly sampled from .

4. The rain scene image synthesis method learning from rendering of claim 3, wherein, The guided diffusion model is trained on the implicit variable of the low-resolution image, and the implicit variable of the low-resolution rain scene image is roughly predicted; in the high-resolution image generation process, the implicit variable predicted by the guided diffusion model is used as the condition of the reverse process of the diffusion model to guide image generation; the specific steps include: down-sampling the input background image to obtain a low resolution image combining the low resolution image with the mask image to obtain a low resolution mask image encoding the low resolution image and the low resolution mask image into a latent space to obtain latent variables and and inputting the latent variables to a guided diffusion model to predict a low resolution rain scene image , the objective function of the guided diffusion model GDM is represented as: wherein, is the latent variable of the real low-resolution rain scene image, and the backbone of the guided diffusion model is implemented using UNet.

5. The rain scene image synthesis method learning from rendering of claim 4, wherein, The specific steps of using the conditional mechanism for constraint in the reverse process include: Using the guided model to predict the implicit variable of the low-resolution rain scene image; Based on the cross-attention mechanism, the implicit variable of the low-resolution rain scene image is used to enhance the bottom backbone of the diffusion model; The high-resolution background image and the high-resolution rain layer mask are combined into a high-resolution mask image, and the implicit variable is obtained by compression through the encoder, and the concatenation mechanism is used as the condition of the reverse process.

6. The rain scene image synthesis method learning from rendering of claim 5, wherein, The specific steps of constraint based on the cross-attention mechanism include: modeling the inverse process of the implicit diffusion model as a conditional distribution using a conditional denoising network constraining the inverse process; cross attention mechanism As a condition, we have mapped to the middle layer of the diffusion model base UNet through the cross attention layer, denoted as: wherein, , , ; represents an intermediate representation of a UNet implementation of , is a learnable projection matrix, is a query vector and a key vector have the same dimension.

7. The rain scene image synthesis method learning from rendering of claim 5, wherein, The specific steps of constraint based on the concatenation mechanism include: Combining a high resolution background image and a rain layer mask into a mask image , by an encoder compressing into latent variables concatenating the latent variables with the input hidden variables as a condition for the reverse process, in particular, the variable input for the reverse process is ; Based on the concatenation and cross-attention conditional mechanism, the conditional implicit diffusion model LDM is learned through the following target function: wherein denotes the variable representing the input of the inverse process, denotes the latent variable representing the low-resolution rain scene image predicted by the guidance model.

8. The rain scene image synthesis method learning from rendering of claim 1, wherein, After generating the rain scene image through the model, the generated rain scene image is post-processed: The generated rain scene image and the background image are mixed using the mixing mode of Lighten, and the rain layer mask is used to keep the pixel color outside the rain range unchanged: wherein, represents a rain mask layer, represents element-wise multiplication, represents generating a rain scene image, represents a high resolution background image.

9. A rain scene image synthesis apparatus that learns from rendering, comprising a memory, a processor, and a program stored in the memory, characterized by, The processor implements the rain scene image synthesis method learned from rendering when the program is executed.

10. A storage medium having stored thereon a program, characterized by The program is executed to implement the rain scene image synthesis method learned from rendering.

Images