A method for realistic and controllable generation of rainy day images based on attribute and style domain transfer.

Abstract

This invention discloses a method for realistically controllable generation of rainy day images based on dual domain transfer of attributes and style. The method includes multiple network models. The first network decomposes measured and simulated rain images into background and rain layers, respectively. The second and third networks take simulated and measured rain layer images as outputs, regressing the corresponding physical attributes, and are iteratively trained alternately through semi-supervised weight momentum updates to achieve domain alignment in the physical attribute space. The fourth network takes the measured rain layer as input, explicitly models and regresses the observed style in the latent space, and achieves domain alignment between simulated and measured styles through resampling. The simulated and measured background-attribute-style images are packaged and input into a fifth network to reconstruct the measured and simulated rain layers, respectively. Pixel consistency loss and attribute consistency loss are used for constraints to obtain rainy day images that combine realism and diversity. This method generates degraded images that combine realism and diversity.

Description

Technical Field

[0001] This invention belongs to the field of image processing and computer vision technology, and more specifically, relates to a method for realistically controllable generation of rainy day images based on dual domain transfer of attributes and style. Background Technology

[0002] Rainy weather, a common and severe weather phenomenon, significantly degrades image contrast and scene details, severely limiting the perception capabilities of unmanned systems. Rainy day image restoration, by recovering a clear background from degraded images, is an effective means to improve the system's perception capabilities. Unlike high-level perception tasks, it is difficult to obtain clear image labels for degraded rainy day images in real-world scenarios. Furthermore, the large distribution difference between simulation data and the actual measurement domain makes it difficult for algorithm models trained on simulation data to generalize in real-world rainy day scenarios. Unlike existing methods that address the rain removal problem from the perspective of models or loss functions, this patent aims to solve the problem of poor generalization performance of supervised methods in real-world scenarios from the perspective of data generation. Existing rainy day image dataset generation methods mainly fall into three categories: generation methods based on simplified physical mechanism models, generation methods based on generative network models, and generation methods based on video processing.

[0003] To bridge the gap between simulated and measured data, some research has begun to use generative models to generate rainy images. For example, Ye et al.'s paper "Closing the loop: Joint rain generation and removal via disentangled image translation," published in the 2021 IEEE Conference on Computer Vision and Pattern Recognition, was the first to introduce a GAN-based generative model into rainy image generation, utilizing both simulated and measured data. Essentially, their method decomposes the rain image into a background and a rain layer, then uses a recurrent generative adversarial network (GAN) to facilitate the transfer between simulated and measured rain layers. The simulated rain layer serves as input to the generator, producing a rain layer with the style of measured rain, and a generative adversarial loss function is used to constrain the generation of realistic rain images. While this type of method can generate rain images with good appearance realism, the uncontrollable generation process of the GAN results in a lack of diversity in the generated degradation patterns. This can increase the bias in the distribution of different types of training data, causing the rain removal model to overfit only to a certain type of degradation, thus limiting the generalization of the rain removal method in real-world testing.

[0004] Another type of rainy day image dataset generation method is based on video processing. This type of method processes real rainy day videos using temporal information and uses the processed video results as labels to construct a large dataset of real rainy-clear images, which serves as the training set for single-image rain deraining methods. For example, the SPA dataset proposed by Wang et al. in their 2019 paper "Spatialattentive single-image deraining with a high-quality real rain dataset" published in the IEEE Conference on Computer Vision and Pattern Recognition, obtains corresponding rainy day videos by continuously shooting specific static scenes. Since the raindrops in the video images are in constant motion while the background is stationary, simple filtering can remove the moving rain degradation in the video. Further frame segmentation is then used to obtain corresponding rainy-clear image data pairs, thus forming a large, experimentally measured rainy day dataset. Although this type of method obtains rainy day images containing real rain degradation, its corresponding "clear image labels" have limitations. Essentially, it is the restoration result of the video deraining algorithm, and the quality of the labels depends on the performance of the video deraining algorithm.

[0005] In general, current rain map simulation models, whether based on simplified physical degradation mechanisms, video-based background generation models, or generative network-based models, each have their own advantages and disadvantages. Some focus only on the diversity of generated results, lacking modeling of the true distribution of measured data; others focus only on the appearance of measured rain maps, lacking design for the diverse variations in measured data. Existing methods cannot simultaneously achieve both diversity and realism in the generated degradation maps. Summary of the Invention

[0006] To address the shortcomings of related technologies, the present invention aims to provide a realistic and controllable method for generating rainy day images based on dual domain transfer of attributes and style, which aims to solve the problem of degenerate generated images that cannot take into account both diversity and realism in existing methods.

[0007] To achieve the above objectives, this invention provides a method for realistically controllable generation of rainy day images based on dual domain transfer of attributes and style, comprising:

[0008] S1. Train the first network with simulated rain images and corresponding clear backgrounds. Input the simulated rain images and measured rain images into the trained first network, and output the simulated rain layer image, the simulated clear background image, the measured rain layer image, and the measured clear background image.

[0009] S2. The second network is trained using the simulated rain layer map and its corresponding physical properties, so that the second network can adaptively regress its corresponding first physical property based on the input simulated rain layer map.

[0010] S3. Input the measured rain layer image, use the second network as the initial weight of the third network, and constrain the third network to train based on the semi-supervised momentum weight update method. After the third network adaptively regresses, the second physical attribute of the rain strip in the measured rain layer image is obtained. The data distribution of the first physical attribute and the second physical attribute is consistent.

[0011] S4. Obtain the mean and variance data of the style from the measured rain layer map, and train the fourth network by constraining it with a prior Gaussian normal distribution. For the style distribution obtained by regression based on the measured rain layer map, obtain two sampling points by resampling to represent the measured style of the measured rain layer map and the style of the simulated rain layer, respectively, so as to achieve the alignment of the apparent styles of the two.

[0012] S5. The simulated clear background image and the measured clear background image, the first physical attribute and the second physical attribute of the rain layer, the measured style of the measured rain layer image and the simulated style of the simulated rain layer image are packaged separately. The packaged data is input into the fifth network. The fifth network is trained by constraining the pixel consistency and attribute consistency loss to obtain a controllable mapping from background-attribute-style to the measured rainy day image space, thereby obtaining a rainy day image that combines realism and diversity.

[0013] Optionally, S1 specifically includes:

[0014] S11. The simulated rain map and the corresponding clear image labels constitute a simulation dataset for training the first network. The loss function is composed of the mean squared error.

[0015]

[0016] Among them, B Syn To simulate a clear background image, B GT Label the clear image;

[0017] S12. Input the simulated rain map into the trained first network to obtain the simulated rain layer map R. Syn And a clear simulation background image B Syn ;

[0018] S13. Input the measured rain map into the trained first network, and use the network weights trained based on the simulation dataset to obtain the measured rain layer map R. Real Clear background image B (based on actual measurement) Real The simulated rain map and the measured rain map share network weights in the first network.

[0019] Optionally, S2 specifically includes:

[0020] Using a simulated rain layer image as input and the physical properties of rain streaks in the simulated rain layer image as attribute labels, the corresponding parameters are regressed through multi-channel global pooling. A second network is then trained using mean-variance constraints, enabling it to adaptively regress the corresponding first physical property based on the input simulated rain layer image. The loss function is as follows:

[0021]

[0022] Among them, For network transformation, W attr R represents the parameters of the second network. syn For the data used to simulate rain layer maps, A GT These are the attribute labels corresponding to the simulated rain layer diagram; the physical properties of the rain streaks include length, density, angle, and brightness.

[0023] Optionally, S3 specifically includes:

[0024] S31. Using the weights trained based on the paired simulated rain layer map-first physical attribute dataset as initial weights, a large supervised loss function constructed with the first physical attribute labels and a self-reconstruction loss function of the measured rain map are used to constrain the training of the third network. The two loss functions are used to update the momentum of the weights in an alternating iterative manner. The momentum update formula is as follows:

[0025]

[0026]

[0027] Where ξ represents the simulated attribute regressor network parameters, θ represents the measured attribute regressor network parameters, v1 and v2 are the initial update iteration rates, t represents the current update step, and t-1 represents the update step in the previous stage. The attribute regression loss function in S21 This represents the self-reconstruction loss of the measured rain map;

[0028] S32, Self-reconstruction loss of the measured rain map for:

[0029]

[0030] Among them, R Real B is a measured rain layer diagram. Real To obtain a clear background image for actual testing, W attri W sty W MFT These are the measured network weights for the third, fourth, and fifth networks, respectively.Real This is a measured rainfall map.

[0031] Optionally, S4 specifically includes:

[0032] S41. Obtain the mean and variance data of the style in the measured rain layer map, and analyze the apparent style S of the measured rain layer map in the feature latent space. Real Explicit modeling is performed to obtain the fourth network;

[0033] S42. Apply a Gaussian prior distribution to the fourth network and use KL divergence to constrain the training of the fourth network, limiting the style of the measured rain layer map to a range close to the prior distribution p(z). The specific loss function is:

[0034]

[0035] in, Where 's' represents the style of the rain layer, and I t Let N be the identity matrix, and M be the number of samples. z Dimensions representing rain layer style;

[0036] S43. By repeatedly sampling the measured styles of different measured rain layer maps, multiple sets of measured styles with the same distribution are obtained. The specific sampling formula is as follows:

[0037] S Real =μ+σ*∈

[0038] ∈~N(0,I t )

[0039] Where μ and σ are the mean and variance of the style regressor output, respectively, and I t It is the identity matrix;

[0040] S44. The measured styles obtained from resampling are labeled as Syn.Sty and Real.Sty respectively, and used as the simulation style of the aligned simulated rain layer map and the measured style of the measured rain layer map.

[0041] Optionally, S5 specifically includes:

[0042] S51. Package the simulated clear background image and the measured clear background image, the first physical property and the second physical property of the rain layer, the measured style of the measured rain layer image and the simulation style of the simulated rain layer image respectively to obtain the simulation decomposition variable Param.Syn and the measured decomposition variable Param.Real.

[0043] S52. Based on the pixel consistency loss and generative adversarial loss of the measured rain image, the generator is jointly constrained to generate results with the measured style. The generative adversarial loss is as follows:

[0044]

[0045] Among them, I gen Based on the physical properties of simulated rain layer images, clear backgrounds, and measured styles, I Rec This is the reconstruction result of the measured rainfall map;

[0046] S53. Based on the physical property consistency loss of the simulated rain layer diagram, the result of the constraint generator has the expected properties, realizing the controllability of the generated result. The specific loss function is as follows:

[0047]

[0048]

[0049] Among them, R Real B is a measured rain layer diagram. Real To obtain a clear background image for actual testing, I Real For measured rainfall maps, I Syn To simulate rain patterns, I Gen W is a rain map generated based on a clear simulated background image and physical properties. attri W sty W MFT These are the measured network weights for the third, fourth, and fifth networks, respectively.

[0050] S54. The trained fifth network obtains features F from the simulated clear background image and the measured clear background image. b Constructing a mapping relationship between physical properties and feature control parameters. (attri,style)→(α,γ,β), where the adjustment parameter α is obtained through the attribute condition attri mapping, and the transformation parameter (γ,β) is obtained through the style condition style mapping;

[0051] S55. Obtain the control parameters and transformation parameters, control the features of the background image, transform the controlled features, and coordinate the attributes and styles to obtain rainy day images that combine realism and diversity.

[0052] Secondly, the present invention provides a realistic and controllable rainy day image generation system based on attribute and style dual domain transfer, realizing the realistic and controllable rainy day image generation method based on attribute and style dual domain transfer as described in any of the first aspects, including:

[0053] The rain layer decomposition module is used to train the first network with simulated rain images and corresponding clear backgrounds. The simulated rain images and measured rain images are input into the trained first network, and the output is a simulated rain layer image, a simulated clear background image, a measured rain layer image, and a measured clear background image.

[0054] The simulation attribute regression module is used to train the second network with the simulation rain layer map and the corresponding physical attributes, so that the second network can adaptively regress its corresponding first physical attribute based on the input simulation rain layer map.

[0055] The measured attribute regression module is used to input the measured rain layer map, use the second network as the initial weight of the third network, and constrain the third network to train based on the semi-supervised momentum weight update method. The third network adaptively regresses to obtain the second physical attribute of the rain strips in the measured rain layer map. The data distribution of the first physical attribute and the second physical attribute is consistent.

[0056] The style autoencoder module is used to obtain the mean and variance data of the style from the measured rain layer map, and train the fourth network constrained by the prior Gaussian normal distribution. The style distribution obtained by regression based on the measured rain layer map is resampled to obtain two sampling points that represent the measured style of the measured rain layer map and the style of the simulated rain layer, respectively, so as to achieve the alignment of the apparent styles of the two.

[0057] The controllable generation module is used to package the simulated clear background image and the measured clear background image, the first physical attribute and the second physical attribute of the rain layer, the measured style of the measured rain layer image and the simulated style of the simulated rain layer image, respectively. The packaged data is input into the fifth network, and the fifth network is trained by constraining the pixel consistency and attribute consistency loss to obtain a controllable mapping from background-attribute-style to the measured rainy day image space, thereby obtaining rainy day images that combine realism and diversity.

[0058] In summary, the technical solutions conceived in this invention have the following advantages compared with the prior art:

[0059] 1. The present invention provides a realistic and controllable method for generating rainy day images based on attribute and style dual domain transfer. The generated degraded data appears realistic. Compared with existing rain map data based on simplified atmospheric physics model simulation, the generated results are closer to the appearance of rainy day degradation captured in the actual field.

[0060] 2. The present invention provides a method for realistically controllable generation of rainy day images based on dual domain transfer of attributes and style. The generated degraded data has diverse forms. The physical attribute parameters of the degradation are explicitly modeled in the attribute space. By adjusting the parameters, different forms of rainy day degraded images can be generated in a fine-grained and controllable manner, which significantly enhances the diversity of the generated results.

[0061] 3. This invention provides a method for realistically controllable generation of rainy images based on dual domain transfer of attributes and style. The generated data can effectively promote the generalization of supervised methods in real-world scenarios. Existing supervised image deraining methods can be effectively improved in real-world scenarios by training and augmenting on the realistic, diverse, clear-degraded pairwise dataset generated in this application, greatly alleviating the predicament of poor restoration performance in real-world scenarios. Attached Figure Description

[0062] Figure 1 This is a flowchart illustrating a method for realistically controllable generation of rainy day images based on dual domain transfer of attributes and style, provided in an embodiment of the present invention.

[0063] Figure 2 This is a network diagram illustrating a method for realistically controllable generation of rainy day images based on dual domain transfer of attributes and style, provided in an embodiment of the present invention.

[0064] Figure 3 This is a network diagram of the controllable generator provided in an embodiment of the present invention;

[0065] Figure 4 This is an example of a rain map generated by a rain image generation method based on attribute and style dual domain transfer provided in this embodiment of the invention.

[0066] Figure 5 This is a schematic diagram comparing the rain image generation method based on attribute and style dual domain transfer provided by an embodiment of the present invention with the removal effect on the measured rain image before and after augmentation of the training set of existing rain removal methods. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. Furthermore, the technical features involved in the various embodiments of this invention described below can be combined with each other as long as they do not conflict with each other.

[0068] The following description, in conjunction with a preferred embodiment, illustrates the content involved in the above embodiments.

[0069] Example 1

[0070] like Figure 1 As shown, a method for realistically controllable generation of rainy day images based on dual domain transfer of attributes and style includes:

[0071] S1. Train the first network with simulated rain images and corresponding clear backgrounds. Input the simulated rain images and measured rain images into the trained first network, and output the simulated rain layer image, the simulated clear background image, the measured rain layer image, and the measured clear background image.

[0072] S2. The second network is trained with the simulated rain layer map and its corresponding physical properties, so that the second network can adaptively regress its corresponding first physical property based on the input simulated rain layer map.

[0073] S3. Input the measured rain layer image, use the second network as the initial weight of the third network, and constrain the third network to train based on the semi-supervised momentum weight update method. After the third network adaptively regresses, the second physical attribute of the rain strip in the measured rain layer image is obtained. The data distribution of the first physical attribute is consistent with that of the second physical attribute.

[0074] S4. Obtain the mean and variance data of the style from the measured rain layer map, and train the fourth network by constraining it with a prior Gaussian normal distribution. For the style distribution obtained by regression based on the measured rain layer map, obtain two sampling points by resampling to represent the measured style of the measured rain layer map and the style of the simulated rain layer, respectively, so as to align the apparent styles of the two.

[0075] S5. The simulated clear background image and the measured clear background image, the first physical attribute and the second physical attribute of the rain layer, the measured style of the measured rain layer image and the simulated style of the simulated rain layer image are packaged separately. The packaged data is input into the fifth network. The fifth network is trained by constraining the pixel consistency and attribute consistency loss to obtain a controllable mapping from background-attribute-style to the measured rainy day image space, thereby obtaining rainy day images that are both realistic and diverse.

[0076] refer to Figure 2 Since simulated rain images are degraded simulations based on simplified atmospheric imaging mechanisms on clear background images, they possess corresponding clear image labels. The clear background of the simulated rain image is used as a label constraint for training the first network, which is then used to decompose the rain layer. For measured rain images lacking clear background labels, the rain layer decomposition network trained on simulated data is used to directly regress the corresponding rain layer and background layer, thus obtaining the simulated rain layer image, the simulated clear background image, the measured rain layer image, and the measured clear background image, respectively.

[0077] The physical properties in the simulated rain layer image are defined, including the length, angle, density, position, and brightness of rain streaks during rain layer image degradation. The physical properties corresponding to the simulated rain layer image are used as labels to constrain the training of a second network, constructing a second network for acquiring the physical properties of the simulated rain layer image. The physical properties of the simulated rain layer image are set as the first physical properties.

[0078] A third network is constructed to obtain the physical properties of the measured rain layer map. The properties of the third network should be the same as those of the second network, and both can be implemented using an attribute regression encoder. However, since the measured rain layer map does not have corresponding degenerate attribute labels, its training method differs from that of the simulated attribute regression module. This application proposes a semi-supervised momentum weight update method to regress the measured rain layer properties. The two encoders are trained alternately and iteratively through semi-supervised weight momentum updates, achieving domain alignment between the simulated and measured rain layer maps in the physical property space.

[0079] Using measured rain layers as input, the mean and variance data of styles are obtained from the measured rain layer images. The apparent style of the measured rain layers is explicitly modeled in the latent space of the features. A prior hypothesis is introduced: the measured rain layers follow a standard normal distribution in the style representation space. This hypothesis constrains the training of a fourth network, which encodes styles. Simultaneously, distribution resampling is used to resample the measured style distribution of the regressed measured rain layer images, making it the simulated style of the simulated rain layer images, thus achieving alignment between measured and simulated styles. Here, style represents the style of the apparent morphology. For example, for rain bars with the same attributes, the simulated style might only be a bright straight line, but the shape and brightness in the measured style are much more complex.

[0080] The simulated and measured background-attribute-style data are packaged and input into the fifth network, which is a controllable generator. The measured and simulated rain layers are reconstructed respectively. Based on the alignment of attributes and styles, as well as the weight sharing of the controllable generator network model, the results generated by the simulated branch and the measured branch naturally have distribution consistency. Furthermore, pixel consistency loss and attribute consistency loss are used to constrain the two respectively, and finally, a rainy day image generation effect with both realism and diversity is achieved.

[0081] This invention decomposes simulated and measured rain images into three parts: physically meaningful background, rain attributes, and rain appearance style. It employs a semi-supervised weight update strategy and an explicit distribution modeling method to collaboratively align the simulated and measured attributes and styles. These variables are then input into the subsequent generator, and image-level and attribute-level consistency loss is used to achieve realistic and controllable rain image generation. The proposed rain image generation method based on attribute and style dual-domain transfer produces results that closely resemble real-world rain degradation images in appearance, solving the problem of existing methods failing to balance diversity and realism in degradation generation. Training and augmentation on the generated realistic and diverse clear-degraded pair dataset effectively improves generalization ability in real-world scenarios, significantly alleviating the predicament of poor real-world scene restoration performance.

[0082] Based on the above embodiments, optionally, S1 specifically includes:

[0083] S11. The simulated rain map and the corresponding clear image labels constitute a simulation dataset for training the first network. The loss function is composed of the mean squared error.

[0084]

[0085] The simulated rain map, based on a simplified atmospheric imaging mechanism, simulates degradation on a clear background image and possesses corresponding clear image labels; B Syn To simulate a clear background image, B GT Label the clear image;

[0086] S12. Input the simulated rain map into the trained first network to obtain the simulated rain layer map R. Syn And a clear simulation background image B Syn ;

[0087] S13. Input the measured rain map into the trained first network, and use the network weights trained based on the simulation dataset to obtain the measured rain layer map R. Real Clear background image B (based on actual measurement) Real The simulated rain map and the measured rain map share network weights in the first network.

[0088] The first network consists of nine residual modules and is trained using pairs of simulated rain images and clear backgrounds; this first network is the rain layer decomposition network. Simulated rain image data has corresponding clear image labels, so the rain layer decomposition network is trained directly using these data pairs. For measured rain images lacking clear background labels, the rain layer decomposition network trained on simulated rain image data directly regresses the corresponding rain layer and background, thus obtaining the simulated rain layer image R. Syn And a clear simulation background image B Syn And measured rain layer map R Real Clear background image B (based on actual measurement) Real Although the difference between the simulation and the measured domain may lead to residual degradation in the estimated measured background, the purpose of this decomposition is to better model the style distribution of the rain layer and its corresponding degradation properties from the incomplete measured rain layer map. Therefore, the impact of the degradation caused by the residual background is relatively small.

[0089] Based on the above embodiments, optionally, S2 specifically includes:

[0090] Using a simulated rain layer image as input and the physical properties of rain streaks in the simulated rain layer image as attribute labels, the corresponding parameters are regressed through multi-channel global pooling. A second network is then trained using mean-variance constraints, enabling it to adaptively regress the corresponding first physical property based on the input simulated rain layer image. The loss function is as follows:

[0091]

[0092] Among them, For network transformation, W attr R represents the parameters of the second network. syn For the data used to simulate rain layer maps, A GT The attribute labels are for the simulated rain layer map; the physical properties of the rain streaks include length, density, angle, and brightness.

[0093] When simulating rain maps using atmospheric physics simulation models, a one-to-one correspondence between physical property parameters and degraded rain layers can be achieved by explicitly adjusting and recording the corresponding simulation parameters. In actual training, the second network is a simulation attribute regression network, which directly uses the simulated rain layer map as input. The network consists of UNet and global pooling layers. Through multi-channel global pooling, the corresponding parameters are regressed, and the network is trained using mean-variance constraints. The attributes of the constructed simulation attribute regression network include the length, angle, density, position, and brightness of rain streaks in the rain map degradation.

[0094] Based on the above embodiments, optionally, S3 specifically includes:

[0095] S31. Using the weights trained based on the paired simulated rain layer map-first physical attribute dataset as initial weights, a third network is trained by constraining a large supervised loss function constructed with the first physical attribute labels and its own image reconstruction self-supervised loss function. The two loss functions are used to update the momentum of the weights in an alternating iterative manner. The momentum update formula is:

[0096]

[0097]

[0098] Where ξ and θ are the simulated and measured attribute regressor parameters, respectively; v1 and v2 are the initial update iteration rates, respectively; t represents the current update step; and t-1 represents the update step in the previous stage. The attribute regression loss function in S21 This represents the self-reconstruction loss of the measured rain map;

[0099] S32, Self-reconstruction loss of the measured rain map for:

[0100]

[0101] Among them, R Real For the measured rain layer diagram, C Real To obtain a clear background image for actual testing, W attri W sty W MFT These are the measured network weights for the third, fourth, and fifth networks, respectively. Real This is a measured rainfall map.

[0102] Compared with the simulated rain layer map, the measured rain layer map itself lacks corresponding attribute labels. Furthermore, the simulated rain layer data based on the physical model has a large domain difference with the measured rain layer data. In this case, the network weights trained based on the simulated rain layer data are difficult to generalize to the measured rain layer. Therefore, this application proposes a method based on semi-supervised momentum weight update to realize the regression of the measured rain layer attributes.

[0103] To improve the quality of subsequent measured rain layer image reconstruction, accurate regression of physical attributes from the measured rain layer images is crucial. Therefore, in addition to using the physical attribute labels from the simulated rain layer data as a supervised loss function, the image consistency loss from the subsequent image generation results (measured rain layer image reconstruction results) is also used to simultaneously constrain the gradient calculation of the attribute regressor. These two methods influence the dynamic update of the attribute regressor parameters through a momentum iteration mechanism. The constructed third network is the measured attribute regression network, with the same attributes and network structure as the simulated attribute regression network. It is initialized using weights trained on paired simulated rain layer data and trained with the supervised loss of the simulated attributes and its own image reconstruction self-supervised loss as constraints.

[0104] Based on the above embodiments, optionally, S4 specifically includes:

[0105] S41. Obtain the mean and variance data of the style in the measured rain layer map, and analyze the apparent style S of the measured rain layer map in the feature latent space. Real Explicit modeling is performed to obtain the fourth network;

[0106] S42. Apply a Gaussian prior distribution to the fourth network and use KL divergence to constrain the training of the fourth network, limiting the style of the measured rain layer map to a range close to the prior distribution p(z). The specific loss function is:

[0107]

[0108] in, Where 's' represents the style of the rain layer, and I t Let N be the identity matrix, and M be the number of samples. z Dimensions representing rain layer style;

[0109] S43. By repeatedly sampling the measured styles of different measured rain layer maps, multiple sets of measured styles with the same distribution are obtained. The specific sampling formula is as follows:

[0110] S Real =μ+σ*∈

[0111] σ*∈~N(0,I t )

[0112] Where μ and σ are the mean and variance of the style regressor output, respectively, and I t It is the identity matrix;

[0113] S44. The measured styles obtained from resampling are labeled as Syn.Sty and Real.Sty respectively, and used as the simulation style of the aligned simulated rain layer map and the measured style of the measured rain layer map.

[0114] Although simulated and measured rain layer data share the same distribution in terms of rain streak physical properties, they differ significantly in their apparent style, which is difficult to characterize using an explicit mathematical distribution. To make the generated results more realistic, it is necessary to focus on the observed style rather than the simulated one. Therefore, to explicitly model the style of measured rainy days, a fourth network, the style encoder, is constructed to regress a new latent variable S. Real To explicitly characterize the style of the measured rain map, embodiments of this application apply a Gaussian prior distribution to it and use KL divergence to restrict the style of the measured rain to a range close to the prior distribution p(z).

[0115] The fourth network consists of four downsampling layers and fully connected modules. It takes the measured rain layer map as input and outputs two two-dimensional vectors representing the mean μ and variance σ of the measured rain layer distribution, respectively. This embodiment of the invention applies a Gaussian prior distribution to the measured apparent style S. Real Explicit modeling is performed using a KL divergence-constrained style encoder for training.

[0116] Based on the above embodiments, optionally, S5 specifically includes:

[0117] S51. Package the simulated clear background image and the measured clear background image, the first physical property and the second physical property of the rain layer, the measured style of the measured rain layer image and the simulated style of the simulated rain layer image respectively to obtain Param.Syn and Param.Real;

[0118] S52. Based on the pixel consistency loss and generative adversarial loss of the measured rain image, the generator is jointly constrained to generate results with the measured style. The generative adversarial loss is as follows:

[0119]

[0120] Among them, I gen Based on the physical properties of simulated rain layer images, clear backgrounds, and measured styles, I Rec This is the reconstruction result of the measured rainfall map;

[0121] S53. Based on the physical property consistency loss of the simulated rain layer diagram, the result of the constraint generator has the expected properties, realizing the controllability of the generated result. The specific loss function is as follows:

[0122]

[0123]

[0124] Among them, R Real B is a measured rain layer diagram. Real To obtain a clear background image for actual testing, I Real For measured rainfall maps, I Syn To simulate rain patterns, I Gen W is a rain map generated based on a clear simulated background image and physical properties. attri W sty W MFT These are the measured network weights for the third, fourth, and fifth networks, respectively.

[0125] S54. The trained fifth network obtains features F from the simulated clear background image and the measured clear background image. b Constructing a mapping relationship between physical properties and feature control parameters. (attri,style)→(α,γ,β), where the adjustment parameter α is obtained through the attribute condition attri mapping, and the transformation parameter (γ,β) is obtained through the style condition style mapping;

[0126] S55. Obtain the control parameters and transformation parameters, control the features of the background image, transform the controlled features, and coordinate the attributes and styles to obtain rainy day images that combine realism and diversity.

[0127] Through the aforementioned decomposition of physical space, the simulated and measured backgrounds, attributes, and styles are packaged into Param.Syn and Param.Real, respectively, as inputs to the subsequent controllable generator. Since the alignment of attributes and styles between the measured and simulated rain layer images has been achieved, and both the measured and simulated clear background images are realistic, they can be considered to have the same distribution. Param.Syn and Param.Real also share the same distribution. Furthermore, the generator weights are shared during the simulation generation and the measured reconstruction processes, so their outputs should naturally have the same distribution. Furthermore, the pixel consistency loss and generative adversarial loss of the measured rain images are used to jointly constrain the generator's generation results.

[0128] To allow explicit control over both attribute parameters and measured style in the generated results, this application presents a network structure that constructs a controllable mapping from background-attribute-style to the measured rainy day image space. For example... Figure 3 As shown, firstly, high-level features F of depth are extracted from the background using a residual module. b Subsequently, the MFT learns a mapping relationship between explicit physical properties and feature modulation parameters. (attri,style)→(α,γ,β), where the adjustment parameter α is obtained through attribute conditions, such as... The transformation parameters (γ, β) are obtained from the style of the measured rain map, such as... in, and This is an embedded module. After obtaining the modulation and transformation parameters, the MFT module first modulates the background feature map, such as: To transform the features to the experimental style, AdaIN is used to transform the modulated features. The specific transformation formula is shown below:

[0129]

[0130] Where μ(·) and σ(·) are the operations of calculating the mean and standard deviation of the features, respectively. Combined with the MFT module, the measured image reconstruction loss, and the simulated attribute reconstruction loss, it is possible to generate realistic rain maps by simultaneously coordinating attributes and style.

[0131] The proposed rainy day image generation method based on attribute and style dual domain transfer in this embodiment explicitly models the degraded physical attribute parameters in the attribute space. By adjusting these parameters, it can generate rainy day degraded images of different forms with fine-grained control, significantly enhancing the diversity of the generated results, compared with existing rain image simulation methods based on generative models and video processing.

[0132] refer to Figure 4The first column is a given clear image, and the second to fourth columns are rain-degraded images with different attributes generated sequentially by adjusting different attribute parameters. Each row represents a specific attribute that is individually adjusted by the technical solution of this invention.

[0133] refer to Figure 5 This paper compares the restoration performance of existing rain removal methods trained on their original data (-O) and the dataset generated in this paper (-P) on real-world rainy images. Specifically, DDN ("Removing rain from single images via a deep detail network," published by Fu et al. in 2017 at the IEEE Conference on Computer Vision and Pattern Recognition), JORDER_E ("Jointrain detection and removal from a single image with contextualized deep networks," published by Yang et al. in 2019 at IEEE Transactions on Pattern Analysis and Machine Intelligence), and MPRNet ("Multi-stage progressive image restoration," published by Zamir et al. in 2021 at the IEEE Conference on Computer Vision and Pattern Recognition) all demonstrate that training on the data generated by the technical solution in this application achieves better rain streak removal and better detail preservation in real-world rain image restoration.

[0134] Example 2

[0135] This invention provides a realistic and controllable rain image generation system based on attribute and style dual domain transfer, realizing the realistic and controllable rain image generation method based on attribute and style dual domain transfer as described in any of Embodiment 1, including:

[0136] The rain layer decomposition module is used to train the first network with simulated rain images and corresponding clear backgrounds. The simulated rain images and measured rain images are input into the trained first network, and the output is a simulated rain layer image, a simulated clear background image, a measured rain layer image, and a measured clear background image.

[0137] The simulation attribute regression module is used to train the second network with the simulation rain layer map and the corresponding physical attributes, so that the second network can adaptively regress its corresponding first physical attribute based on the input simulation rain layer map.

[0138] The measured attribute regression module is used to input the measured rain layer map, use the second network as the initial weight of the third network, and constrain the third network to train based on the semi-supervised momentum weight update method. The third network adaptively regresses to obtain the second physical attribute of the rain strips in the measured rain layer map. The data distribution of the first physical attribute and the second physical attribute is consistent.

[0139] The style autoencoder module is used to obtain the mean and variance data of the style from the measured rain layer map, and train the fourth network constrained by the prior Gaussian normal distribution. The style distribution obtained by regression based on the measured rain layer map is resampled to obtain two sampling points that represent the measured style of the measured rain layer map and the style of the simulated rain layer, respectively, so as to achieve the alignment of the apparent styles of the two.

[0140] The controllable generation module is used to package the simulated clear background image and the measured clear background image, the first physical attribute and the second physical attribute of the rain layer, the measured style of the measured rain layer image and the simulated style of the simulated rain layer image, respectively. The packaged data is input into the fifth network, and the fifth network is trained by constraining the pixel consistency and attribute consistency loss to obtain a controllable mapping from background-attribute-style to the measured rainy day image space, thereby obtaining rainy day images that combine realism and diversity.

[0141] The rainy day image realistic and controllable generation system based on attribute and style dual domain transfer provided in the embodiments of the present invention can execute the rainy day image realistic and controllable generation method based on attribute and style dual domain transfer provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0142] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for realistically controllable generation of rainy day images based on dual domain transfer of attributes and style, characterized in that, include: S1. Train the first network with simulated rain images and corresponding clear backgrounds. Input the simulated rain images and measured rain images into the trained first network, and output the simulated rain layer image, the simulated clear background image, the measured rain layer image, and the measured clear background image. S2. The second network is trained using the simulated rain layer map and its corresponding physical properties, so that the second network can adaptively regress its corresponding first physical property based on the input simulated rain layer map. S3. Input the measured rain layer image, use the second network as the initial weight of the third network, and constrain the third network to train based on the semi-supervised momentum weight update method. After the third network adaptively regresses, the second physical attribute of the rain strip in the measured rain layer image is obtained. The data distribution of the first physical attribute and the second physical attribute is consistent. S4. Obtain the mean and variance data of the style from the measured rain layer map, and train the fourth network by constraining it with a prior Gaussian normal distribution. For the style distribution obtained by regression based on the measured rain layer map, obtain two sampling points by resampling to represent the measured style of the measured rain layer map and the style of the simulated rain layer, respectively, so as to achieve the alignment of the apparent styles of the two. S5. The simulated clear background image and the measured clear background image, the first physical attribute and the second physical attribute of the rain layer, the measured style of the measured rain layer image and the simulated style of the simulated rain layer image are packaged separately. The packaged data is input into the fifth network. The fifth network is trained by constraining the pixel consistency and attribute consistency loss to obtain a controllable mapping from background-attribute-style to the measured rainy day image space, thereby obtaining a rainy day image that combines realism and diversity.

2. The method as described in claim 1, characterized in that, S1 specifically includes: S11. The simulated rain map and the corresponding clear image labels constitute a simulation dataset for training the first network. The loss function is composed of the mean squared error. Among them, B Syn To simulate a clear background image, B GT Label the clear image; S12. Input the simulated rain map into the trained first network to obtain the simulated rain layer map R. Syn and a clear simulation background image B Syn ; S13. Input the measured rain map into the trained first network, and use the network weights trained based on the simulation dataset to obtain the measured rain layer map R. Real Clear background image B (based on actual measurement) Real The simulated rain map and the measured rain map share network weights in the first network.

3. The method as described in claim 1, characterized in that, S2 specifically includes: Using a simulated rain layer image as input and the physical properties of rain streaks in the simulated rain layer image as attribute labels, the corresponding parameters are regressed through multi-channel global pooling. A second network is then trained using mean-variance constraints, enabling it to adaptively regress the corresponding first physical property based on the input simulated rain layer image. The loss function is as follows: Where F(°) is the network transformation, W attr R represents the parameters of the second network. syn For the data used to simulate rain layer maps, A GT These are the attribute labels corresponding to the simulated rain layer map; the physical properties of the rain streaks include length, density, angle, and brightness.

4. The method as described in claim 1, characterized in that, S3 specifically includes: S31. Using the weights trained based on the paired simulated rain layer map-first physical attribute dataset as initial weights, a large supervised loss function constructed with the first physical attribute labels and a self-reconstruction loss function of the measured rain map are used to constrain the training of the third network. The two loss functions are used to update the momentum of the weights in an alternating iterative manner. The momentum update formula is as follows: Where ξ represents the simulated attribute regressor network parameters, θ represents the measured attribute regressor network parameters, v1 and v2 are the initial update iteration rates, t represents the current update step, and t-1 represents the update step in the previous stage. The attribute regression loss function in S21 This represents the self-reconstruction loss of the measured rain map; S32, Self-reconstruction loss of the measured rain map for: Among them, R Real B is a measured rain layer diagram. Real To obtain a clear background image for actual testing, W attri W sty W MFT These are the measured network weights for the third, fourth, and fifth networks, respectively. Real This is a measured rainfall map.

5. The method as described in claim 1, characterized in that, S4 specifically includes: S41. Obtain the mean and variance data of the style in the measured rain layer map, and analyze the apparent style S of the measured rain layer map in the feature latent space. Real Explicit modeling is performed to obtain the fourth network; S42. Apply a Gaussian prior distribution to the fourth network and use KL divergence to constrain the training of the fourth network, limiting the style of the measured rain layer map to a range close to the prior distribution p(z). The specific loss function is: in, Where 's' represents the style of the rain layer, and I t Let N be the identity matrix, and M be the number of samples. z Dimensions representing rain layer style; S43. By repeatedly sampling the measured styles of different measured rain layer maps, multiple sets of measured styles with the same distribution are obtained. The specific sampling formula is as follows: ∈~N(0,I t ) Where μ and σ are the mean and variance of the style regressor output, respectively, and I t It is the identity matrix; S44. The measured styles obtained from resampling are labeled as Sty.Sty and Real.Sty, respectively, and used as the simulation style of the aligned simulated rain layer map and the measured style of the measured rain layer map.

6. The method as described in claim 1, characterized in that, S5 specifically includes: S51. Package the simulated clear background image and the measured clear background image, the first physical property and the second physical property of the rain layer, the measured style of the measured rain layer image and the simulation style of the simulated rain layer image respectively to obtain the simulation decomposition variable Param.Sty and the measured decomposition variable Param.Real. S52. Based on the pixel consistency loss and generative adversarial loss of the measured rain image, the generator is jointly constrained to generate results with the measured style. The generative adversarial loss is as follows: Among them, I gen Based on the physical properties of simulated rain layer images, clear backgrounds, and measured styles, I Rec This is the reconstruction result of the measured rainfall map; S53. Based on the physical property consistency loss of the simulated rain layer diagram, the result of the constraint generator has the expected properties, realizing the controllability of the generated result. The specific loss function is as follows: Among them, R Real B is a measured rain layer diagram. Real To obtain a clear background image for actual testing, I Real For measured rainfall maps, I Syn To simulate rain patterns, I Gen W is a rain map generated based on a clear simulated background image and physical properties. attri W sty W MFT These are the measured network weights for the third, fourth, and fifth networks, respectively. S54. The trained fifth network obtains features F from the simulated clear background image and the measured clear background image. b Constructing a mapping relationship between physical properties and feature control parameters. (attri,style)→(α,γ,β), where the adjustment parameter α is obtained through the attribute condition attri mapping, and the transformation parameter (γ,β) is obtained through the style condition style mapping; S55. Obtain the control parameters and transformation parameters, control the features of the background image, transform the controlled features, and coordinate the attributes and styles to obtain rainy day images that combine realism and diversity.

7. A system for realistically and controllably generating rainy images based on attribute and style dual-domain transfer, realizing the method for realistically and controllably generating rainy images based on attribute and style dual-domain transfer as described in any one of claims 1-6, characterized in that, include: The rain layer decomposition module is used to train the first network with simulated rain images and corresponding clear backgrounds. The simulated rain images and measured rain images are input into the trained first network, and the output is a simulated rain layer image, a simulated clear background image, a measured rain layer image, and a measured clear background image. The simulation attribute regression module is used to train the second network with the simulation rain layer map and the corresponding physical attributes, so that the second network can adaptively regress its corresponding first physical attribute based on the input simulation rain layer map. The measured attribute regression module is used to input the measured rain layer map, use the second network as the initial weight of the third network, and constrain the third network to train based on the semi-supervised momentum weight update method. The third network adaptively regresses to obtain the second physical attribute of the rain strips in the measured rain layer map. The data distribution of the first physical attribute and the second physical attribute is consistent. The style autoencoder module is used to obtain the mean and variance data of the style from the measured rain layer map, and train the fourth network constrained by the prior Gaussian normal distribution. The style distribution obtained by regression based on the measured rain layer map is resampled to obtain two sampling points that represent the measured style of the measured rain layer map and the style of the simulated rain layer, respectively, so as to achieve the alignment of the apparent styles of the two. The controllable generation module is used to package the simulated clear background image and the measured clear background image, the first physical attribute and the second physical attribute of the rain layer, the measured style of the measured rain layer image and the simulated style of the simulated rain layer image, respectively. The packaged data is input into the fifth network, and the fifth network is trained by constraining the pixel consistency and attribute consistency loss to obtain a controllable mapping from background-attribute-style to the measured rainy day image space, thereby obtaining rainy day images that combine realism and diversity.

Images