Unsupervised bidirectional variational auto-encoding intrinsic image decomposition network, method and application

Abstract

The present application relates to a kind of unsupervised bidirectional variational self-encoding essential image decomposition network, method and application, belong to image recognition technical field.The network uses parallel encoding and decoding strategy, i.e. using two sets of same but not associated by encoder and decoder consisting of deep learning network for reflectogram or illumination map image generation.Encoder and decoder are composed of network module comprising down-sampling or up-sampling;Mean and standard deviation of latent vector distribution are obtained by encoder from the feature between encoder and decoder, and then the feature meeting specific distribution is sampled from it.The method proposes to introduce the idea of variational self-encoding, constrain the intermediate vector in the encoding and decoding process, improve the robustness of the model;At the same time, an asymmetric generative adversarial framework is proposed, which improves the convergence performance of the model.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of image recognition, and relates to a network framework method for completing intrinsic image decomposition in an unsupervised form by a generative adversarial network and a variational autoencoder in combination with a self-attention mechanism and a convolutional neural network for a single image. BACKGROUND

[0002] The human visual system can easily identify targets from images, but images are just a set of data for computers, which cannot be directly understood and processed in a biological way. After the feature extraction method based on deep learning such as convolutional neural network was proposed, many tasks of computer vision once ushered in the dawn of victory. However, computer vision tasks still face many difficulties, such as the change of shooting environment light state, the change of background, the change of shooting angle and distance, the change of target posture and shape, and the change of mutual position between targets. Among them, the inconsistency of the apparent state of image data has always been one of the main factors restricting the performance improvement of image processing algorithm. To solve this problem, one solution is to extract the features in the image that do not change with the environment. That is, if the texture, color, material and other inherent information of the target can be separated from the environmental information in the image, and the part affected by the environment is filtered out, then a more accurate feature description of the target can be obtained, which is conducive to the performance improvement of other visual perception tasks. The goal of intrinsic image decomposition is to realize the extraction of inherent features, and its method is to decompose the image into a reflection map and a lighting map, which respectively contain texture, color, material and shape information, and lighting information.

[0003] The method based on deep learning is the mainstream algorithm for intrinsic image decomposition at present, but the existing deep learning method greatly depends on the data set and is also limited by the data set. For a deep network based on supervised learning, a large amount of labeled data is needed to train the network. The production of data labels often consumes time and effort, and at the same time, it is often impossible to provide labels for some image transformation tasks. For example, for a photo taken in a natural scene, it is difficult to provide pixel-level labels for the reflection map and the lighting map. Therefore, there is a more extensive demand for building an unsupervised learning network structure.

[0004] In recent years, there are three kinds of unsupervised generative models that perform well in computer vision tasks, including variational autoencoder, generative adversarial network and autoregressive model. The variational autoencoder (VAE) pairs the differentiable encoder with the encoder / generator network, and trains through the injection of noise; the generative adversarial network trains the generative model and the discriminative model at the same time, and solves the unsupervised problem in the way of adversarial learning. SUMMARY

[0005] The technical problems to be solved by the present application are:

[0006] In view of the problems that the supervised method is highly dependent on artificial labeling and the unsupervised method is insufficiently constrained by labels, the present application is inspired by the unsupervised learning method, and the implementation ideas of the generative adversarial network and the variational autoencoder unsupervised learning are used to propose an unsupervised learning framework combined with unsupervised and self-supervised ideas and trained on a non-paired data set, namely a bidirectional variational autoencoder generative adversarial network and an essential image decomposition method based on the network.

[0007] To solve the above technical problems, the technical scheme adopted by the present application is:

[0008] An unsupervised bidirectional variational autoencoder essential image decomposition network, characterized in that the network uses a parallel encoding and decoding strategy, that is, two sets of identical but mutually independent deep learning networks composed of encoders and decoders are used to generate reflection images or illumination images.

[0009] The further technical scheme of the present application: contains two generation paths from X to Y' and from Y to X', and the two generation paths use different generation models, wherein the path from X to Y' uses a generation network combining convolution and self-attention, and the path from Y to X' uses a generation network composed of only convolution modules, and this strategy is called an asymmetric bidirectional generation network; wherein X and Y represent the real images of the original image and the reflection image / illumination image respectively, and X' and Y' are the predicted images generated by the generator; E, G and E', G' are the encoders and decoders from the X domain to the Y domain and from the Y domain to the X domain respectively, and D and D' are the corresponding discriminators.

[0010] The further technical scheme of the present application: the encoder and the decoder are composed of network modules containing downsampling or upsampling; the features between the encoder and the decoder are encoded to obtain the mean and standard deviation of the latent vector distribution, and then sampled to features conforming to a specific distribution.

[0011] An unsupervised bidirectional variational autoencoder essential image decomposition method, characterized by the following steps:

[0012] Step 1: Data preparation

[0013] Use the original image as the input of the essential image decomposition model, and use the reflection image or the illumination image as the label during model training; randomly extract a certain number of input images in the test image data set, and then extract the corresponding reflection image or illumination image;

[0014] Step 2: Construct a self-attention and convolution generator

[0015] The self-attention and convolution generator is composed of an encoder and a decoder comprising five layers of modules, the encoder performs down-sampling layer by layer, the resolution of the feature map in the network is correspondingly reduced, the decoder performs up-sampling layer by layer, and the resolution of the feature map is correspondingly increased;

[0016] The encoder is composed of a "self-attention / convolution module + down-sampling" structure, and the decoder is obtained by stacking a "self-attention / convolution module + up-sampling" structure; the down-sampling or up-sampling is completed by convolution or deconvolution operation in the convolution module; the latent vector obtained by sampling is transmitted to the stacked module together with the features transmitted by the skip connection for decoding and restoring the resolution layer by layer, and finally a predicted image is obtained;

[0017] Step 3: Constructing an asymmetric bidirectional generation network

[0018] The asymmetric bidirectional generation network refers to that in the generative adversarial network, the mapping from the source data domain to the target domain uses a generator based on a self-attention mechanism, and the corresponding inverse generator uses a convolutional neural network;

[0019] The mapping from the source data domain to the target domain in the asymmetric bidirectional generation network uses the self-attention and convolution generator in step 2, and the corresponding mapping from the target domain to the source data domain replaces all the self-attention modules of the self-attention and convolution generator in step 2 with convolution modules;

[0020] Step 4: Setting a loss function, inputting a training data set to train the asymmetric bidirectional generation network, using a reflectance map label in the training of the reflectance map decomposition network, and using an illumination map label in the training of the illumination map decomposition network;

[0021] Step 5: Using the trained asymmetric bidirectional generation network to perform image decomposition to obtain a reflectance map or an illumination map.

[0022] Further technical solutions of the present application: the training data set is an unsupervised data set made on the basis of the ShapeNet data set, the images in the data set are processed into original images, reflectance maps and illumination maps respectively, and are used as inputs and two kinds of labels in training.

[0023] Further technical solutions of the present application: the self-attention module uses a convolution or deconvolution with a convolution kernel size of 1 and a step of 2 as a down-sampling or up-sampling module.

[0024] Further technical solutions of the present application: using mean square error, local mean square error and structural dissimilarity as evaluation indexes of the output.

[0025] The further technical scheme of the present application is: in training, a random seed is used for fixed initialization parameters, an Adam optimizer is uniformly used for network optimization, the parameters beta1 and beta2 of the optimizer are respectively set to 0.9 and 0.999, the weight decay rate is set to 1e-8, the network epoch number is set to 250 times, and the initial learning rate is 0.0002.

[0026] The further technical scheme of the present application is: the loss function in training includes two parts of the loss function of the generator and the loss function of the discriminator, wherein the loss function of the generator includes an adversarial loss, a cycle consistency loss and a KL loss; the loss function of the discriminator includes an adversarial loss and a distribution loss.

[0027] An application of the image decomposition method, characterized in that the texture, color, material, shape information, illumination information and environmental information of the target are separated from the image, the part affected by the environment in the image is filtered out, the more accurate feature description of the target is obtained, and the performance of other visual perception tasks is improved.

[0028] The present application has the following advantages:

[0029] The unsupervised bidirectional variational auto-encoding adversarial network provided by the present application realizes intrinsic image decomposition, and the traditional supervised method needs to use paired samples for training, which has high cost, the present application uses non-paired samples for unsupervised training based on a generative adversarial network, and the training cost is reduced.

[0030] In view of the weak constraint problem of the unsupervised method, the idea of variational auto-encoding is introduced to constrain the intermediate vector in the encoding and decoding process, and the robustness of the model is improved.

[0031] In view of the convergence difficulty problem of the generative adversarial network, an asymmetric generative adversarial framework is proposed, and the convergence performance of the model is improved.

[0032] In view of the problem that the traditional convolutional neural network lacks attention to long-distance dependency relationship, a network module combining self-attention mechanism and convolutional neural network is proposed. BRIEF DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings are for the purpose of illustrating preferred embodiments of the present application and are not to be construed as limiting the present application, wherein the same reference numerals in the drawings denote the same elements.

[0034] Figure 1 Unsupervised bidirectional variational auto-encoding adversarial network schematic diagram;

[0035] Figure 2 Self-attention and convolution-based generation module;

[0036] Figure 3Visualization results of different methods on MPI-Sintel dataset. DETAILED DESCRIPTION

[0037] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be further described in detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0038] The embodiment of the present application proposes an unsupervised bidirectional variational autoencoder intrinsic image decomposition network, which uses a parallel encoding and decoding strategy, that is, uses two sets of the same but mutually independent encoders and decoders to generate reflectance and illumination images. Its structure is as shown in the figure Figure 1 X and Y in the figure respectively represent the original image and the real image of the reflectance / illumination image, and X' and Y' are the predicted images generated by the generator. E, G and E', G' are the encoders (inference network) and decoders (generation network) from the X domain to the Y domain and from the Y domain to the X domain, and D and D' are the corresponding discriminators. The encoder and the decoder are composed of network modules containing downsampling or upsampling. The features between the encoder and the decoder get the mean and standard deviation of the latent vector distribution through the inference network, and then sample the latent variables conforming to the specific distribution therefrom. The figure contains two generation paths from X to Y' and from Y to X', and the two generation paths use different generation models, wherein the path from X to Y' uses a generation network combining convolution and self-attention, and the path from Y to X' uses a generation network composed of only convolution modules, and this strategy is called asymmetric bidirectional generation network.

[0039] The embodiment of the present application further proposes an unsupervised bidirectional variational autoencoder intrinsic image decomposition method based on the above network, and the specific steps are as follows:

[0040] Step one: data preparation

[0041] The present application uses the original image as the input of the intrinsic image decomposition model, and uses any reflectance image and illumination image that do not correspond to the input image as the label during model training. A certain number of input images are randomly extracted in the test image data set, and then the corresponding reflectance image and illumination image are extracted. The image embedding is performed on each extracted input image, and the input token of the subsequent Transformer network is processed.

[0042] Step two: build self-attention convolution generator

[0043] The generator based on self-attention and convolution contains five layers of networks with different scales and is one-to-one corresponding in the encoding network and the decoding network. The generator mixes the convolutional layer and the self-attention mechanism in the model. As shown in Figure 2 The self-attention based generator is composed of an encoder and a decoder containing five layers of modules. The encoder completes down-sampling layer by layer, and the resolution of the feature map in the network decreases accordingly. The decoder completes up-sampling layer by layer, and the resolution of the feature map increases accordingly. Among them, the three layers with higher resolution are convolutional layers, and the two layers with lower resolution are composed of self-attention layers. The advantage of this structure is that the neural network can accept larger input images without dramatic changes in scale. The convolutional layer in the high-resolution part uses a fixed-size convolution kernel (3x3), so it is less sensitive to scale changes than the self-attention mechanism.

[0044] The self-attention and convolution based generation network is "U" shaped in general structure, that is, it is encoded and then decoded according to different scales, but there is a big difference in the design of the bottleneck layer. In the classic encoding and decoding network, the bottleneck layer usually uses a convolutional layer to further encode the features between the encoding network and the decoding network. The bottleneck layer of the generator in the bidirectional variational auto-encoder is replaced by a latent vector layer similar to the variational auto-encoder. The latent vector layer predicts the mean and standard deviation of the latent vector distribution through two fully connected layers, and then generates a standard normal distribution using the mean and standard deviation sampled from the encoder. The features sampled from the latent vector layer conforming to a specific distribution will replace the bottleneck layer output as the input of the decoding network.

[0045] The encoder network is composed of "self-attention / convolution module + down-sampling", and the decoder network is stacked by "self-attention / convolution module + up-sampling" structure. Down-sampling or up-sampling is completed by convolution or deconvolution operation in the convolution module, and convolution kernel size of 1 and step of 2 are used in the self-attention module. The latent vector obtained by sampling will be transmitted together with the features transmitted by the skip connection into the stacked module for decoding and restoring the resolution level by level, and finally the predicted image is obtained.

[0046] Step three: build an asymmetric bidirectional generation network

[0047] The asymmetric bidirectional generation network of the present application refers to that the mapping from the source data domain to the target domain in the generative adversarial network uses a generator based on a self-attention mechanism, while the corresponding inverse generator uses a convolutional neural network. For the intrinsic image decomposition task, only the generator from the source data domain to the target domain is actually needed, and the other generators and discriminators only play a role in the training process. The generator designed based on the self-attention mechanism has good long-distance dependence in the intrinsic image decomposition task, so it is still the first choice for the generator in unsupervised learning. Correspondingly, using a convolutional neural network as the inverse generator can not only reduce the computational amount of the model, but more importantly, it can counteract the training difficulty caused by the increase in the number of parameters.

[0048] In the bidirectional generation structure using the self-attention network as the generator, the training difficulty mentioned above not only lies in the disappearance of the gradient, but the sudden increase in the number of parameters means that the process of fitting the mapping is more difficult to learn. The idea of asymmetric generation is to deliberately make the learning speed of the two generators deviate, forcing one of the generators to catch up with the corresponding inverse generator for training. The classic generative adversarial network uses a discriminator as the driving force for model training, and after the introduction of the bidirectional generation structure in Cycle-GAN, the thrust of the discriminator is diluted, and the self-attention mechanism further burdens the model convergence. The asymmetric generation strategy can provide acceleration for the cyclic generation, and form a combined force with the supervision of the discriminator.

[0049] The mapping from the source data domain to the target domain in the asymmetric bidirectional generation network uses the generator in step two, and the corresponding mapping from the target domain to the source data domain replaces all the self-attention modules of the generator in step two with convolution modules.

[0050] Step four: define the loss function

[0051] The loss function of the bidirectional variational autoencoder includes the loss function of the generator and the loss function of the discriminator. The loss function of the generator is:

[0052] L G =L GAN +αL cyc +βL kld

[0053] In the formula, L G is the generator loss, L GAN is the adversarial loss, L cyc is the cycle consistency loss, L kld is the KL loss, and α and β are the weights of the cycle consistency loss and the KL loss, respectively.

[0054] The adversarial loss is:

[0055] L GAN =LGAN (G, DY, X, Y) + L GAN (F, DX, Y, X)

[0056] The bidirectional VAE contains two generators G and F, corresponding to two discriminators DX and DY. The input of generator G of the bidirectional VAE is the picture x of domain X, and the goal of generator G is to make the generated sample G(x) as similar as possible to the sample in domain Y. The input of discriminator DY has two kinds: the sample y of domain Y and G(x), and the optimization goal of discriminator DY is to distinguish y and G(x) as much as possible. That is:

[0057]

[0058] For generator F and discriminator DX, the principle is the same as above: y input generator F gets output G(y), and the goal is to make G(y) as similar as possible to the real X domain data x. And the input of discriminator DX is G(y) and x, and the optimization goal is to distinguish them as much as possible.

[0059] The cycle consistency loss constraint keeps the image unchanged after two mappings, and the formula is as follows:

[0060]

[0061] In the generator, the KL divergence is introduced to calculate the difference between the distribution of the latent variable and the standard Gaussian distribution. The formula is as follows:

[0062]

[0063] In the formula, KL represents the calculation of the KL divergence between the generation model p θ and the inference model .

[0064] The loss function of the discriminator is composed of the adversarial loss and the distribution loss. The adversarial loss is similar to that in the generator, which is used to calculate the similarity between the restored image and the real image. The distribution loss calculates the distribution difference between the restored image and the real image according to the mu and sigma of the Gaussian distribution, and introduces the standard normal distribution as a regularization term. The formula is as follows:

[0065]

[0066] In the formula, z g and z r respectively represent the mean and standard deviation of the latent variable generated by the distribution discriminator from the generated image and the real image, and z n represents the latent variable conforming to the standard normal distribution.

[0067] The final loss function of the discriminator can be expressed as:

[0068] L D =L GAN +L Gussian

[0069] Step five: network training

[0070] For all experiments, the present application uses a random seed to fix the initialization parameters. The network optimizer uniformly uses the Adam optimizer, and the parameters β1 and β2 of the optimizer are set to 0.9 and 0.999 respectively, and the weight decay rate is set to 1e-8. The network epochs is set to 250 times, and the initial learning rate is 0.0002. The batchsize of model training is set to 24. Among them, the input image resolution of training and testing is 256x256x3.

[0071] Embodiment:

[0072] Step one: data preparation and index selection

[0073] The bidirectional variational autoencoder uses unpaired labels as the data set in training, and existing data sets are difficult to directly meet this condition, so the present application makes an unsupervised data set on the basis of the ShapeNet data set. The present application selects 7 categories in the ShapeNet data set, and selects unpaired images in each category of objects as training samples, a total of 1000 pairs of pictures.

[0074] The images in the data set are respectively processed into original images, reflection maps and illumination maps, which are respectively used as inputs and two kinds of labels in training. The size of the picture is uniformly processed to 256x256x3.

[0075] Then select appropriate evaluation indexes. The present application adopts mean square error, local mean square error and structural dissimilarity as evaluation indexes.

[0076] 1) Mean square error (MSE)

[0077] Mean square error is often used to evaluate the difference between the predicted image and the real image in the global range in visual tasks, and its mathematical definition is shown in the following formula:

[0078]

[0079] The present application normalizes the image before calculating the MSE, and the final MSE value range is (0, 1).

[0080] 2) Local mean square error (LMSE)

[0081] The mean square error can roughly evaluate the difference degree of the image in the global range, but cannot accurately measure the difference of the local image. In order to evaluate the difference degree of the local image, the local mean square error (LMSE) is proposed on the basis of the mean square error. The LMSE sets a sliding window in the image and calculates the mean square error of the image block in each window, and then the mean square error of all windows is comprehensively calculated to obtain the LMSE. The scale-invariant MSE is used in the calculation of the LMSE, and the scale-invariant covariance is adjusted by introducing the global brightness of the predicted image. The formula is as follows:

[0082]

[0083] Specifically, the LMSE selects a sliding window with a size of k×k in the calculation, and then slides on the image with a step of k / 2. The scale-invariant covariance is calculated in a local area ω each time. If the set of local areas of the image is represented as Ω, then the LMSE is equal to the cumulative sum of the scale-invariant covariances of all local areas in the set Ω, which is expressed in mathematical formula as follows:

[0084] LMSE k (x,x′)=∑ ω∈Ω siMSE(x ω ,x′ ω )

[0085] In actual calculation, the window size k is set to 20 in the present application. The value range of the LMSE is (0, 1), and the smaller the value in the value range, the smaller the local difference of the essential image.

[0086] 3) Structural dissimilarity (DSSIM)

[0087] The structural similarity (SSIM) is a quantitative index for measuring the similarity of images in structure, contrast and brightness. Given two images X and Y, the structural similarity of X and Y can be derived by the following formula:

[0088] SSIM(X,Y)=L(X,Y)*C(X,Y)*S(X,Y)

[0089]

[0090]

[0091]

[0092] Where μ X and μ Y represent the mean of the image, σ X and σ YStandard deviation of the image, σ XY Covariance of the image, C1,C2 and C2 are constants. To avoid the occurrence of zero division phenomenon, usually take C1=(k1L) 2 , C2=(k2L) 2 , Where k1=0.01, k2=0.03, L=255. The formula of L(X,Y), C(X,Y) and S(X,Y) is brought into the structural similarity formula, and the complete formula of the structural similarity is as follows.

[0093]

[0094] The greater the structural similarity, the more similar the prediction result is to the label; on the contrary, the smaller the structural similarity, the more dissimilar the generated image is to the label image. In order to make this index as the other two indexes, the smaller the value is, the better, the present application uses structural dissimilarity instead of structural similarity.

[0095] The structural dissimilarity (DSSIM) is a linear mapping of the structural similarity, and its conversion mode can be described by the following formula:

[0096]

[0097] The value range of DSSIM is [0,1], and the smaller the value is, the more approximate the prediction result is to the label.

[0098] Step two: building an unsupervised bidirectional variational auto-encoding essential image decomposition network

[0099] The present application builds an unsupervised bidirectional variational auto-encoding essential image decomposition network through Pytorch, and uses the data set established in step one for training. The model is trained for 250 epochs.

[0100] Step three: analysis of experimental results

[0101] 1) Experimental results on MPI data set

[0102] The present application compares the essential image decomposition effects of bidirectional variational auto-encoding adversarial network and other recent works. In order to ensure the accuracy of the experimental results, all the training and testing are completed on the MPI-Sintel data set. It is particularly noted that MSCR and FY18 are trained by using a full supervision method, as a control group of the remaining unsupervised methods.

[0103] According to the experimental results in Table 1, it can be seen that the bidirectional variational auto-encoding adversarial network proposed in the present application is superior to other unsupervised methods in terms of the MSE results of the reflectance map and the illumination map recovery, proving the effectiveness of the proposed method. In addition, the generator in the proposed method is superior to the supervised learning method as a control in the case of supervised training, further verifying the advantage of introducing the self-attention mechanism in the generator.

[0104] Figure 3 The visualization results of different intrinsic image decomposition methods are shown, and two groups of scenes are selected as examples in the figure, and the first row of each group of scenes is the reflectance map recovery result, and the second row is the illumination map recovery result. The method in the second column is the result of the self-attention generator in the bidirectional variational auto-encoding adversarial network trained using the full supervision method, the third column (FY18) is the full supervision algorithm, and the fourth column (LS18) is the unsupervised algorithm. As can be seen from the figure, the recovered reflectance map of the bidirectional variational auto-encoding adversarial network is better than the previous unsupervised method in removing the illumination information, and the content clarity of the recovered reflectance map has improved. In terms of the recovery effect of the illumination map, the results of the proposed method are closer to the true label. By comparing the visualization effects of the bidirectional variational auto-encoding adversarial network under unsupervised and full supervision training, it can be seen that the results obtained by unsupervised training still have deficiencies, and one of the more prominent points is that the local color of the recovered reflectance map of the two groups of unsupervised methods has changed significantly compared with the label, and the bidirectional variational auto-encoding adversarial network has not solved this point compared with other unsupervised methods.

[0105] Table 1 Performance comparison of different methods on MPI dataset

[0106]

[0107]

[0108] 2) Comparison experiment of unsupervised strategy

[0109] The bidirectional variational autoencoder adversarial network is proposed by referring to the ideas of Cycle-GAN and variational autoencoder. In order to prove that the performance of the proposed model is better than using only Cycle-GAN or variational autoencoder, the present application designs a set of control experiments. Four groups of control models are set in the experiment, and the "unsupervised + self-supervised" item in the experimental group represents the bidirectional variational autoencoder adversarial network, the "unsupervised" item represents the Cycle-GAN, and the "self-supervised" item represents the variational autoencoder. The control group is the generator trained by the full supervision method. In order to make the results of the control experiment only reflect the influence of the supervision training strategy, the present application controls the other variable conditions to be unchanged, and the image generation model of the four groups of experiments is implemented by using the same kind of encoder-decoder based on the self-attention mechanism. It should be particularly noted that the encoder-decoder in the "unsupervised" method uses the latent variable layer in the bottleneck layer, while the "self-supervised" method, the "unsupervised + self-supervised" method and the "full supervision" method use the latent variable layer in the bottleneck layer. On the training data set, the four groups of methods are trained on the ShapeNet data set of 1k pairs of pictures, wherein the "self-supervised" method uses the input image as the label, the "full supervision" method uses the real reflection image and the light image as the label, and the other two groups use the non-paired reflection image and the light image as the label.

[0110] Table 2 Training results of different supervision strategies

[0111]

[0112] According to the experimental results in Table 2, it can be seen that the autoencoder using only the self-supervised method is not very good in essence image decomposition because it uses the input image itself as supervision in training. The bidirectional variational autoencoder network combining "unsupervised" and "self-supervised" is better than Cycle-GAN trained on the non-paired ShapeNet data set, which proves that the bidirectional variational autoencoder adversarial network proposed in the present application is effective in unsupervised training. In addition, compared with the results of the full supervision method and the bidirectional variational autoencoder adversarial network, it can be seen that the performance of the unsupervised method still lags behind the full supervision method, especially in the recovery performance of the reflection image and the limit of the generation model.

[0113] 3) Evaluation of training strategy

[0114] For the training strategy of the neural network model, the present application proposes an asymmetric generative adversarial framework. Under the asymmetric generative adversarial framework, the bidirectional variational autoencoder network is implemented based on the self-attention mechanism from the original image domain to the reflection image / light image generation network, and the reverse generation network is implemented based on the convolutional neural network. In order to verify the effectiveness of the asymmetric generative adversarial framework, the present application designs a set of control experiments containing three modes of symmetric generation and asymmetric generation, which are:

[0115] (a) Convolutional generative network + Convolutional generative network

[0116] (b) Self-attention generative network + Self-attention generative network

[0117] (c) Self-attention generative network + Convolutional generative network

[0118] Table 3 Experimental results of different training strategies

[0119]

[0120] According to the experimental results in Table 3, under the condition that other variables are the same, the method of asymmetric generative confrontation "Transformer+Conv" is better than the two groups of experiments of "Conv+Conv" and "Transformer+Transformer" using symmetric generation in terms of MSE and LMSE.

[0121] The above merely describes the specific embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements shall be encompassed within the protection scope of the present application.

Claims

1. An unsupervised bidirectional variational autoencoder-based essential image decomposition method, characterized in that... The steps are as follows: Step 1: Data Preparation The original image is used as the input to the essential image decomposition model, and the reflection map or illumination map is used as the label during model training. A certain number of input images are randomly selected from the test image dataset, and then the corresponding reflection map or illumination map is extracted. Step 2: Construct self-attention and convolution generators The self-attention and convolution generator consists of an encoder and a decoder, each with five layers. The encoder performs downsampling layer by layer, resulting in a decrease in the resolution of the feature maps in the network. The decoder performs upsampling layer by layer, resulting in an increase in the resolution of the feature maps. The encoder consists of a "self-attention and convolution module + downsampling" and the decoder is obtained by stacking a "self-attention and convolution module + upsampling" structure. Downsampling or upsampling is performed in the convolution module by convolution or deconvolution operations. The latent vector obtained by sampling is fed into the stacked module together with the features passed by the skip connections to decode step by step and restore the resolution, and finally obtain the predicted image. Step 3: Construct an asymmetric bidirectional generative network The aforementioned asymmetric bidirectional generative network refers to a generative adversarial network in which the mapping from the source data domain to the target domain uses a generator based on a self-attention mechanism, while the corresponding inverse generator is implemented using a convolutional neural network. In the asymmetric bidirectional generative network, the mapping from the source data domain to the target domain uses the self-attention and convolution generator in step 2, while the corresponding mapping from the target domain to the source data domain replaces all the self-attention modules of the self-attention and convolution generator in step 2 with convolution modules. The asymmetric bidirectional generative network described herein includes a path from X to Y. and from Y to X Two generation paths, each using a different generation model, where the path from X to Y... The path uses a generative network that combines convolutions and self-attention, from Y to X. The path uses a generative network consisting only of convolutional modules; this strategy is called an asymmetric bidirectional generative network. Here, X and Y represent the original image and the real image (reflection map or illumination map), respectively. and Y These are the predicted images generated by the generator; E, G, and E G These are the encoder and decoder, respectively, from the X domain to the Y domain and from the Y domain to the X domain, D and D That is the corresponding discriminator; Step 4: Set the loss function, input the training dataset to train the asymmetric bidirectional generative network, use reflection map labels in the training of the reflection map decomposition network, and use illumination map labels in the training of the illumination map decomposition network. Step 5: Use the trained asymmetric bidirectional generative network to decompose the image and obtain the reflection map or illumination map.

2. The unsupervised bidirectional variational autoencoder-based image decomposition method according to claim 1, characterized in that, The training dataset is an unsupervised dataset created based on the ShapeNet dataset. The images in the dataset are processed into original images, reflection maps, and illumination maps, which are used as inputs and two types of labels in the training process, respectively.

3. The unsupervised bidirectional variational autoencoder-based image decomposition method according to claim 1, characterized in that, The self-attention module uses convolution or deconvolution with a kernel size of 1 and a stride of 2 as a downsampling or upsampling module.

4. The unsupervised bidirectional variational autoencoder-based image decomposition method according to claim 1, characterized in that, Mean square error, local mean square error, and structural dissimilarity are used as evaluation metrics for the output.

5. The unsupervised bidirectional variational autoencoder-based image decomposition method according to claim 1, characterized in that, During training, a random seed is used for fixed initialization parameters, and the Adam optimizer is used uniformly. The optimizer's parameters... and The values ​​were set to 0.9 and 0.999 respectively, the weight decay rate was set to 1e-8, the number of network epochs was set to 250, and the initial learning rate was 0.0002.

6. The unsupervised bidirectional variational autoencoder-based image decomposition method according to claim 1, characterized in that, The loss function in training consists of two parts: the generator loss function and the discriminator loss function. The generator loss function includes adversarial loss, cycle-consistent loss, and KL loss; the discriminator loss function includes adversarial loss and distribution loss.

7. An application of the image decomposition method according to claim 1, characterized in that, The texture, color, material, shape, lighting information, and environmental information of the target are separated from the image, and the parts of the image affected by the environment are filtered out, resulting in a more accurate feature description of the target.

Images