Room decoration style migration method based on semantic and style information guidance

Abstract

The invention provides a room decoration style migration method based on semantic and style information guidance. The method comprises the following steps: firstly, extracting semantic analysis features of an original room picture, then taking the original room picture, a reference style picture and the semantic analysis features as training samples together, and taking a room picture with a changed style as a label to construct a training set; a room picture feature and a reference picture feature are extracted by using a generator network, a normalization process is adjusted by using the reference picture feature through a double-branch normalization module to obtain space structure information and texture detail information, and a room image with a reference style is finally output through AFF feature fusion. And a discriminator network is used to discriminate the generated image by taking the label as a reference, and network training is completed. And finally, inputting the original room picture, the semantic analysis characteristics of the original room picture and the reference style picture into the trained generator network, generating a room image with a reference style, and completing decoration style migration.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and in particular relates to a room decoration style migration method based on semantic and style information guidance. Background Art

[0002] In recent years, computer vision has been increasingly used in the interior decoration industry. The overall style of a room determines the visual effect and atmosphere of the space. Style transfer of room images helps to achieve personalized and diversified decoration design. Traditional non-parametric image style transfer methods are mainly based on stroke rendering, image classification or filtering methods, and texture synthesis. These methods can usually only extract the underlying features of the image and it is difficult to generate high-level abstract features. Therefore, they are only suitable for artistic image style transfer, and are not effective in modern home design, and it is difficult to meet actual needs.

[0003] The emergence of deep learning has brought new image style transfer methods. Gatys proposed an image style transfer algorithm based on convolutional neural network (CNN), which extracts image features at different levels to achieve the separation of style and content. However, the images generated by this method often have problems such as noise and unclearness, and require a large number of paired training images. The training speed is slow and takes up a lot of memory. The cycle generative adversarial network (CycleGAN) proposed by Zhu et al. combines GAN and dual learning, uses two generators and two discriminators to convert between two style domains, and retains the content information of the image through cycle consistency loss, but this method still has problems in structural integrity and rationality of texture details. Modern home design is complex and emphasizes realism and verisimilitude. Existing methods are difficult to effectively retain the structural information of images and generate real textures. Therefore, it is of great significance to develop an image style transfer method that can simultaneously retain the structural information of room pictures and generate real style textures. Summary of the invention

[0004] In view of the shortcomings of the existing technology, the present invention proposes a room decoration style migration method based on semantic and style information guidance. This method uses a dual-branch structure to independently process the spatial structure and texture details, so that the generated room image is delicate and realistic in local details, and at the same time has consistency and coherence in the whole picture.

[0005] A room decoration style migration method based on semantic and style information guidance, the specific method is as follows:

[0006] Step 1: Use the pre-trained model to extract the features of the original room image and obtain the corresponding semantic parsing features. Use the original room image, reference style image, and semantic parsing features as training samples, use the room image after the style change as a label, and construct a training set.

[0007] Step 2: Construct a style transfer generative adversarial network model based on the reference style of room decoration; the generative adversarial network includes a generator network and a discriminator network. The generator network includes a generated room image encoder, a reference style image encoder, a dual-branch normalization module (DBN) and a decoder.

[0008] The room picture encoder and the reference style picture encoder have the same structure and are used to extract features from the original room picture and the reference style picture respectively to obtain room photo features and reference picture features.

[0009] The dual-branch normalization module includes a class adaptive normalization module, an instance adaptive normalization module and an attention feature fusion AFF module (Attentional Feature Fusion, AFF).

[0010] The class adaptive normalization module normalizes the reference image features according to the semantic parsing features to provide spatial structure information. First, the input semantic parsing features and reference image features are subjected to class adaptive normalization Clade processing, and after passing through the ReLU activation function, they are passed through a convolution layer with a convolution kernel size of 3, a padding of 1, and a stride of 1. The convolved features are subjected to Clade processing together with the semantic parsing features, and after passing through the ReLU activation function and a convolution layer with a convolution kernel size of 3, a padding of 1, and a stride of 1, the obtained results are residually connected with the reference image features to output the spatial structure information.

[0011] The instance adaptive normalization module uses the information in the reference image features to adjust the normalization process of the room photo features to provide texture detail information. First, the input room photo features and the reference image features are subjected to instance adaptive normalization Adain processing, and then pass through a convolution layer with a convolution kernel size of 3, a padding of 1, a stride of 1 and a ReLU activation function, and then pass through a convolution layer with a convolution kernel size of 3, a padding of 1, a stride of 1 and a ReLU activation function, and then perform a residual connection with the input room photo features, and then after passing through the ReLU activation function, output the texture detail information.

[0012] The AFF feature fusion module is used to perform feature fusion on the spatial structure information output by the class adaptive normalization module and the texture detail information output by the instance adaptive normalization module.

[0013] The decoder decodes the features fused by the AFF feature fusion module to obtain a room image with a reference style.

[0014] The discriminator network is used to discriminate the image output by the decoder, so as to make the image generated by the generator network more realistic and with more delicate details.

[0015] Step 3: Input the training set data in step 1 into the generative adversarial network model constructed in step 2, and use the Adam optimization algorithm to optimize the network model. The loss functions used include the perceptual loss, adversarial loss, and cycle consistency loss of the generated image, and save the model parameters obtained through training.

[0016] Step 4: Select a room image that needs to be modified in decoration style, extract the corresponding semantic parsing features through the pre-trained model, and then input the room image, semantic parsing features and the reference image with the migration target style into the generator network trained in step 3 to obtain the room image after the decoration style is migrated.

[0017] The present invention has the following beneficial effects:

[0018] This method uses a dual-branch normalization module to decouple spatial information and detail texture information respectively: the first branch uses the reference style to generate instance adaptation weights, and applies the instance adaptation weights to instance normalize the input image, making the generated room image more delicate and realistic in local details. The second branch uses the semantic analysis of the room image to generate class adaptation weights, and uses the generated weights to perform class adaptation normalization on the reference image, thereby ensuring the consistency and coherence of the generated room image as a whole; the two are processed separately, fully considering the influence of the independence of texture information and spatial information. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the room decoration style transfer method guided by semantic and style information;

[0020] Figure 2 Schematic diagram of the generator network structure constructed in the embodiment;

[0021] Figure 3 Schematic diagram of the structure of the dual-branch normalization module constructed in the embodiment;

[0022] Figure 4 is a schematic diagram of the structure of a feature fusion module based on AFF in an embodiment;

[0023] Figure 5 Schematic diagram of the discriminator network structure constructed in the embodiment. DETAILED DESCRIPTION

[0024] The present invention will be further explained below with reference to the accompanying drawings;

[0025] like Figure 1 As shown, a room decoration style migration method based on semantic and style information guidance specifically includes the following steps:

[0026] Step 1: Use the pre-trained VGG network to extract the features of the original room image and obtain the corresponding semantic analysis features. Collect room images with various decoration styles as reference style images. Use the original room images, reference style images, and semantic analysis features as training samples, and use the room images with changed styles as labels to construct a training set.

[0027] Step 2: Construct a style transfer generative adversarial network model based on the reference style of room decoration; the generative adversarial network includes a generator network and a discriminator network. Figure 2 As shown, the generator network includes a generated room picture encoder, a reference style picture encoder, a dual-branch normalization module (DBN) and a decoder.

[0028] The room picture encoder and the reference style picture encoder have the same structure, both of which include 4 cascaded encoding units, each of which includes a convolution layer with a convolution kernel of 3, a step size of 2, and a padding of 1, an instance normalization layer, and an activation function ReLU. The room picture encoder and the reference style picture encoder are used to extract features of the original room picture and the reference style picture, respectively, to obtain room photo features and reference picture features, wherein the room picture encoder is used to obtain contextual structural information of the original room picture, and the reference style picture encoder is used to obtain contextual information of the color style of the reference style picture.

[0029] like Figure 3 As shown, the dual-branch normalization module includes a class adaptive normalization module, an instance adaptive normalization module and an attention feature fusion AFF module.

[0030] The class adaptive normalization module first performs semantic parsing on the input features M p , reference image feature F r After Clade processing, after ReLU activation function, it passes through a convolution layer with a kernel size of 3, padding of 1, and stride of 1. The convolved features and semantic parsing features are Clade processed again, and after passing through ReLU activation function and a convolution layer with a kernel size of 3, padding of 1, and stride of 1, the result is residually connected with the reference image feature to output spatial structure information. The Clade processing method is:

[0031]

[0032] Among them, F CLADE It represents the features obtained after Clade processing, μ() represents the mean of the features, and σ() represents the variance of the features. represents the scale parameter of the i-th category in the semantic parsing graph, represents the offset parameter of the i-th category in the semantic parsing graph, and both are optimized as a regular network parameter through a modulation parameter library.

[0033] The instance adaptive normalization module first performs a room photo feature F on the input p and reference image feature F r Adain processing is performed, and then a convolution layer with a convolution kernel size of 3, padding of 1, stride of 1 and ReLU activation function is passed through, and then a convolution layer with a convolution kernel size of 3, padding of 1, stride of 1 and ReLU activation function is passed through, and then a residual connection is made with the input room photo features, and then after the ReLU activation function, the texture detail information is output. The Adain processing method is:

[0034]

[0035] Among them, F AdaIN Represents the features after Adain processing.

[0036] The two processes local texture details and overall spatial information separately to fully consider the influence of the independence of texture information and spatial information. Since the decoupling of structure and texture will inevitably produce redundancy, in order to avoid information redundancy and further utilize the complementary advantages of the two branches, the following is adopted: Figure 4 The AFF fusion module shown performs feature fusion on the spatial structure information output by the class adaptive normalization module and the texture detail information output by the instance adaptive normalization module. The AFF fusion module first concatenates the input spatial structure information and texture detail information, calculates global attention and local attention respectively, concatenates the two attention calculation results, activates them through the sigmoid function, multiplies the activation results with the input spatial structure information and texture detail information respectively, and then concatenates them to output the fused features.

[0037] The decoder decodes the features fused by the AFF feature fusion module to obtain a room image with a reference style. The decoder includes 4 decoding units, a convolution layer with a convolution kernel size of 3, a padding of 1, a stride of 1, and an activation function Tanh. Each decoding unit contains a transposed convolution layer with a convolution kernel of 3, a stride of 2, and a padding of 1, an instance normalization layer, and an activation function ReLU, which is used to output an upsampled feature map and splice it with the output feature map of the room image encoder at the same level in the channel dimension as the input of the next decoding unit.

[0038] like Figure 5As shown in the figure, the discriminator network is used to discriminate the image output by the decoder, making the image generated by the generator network more realistic and with finer details. The discriminator network includes 5 groups of convolutional layers with a kernel size of 3, a stride of 2, and a padding of 2, a batch normalization layer module, and a LeakeyReLU with a slope of 0.2.

[0039] Step 3: Use normal distribution to initialize the generative adversarial network model constructed in step 2, and its weight W satisfies the following distribution:

[0040] W~N(μ,σ 2 )(3)

[0041] Among them, μ = 0, is the mean of the distribution, σ 2 =0.02, which is the variance.

[0042] Input the training set data in step 1 into the initialized generative adversarial network model, use the Adam optimization algorithm to optimize the network model, and set the total loss function to:

[0043] L total =λ1L adv +λ2L cyc +λ3L per (4)

[0044]

[0045] Among them, λ1=1, λ2=0.5, λ3=0.5 are the weight coefficients of the loss function, L adv , L cyc , L per denote adversarial loss, cycle consistency loss, and perceptual loss, respectively. x is the input original room image, y is the room image after style transformation, r is the reference style image, M(x) is the semantic parsing feature corresponding to the original room image, G() is the generator network, F() is the generator network transformed back to the source domain, φ j () indicates that the j-th layer intermediate features of the pre-trained VGG network are used to calculate the perceptual distance between features, C j H represents the number of channels of the j-th layer feature map of the VGG network, j Represents the width of the feature map of the jth layer of the VGG network, W j Represents the height of the feature map of the jth layer of the VGG network, r x represents the reference style image related to the original room image x, r y represents the reference style image related to the room image y after style transformation, E x represents the expectation of the data distribution of the original room image x, E yrepresents the expectation of the data distribution of the room image y after style transformation, E x～pdata(x) represents the expected value calculated according to the data distribution pdata(x), pdata(x) represents a real data distribution about x, and D(y) represents the probability of the discriminator network output.

[0046] Specify the number of iterations to update the network parameters and save the trained model parameters.

[0047] Step 4: Select a room image that needs to be modified in decoration style, extract the corresponding semantic parsing features through the pre-trained VGG network, and then input the room image, semantic parsing features and the reference image with the migration target style into the generator network trained in step 3 to obtain the room image after the decoration style is migrated.

Claims

1. A room decoration style transfer method based on semantic and style information guidance, Features: The specific steps include: Step 1: Extract the semantic parsing features of the original room image, use the original room image, reference style image, and semantic parsing features as training samples, use the room image after changing the style as a label, and construct a training set; Step 2: construct a style transfer generative adversarial network model based on the reference style of room decoration; the generative adversarial network includes a generator network and a discriminator network; the generator network includes a generated room picture encoder, a reference style picture encoder, a dual-branch normalization module and a decoder; The room picture encoder and the reference style picture encoder are used to extract features from the original room picture and the reference style picture respectively to obtain room picture features and reference picture features; The dual-branch normalization module includes a class adaptive normalization module, an instance adaptive normalization module and an attention feature fusion AFF module; The decoder decodes the features fused by the AFF feature fusion module to obtain a room image with a reference style; The discriminator network is used to discriminate the image output by the decoder; Step 3: Input the training set data in step 1 into the generative adversarial network model constructed in step 2 for training, and save the model parameters obtained by training; Step 4: Select a room image that needs to be modified in decoration style, extract the corresponding semantic parsing features, and then input the room image, semantic parsing features and the reference image with the migration target style into the generator network trained in step 3 to obtain the room image after the decoration style is migrated.

2. A room decoration style migration method based on semantic and style information guidance as claimed in claim 1, Features: Use the pre-trained VGG network to extract semantic parsing features of the original room image.

3. A room decoration style migration method based on semantic and style information guidance as claimed in claim 1, Features: The room image encoder and the reference style image encoder have the same structure, both including four cascaded encoding units, each encoding unit including a convolution layer with a convolution kernel of 3, a stride of 2, and a padding of 1, an instance normalization layer and an activation function ReLU.

4. A room decoration style migration method based on semantic and style information guidance as claimed in claim 1, Features: The class adaptive normalization module first performs class adaptive normalization Clade processing on the input semantic parsing features and reference image features, and then passes through the ReLU activation function and the convolution layer. Then, the convolved features are Clade processed together with the semantic parsing features, and then pass through the ReLU activation function and the convolution layer. The obtained results are residually connected with the reference image features to output spatial structure information.

5. A room decoration style migration method based on semantic and style information guidance as claimed in claim 1, Features: The instance adaptive normalization module first performs instance adaptive normalization Adain processing on the room photo features and the reference image features, and then passes through a convolution layer and a ReLU activation function, and then performs a residual connection with the input room photo features after passing through a convolution layer and a ReLU activation function, and then outputs texture detail information after passing through a ReLU activation function.

6. A room decoration style migration method based on semantic and style information guidance as claimed in claim 1, Features: The AFF fusion module first splices the spatial structure information and texture detail information, calculates the global attention and the local attention respectively, splices the two attention calculation results and activates them through the sigmoid function, multiplies the activation results with the input spatial structure information and texture detail information respectively, and then splices them to obtain the fused features.

7. A room decoration style migration method based on semantic and style information guidance as claimed in claim 3, Features: The decoder includes 4 decoding units, a convolution layer with a convolution kernel size of 3, a padding of 1, a stride of 1, and an activation function Tanh; each decoding unit includes a transposed convolution layer with a convolution kernel of 3, a stride of 2, and a padding of 1, an instance normalization layer, and an activation function ReLU, which is used to output an upsampled feature map and concatenate it with the output feature map of the room picture encoder at the same level in the channel dimension as the input of the next decoding unit.

8. A room decoration style migration method based on semantic and style information guidance as claimed in claim 1, Features: Set the total loss function to: L total =λ 1 L adv +λ 2 L cyc +λ 3 L per Among them, λ 1 =1,λ 2 =0.5,λ 3 =0.5, is the weight coefficient of the loss function, L adv , L cyc , L per They represent adversarial loss, cycle consistency loss, and perceptual loss respectively.

9. A room decoration style migration method based on semantic and style information guidance as claimed in claim 1 or 8, Features: The Adam optimization algorithm is used to optimize the parameters of the generative adversarial network model.

Images