A room decoration style transfer method guided by semantic and style information

Abstract

The present invention proposes a room decoration style migration method based on semantic and style information guidance. The method first extracts the semantic parsing features of the original room image, and then uses the original room image, reference style image and semantic parsing features as training samples, and uses the room image after changing the style as a label to construct a training set. The generator network is used to extract room photo features and reference image features. The dual-branch normalization module is used to adjust the normalization process with the reference image features to obtain spatial structure information and texture detail information. Then, through AFF feature fusion, the room image with the reference style is finally output. The discriminator network is used to discriminate the generated image based on the label to complete the network training. Finally, the original room image and its semantic parsing features, as well as the reference style image, are input into the trained generator network to generate a room image with the reference style, completing the decoration style migration.

Description

Technical Field

[0001] The present invention belongs to the field of image processing technology, and in particular relates to a room decoration style migration method guided by semantic and style information. Background Art

[0002] In recent years, the application of computer vision in the interior decoration industry has become increasingly widespread. A room's overall style determines its visual impact and atmosphere. Transferring the style of room images helps achieve personalized and diverse interior design. Traditional non-parametric image style transfer methods are primarily based on stroke rendering, image classification or filtering, and texture synthesis. These methods typically only extract low-level image features and struggle to generate high-level, abstract features. Therefore, they are only suitable for artistic image style transfer and are ineffective in modern home design, failing to meet practical needs.

[0003] The emergence of deep learning has brought about new image style transfer methods. Gatys proposed an image style transfer algorithm based on convolutional neural networks (CNNs), which extracts image features at different levels to achieve the separation of style and content. However, the images generated by this method often have problems such as noise and unclearness, and require a large number of paired training images, which is slow to train and takes up a lot of memory. The Cycle Generative Adversarial Network (CycleGAN) proposed by Zhu et al. combines GAN and dual learning, using two generators and two discriminators to convert between two style domains and retaining the content information of the image through cycle consistency loss. However, this method still has problems with structural integrity and reasonable texture details. Modern home design is complex and emphasizes realism and verisimilitude. Existing methods have difficulty in effectively preserving the structural information of images and generating realistic textures. Therefore, it is of great significance to develop an image style transfer method that can simultaneously preserve the structural information of room images and generate realistic style textures. Summary of the Invention

[0004] To address the shortcomings of the existing technology, the present invention proposes a room decoration style transfer method guided by semantic and style information. This method uses a dual-branch structure to independently process spatial structure and texture details, so that the generated room images are delicate and realistic in local details, while having consistency and coherence in the overall picture.

[0005] A room decoration style transfer method based on semantic and style information guidance, the specific method is as follows:

[0006] Step 1: Use the pre-trained model to extract features of the original room image and obtain the corresponding semantic parsing features. Use the original room image, reference style image, and semantic parsing features as training samples, and use the room image after the style change as the label to construct a training set.

[0007] Step 2: Build a generative adversarial network model for style transfer based on the reference style of the room decoration. The generative adversarial network includes a generator network and a discriminator network. The generator network includes a generated room image encoder, a reference style image encoder, a dual-branch normalization module (DBN), and a decoder.

[0008] The room picture encoder and the reference style picture encoder have the same structure and are used to extract features from the original room picture and the reference style picture respectively to obtain room photo features and reference picture features.

[0009] The dual-branch normalization module includes a class adaptive normalization module, an instance adaptive normalization module and an attention feature fusion AFF module (Attentional Feature Fusion, AFF).

[0010] The class-adaptive normalization module normalizes the reference image features based on the semantic parsing features to provide spatial structure information. It first performs class-adaptive normalization Clade processing on the input semantic parsing features and reference image features, then passes them through a ReLU activation function and a convolutional layer with a kernel size of 3, padding of 1, and stride of 1. The convolved features are then Clade processed together with the semantic parsing features, and then passed through a ReLU activation function and a convolutional layer with a kernel size of 3, padding of 1, and stride of 1. The resulting result is then residually connected with the reference image features to output the spatial structure information.

[0011] The instance adaptive normalization module uses information in the reference image features to adjust the normalization process of the room photo features to provide texture detail information. First, the input room photo features and reference image features are subjected to instance adaptive normalization Adain processing. Then, they pass through a convolution layer with a convolution kernel size of 3, a padding of 1, and a stride of 1 and a ReLU activation function. Then, after passing through a convolution layer with a convolution kernel size of 3, a padding of 1, and a stride of 1 and a ReLU activation function, a residual connection is performed with the input room photo features. After passing through the ReLU activation function, texture detail information is output.

[0012] The AFF feature fusion module is used to perform feature fusion on the spatial structure information output by the class adaptive normalization module and the texture detail information output by the instance adaptive normalization module.

[0013] The decoder decodes the features fused by the AFF feature fusion module to obtain a room image with a reference style.

[0014] The discriminator network is used to discriminate the image output by the decoder, making the pictures generated by the generator network more realistic and with more delicate details.

[0015] Step 3: Input the training set data in step 1 into the generative adversarial network model constructed in step 2, and use the Adam optimization algorithm to optimize the network model. The loss functions used include the perceptual loss, adversarial loss, and cycle consistency loss of the generated image, and save the model parameters obtained by training.

[0016] Step 4: Select an image of a room whose decoration style needs to be modified, extract the corresponding semantic parsing features through the pre-trained model, and then input the room image, semantic parsing features, and a reference image with the transfer target style into the generator network trained in step 3 to obtain the room image after the decoration style is transferred.

[0017] The present invention has the following beneficial effects:

[0018] This method decouples spatial information from detailed texture information through a two-branch normalization module. The first branch uses the reference style to generate instance-adaptive weights, which are then applied to the input image for instance normalization, resulting in a more refined and realistic generated room image with local details. The second branch uses semantic parsing of the room image to generate class-adaptive weights, which are then applied to the reference image for class-adaptive normalization, ensuring the overall consistency and coherence of the generated room image. These two separate processes fully account for the independent influence of texture and spatial information. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of the room decoration style transfer method guided by semantic and style information;

[0020] Figure 2 Schematic diagram of the generator network structure constructed in the embodiment;

[0021] Figure 3 Schematic diagram of the dual-branch normalization module structure constructed in the embodiment;

[0022] Figure 4 2 is a schematic diagram of the structure of a feature fusion module based on AFF in an embodiment;

[0023] Figure 5 Schematic diagram of the discriminator network structure constructed in the embodiment. DETAILED DESCRIPTION

[0024] The present invention will be further explained below with reference to the accompanying drawings;

[0025] like Figure 1 As shown in FIG, a room decoration style transfer method based on semantic and style information guidance includes the following steps:

[0026] Step 1: Use a pre-trained VGG network to extract features from the original room image and obtain corresponding semantic parsing features. Collect room images with various decor styles as reference style images. Use the original room images, reference style images, and semantic parsing features as training samples, and use the style-modified room images as labels to construct a training set.

[0027] Step 2: Construct a style transfer generative adversarial network model based on the reference style of room decoration; the generative adversarial network includes a generator network and a discriminator network. Figure 2 As shown, the generator network includes a generative room picture encoder, a reference style picture encoder, a dual-branch normalization module (DBN) and a decoder.

[0028] The room image encoder and reference style image encoder share the same structure, consisting of four cascaded encoding units. Each encoding unit includes a convolutional layer with a kernel size of 3, a stride of 2, and a padding of 1, an instance normalization layer, and a Reluctant Unified Unit (ReLU) activation function. The room image encoder and reference style image encoder are used to extract features from the original room image and reference style image, respectively, to obtain room photo features and reference image features. The room image encoder is used to obtain contextual structural information of the original room image, while the reference style image encoder is used to obtain contextual information about the color style of the reference style image.

[0029] like Figure 3 As shown, the dual-branch normalization module includes a class adaptive normalization module, an instance adaptive normalization module and an attention feature fusion AFF module.

[0030] The class adaptive normalization module first performs semantic parsing on the input features M p , reference image feature F r After Clade processing, the image passes through a ReLU activation function and then a convolution layer with a kernel size of 3, padding of 1, and stride of 1. The convolved features are then Clade processed together with the semantic parsing features. After another ReLU activation function and a convolution layer with a kernel size of 3, padding of 1, and stride of 1, the result is residually connected with the reference image features to output spatial structure information. The Clade processing method is as follows:

[0031]

[0032] Among them, F CLADE Represents the features obtained after Clade processing, μ() represents the mean of the features, and σ() represents the variance of the features. represents the scale parameter of the i-th category in the semantic parsing graph, Represents the offset parameter of the i-th category in the semantic parsing graph. Both are optimized as a regular network parameter through a modulation parameter library.

[0033] The instance adaptive normalization module first performs a normalization on the input room photo features F p and reference image features F r Adain processing is performed, and then a convolution layer with a kernel size of 3, padding of 1, and stride of 1 is passed through, and a ReLU activation function is used. After passing through a convolution layer with a kernel size of 3, padding of 1, and stride of 1 and a ReLU activation function, a residual connection is made with the input room photo features, and after another ReLU activation function, texture detail information is output. The Adain processing method is:

[0034]

[0035] Among them, F AdaIN Represents the features after Adain processing.

[0036] The two processes local texture details and overall spatial information separately to fully consider the influence of the independence of texture information and spatial information. Since the decoupling of structure and texture will inevitably produce redundancy, in order to avoid information redundancy and further utilize the complementary advantages of the two branches, the following is adopted: Figure 4 The AFF fusion module shown in the figure performs feature fusion on the spatial structure information output by the class adaptive normalization module and the texture detail information output by the instance adaptive normalization module. The AFF fusion module first concatenates the input spatial structure information and texture detail information, calculates global attention and local attention respectively, concatenates the two attention calculation results, activates them through a sigmoid function, multiplies the activation results with the input spatial structure information and texture detail information respectively, and then concatenates them to output the fused features.

[0037] The decoder decodes the features fused by the AFF feature fusion module to obtain a room image with a reference style. The decoder includes four decoding units, a convolutional layer with a kernel size of 3, a padding of 1, and a stride of 1, and an activation function Tanh. Each decoding unit contains a transposed convolutional layer with a kernel size of 3, a stride of 2, and a padding of 1, an instance normalization layer, and an activation function ReLU. It is used to output an upsampled feature map, which is concatenated with the output feature map of the room image encoder at the same level in the channel dimension and serves as the input of the next decoding unit.

[0038] like Figure 5As shown in the figure, the discriminator network is used to discriminate the image output by the decoder, making the images generated by the generator network more realistic and detailed. The discriminator network includes five groups of convolutional layers with a kernel size of 3, a stride of 2, and a padding of 2, a batch normalization layer module, and a LeakeyReLU with a slope of 0.2.

[0039] Step 3: Use the normal distribution (Normal) to initialize the generative adversarial network model constructed in step 2, and its weight W satisfies the following distribution:

[0040] W~N(μ,σ 2 )(3)

[0041] Among them, μ = 0, is the mean of the distribution, σ 2 =0.02, which is the variance.

[0042] Input the training set data in step 1 into the initialized generative adversarial network model, use the Adam optimization algorithm to optimize the network model, and set the total loss function to:

[0043] L total =λ1L adv +λ2L cyc +λ3L per (4)

[0044]

[0045] Among them, λ1=1, λ2=0.5, λ3=0.5 are the weight coefficients of the loss function, L adv , L cyc , L per denotes adversarial loss, cycle consistency loss, and perceptual loss, respectively. x is the input original room image, y is the room image after style transformation, r is the reference style image, M(x) is the semantic parsing feature corresponding to the original room image, G() is the generator network, F() is the generator network transformed back to the source domain, φ j () indicates that the j-th layer intermediate features of the pre-trained VGG network are used to calculate the perceptual distance between features, C j H represents the number of channels of the j-th layer feature map of the VGG network, j Indicates the width of the j-th layer feature map of the VGG network, W j Represents the height of the feature map of the jth layer of the VGG network, r x represents the reference style image related to the original room image x, r y represents the reference style image related to the room image y after style transformation, E x represents the expectation of the data distribution of the original room image x, E yrepresents the expectation of the data distribution of the room image y after style transformation, E x～pdata(x) represents the expected value calculated based on the data distribution pdata(x), pdata(x) represents a real data distribution about x, and D(y) represents the probability output by the discriminator network.

[0046] Specify the number of iterations to update the network parameters and save the trained model parameters.

[0047] Step 4: Select an image of a room whose decoration style needs to be modified, extract the corresponding semantic parsing features through the pre-trained VGG network, and then input the room image, semantic parsing features, and the reference image with the transfer target style into the generator network trained in step 3 to obtain the room image after the decoration style is transferred.

Claims

1. A room decoration style transfer method based on semantic and style information guidance, characterized by: The specific steps include: Step 1: Extract semantic parsing features from the original room image. Use the original room image, reference style image, and semantic parsing features as training samples, and use the style-modified room image as a label to construct a training set. Step 2: Construct a style transfer generative adversarial network model based on the reference style of room decoration; the generative adversarial network includes a generator network and a discriminator network; the generator network includes a generated room image encoder, a reference style image encoder, a dual-branch normalization module and a decoder; The room picture encoder and the reference style picture encoder are used to extract features from the original room picture and the reference style picture respectively to obtain room photo features and reference picture features; The dual-branch normalization module includes a class adaptive normalization module, an instance adaptive normalization module and an attention feature fusion AFF module; The decoder decodes the features fused by the attention feature fusion AFF module to obtain a room image with a reference style; The discriminator network is used to discriminate the image output by the decoder; Step 3: Input the training set data in step 1 into the generative adversarial network model constructed in step 2 for training, and save the model parameters obtained by training; Step 4: Select an image of a room whose decoration style needs to be modified, extract the corresponding semantic parsing features, and then input the room image, semantic parsing features, and a reference image with the target style into the generator network trained in step 3 to obtain the room image after the decoration style is transferred.

2. The method for room decoration style transfer based on semantic and style information guidance as claimed in claim 1, characterized in that: Use the pre-trained VGG network to extract semantic parsing features of the original room image.

3. The method for room decoration style transfer based on semantic and style information guidance as claimed in claim 1, characterized in that: The room image encoder and the reference style image encoder have the same structure, both including four cascaded encoding units. Each encoding unit includes a convolution layer with a convolution kernel of 3, a stride of 2, and a padding of 1, an instance normalization layer, and an activation function ReLU.

4. The method for room decoration style transfer based on semantic and style information guidance as claimed in claim 1, characterized in that: The class-adaptive normalization module first performs class-adaptive normalization Clade processing on the input semantic parsing features and reference image features, then passes through the ReLU activation function and convolution layer, and then performs Clade processing together with the convolved features and the semantic parsing features, and then passes through the ReLU activation function and convolution layer. The obtained results are residually connected with the reference image features to output spatial structure information.

5. The method for room decoration style transfer based on semantic and style information guidance as claimed in claim 1, characterized in that: The instance adaptive normalization module first performs instance adaptive normalization Adain processing on the room photo features and the reference image features, then passes through a convolutional layer and a ReLU activation function, and then performs a residual connection with the input room photo features after passing through another convolutional layer and a ReLU activation function. After another ReLU activation function, texture detail information is output.

6. The method for room decoration style transfer based on semantic and style information guidance as claimed in claim 1, characterized in that: The attention feature fusion AFF module first splices the spatial structure information and texture detail information, calculates the global attention and local attention respectively, splices the two attention calculation results and activates them through the sigmoid function, multiplies the activation results with the input spatial structure information and texture detail information respectively, and then splices them to obtain the fused features.

7. The method for room decoration style transfer based on semantic and style information guidance as claimed in claim 3, characterized in that: The decoder includes 4 decoding units, a convolution layer with a convolution kernel size of 3, a padding of 1, and a stride of 1, and an activation function Tanh; each decoding unit contains a transposed convolution layer with a convolution kernel size of 3, a stride of 2, and a padding of 1, an instance normalization layer, and an activation function ReLU, which is used to output an upsampled feature map, which is concatenated with the output feature map of the room image encoder at the same level in the channel dimension as the input of the next decoding unit.

8. The method for room decoration style transfer based on semantic and style information guidance as claimed in claim 1, characterized in that: Set the total loss function to: ； in, , is the weight coefficient of the loss function, denote adversarial loss, cycle consistency loss, and perceptual loss, respectively.

9. The method for room decoration style transfer based on semantic and style information guidance according to claim 1 or 8, characterized in that: The Adam optimization algorithm is used to optimize the parameters of the generative adversarial network model.

Images