Household scene image generation method based on Stable Diffusion

Abstract

The invention discloses a home scene image generation method based on Stable Diffusion. The method comprises the following steps: firstly collecting a multi-style home scene image-text data set for a home scene generation task; then, on the basis of the data set, a Stable Diffusion model of the text graph is trained by adopting a LoRA training strategy; meanwhile, the content structure of the generated home scene image is controlled in combination with a ControlNet network, so that a Stable Diffusion model which effectively meets a home scene image generation task is obtained; the method comprises the following steps: combining a Stable Diffusion model network with a home scene generation problem, taking a multi-style home scene graph as initial noise, and taking a design constraint as a condition for generating a structural scene graph; according to the invention, a small sample training set is adopted, so that various home scene designs can be generated; meanwhile, the skill requirement for model training of the user is low, and common users can also be well trained and applied.

Description

Technical Field

[0001] The present invention relates to the field of generative artificial intelligence, and in particular to a method for generating home scene images based on Stable Diffusion. Background Art

[0002] Intelligent home scene design is a new technical direction and development trend in the field of architectural design. Traditional home scene design methods often require a lot of manpower and time investment, and it is difficult to meet the personalized needs of users. Therefore, how to quickly and efficiently intelligently generate home scene images with a specific design style has become a hot research issue in the current field of intelligent buildings.

[0003] In recent years, with the rapid development of artificial intelligence technology (AIGC), the text-driven image generation model has been applied in many fields. However, there is little research in the field of home scene design. Therefore, this patent adopts the Stable Diffusion model, combined with the LoRA training strategy and the ControlNet network module, to propose a method for generating home scene images based on Stable Diffusion. This method uses deep neural network technology to model and learn the style of home scene images. Through the input of text description and structural control information of the home scene, the proposed text-driven image model can automatically generate home scene images with a specific style, thereby simplifying the manual design process, improving design efficiency, lowering the design threshold, and providing users with a personalized home scene design experience.

[0004] The Stable Diffusion model is a representative deep network model of cultural graphs and has advanced image generation capabilities. However, the images generated by this model are diverse, difficult to control, and random, which leads to its insufficient generation effect in a specific field. Summary of the invention

[0005] The purpose of the present invention is to provide a method for generating home scene images based on Stable Diffusion to solve the following technical problems:

[0006] How to solve the problem of low style quality faced by the Wensheng graph model in generating home scene images, and improve the ability to generate home scene images based on Stable Diffusion.

[0007] The purpose of the present invention can be achieved through the following technical solutions:

[0008] (1) Collect home images and construct home scene image-text datasets of various home decoration styles;

[0009] (2) Setting model inference parameters;

[0010] (3) Use the home scene image-text dataset to train the Stable Diffusion model. When training the Stable Diffusion model, LoRA is used as the attention mechanism for fine-tuning the cross-attention layer between the encoder and decoder to obtain a fine-tuned diffusion model specifically for home scene images;

[0011] (4) using 3D modeling software to construct a texture-free home scene model and intercepting 2D images of the model from multiple perspectives to form a home scene preview image;

[0012] (5) Using the Canny operator to extract the edges of the home scene preview image to obtain a structural reference control benchmark image, and processing the home scene structure control information through the ControlNet network; inputting the home scene text prompt phrase or sentence description into the home scene image-specific fine-tuning diffusion model obtained based on step (3), and obtaining an initial home scene image that satisfies the text semantics; then inputting the home scene structure control information into the home scene image-specific fine-tuning diffusion model, and obtaining a home scene generation network that can be structurally controlled;

[0013] (6) The home scene generation network performs reasoning based on text description and structural control information to generate batches of target home scene images.

[0014] Prioritize, collect home images, and construct home scene image-text datasets of various home decoration styles. Specifically, collect high-quality home images of different home decoration styles from massive online home images, including new Chinese style, classical Chinese style, Nordic style, Bohemian style, Mediterranean style, American country style, French light luxury style, Japanese style, industrial style, Southeast Asian style and other N major categories. Each major category contains n images. After the collected images are cropped to 512×512 pixels in width and height, the WD1.4 labeler is used to annotate each image with text to form paired home scene image-text data.

[0015] Preferably, the preset Stable Diffusion basic model is trained using the home scene image-text dataset. Specifically, the Stable Diffusion V1.5 basic model is trained using the dataset obtained in (1) to obtain a fine-tuned diffusion model exclusive to home scenes.

[0016] Furthermore, the Stable Diffusion model is a potential diffusion model with a conditional mechanism. The potential diffusion model is a probabilistic model that learns the data distribution p(x) by gradually denoising a normally distributed variable, which is equivalent to the inverse process of a Markov chain of length T. In the field of image synthesis, the variational lower bound reweighted variant of p(x) is the mainstream method, which can be regarded as a series of denoising autoencoders ∈ θ (x t ,t); t = 1…T, trained to predict its input x t , where x t is a noisy version of the input x, t is uniformly sampled from {1,…,T}. Given an image x∈R in RGB space H×W×3 , the encoder ε encodes x into a latent representation z = ε(x), and the decoder Reconstruct the image in the latent space and get where z∈R h×w×c . The corresponding goal can be expressed as:

[0017]

[0018] The neural backbone of the model ∈ θ (z t ,t) is implemented in the form of a time-conditional UNet. Since the forward process is fixed, z can be effectively obtained from ε during training. t , just pass the decoder Decode p(z) into image space.

[0019] The diffusion model can in principle simulate the conditional distribution p(z│y), which requires a conditional denoising autoencoder ∈ θ (z t ,t,y) to achieve, where y can be text, semantic graph or other image-to-image translation tasks. For this purpose, a domain-specific encoder τ is introduced θ Let y be And mapped to the original UNet to form a Stable Diffusion model, the corresponding target is changed to:

[0020]

[0021] Among them, τ θ and ∈ θ This conditional mechanism is very flexible because τ θ It can be parameterized in specific fields, for example, when y is text, it can be parameterized using Transformer.

[0022] The cross-attention layer of the StableDiffusion model is fine-tuned using the matrix low-rank adaptation (LoRA) training strategy. The weight matrix of the pre-trained cross-attention layer is denoted as W0∈R k×d , d and k represent the input dimension and output dimension of the cross-attention layer respectively; for each cross-attention layer, an additional parallel network structure is designed, which includes two modules, and their weight matrices are recorded as A and B respectively; low-rank matrix decomposition ΔW = AB, where ΔW has the same size as W0, A∈R r×d , B∈R k×r And r<<min(d,k); W0 is frozen during training, A and B contain trainable parameters, for the input x∈R of the cross attention layer d×1 , its output h is:

[0023] h=W0x+ΔWx=W0x+BAx

[0024] Among them, the weight matrix A is initialized by random Gaussian distribution, and the weight matrix B is initialized to 0, so ΔW=AB is a 0 matrix when training starts.

[0025] Preferably, a texture-free home scene model is constructed using 3D modeling software and two-dimensional images of the model under multiple perspectives are captured to form a home scene preview image. Specifically, a 3D modeling software specially designed for building information modeling is used to perform structural design of the home scene, and the 3D model of the home scene is captured from multiple angles, and the obtained two-dimensional image is used as the home scene preview image.

[0026] Preferably, Canny is used to extract the edges of the home scene preview image to obtain a structural reference control benchmark image, and the home scene structural control information is obtained by processing it through the ControlNet network; the home scene text prompt phrase or sentence description is input into the home scene image-specific fine-tuning diffusion model obtained based on step (3), and an initial home scene image that satisfies the text semantics can be obtained; then the home scene structural control information is input into the home scene image-specific fine-tuning diffusion model to obtain a structurally controllable home scene generation network. Specifically, the style text description of the target home scene image is used to input the text encoder τ θ , the texture-free home scene model preview image x0 obtained in step (4) is edge-processed to obtain the structural reference control benchmark image c i Input the image encoder ε. Input the contents generated by the text encoder and the image encoder into the structure-controllable home scene generation network respectively, and obtain the home scene graph that conforms to the original structure reference control benchmark graph, and finally generate batches of high-quality target home scene renderings.

[0027] Prioritizing, the home scene generation network performs reasoning based on text description and structural control information to generate batches of target home scene images. Specifically, multi-phrase or sentence descriptions are designed to describe the required home scene, and then reasoning is performed in the structurally controllable home scene generation network through the DPM++2M Karras sampler to finally obtain the required home scene images in batches.

[0028] Beneficial effects of the present invention:

[0029] (1) The present invention aims at the task of generating home scenes. It is necessary to first collect a multi-style home scene image-text dataset; then, based on the dataset, the LoRA training strategy is used to train the Stable Diffusion model of the text graph; at the same time, the content structure of the home scene image is generated by combining the ControlNet network control, so as to obtain a Stable Diffusion model that effectively meets the task of generating home scene images; by combining the Stable Diffusion model network with the home scene generation problem, the multi-style home scene graph is used as the initial noise, and the design constraints are used as the conditions for generating the structural scene graph;

[0030] (2) The present invention uses a small sample training set to generate a variety of home scene designs. At the same time, the skill requirements for users to train the model are relatively low, and ordinary users can also train and apply it well. The design can basically meet the design requirements and has similar design effects as other generative methods. After fine-tuning, better results will be obtained, achieving the purpose of auxiliary design.

[0031] (3) The present invention uses a method that combines computer graphics and artificial intelligence. Designers only need to complete the design of several keywords in advance to automatically generate home scene images with target style text descriptions in batches. Compared with existing methods, this method has the characteristics of high efficiency, simple operation, and high generation quality. In addition, the images generated by this method have higher resolution, better quality, and less noise, which better meet design requirements.

[0032] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0034] Figure 1This is a flow chart of a method for generating a home scene image based on Stable Diffusion according to the present invention;

[0035] Figure 2 This is a schematic diagram of the network structure based on the Stable Diffusion model of the present invention;

[0036] Figure 3 This is an example of a home scene image and label in an embodiment of the present invention;

[0037] Figure 4 It is a preview image of a home scene and a control background image in an embodiment of the present invention;

[0038] Figure 5 This is a schematic diagram of the ControlNet network operation mode in the present invention;

[0039] Figure 6 An example of generating a new Chinese style home scene image in an embodiment of the present invention;

[0040] Figure 7 This is an example of generating a Japanese-style home scene image in an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0042] See also Figure 1 As shown, the present invention is a method for generating a home scene image based on Stable Diffusion, comprising the following steps:

[0043] S01, collect home images and build home scene image-text datasets of various home decoration styles.

[0044] This step is divided into two small steps. First, high-quality home scene images of various home decoration styles are collected from massive online home images, including 10 major categories: new Chinese style, classical Chinese style, Nordic style, Bohemian style, Mediterranean style, American country style, French light luxury style, Japanese style, industrial style, and Southeast Asian style. Each major category contains 30 images; then the images are uniformly cropped to 512*512 pixel size, and the WD1.4 labeler is used to annotate each image with text. The parameter threshold is selected as 0.4, and manual modification and adjustment are performed on individual inaccurate or erroneous annotations to obtain the best text annotation effect.

[0045] S02, setting model inference parameters.

[0046] The inference parameters are designed by using the recommended values; the width and height of the generated image are set to 512 pixels, the CFG Scale is set to 7, and the number of random seeds is set to -1. In the inference sampling stage, the DPM++2M Karras sampler is selected to generate home scene images. This sampler has a slow convergence speed but high accuracy. At the same time, after multiple verifications, it is found that the generation quality of home scene images is better when the number of Clip termination layers is set to 4.

[0047] S03, using the home scene image-text dataset, train the Stable Diffusion model. When training the Stable Diffusion model, LoRA is used as the attention mechanism for fine-tuning the cross attention layer between the encoder and the decoder to obtain a fine-tuned diffusion model specifically for home scene images.

[0048] like Figure 2 As shown in Figure 1, the Stable Diffusion model is a potential diffusion model with a conditional mechanism. The potential diffusion model is a probabilistic model that learns the data distribution p(x) by gradually denoising a normally distributed variable, which is equivalent to the inverse process of a Markov chain of length T. In the field of image synthesis, the variational lower bound reweighted variant of p(x) is the mainstream method, which can be regarded as a series of denoising autoencoders ∈ θ (x t ,t); t = 1…T, trained to predict its input x t , where x t is a noisy version of the input x, t is uniformly sampled from {1,…,T}. Given an image x∈R in RGB space H×W×3 , the encoder ε encodes x into a latent representation z = ε(x), and the decoder Reconstruct the image in the latent space and get where z∈R h×w×c The corresponding goal can be expressed as

[0049]

[0050] The neural backbone of the model ∈ θ (z t ,t) is implemented in the form of a time-conditional U-Net. Since the forward process is fixed, z can be effectively obtained from ε during training. t , just by p(Z) can be decoded into image space.

[0051] The diffusion model can in principle simulate the conditional distribution p(z│y), which requires a conditional denoising autoencoder ∈ θ (z t ,t,y) to achieve this, where y can be text, semantic graph or other image-to-image translation tasks. For this purpose, a domain-specific encoder τ is introduced θ Let y be And mapped to the original U-Net to form a large model StableDiffusion, the corresponding target is also changed to

[0052]

[0053] Among them, τ θ and ∈ θ This conditional mechanism is very flexible because τ θ It can be parameterized in specific fields, for example, when y is text, it can be parameterized using Transformer.

[0054] The Stable Diffusion model includes VAE, a text encoder, an image decoder and an image information generator. VAE maps the home scene graph to the latent space to obtain a pure noise vector, and the text encoder converts the text description into a semantic vector. The pure noise vector and the semantic vector are concatenated as a conditional vector. The image information generator uses a denoising U-Net network and a sampler algorithm to perform a diffusion process in the latent space, and gradually diffuses from the latent space to the original space through the diffusion process, and gradually generates a latent vector containing semantic information from the pure noise vector. In each step of diffusion, an image decoder is used to generate a denoising function based on the conditional vector and the current image state to repair the noise in the image and make the image closer to the target image. After multiple steps of diffusion, VAE decodes the latent vector into the final home scene image. This embodiment uses LoRA to fine-tune the Stable Diffusion model, and the weight matrix of the pre-trained cross-attention layer is denoted as W0∈R k×d , d and k represent the input dimension and output dimension of the cross attention layer respectively; for each cross attention layer, a parallel network structure is designed, which consists of two modules, whose weight matrices are recorded as A and B respectively; low-rank matrix decomposition ΔW = AB, where ΔW has the same size as W0, A∈R r×d , B∈R k×r And r<<min(d,k); W0 is frozen during training, A and B contain trainable parameters, for the input x∈R of the cross attention layer d×1 , its output h is:

[0055] h=W0x+ΔWx=W0x+BAx (3)

[0056] Among them, the weight matrix A is initialized by random Gaussian distribution, and the weight matrix B is initialized to 0, so ΔW=AB is a 0 matrix when training starts.

[0057] In addition, since Stable Diffusion is more complex and flexible, there are many parameters that need to be adjusted. After repeated experiments, some important parameters are as follows: The Stable Diffusion model version selects the V1-5-pruned model, which has been verified to be suitable for the design of home scenes. It is also found by experiments that satisfactory results can be obtained when the number of repetitions is 8 and the number of training rounds is 16, that is, a single image is trained 128 times. It should be noted that overfitting or underfitting may occur during the training process, so the LoRA model at different training stages (every two rounds) will be saved, which makes it easier for users to choose the most appropriate fine-tuning model.

[0058] S04, using 3D modeling software to construct a home scene model without texture mapping and intercepting 2D images of the model under multiple viewing angles to form a home scene preview image.

[0059] Revit, a software specially designed for BIM building information modeling, is used for the structural design of the home scene. A building template is established in Revit, and a three-dimensional home scene model is established by combining self-built home scene elements with home elements in the expansion dock. The two-dimensional images obtained by multi-angle capture are used as preview images of the home scene.

[0060] S05, use the Canny operator to extract the edges of the home scene preview image to obtain a structural reference control benchmark image, and obtain the home scene structural control information through ControlNet network processing; input the home scene text prompt phrase or sentence description into the home scene image exclusive fine-tuning diffusion model obtained based on step S03, and obtain the initial home scene image that meets the text semantics; then input the home scene structural control information into the home scene image exclusive fine-tuning diffusion model to obtain a structurally controllable home scene generation network.

[0061] like Figure 5 As shown in Figure 2, ControlNet works in a similar way to LoRA, that is, an additional branch is added to the pre-trained network, and the style text description of the target home scene image is used as the input to the text encoder τ θ , using the texture-free home scene model preview image x0 to obtain the structural reference control benchmark image c after edge processing iInput image encoder ε. By supplementing the large model with end-to-end training, additional control information for specific tasks, such as depth map, semantic segmentation map, key points, etc., can be learned while supporting a small data set of less than 100k. This step S05 uses the control base map as the structural reference control information, and based on the "home scene image-specific fine-tuning diffusion model" obtained in step S03, integrates the ControlNet that uses the structural reference control benchmark map for structural control; the resulting model of this step S05 is called the "structurally controllable home scene generation network".

[0062] Use the style text description of the target home scene image to input the text encoder, use the structural reference control benchmark map to input the image encoder, and input the contents generated by the text encoder and image encoder into the structurally controllable home scene generation network respectively to obtain a home scene map that conforms to the original structural reference control benchmark map, and finally generate batches of high-quality target home scene renderings.

[0063] S06, the home scene generation network performs reasoning based on the text description and structural control information to generate batches of target home scene images.

[0064] Design multi-word and sentence descriptions to describe the required home scenes, and then use the DPM++2M Karras sampler to perform inference in the home scene-specific fine-tuned diffusion model to finally obtain the home scene images required for batching. Figure 6 , Figure 7 As shown in the figure, the unified field "A living room with leather sofas, mahogany dining table and chairs, a TV cabinet, and artistic potted plants" and the special field "New Chinese Style" or "Japanese Style" are designed to generate home scene images, and two types of home scene images of new Chinese style and Japanese style are obtained. The two types of images are consistent in the picture structure, which is consistent with the control base map constructed by the modeling software Revit. At the same time, they have a good reflection of the content of the unified field. The style differentiation comes from the different designs of special fields.

[0065] For each inference, the input data of the model includes: the text description of the target home scene and the target home scene structure reference control benchmark map. The specific input data corresponds to the following modules: the target home scene image is generated using the text description of the target home scene and the target home scene LoRA model, and the target home scene structure reference control benchmark map is used as the control information bitmap of the structure-controllable home scene generation network. At the same time, the style text description of the target home scene is input into the text encoder, and the target home scene structure reference control benchmark map that has been designed for the target home scene is input into the image encoder to obtain the final result.

[0066] The above contents are merely examples and explanations of the concept of the present invention. The technicians in this technical field may make various modifications or additions to the specific embodiments described or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the claims, they should all fall within the protection scope of the present invention.

Claims

1. A method for generating home scene images based on Stable Diffusion, characterized in that: The following steps are involved: S1. Collect home images and build home scene image-text datasets of various home decoration styles; S2, setting model inference parameters; S3. Use the home scene image-text dataset to train the preset Stable Diffusion basic model to obtain a fine-tuned diffusion model exclusive to home scene images; S4, using 3D modeling software to construct a home scene model without texture mapping and intercepting 2D images under multiple viewing angles to form a corresponding home scene preview image; S5. Use the Canny operator to extract the edge of the preview image to obtain a structural reference control benchmark image, and obtain the home scene structural control information through ControlNet network processing; Input the home scene text prompt phrase or sentence description into the home scene image-specific fine-tuning diffusion model to obtain an initial home scene image that meets the text semantics; then input the home scene structure control information into the home scene image-specific fine-tuning diffusion model to obtain a structure-controllable home scene generation network; S6. The home scene generation network performs reasoning based on text description and structural control information to generate batches of target home scene images.

2. The method for generating home scene images based on Stable Diffusion according to claim 1, characterized in that: In step S1, the home scene image-text datasets of various home decoration styles include N major categories, namely, new Chinese style, classical Chinese style, Nordic style, Bohemian style, Mediterranean style, American country style, French light luxury style, Japanese style, industrial style, and Southeast Asian style, and each major category contains n images.

3. The method for generating home scene images based on Stable Diffusion according to claim 1, characterized in that: In step S2, the model inference parameters are designed using the image inference recommended values ​​set by the Stable Diffusion model by default, that is, the width and height of the generated image are set to 512, the prompt word guidance coefficient is set to 7, and the random seed number is set to -1; in the inference sampling stage, the DPM++2M Karras sampler is selected to generate the home scene image.

4. The method for generating home scene images based on Stable Diffusion according to claim 1, characterized in that: The Stable Diffusion model in step S3 includes: a variational autoencoder, a text encoder and an image information generator; The variational autoencoder maps the home scene graph to a latent space to obtain a pure noise vector; The text encoder converts the text description into a semantic vector, and concatenates the pure noise vector with the semantic vector as a conditional vector; The image information generator uses a denoising U-Net network and a sampler algorithm to perform an inverse diffusion process on the conditional vector in the latent space, and gradually diffuses from the latent space to the original space through the inverse diffusion process, so as to gradually generate a latent vector containing semantic information from the conditional vector; After multiple steps of inverse diffusion, the variational autoencoder decodes the latent vector into a home scene image.

5. The method for generating home scene images based on Stable Diffusion according to claim 1, characterized in that: In step S3, the cross attention layer of the Stable Diffusion model is fine-tuned by adopting a matrix low-rank adaptive training strategy: The weight matrix of the pre-trained cross attention layer is denoted as W0∈R k×d , d and k represent the input dimension and output dimension of the cross-attention layer respectively; for each cross-attention layer, an additional parallel network structure is designed, which includes two modules, and their weight matrices are recorded as A and B respectively; low-rank matrix decomposition ΔW = AB, where ΔW has the same size as W0, A∈R r×d , B∈R k×r And r<<min(d,k); W0 is frozen during training, A and B contain trainable parameters, for the input x∈R of the cross attention layer d×1 , its output h is: h=W0x+ΔWx=W0x+BA x Among them, the weight matrix A is initialized by random Gaussian distribution, and the weight matrix B is initialized to 0, so ΔW=AB is a 0 matrix when training starts.

6. The method for generating home scene images based on Stable Diffusion according to claim 1, characterized in that: The step S4 includes: The home scene model is constructed using three-dimensional modeling software, and two-dimensional images of the model under multiple perspectives are captured to form the corresponding home scene preview image.

7. The method for generating home scene images based on Stable Diffusion according to claim 1, characterized in that: The step S5 includes: The Canny operator is used to extract the edges of the home scene preview image to obtain a structural reference control benchmark image, and the home scene structural control information is obtained through ControlNet network processing; the home scene text prompt phrases or sentence descriptions are input into the text encoder to obtain a semantic vector, and then input into the home scene image-specific fine-tuning diffusion model to obtain an initial home scene image that satisfies the text semantics; then the home scene structural control information is input into the home scene image-specific fine-tuning diffusion model to obtain a structurally controllable home scene generation network.

8. The method for generating home scene images based on Stable Diffusion according to claim 1, characterized in that: The step S6 includes: The home scene generation network performs reasoning based on text description and structural control information. After sampling by the sampler, batches of target home scene images can be inferred and generated in the structurally controllable home scene generation network.

Images