Method and device for generating interior design rendering graph based on diffusion model

Abstract

The invention discloses an interior design rendering graph generation method and device based on a diffusion model, and the method comprises the steps: collecting the data of a historical interior design decoration rendering graph, carrying out the style classification, and carrying out the fine adjustment of a LoRA adapter of each style; extracting Canny operator features and depth image features of the indoor blank image, and generating control conditions; identifying a fixed structure in the indoor blank image by using the image segmentation model, and marking mask data; a control condition, mask data and prompt are input, and a decoration rendering graph fitting the indoor structure is generated; setting a selectable decoration style according to the LoRA adapter, and inserting the LoRA in the workflow to adjust the style of the rendering graph; and setting and adjusting hyper-parameters to ensure that the structural layout of the rendered image is consistent with the indoor blank image and improve the detail information and the image quality of the rendered image. The method can improve the generation effect and practicability of the interior design rendering graph, and can be widely applied to the technical field of computers.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to a method and device for generating interior design renderings based on a diffusion model. Background Art

[0002] In the field of modern interior design, rough house design has always been an important and challenging topic. The design of rough houses needs to take into account multiple aspects such as spatial layout, functional planning, and style matching, while traditional design methods often rely on the designer's experience and manual drawing, which is time-consuming and laborious. At the same time, with the increasing demand of consumers for personalized and high-quality indoor environments, designers are facing greater pressure and challenges. How to efficiently generate high-quality interior design renderings has become an urgent problem to be solved in the industry. With the rapid development of artificial intelligence and deep learning technologies, computer vision has made significant progress in the fields of image generation and image editing. Among them, the Stable Diffusion model, as an advanced diffusion model, generates high-quality images through a gradual denoising process and performs well in many image generation tasks. However, relying solely on the pre-trained Stable Diffusion model for rough house interior design rendering still faces some limitations, such as the inability to accurately control the specific elements and styles in the image. Summary of the invention

[0003] The main purpose of the embodiments of the present invention is to propose a method and device for generating interior design renderings based on a diffusion model, which can improve the generation effect and practicality of interior design renderings.

[0004] To achieve the above object, an embodiment of the present invention provides a method for generating an interior design rendering based on a diffusion model, comprising the following steps:

[0005] Collect data on historical interior design and decoration renderings and classify their styles, and fine-tune LoRA adapters for each style;

[0006] Input an indoor rough image to be decorated, extract the Canny operator features and depth image features of the indoor rough image, and use the Canny operator features and depth image features as control conditions for subsequent rendering generation;

[0007] Using an image segmentation model to identify fixed structures in the indoor roughcast image and mark mask data;

[0008] Using the pre-trained Stable Diffusion model, combined with Controlnet and Stable Inpainting workflows, input the control conditions, the mask data and prompt to generate a decoration rendering that fits the interior structure;

[0009] Set the optional decoration style according to the LoRA adapter, and insert LoRA into the workflow to adjust the rendering style according to the style options;

[0010] Set and adjust hyperparameters to ensure that the structural layout of the rendering is consistent with the rough interior image and to improve the details and image quality of the rendering.

[0011] In some embodiments, the process of collecting historical interior design and decoration rendering data and classifying the styles, and fine-tuning LoRA adapters of different styles respectively, includes the following steps:

[0012] Classifying the historical interior design and decoration renderings according to products and design styles to obtain style data of each style series, wherein the design styles include pastoral style, new Chinese style, and Mediterranean style;

[0013] The style data is subdivided according to the space type of the apartment, where the space type includes bedroom, living room, kitchen, and bathroom;

[0014] Based on the style and room type, fine-tune to get the LoRA adapter corresponding to "style_room type".

[0015] In some embodiments, the fine-tuning to obtain a LoRA adapter corresponding to "style_room type" based on style and room type includes the following steps:

[0016] Divide the data into training set and validation set, and load the pre-trained model of Stable Diffusion;

[0017] Insert the LoRA adapter into the convolutional layers and self-attention layers of the U-Net part of the Stable Diffusion model;

[0018] Initialize two low-rank matrices in the LoRA adapter and set the scaling factor;

[0019] The training set data was iteratively trained. The model selected MSE as the loss function, used the Adam optimizer, the initial learning rate was 0.0001, the distress descent learning rate decay method was adopted, the batch size was set to 16, the gradient accumulation step number was 1, and a total of 1000 rounds of training were performed.

[0020] Use the validation set to evaluate performance and adjust model parameters to ensure the generalization ability of the model;

[0021] Save the fine-tuned LoRA adapter and establish corresponding management and usage mechanisms.

[0022] In some embodiments, the input of the indoor rough image to be decorated and the extraction of the Canny operator features and the depth image features of the indoor rough image are specifically as follows:

[0023] Input the rough-finished image of the interior to be decorated, and use the Canny operator to extract the edge features of the image as the first control condition of the spatial structure;

[0024] The DPT-Hybrid-MiDas model is used to estimate the depth features of the input image and obtain the second control condition in the space;

[0025] The first control condition and the second control condition are saved and displayed visually.

[0026] In some embodiments, the method of using an image segmentation model to identify fixed structures in the indoor roughcast image and marking mask data includes the following steps:

[0027] Using the SAM model to identify the indoor rough image to be decorated, the position and size of the fixed structure are automatically segmented, and the fixed structure includes doors and windows;

[0028] A post-processing operation is performed on the segmented mask data to obtain mask data, which is recorded as Mask and saved. The post-processing operation includes corrosion and expansion.

[0029] In some embodiments, the use of the pre-trained Stable Diffusion model, combined with Controlnet and Stable Inpainting workflows, inputting the control conditions, the mask data, and the prompt, and generating a decoration rendering that fits the interior structure includes the following steps:

[0030] Load the pre-trained Stable Diffusion model and insert it into the Controlnet branch and the StableInpainting branch;

[0031] Load the control conditions and add them into the Depth model of Controlnet;

[0032] Loading the mask data, and inputting the mask data into the Stable Inpainting branch;

[0033] Enter the description related to interior design as the positive text prompt and negative prompt, and set the number of steps and resolution for generation;

[0034] Start the model to generate renderings and display the generated renderings.

[0035] In some embodiments, the optional decoration style is set according to the LoRA adapter, and the LoRA is inserted into the workflow to adjust the rendering style according to the style option, specifically:

[0036] Provide optional decoration styles and room function types for users to choose from, and change the overall decoration style based on the generated decoration renderings;

[0037] Insert the LoRA adapter corresponding to the style and room type into the Unet of the Stable Diffusion model, change the design style of the decoration rendering and display it on the front end.

[0038] In some embodiments, the hyperparameters are set and adjusted to ensure that the structural layout of the rendering is consistent with the rough indoor image and to improve the detail information and image quality of the rendering, specifically:

[0039] Set the control intensity parameters and control early stopping parameters according to the rendering;

[0040] The layout fit of the rendering image is controlled according to the control strength parameter, and the detail features and image quality of the model are controlled according to the control early stopping parameter.

[0041] Another aspect of the embodiment of the present invention further provides a device for generating interior design renderings based on a diffusion model, comprising:

[0042] The first module is used to collect data on historical interior design and decoration renderings and classify their styles, and fine-tune the LoRA adapters for each style;

[0043] The second module is used to input the indoor rough image to be decorated, and extract the Canny operator features and depth image features of the indoor rough image, and the Canny operator features and depth image features are used as control conditions for subsequent rendering generation;

[0044] The third module is used to identify the fixed structure in the indoor roughcast image by using the image segmentation model and mark the mask data;

[0045] The fourth module is used to use the pre-trained Stable Diffusion model, combined with Controlnet and Stable Inpainting workflow, input the control conditions, the mask data and prompt, and generate a decoration rendering that fits the indoor structure;

[0046] The fifth module is used to set the optional decoration style according to the LoRA adapter, and insert LoRA into the workflow to adjust the rendering style according to the style option;

[0047] The sixth module is used to set and adjust hyperparameters to ensure that the structural layout of the rendering is consistent with the interior rough image and to improve the detail information and image quality of the rendering.

[0048] Another aspect of an embodiment of the present invention further provides an electronic device, including a processor and a memory;

[0049] The memory is used to store programs;

[0050] The processor executes the program to implement the method described above to achieve the above purpose.

[0051] To achieve the above objective, another aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the above-mentioned method is implemented.

[0052] The embodiment of the present invention also discloses a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. A processor of a computer device can read the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device executes the above method.

[0053] The embodiments of the present invention include at least the following beneficial effects: the present invention provides a method and device for generating interior design renderings based on a diffusion model, the present invention collects data of historical interior design decoration renderings and performs style classification, and fine-tunes LoRA adapters of various styles respectively; inputs an indoor rough image to be decorated, extracts Canny operator features and depth image features of the indoor rough image, and the Canny operator features and depth image features are used as control conditions for subsequent rendering generation; uses an image segmentation model to identify fixed structures in the indoor rough image and marks mask data; uses a pre-trained Stable Diffusion model, combines Controlnet and Stable Inpainting workflows, inputs the control conditions, the mask data, and prompt, and generates a decoration rendering that fits the indoor structure; sets an optional decoration style according to the LoRA adapter, and inserts LoRA into the workflow to adjust the rendering style according to the style options; sets and adjusts hyperparameters to ensure that the structural layout of the rendering is consistent with the indoor rough image and improves the detail information and image quality of the rendering. The present invention can improve the generation effect and practicality of interior design renderings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 It is a schematic diagram of an implementation environment provided by an embodiment of the present invention;

[0055] Figure 2 is a flow chart of overall steps provided by an embodiment of the present invention;

[0056] Figure 3 It is a schematic diagram of a model of a method for generating an interior design rendering provided by an embodiment of the present invention;

[0057] Figure 4 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0058] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention. They are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0059] It is understood that the terms "first", "second", etc. used in the present invention may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiments of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0060] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0062] The method and device for generating interior design renderings based on a diffusion model provided in an embodiment of the present invention relate to the field of computer technology. The method for generating interior design renderings based on a diffusion model provided in an embodiment of the present invention can be applied to a terminal, can be applied to a server, or can be software running in a terminal or a server. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, and a car terminal, etc., but is not limited to this; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server can also be a node server in a blockchain network; the software can be an application that implements a method for generating interior design renderings based on a diffusion model, etc., but is not limited to the above forms.

[0063] The present invention can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present invention can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0064] like Figure 1 FIG. 1 is a schematic diagram of an implementation environment provided by an embodiment of the present invention. Figure 1 The implementation environment includes at least one terminal 102 and a server 101. The terminal 102 and the server 101 can be connected to a network wirelessly or wired to complete data transmission and exchange.

[0065] Server 101 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.

[0066] In addition, the server 101 can also be a node server in the blockchain network. Blockchain is a new application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanism, and encryption algorithm.

[0067] The terminal 102 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc. The terminal 102 may also be a vehicle-mounted terminal of various device types described above, but is not limited thereto. The terminal 102 and the server 101 may be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment of the present invention.

[0068] Based on the example Figure 1 In the implementation environment shown, an embodiment of the present invention provides a method for generating interior design renderings based on a diffusion model. The following is an example of applying the method for generating interior design renderings based on a diffusion model to a server 101. It can be understood that the method can also be applied to a terminal 102.

[0069] In order to improve the generation effect and practicality of interior design renderings, the present invention introduces Controlnet branch technology. Controlnet is a method that achieves fine control of generated images by introducing additional control conditions in a pre-trained model. By integrating the Controlnet branch in the Stable Diffusion model, designers can use comprehensive control conditions to generate interior design renderings that are more in line with actual needs. In addition, combined with the Stable Inpainting method, the door and window structure in the room can be fixed during the generation process, further improving the practicality and reference value of the rendering. The present invention also utilizes the LoRA (Low-Rank Adaptation) adapter technology to meet the needs of different styles of design. LoRA is an efficient model fine-tuning method that achieves efficient parameter updates by inserting low-rank matrices into specific layers of the pre-trained model. Using the style-fine-tuning LoRA adapter, the design style of the rendering can be quickly switched to generate a variety of interior design solutions without significantly increasing the computing cost.

[0070] Reference Figure 2, Figure 2 The flowchart of the method for generating interior design renderings based on the diffusion model applied to the server provided by the embodiment of the present invention, the execution subject of the method can be any of the aforementioned computer devices (including servers or terminals). Figure 2 , the method may include the following steps:

[0071] Collect data on historical interior design and decoration renderings and classify their styles, and fine-tune LoRA adapters for each style;

[0072] Input an indoor rough image to be decorated, extract the Canny operator features and depth image features of the indoor rough image, and use the Canny operator features and depth image features as control conditions for subsequent rendering generation;

[0073] Using an image segmentation model to identify fixed structures in the indoor roughcast image and mark mask data;

[0074] Using the pre-trained Stable Diffusion model, combined with Controlnet and Stable Inpainting workflows, input the control conditions, the mask data and prompt to generate a decoration rendering that fits the interior structure;

[0075] Set the optional decoration style according to the LoRA adapter, and insert LoRA into the workflow to adjust the rendering style according to the style options;

[0076] Set and adjust hyperparameters to ensure that the structural layout of the rendering is consistent with the rough interior image and to improve the details and image quality of the rendering.

[0077] In some embodiments, the process of collecting historical interior design and decoration rendering data and classifying the styles, and fine-tuning LoRA adapters of different styles respectively, includes the following steps:

[0078] Classifying the historical interior design and decoration renderings according to products and design styles to obtain style data of each style series, wherein the design styles include pastoral style, new Chinese style, and Mediterranean style;

[0079] The style data is subdivided according to the space type of the apartment, where the space type includes bedroom, living room, kitchen, and bathroom;

[0080] Based on the style and room type, fine-tune to get the LoRA adapter corresponding to "style_room type".

[0081] In some embodiments, the fine-tuning to obtain a LoRA adapter corresponding to "style_room type" based on style and room type includes the following steps:

[0082] Divide the data into training set and validation set, and load the pre-trained model of Stable Diffusion;

[0083] Insert the LoRA adapter into the convolutional layers and self-attention layers of the U-Net part of the Stable Diffusion model;

[0084] Initialize two low-rank matrices in the LoRA adapter and set the scaling factor;

[0085] The training set data was iteratively trained. The model selected MSE as the loss function, used the Adam optimizer, the initial learning rate was 0.0001, the distress descent learning rate decay method was adopted, the batch size was set to 16, the gradient accumulation step number was 1, and a total of 1000 rounds of training were performed.

[0086] Use the validation set to evaluate performance and adjust model parameters to ensure the generalization ability of the model;

[0087] Save the fine-tuned LoRA adapter and establish corresponding management and usage mechanisms.

[0088] In some embodiments, the input of the indoor rough image to be decorated and the extraction of the Canny operator features and the depth image features of the indoor rough image are specifically as follows:

[0089] Input the rough-finished image of the interior to be decorated, and use the Canny operator to extract the edge features of the image as the first control condition of the spatial structure;

[0090] The DPT-Hybrid-MiDas model is used to estimate the depth features of the input image and obtain the second control condition in the space;

[0091] The first control condition and the second control condition are saved and displayed visually.

[0092] In some embodiments, the method of using an image segmentation model to identify fixed structures in the indoor roughcast image and marking mask data includes the following steps:

[0093] Using the SAM model to identify the indoor rough image to be decorated, the position and size of the fixed structure are automatically segmented, and the fixed structure includes doors and windows;

[0094] A post-processing operation is performed on the segmented mask data to obtain mask data, which is recorded as Mask and saved. The post-processing operation includes corrosion and expansion.

[0095] In some embodiments, the use of the pre-trained Stable Diffusion model, combined with Controlnet and Stable Inpainting workflows, inputting the control conditions, the mask data, and the prompt, and generating a decoration rendering that fits the interior structure includes the following steps:

[0096] Load the pre-trained Stable Diffusion model and insert it into the Controlnet branch and the StableInpainting branch;

[0097] Load the control conditions and add them into the Depth model of Controlnet;

[0098] Loading the mask data, and inputting the mask data into the Stable Inpainting branch;

[0099] Enter the description related to interior design as the positive text prompt and negative prompt, and set the number of steps and resolution for generation;

[0100] Start the model to generate renderings and display the generated renderings.

[0101] In some embodiments, the optional decoration style is set according to the LoRA adapter, and the LoRA is inserted into the workflow to adjust the rendering style according to the style option, specifically:

[0102] Provide optional decoration styles and room function types for users to choose from, and change the overall decoration style based on the generated decoration renderings;

[0103] Insert the LoRA adapter corresponding to the style and room type into the Unet of the Stable Diffusion model, change the design style of the decoration rendering and display it on the front end.

[0104] In some embodiments, the hyperparameters are set and adjusted to ensure that the structural layout of the rendering is consistent with the rough indoor image and to improve the detail information and image quality of the rendering, specifically:

[0105] Set the control intensity parameters and control early stopping parameters according to the rendering;

[0106] The layout fit of the rendering image is controlled according to the control strength parameter, and the detail features and image quality of the model are controlled according to the control early stopping parameter.

[0107] The present invention realizes the rapid and efficient generation of interior design renderings in rough scenes by introducing Controlnet branches and Stable Inpainting methods, combined with LoRA adapter technology. This method not only improves the quality and practicality of renderings, ensures the reference value of design solutions, but also greatly improves the scarcity of interior design reference cases for rough houses. With this technology, designers can more conveniently generate design drawings that are close to real scenes, meet consumers' personalized and high-quality interior design needs, and promote the development and progress of the interior design industry.

[0108] The implementation process of the present invention is described in detail below using a specific application scenario as an example:

[0109] refer to Figure 3 The embodiment of the present invention provides a method for generating an interior design rendering based on a diffusion model, comprising the following steps:

[0110] 1. Collect historical interior design and decoration rendering data and classify the styles, fine-tune the LoRA adapters of each style and save them;

[0111] 2. Input the rough-finished image of the interior to be decorated, extract the Canny operator features and deep image features of the image and use them as control conditions for the subsequent rendering generation;

[0112] 3. Use the image segmentation model to identify fixed structures such as doors and windows in the rough image, mark the mask data and save it to protect the indoor structure;

[0113] 4. Use the pre-trained Stable Diffusion model, supplemented by Controlnet and Stable Inpainting workflows, input control conditions, mask data, and prompts to generate decoration renderings that fit the interior structure;

[0114] 5. Set the optional decoration style according to the LoRA adapter fine-tuned in the first step, and insert LoRA into the workflow to adjust the rendering style according to the style options;

[0115] 6. Set and adjust hyperparameters to ensure that the structure layout of the rendering is consistent with the rough Figure 1 Improves the detail and quality of rendered images.

[0116] The first step is to collect historical interior design and decoration rendering data and classify the styles, and then fine-tune the LoRA adapters for each style and save them.

[0117] First, the historical interior design rendering data is classified according to product and design style to obtain data for each style series, such as pastoral style, new Chinese style, Mediterranean style, etc. The style data is subdivided according to the spatial type of the house, covering bedrooms, living rooms, kitchens, bathrooms and other types.

[0118] The above-organized rendering data is divided into "style_room type" units, and the corresponding LoRA adapters are fine-tuned. LoRA is an efficient model fine-tuning method that inserts low-rank matrices into specific layers of the pre-trained model to achieve efficient parameter updates. The following processing is performed on each LoRA adapter:

[0119] a. First, divide the data into training set and validation set, and load the pre-trained model of Stable Diffusion;

[0120] b. Insert the LoRA adapter into the convolutional layers and self-attention layers of the U-Net part of the Stable Diffusion model;

[0121] c. Initialize two low-rank matrices in the LoRA adapter and set the scaling factor;

[0122] d. Perform iterative training on the training set data. The model uses MSE as the loss function, uses the Adam optimizer, the initial learning rate is 0.0001, the distress descent learning rate decay method is used, the batch size is set to 16, the gradient accumulation step is 1, and a total of 1000 rounds of training are performed;

[0123] e. Use the validation set to evaluate performance and adjust model parameters to ensure the generalization ability of the model;

[0124] Save the fine-tuned LoRA adapter and establish an effective management and usage mechanism.

[0125] In the second step, the image of the interior to be decorated is input, and the Canny operator features and deep image features of the image are extracted to serve as control conditions for the subsequent generation of renderings.

[0126] Input the rough-finished image of the interior to be decorated, use the Canny operator to extract the edge features of the image as one of the control conditions of the spatial structure; use the DPT-Hybrid-MiDas model to estimate the depth features of the input image and obtain the depth control condition in the space. Save the above two control conditions and visualize them.

[0127] The third step is to use the image segmentation model to identify fixed structures such as doors and windows in the rough image, mark the mask data and save it to protect the indoor structure.

[0128] The SAM model is used to identify the rough-finished indoor image to be decorated input in the second step, and the position and size of fixed structures such as doors and windows are automatically segmented. The segmented mask data is subjected to post-processing operations such as corrosion and expansion to obtain relatively complete mask data, which is recorded as Mask. The Mask is saved and displayed.

[0129] In the fourth step, use the pre-trained Stable Diffusion model, supplemented by Controlnet and Stable Inpainting workflows, input control conditions, mask data, and prompts to generate decoration renderings that fit the interior structure.

[0130] Integrate the model to generate a decoration rendering that fits the interior structure. The specific operation process is as follows:

[0131] a. Load the pre-trained Stable Diffusion model and insert the Controlnet branch and the StableInpainting branch;

[0132] b. Load the control conditions saved in the second step, add the control conditions and input them into the Depth model of Controlnet;

[0133] c. Load the Mask saved in step 3 and input the Mask into the Stable Inpainting branch;

[0134] d. Enter the description related to interior design as the positive text prompt, as well as the conventional negative prompt, and set the number of steps and resolution for generation;

[0135] e. Start the model to generate renderings and display the generated renderings.

[0136] Step 5. Set the optional decoration style based on the LoRA adapter fine-tuned in the first step, and insert LoRA into the workflow to adjust the rendering style according to the style options.

[0137] Provide optional decoration styles and room function types for users to choose from, and change the overall decoration style based on the rendering generated in step 4. Insert the LoRA adapter corresponding to the style and room type into the Unet of the above Stable Diffusion pre-trained model, change the design style of the rendering and display it on the front end.

[0138] Step 6: Set and adjust hyperparameters to ensure that the structure layout of the rendering is consistent with the original image. Figure 1 Improves the detail and quality of rendered images.

[0139] Check the interior design rendering generated by the above process, set the control strength parameter and control early stopping parameter, use the control strength parameter to control the layout fit of the rendering, and use the control early stopping parameter to control the detail features and image quality of the model.

[0140] In summary, the present invention can assist designers to quickly render the interior design and decoration effects of rough houses, present various available decoration styles to users for reference, improve the efficiency of client interior design solutions, greatly save the design time of unfamiliar scenes, save rendering resources, and to a certain extent improve the conversion of related goods and services. The highlight of this method and device is to apply the Wensheng map capability of the diffusion model to the generation of interior design renderings, solving the existing problems of long time-consuming cycle and high rendering resource consumption of interior design rendering previews. On the basis of prompt control Wensheng map, Controlnet model and Stable Inpainting model are added to control and fix the spatial structure, which fits the actual design scene and has practicality and reference value. Use historical data to supervise and fine-tune the LoRA adapter, provide interactive options to switch the style of generated renderings, and ensure the diversity of rendering previews in a single scene.

[0141] Another aspect of the embodiment of the present invention further provides a device for generating interior design renderings based on a diffusion model, comprising:

[0142] The first module is used to collect data on historical interior design and decoration renderings and classify their styles, and fine-tune the LoRA adapters for each style;

[0143] The second module is used to input the indoor rough image to be decorated, and extract the Canny operator features and depth image features of the indoor rough image, and the Canny operator features and depth image features are used as control conditions for subsequent rendering generation;

[0144] The third module is used to identify the fixed structure in the indoor roughcast image by using the image segmentation model and mark the mask data;

[0145] The fourth module is used to use the pre-trained Stable Diffusion model, combined with Controlnet and Stable Inpainting workflow, input the control conditions, the mask data and prompt, and generate a decoration rendering that fits the indoor structure;

[0146] The fifth module is used to set the optional decoration style according to the LoRA adapter, and insert LoRA into the workflow to adjust the rendering style according to the style option;

[0147] The sixth module is used to set and adjust hyperparameters to ensure that the structural layout of the rendering is consistent with the interior rough image and to improve the detail information and image quality of the rendering.

[0148] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0149] The embodiment of the present invention further provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the above-mentioned method for generating interior design renderings based on a diffusion model when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0150] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0151] See also Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0152] The processor 401 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0153] The memory 402 may be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 402 may store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 402, and the processor 401 calls and executes the method for generating interior design renderings based on a diffusion model according to the embodiment of the present invention.

[0154] Input / output interface 403, used to implement information input and output;

[0155] Communication interface 404, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0156] Bus 405 , which transmits information between various components of the device (e.g., processor 401 , memory 402 , input / output interface 403 , and communication interface 404 );

[0157] The processor 401 , the memory 402 , the input / output interface 403 and the communication interface 404 are connected to each other in communication within the device via the bus 405 .

[0158] An embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-mentioned method for generating interior design renderings based on a diffusion model.

[0159] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0160] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0161] The embodiments described in the embodiments of the present invention are intended to more clearly illustrate the technical solutions of the embodiments of the present invention, and do not constitute a limitation on the technical solutions provided by the embodiments of the present invention. Those skilled in the art can appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of the present invention are also applicable to similar technical problems.

[0162] Those skilled in the art will appreciate that the technical solutions shown in the figures do not limit the embodiments of the present invention and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0163] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0164] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0165] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0166] It should be understood that in the present invention, "at least one (item)" refers to one or more, and "plurality" refers to two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can represent: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following items" or similar expressions refers to any combination of these items, including any combination of single items or plural items. For example, at least one of a, b or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0167] In the several embodiments provided by the present invention, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0168] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0169] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0170] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including multiple instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., various media that can store programs.

[0171] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments of the present invention is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments of the present invention shall be within the scope of the rights of the embodiments of the present invention.

Claims

1. A method for generating interior design renderings based on a diffusion model, characterized in that: The following steps are involved: Collect data on historical interior design and decoration renderings and classify their styles, and fine-tune LoRA adapters for each style; Input an indoor rough image to be decorated, extract the Canny operator features and depth image features of the indoor rough image, and use the Canny operator features and depth image features as control conditions for subsequent rendering generation; Using an image segmentation model to identify fixed structures in the indoor roughcast image and mark mask data; Using the pre-trained Stable Diffusion model, combined with Controlnet and Stable Inpainting workflows, input the control conditions, the mask data and prompt to generate a decoration rendering that fits the interior structure; Set the optional decoration style according to the LoRA adapter, and insert LoRA into the workflow to adjust the rendering style according to the style options; Set and adjust hyperparameters to ensure that the structural layout of the rendering is consistent with the rough interior image and to improve the details and image quality of the rendering.

2. The method for generating interior design renderings based on a diffusion model according to claim 1, characterized in that: The method of collecting historical interior design and decoration rendering data and classifying the styles, and fine-tuning the LoRA adapters of different styles respectively, includes the following steps: Classifying the historical interior design and decoration renderings according to products and design styles to obtain style data of each style series, wherein the design styles include pastoral style, new Chinese style, and Mediterranean style; The style data is subdivided according to the space type of the apartment, where the space type includes bedroom, living room, kitchen, and bathroom; Based on the style and room type, fine-tune to get the LoRA adapter corresponding to "style_room type".

3. The method for generating interior design renderings based on a diffusion model according to claim 2, characterized in that: The method of fine-tuning the LoRA adapter corresponding to "style_room type" by taking style and room type as units includes the following steps: Divide the data into training set and validation set, and load the pre-trained model of Stable Diffusion; Insert the LoRA adapter into the convolutional layers and self-attention layers of the U-Net part of the Stable Diffusion model; Initialize two low-rank matrices in the LoRA adapter and set the scaling factor; The training set data was iteratively trained. The model selected MSE as the loss function, used the Adam optimizer, the initial learning rate was 0.0001, the distress descent learning rate decay method was adopted, the batch size was set to 16, the gradient accumulation step number was 1, and a total of 1000 rounds of training were performed. Use the validation set to evaluate performance and adjust model parameters to ensure the generalization ability of the model; Save the fine-tuned LoRA adapter and establish corresponding management and usage mechanisms.

4. The method for generating interior design renderings based on a diffusion model according to claim 1, characterized in that: The input of the indoor rough image to be decorated is performed, and the Canny operator features and the depth image features of the indoor rough image are extracted, specifically: Input the rough-finished image of the interior to be decorated, and use the Canny operator to extract the edge features of the image as the first control condition of the spatial structure; The DPT-Hybrid-MiDas model is used to estimate the depth features of the input image and obtain the second control condition in the space; The first control condition and the second control condition are saved and displayed visually.

5. The method for generating interior design renderings based on a diffusion model according to claim 1, characterized in that: The method of using the image segmentation model to identify the fixed structure in the indoor roughcast image and marking the mask data comprises the following steps: Using the SAM model to identify the indoor rough image to be decorated, the position and size of the fixed structure are automatically segmented, and the fixed structure includes doors and windows; A post-processing operation is performed on the segmented mask data to obtain mask data, which is recorded as Mask and saved. The post-processing operation includes corrosion and expansion.

6. The method for generating interior design renderings based on a diffusion model according to claim 1, characterized in that: The method of using the pre-trained Stable Diffusion model, combining Controlnet and Stable Inpainting workflow, inputting the control conditions, the mask data and prompt, and generating a decoration rendering that fits the indoor structure includes the following steps: Load the pre-trained Stable Diffusion model and insert it into the Controlnet branch and the Stable Inpainting branch; Load the control conditions and add them into the Depth model of Controlnet; Loading the mask data, and inputting the mask data into the Stable Inpainting branch; Enter the description related to interior design as the positive text prompt and negative prompt, and set the number of steps and resolution for generation; Start the model to generate renderings and display the generated renderings.

7. The method for generating interior design renderings based on a diffusion model according to claim 1, characterized in that: The optional decoration style is set according to the LoRA adapter, and the LoRA is inserted into the workflow to adjust the rendering style according to the style option, specifically: Provide optional decoration styles and room function types for users to choose from, and change the overall decoration style based on the generated decoration renderings; Insert the LoRA adapter corresponding to the style and room type into the Unet of the Stable Diffusion model, change the design style of the decoration rendering and display it on the front end.

8. The method for generating interior design renderings based on a diffusion model according to claim 1, characterized in that: The hyper parameters are set and adjusted to ensure that the structural layout of the rendering is consistent with the rough indoor image and to improve the detail information and image quality of the rendering, specifically: Set the control intensity parameters and control early stopping parameters according to the rendering; The layout fit of the rendering image is controlled according to the control strength parameter, and the detail features and image quality of the model are controlled according to the control early stopping parameter.

9. A device for generating interior design renderings based on a diffusion model, characterized in that: include: The first module is used to collect data on historical interior design and decoration renderings and classify their styles, and fine-tune the LoRA adapters for each style; The second module is used to input the indoor rough image to be decorated, and extract the Canny operator features and depth image features of the indoor rough image, and the Canny operator features and depth image features are used as control conditions for subsequent rendering generation; The third module is used to identify the fixed structure in the indoor roughcast image by using the image segmentation model and mark the mask data; The fourth module is used to use the pre-trained Stable Diffusion model, combined with Controlnet and Stable Inpainting workflow, input the control conditions, the mask data and prompt, and generate a decoration rendering that fits the indoor structure; The fifth module is used to set the optional decoration style according to the LoRA adapter, and insert LoRA into the workflow to adjust the rendering style according to the style option; The sixth module is used to set and adjust hyperparameters to ensure that the structural layout of the rendering is consistent with the interior rough image and to improve the detail information and image quality of the rendering.

10. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 8.

Images