Indoor scene image generation method and device, equipment and storage medium

Abstract

The invention discloses an indoor scene image generation method and device, equipment and a storage medium. The method comprises the following steps: firstly, acquiring a to-be-processed image with furniture and information input by a user, and performing furniture elimination processing on the image to obtain an empty room image; and then generating a cue word for image generation based on a preset cue word template in combination with user input information. Meanwhile, extracting a depth map and a semantic segmentation map of the empty room image, and inputting the depth map and the semantic segmentation map into a pre-trained depth map control model and a pre-trained semantic segmentation control model for feature coding to obtain corresponding depth control features and semantic segmentation control features. And performing parameter adjustment on the large-scale graph generation model subjected to parameter fine adjustment by using the two types of control features to obtain a target graph generation model. And finally, inputting the empty room image and the cue word into the target image generation model to obtain a target image. According to the method, the spatial structure stability and the personalized design requirement of the user on the indoor layout can be considered.

Description

Technical Field

[0001] This application relates to the field of computer technology, specifically to a method, apparatus, device, and storage medium for generating indoor scene images. Background Technology

[0002] Before the advent of Control Networks (ControlNet), artificial intelligence (AI) image generation models such as Stable Diffusion primarily relied on text prompts to guide the generated content, resulting in coarse-grained control and difficulty in precisely locating details. The introduction of ControlNet enables precise, pixel-level control of the generation process, significantly improving the model's responsiveness to input information. However, in applications involving the decoration of non-empty rooms, the process of using ControlNet for spatial structure and furniture layout control presents the following problems: When the control weight is high, the model strictly follows the spatial structure constraints of the input, and can only make stylistic modifications to the existing furniture, while the position and placement of the furniture remain unchanged. This limits the user's personalized design needs for the interior layout. When the control weights are reduced to allow for more varied furniture layouts and types, the model's constraints on the spatial structure weaken, which may lead to unreasonable phenomena such as the disappearance of doors and windows, their addition, or abnormal wall structures, affecting the realism and consistency of the final effect.

[0003] Therefore, how to maintain the rationality of the spatial structure while allowing for flexible adjustments to the layout and types of furniture is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] Based on the above problems, this application provides an indoor scene image generation method, apparatus, device and storage medium, which can ensure the integrity of the original room space structure (such as doors, windows and walls) and realize diversified adjustments to furniture layout and types, thereby taking into account both the stability of the space structure and the user's personalized design needs for the interior layout, and avoiding the contradiction of "inflexible furniture changes" or "abnormal space structure" caused by the imbalance of control weights.

[0005] The embodiments of this application disclose the following technical solutions: An indoor scene image generation method, the method comprising: The system acquires an image to be processed and user input information, and removes furniture from the image to obtain an empty room image; the image to be processed is an interior scene image with furniture; the user input information includes the room type and target decoration style of the image to be processed. Based on the preset prompt word template and the user input information, a raw image prompt word is generated, and the depth map and semantic segmentation map of the empty room image are obtained; The depth map and semantic segmentation map of the empty room image are respectively input into the pre-trained depth map control model and semantic segmentation control model for feature encoding to obtain depth control features and semantic segmentation control features. The parameters of the raw image model are adjusted using the depth control features and the semantic segmentation control features to obtain the target raw image model; the raw image model has been fine-tuned using a parameter fine-tuning model. The empty room image and the raw image prompt are input into the target raw image model to generate the target image.

[0006] In one possible implementation, the training process of the depth map control model includes: Obtain a first indoor image set; the first indoor image set includes multiple first indoor image samples with furniture, each first indoor image sample is labeled with a target label; the target label includes the room type of the first indoor image sample and the furniture category and furniture material of all furniture in the first indoor image sample; Furniture removal is performed on each of the first indoor image samples in the first indoor image set to obtain a first empty room image set; Obtain the depth map of each image in the first empty room image set to obtain a depth image set; Each first indoor image sample in the first indoor image set is paired with a corresponding image in the corresponding depth image set to obtain multiple first training sample pairs; The multiple first training sample pairs are input into the depth map conditional control network for model training to obtain the depth map control model.

[0007] In one possible implementation, the training process of the semantic segmentation control model includes: Acquire a second set of indoor images; the second set of indoor images includes multiple second indoor image samples with furniture, each second indoor image sample is labeled with a target label; the target label includes the room type of the second indoor image sample and the furniture category and furniture material of all furniture in the second indoor image sample; Furniture removal is performed on each of the second indoor image samples in the second indoor image set to obtain a second empty room image set; Obtain semantic segmentation maps of each image in the second empty room image set to obtain a semantic segmentation image set; Each second indoor image sample in the second indoor image set is paired with a corresponding image in the corresponding depth image set to obtain multiple second training sample pairs; The multiple second training sample pairs are input into the semantic segmentation conditional control network for model training to obtain the semantic segmentation control model.

[0008] In one possible implementation, the training process of the parameter fine-tuning model includes: Obtain a third indoor image set; the third indoor image set includes multiple third indoor image samples with furniture, each third indoor image sample is labeled with an auxiliary label; the auxiliary label includes the room type of the third indoor image sample, the furniture category and furniture material of all furniture in the third indoor image sample, and a trigger word; the auxiliary label is used to provide additional semantic description; the trigger word exists as a placeholder to prevent the model from learning the auxiliary label or extracting actual semantics; The third indoor image set is input into the LoRA model for model training to obtain a parameter fine-tuning model.

[0009] An indoor scene image generation device, the device comprising: An acquisition and elimination unit is used to acquire an image to be processed and user input information, and to perform furniture removal on the image to be processed to obtain an empty room image; the image to be processed is an interior scene image with furniture; the user input information includes the room type and target decoration style of the image to be processed; The prompt word generation unit is used to generate image prompt words based on a preset prompt word template and the user input information, and to obtain the depth map and semantic segmentation map of the empty room image; The feature encoding unit is used to input the depth map and semantic segmentation map of the empty room image into the pre-trained depth map control model and semantic segmentation control model respectively for feature encoding to obtain depth control features and semantic segmentation control features. The parameter adjustment unit is used to adjust the parameters of the large raw image model using the depth control features and the semantic segmentation control features to obtain the target raw image model; the large raw image model has been fine-tuned by the parameter fine-tuning model. An image generation unit is used to input the empty room image and the image prompt into the target image model to generate an image and obtain a target image.

[0010] In one possible implementation, the device further includes: The first acquisition unit is used to acquire a first indoor image set; the first indoor image set includes multiple first indoor image samples with furniture, each first indoor image sample is marked with a target label; the target label includes the room type of the first indoor image sample and the furniture category and furniture material of all furniture in the first indoor image sample; The first furniture removal unit is used to remove furniture from each of the first indoor image samples in the first indoor image set to obtain a first empty room image set. The second acquisition unit is used to acquire the depth map of each image in the first empty room image set to obtain a depth image set; The first pairing unit is used to pair each first indoor image sample in the first indoor image set with the corresponding image in the corresponding depth image set to obtain multiple first training sample pairs; The first model training unit is used to input the plurality of first training sample pairs into the depth map conditional control network for model training to obtain the depth map control model.

[0011] In one possible implementation, the device further includes: The third acquisition unit is used to acquire a second indoor image set; the second indoor image set includes multiple second indoor image samples with furniture, each second indoor image sample is marked with a target label; the target label includes the room type of the second indoor image sample and the furniture category and furniture material of all furniture in the second indoor image sample; The second furniture removal unit is used to remove furniture from each of the second indoor image samples in the second indoor image set to obtain a second empty room image set. The fourth acquisition unit is used to acquire the semantic segmentation map of each image in the second empty room image set to obtain a semantic segmentation image set; The second pairing unit is used to pair each second indoor image sample in the second indoor image set with the corresponding image in the corresponding depth image set to obtain multiple second training sample pairs; The second model training unit is used to input the multiple second training sample pairs into the semantic segmentation conditional control network for model training, thereby obtaining the semantic segmentation control model.

[0012] In one possible implementation, the device further includes: The fifth acquisition unit is used to acquire a third indoor image set; the third indoor image set includes multiple third indoor image samples with furniture, and each third indoor image sample is labeled with an auxiliary label; the auxiliary label includes the room type of the third indoor image sample, the furniture category and furniture material of all furniture in the third indoor image sample, and a trigger word; the auxiliary label is used to provide additional semantic description; the trigger word exists as a placeholder to prevent the model from learning the auxiliary label or extracting actual semantics; The third model training unit is used to input the third indoor image set into the LoRA model for model training to obtain a parameter fine-tuning model.

[0013] An indoor scene image generation device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the indoor scene image generation method described above.

[0014] A computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform the indoor scene image generation method as described above. Compared with the prior art, this application has the following beneficial effects: This application provides a method, apparatus, device, and storage medium for generating indoor scene images. Specifically, when executing the indoor scene image generation method provided in this application embodiment, the image to be processed and user input information are first acquired, and furniture removal is performed on the image to obtain an empty room image. The user input information includes the room type and target decoration style, which provides necessary contextual information for subsequent steps. Then, based on a preset prompt word template and user input information, raw image prompt words are generated, and the depth map and semantic segmentation map of the empty room image are further extracted. These images provide rich contextual information, which helps to improve the quality of image generation. Next, the depth map and semantic segmentation map are respectively input into a pre-trained depth map control model and a semantic segmentation control model for feature encoding to obtain depth control features and semantic segmentation control features. These features are used to adjust the parameters of the raw image model to generate a target raw image model. The raw image model has been fine-tuned to further improve the accuracy of image generation. Finally, the empty room image and raw image prompt words are input into the target raw image model to generate the target image. This application allows for flexible adjustment of furniture layout and style while maintaining the stability and rationality of the spatial structure, ensuring the authenticity and consistency of the generated results. Attached Figure Description

[0015] To more clearly illustrate the technical solutions in this embodiment or the prior art, the drawings used in the description of the embodiment or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 A flowchart illustrating an indoor scene image generation method provided in this application embodiment; Figure 2 A schematic diagram of an indoor scene image provided in an embodiment of this application; Figure 3 A schematic diagram of an empty room image provided in an embodiment of this application; Figure 4 A flowchart illustrating a depth map control model training method provided in this application embodiment; Figure 5 A schematic diagram of yet another indoor scene image provided in an embodiment of this application; Figure 6 A schematic diagram of yet another empty room image provided in an embodiment of this application; Figure 7 A schematic diagram of a depth map provided in an embodiment of this application; Figure 8 A flowchart illustrating a semantic segmentation control model training method provided in this application embodiment; Figure 9 A schematic diagram of a semantic segmentation graph provided in an embodiment of this application; Figure 10 This is a schematic diagram of an indoor scene image generation device provided in an embodiment of this application. Detailed Implementation

[0017] To facilitate understanding of the technical solutions provided in the embodiments of this application, the background technology involved in the embodiments of this application will be described below.

[0018] Before ControlNet, AI image generation models like Stable Diffusion primarily relied on text prompts to control the generated content, making it difficult to fine-tune image details. ControlNet, however, provides precise pixel-level control, significantly enhancing the model's responsiveness to input information. But in decorating non-empty rooms, using ControlNet to control spatial structure and furniture layout presents a contradiction: When the control weight is high, the model can only make stylistic modifications to existing furniture, while the furniture positions remain unchanged, which cannot meet the user's needs for layout changes. However, reducing the control weight to achieve more furniture changes can lead to abnormal spatial structures, such as the disappearance of doors and windows or changes in walls, affecting the realism of the image.

[0019] To address this issue, this application provides a method, apparatus, device, and storage medium for generating indoor scene images. By first removing furniture from the image to be processed (with furniture), an empty room image is obtained, effectively avoiding the spatial anomaly risks associated with directly controlling complex spatial structures and furniture layouts. Simultaneously, this method combines user-inputted room type and target decoration style, generating precise image prompts using preset prompt templates, enhancing the semantic relevance of the generated content. Then, the depth map and semantic segmentation map of the empty room image are input into specially trained depth map control models and semantic segmentation control models for feature encoding, extracting fine-grained spatial and semantic information for parameter adjustment of the large-scale image model, significantly improving the model's understanding and control of spatial structure and semantic layout. Finally, the large-scale generation model is further optimized through parameter fine-tuning, enabling better integration of multimodal features. This achieves high-quality indoor image generation that balances spatial structural stability and furniture style diversity, overcoming the limitations of traditional methods that struggle to balance fixed furniture positions or disrupted spatial structures. This meets users' personalized design needs and improves the realism and consistency of the final effect.

[0020] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0021] See Figure 1 This figure is a flowchart of an indoor scene image generation method provided in an embodiment of this application, as shown below. Figure 1 As shown, the indoor scene image generation method may include steps S101-S105: S101: Obtain the image to be processed and user input information, and remove furniture from the image to obtain an empty room image.

[0022] To achieve personalized interior space design and high-quality image generation, the first step is to acquire an interior scene image with furniture as the image to be processed. Simultaneously, user input information is collected, including the room type corresponding to the image and the user's desired interior design style. This is achieved by processing images with furniture (such as...) Figure 2 Furniture removal processing can simplify complex interior environments into images of empty rooms containing only the basic spatial structure (such as...). Figure 3, Figure 3 for Figure 2 The system obtains empty room images after furniture removal, thus eliminating the interference of furniture layout on subsequent generation processes. This allows the model to focus on the rationality and stability of the spatial structure itself. Acquiring empty room images not only provides a clear data foundation for subsequent multimodal feature extraction based on depth maps and semantic segmentation maps, but also creates conditions for customizing decoration styles according to specific user needs. Therefore, while ensuring the accuracy of the spatial geometry, the system can flexibly adjust furniture styles and layouts based on the user-input room type and decoration style, achieving a more personalized and realistic interior scene image generation effect. This process of first removing furniture and then reconstructing it effectively avoids the anomalies caused by the interaction between furniture and spatial structure in traditional methods.

[0023] In one possible implementation, furniture removal can be performed on the image to be processed using, but is not limited to, the following methods: In one possible implementation, furniture removal can be performed on the image to be processed using, but is not limited to, the following methods: First, multimodal large models (such as FLUX.1 Kontext, Google Gemini 2.5 FlashImage, or JiMeng 4.0 and other advanced artificial intelligence generated content (AIGC) image editing models) can be used to perform preliminary processing on interior scene images with furniture. With their powerful cross-modal understanding and editing capabilities, these models can help identify and remove furniture elements to achieve a more natural empty room effect.

[0024] Secondly, a multi-iteration furniture removal model can be used. This model is based on the Stable Diffusion 2 Inpainting large model and has been specially fine-tuned to form a LoRA model, which is optimized for the indoor furniture removal task. This model combines the semantic segmentation SegmentAnything model and the furniture detection and classification model to generate accurate furniture masks. Through multiple rounds of repair iterations, the space with furniture is transformed into an image of a structurally stable and furniture-free empty room.

[0025] In addition, manual processing techniques such as Photoshop can be used to optimize the details of the results generated by the above automated methods, correct areas that the model cannot perfectly handle, and ensure the spatial continuity and visual consistency of the empty room images.

[0026] S102: Generate image prompts based on preset prompt templates and user input information, and obtain the depth map and semantic segmentation map of the empty room image.

[0027] To more accurately guide the image generation process, pre-set prompt templates can be combined with user-inputted room type and decoration style information to generate targeted and clearly expressive image prompts. Simultaneously, depth maps and semantic segmentation maps can be extracted from empty room images, obtaining the depth map and semantic segmentation map of the space respectively. This multimodal information provides fine-grained spatial geometric and semantic constraints for subsequent models, thereby enhancing the structural rationality and stylistic consistency of the generated results.

[0028] S103: Input the depth map and semantic segmentation map of the empty room image into the pre-trained depth map control model and semantic segmentation control model respectively for feature encoding to obtain depth control features and semantic segmentation control features.

[0029] To achieve precise control over spatial structure and semantic information, the system inputs the depth map and semantic segmentation map of the extracted empty room image into pre-trained depth map control models and semantic segmentation control models, respectively. These two models then perform feature encoding on the input data. This process extracts depth control features reflecting the spatial geometry and semantic segmentation control features expressing the semantic distribution of the scene, providing rich and fine-grained guidance for subsequent generation models and ensuring that the final generated image is effectively guaranteed in terms of structural rationality and semantic consistency.

[0030] S104: The parameters of the raw image model are adjusted using the depth control features and the semantic segmentation control features to obtain the target raw image model.

[0031] To improve the accuracy and diversity of indoor scene image generation, the system uses depth control features and semantic segmentation control features extracted from the depth map control model and semantic segmentation control model as key inputs. These are then combined with a pre-trained large-scale generative model, and its parameters are specifically adjusted. This large-scale generative model has already undergone initial fine-tuning through a parameter fine-tuning model, demonstrating strong basic generation capabilities and adaptability. Building upon this, the model parameters are further refined using depth and semantic features, enabling the target generative model to more accurately capture spatial geometry and scene semantic information. This allows for flexible control over details such as furniture style and layout while maintaining the overall structural rationality of the room. Through this parameter adjustment strategy based on multimodal control features, the target generative model not only improves the realism and consistency of the generated results but also meets users' personalized design needs, significantly enhancing the quality and practical value of the generated images.

[0032] S105: Input the empty room image and the raw image prompt into the target raw image model to generate the target image.

[0033] After constructing and adjusting the parameters of the target image model, the system inputs an empty room image along with image prompts generated based on user input. The empty room image provides an accurate and structurally complete spatial foundation, while the image prompts contain specific descriptions of the room type, decoration style, and design details. The combination of these two provides the model with comprehensive and detailed guidance. By deeply fusing these multimodal inputs, the target image model can generate furniture styles and layouts that meet user expectations based on spatial geometric and semantic features, while ensuring the rationality and naturalness of the overall interior scene.

[0034] Based on the content of S101-S105, the process first involves acquiring an image of an interior scene with furniture and user input information, then using furniture removal technology to obtain an image of an empty room. The user input information includes the room type and target decoration style, providing crucial context for subsequent steps. Next, based on a pre-set prompt template and the user input information, raw image prompts are generated, and depth maps and semantic segmentation maps of the empty room image are obtained. These images are further decomposed into depth maps and semantic segmentation maps. These maps are then input into pre-trained depth map control models and semantic segmentation control models for feature encoding, yielding depth control features and semantic segmentation control features. Finally, the extracted depth control features and semantic segmentation control features are used to adjust the parameters of a pre-fine-tuned raw image model, resulting in a target raw image model. The empty room image and raw image prompts are then input into the target raw image model to generate the target image. This application overcomes the limitations of traditional methods that struggle to balance fixed furniture positions with disrupted spatial structures, thus meeting the personalized design needs of users.

[0035] See Figure 4 , Figure 4 A flowchart of a depth map control model training method provided in this application embodiment is shown, which can be implemented through steps S401-S405: S401: Acquire the first indoor image set.

[0036] To construct a depth map control model capable of accurately understanding and processing interior spatial structure and furniture information, it is first necessary to acquire a first set of interior images (such as...). Figure 5This image set contains a large number of interior scene image samples with furniture. Each sample is meticulously labeled with target tags, which not only cover the room type corresponding to the image but also include the specific category and material information of all furniture in the image. By accurately labeling these multi-dimensional attributes, rich and diverse supervision information can be provided for subsequent model training, enabling the model to better learn the spatial distribution patterns of furniture in different room types and the impact of furniture materials on visual representation. This comprehensive and meticulous data preparation process is the foundation for ensuring that the model has strong recognition and generation capabilities, helping to improve the accuracy and robustness of the depth map conditional control network in practical applications, thereby achieving higher-quality interior scene image generation results.

[0037] S402: Perform furniture removal on each of the first indoor image samples in the first indoor image set to obtain a first empty room image set.

[0038] To remove the interference of furniture in indoor images on spatial structure recognition and improve the accuracy of the depth map control model's understanding of room geometry, it is necessary to perform furniture removal processing on each image sample with furniture in the first indoor image set, thereby generating a new image set, namely the first empty room image set (e.g., Figure 6 This collection of images retains only the basic structure and boundary information of the room, completely eliminating the visual influence of furniture. This not only ensures the realism of the spatial layout of the empty room images but also provides clear and reliable input data for subsequent depth map extraction and feature encoding. This effectively improves the model's performance in understanding spatial geometry and layout relationships, thus laying a solid foundation for generating high-quality interior scenes.

[0039] It should be noted that the furniture removal method used in this step is similar to that used in step S101, and will not be described again here.

[0040] S403: Obtain the depth map of each image in the first empty room image set to obtain a depth image set.

[0041] To fully capture the spatial geometric information in the empty room images and improve the model's understanding of 3D structures, depth maps need to be extracted from each image in the first empty room image set. By employing advanced depth estimation techniques or sensor acquisition methods, the 2D empty room images are converted into corresponding depth maps. These depth maps accurately reflect the distance relationships and spatial hierarchy of various locations within the interior space relative to the camera. The generated depth maps not only reveal key geometric features such as the room's height, width, and depth but also provide important spatial constraint information for subsequent models, helping to distinguish different wall, floor, and ceiling areas and enhancing the scene's spatial perception. Finally, by performing this step on the entire empty room image set, a complete and high-quality depth image set (such as...) is constructed. Figure 7 A collection of such images, Figure 7 for Figure 6 The depth map provides rich and accurate multimodal input data for training the depth map conditional control network, which significantly improves the model's performance and generalization ability in complex indoor environments.

[0042] S404: Pair each first indoor image sample in the first indoor image set with the corresponding image in the corresponding depth image set to obtain multiple first training sample pairs.

[0043] To effectively train the depth map-conditional control model, each furniture-laden image sample in the first indoor image set can be precisely paired with the corresponding empty room depth map in the depth image set. This pairing creates multiple training sample pairs, ensuring that each pair contains both rich visual information (including details of the furniture and its materials) and accurate spatial geometric information (provided by the depth map). This combination of multimodal data not only helps the model learn to infer three-dimensional spatial structures from two-dimensional images but also enables it to understand the rational layout and spatial relationships of furniture in specific room types. Furthermore, by clearly labeling the targets, the model can better capture the correlation between semantic information such as room type, furniture category, and material and spatial structure during training, providing a solid data foundation for subsequently generating empty room images or rearranging furniture.

[0044] Pairing the first indoor image with the depth image not only enriches the representation of the training data but also enhances the model's ability to perceive and generate complex indoor scenes, making it a crucial step in building a high-performance depth map control model.

[0045] S405: Input the multiple first training sample pairs into the depth map conditional control network to train the model and obtain the depth map control model.

[0046] To enable the model to effectively grasp the complex relationship between depth information and furniture layout in interior spaces, multiple first training sample pairs are systematically trained using a pre-designed depth map conditional control network (ControlNet Depth). Each training sample pair consists of an interior image with furniture and its corresponding depth map. This multimodal input provides the neural network with rich visual and spatial geometric features, helping the model learn how to accurately infer the three-dimensional spatial structure based on two-dimensional images. Through repeated iterations and optimizations, the network gradually adjusts its internal parameters to capture the intrinsic relationship between semantic information such as room type, furniture category, and material and spatial depth features, thereby achieving a precise understanding and representation of the spatial layout of empty rooms. During training, a loss function is used to measure the difference between the predicted depth and the true depth map, continuously improving the model's generalization ability and prediction accuracy. Finally, after sufficient training, the obtained depth map control model can generate depth information that conforms to the actual spatial geometry.

[0047] See Figure 8 , Figure 8 A flowchart of a semantic segmentation control model training method provided in this application embodiment is shown, which can be implemented through steps S801-S805: S801: Obtain the second indoor image set.

[0048] To train a semantic segmentation control model capable of accurately identifying and segmenting different areas and furniture categories within an interior space, a second set of interior images (such as...) can be acquired first. Figure 6 This image set consists of multiple samples of interior scenes with furniture. Each sample not only showcases rich furniture layouts and spatial structures but also comes with detailed object labels. These labels comprehensively cover the room type information corresponding to the image, such as bedroom, living room, or kitchen, while also specifying the category of all furniture in the image (e.g., sofa, bed, bookcase) and the specific furniture material (e.g., fabric, wood, metal). Through precise annotation of multi-dimensional attributes, the dataset ensures that it contains not only visual appearance information but also the semantic features and material properties of the furniture, providing strong supervision signals for subsequent model training. This meticulous and systematic data preparation helps the semantic segmentation control model improve its ability to identify and segment various types of furniture and spatial areas when dealing with complex interior environments.

[0049] S802: Perform furniture removal on each of the second indoor image samples in the second indoor image set to obtain a second empty room image set.

[0050] To better extract the structural and semantic information of the interior space and reduce the interference of furniture on the spatial outline and background environment, furniture removal processing is required for each image sample with furniture in the second interior image set. This process utilizes advanced image segmentation and inpainting techniques to first accurately identify various furniture regions in the image, including their shape, boundaries, and location. These furniture regions are then removed from the image and intelligently filled using the texture and color information of the surrounding environment, ensuring that the removed image remains visually natural and continuous, avoiding obvious gaps or abnormal areas. After furniture removal processing, the generated second empty room image set (such as...) Figure 7 A collection of such images, Figure 7 for Figure 6 The depth map retains only the basic spatial structure of the room, such as key elements like walls, floors, and ceilings, without any furniture information. This pristine spatial image not only helps subsequent semantic segmentation models learn and understand the semantic boundaries of space more accurately, but also improves the model's ability to distinguish different spatial regions, thus providing a data foundation for high-quality interior scene generation and editing.

[0051] S803: Obtain the semantic segmentation map of each image in the second empty room image set to obtain a semantic segmentation image set.

[0052] To gain a deeper understanding of the various spatial regions and their semantic attributes within the empty room images, and to further enhance the semantic parsing capabilities of indoor scenes, the next step is to perform semantic segmentation processing on each image in the second empty room image set to obtain a semantically segmented image set (e.g., ...). Figure 9 A collection of such images, Figure 9 for Figure 6 (Semantic segmentation images). By employing advanced semantic segmentation algorithms, the model can divide different regions in an image into several categories with clear semantic meanings according to their function and structure, such as basic structural elements like walls, floors, ceilings, doors, and windows, as well as other possible spatial details. This process not only extracts spatial boundary information from the image but also assigns a corresponding semantic label to each pixel, thus forming a high-precision semantic segmentation map. Summarizing the semantic segmentation results of all empty room images yields a complete semantic segmentation image set. This semantic segmentation image set serves as an important input for subsequent model training, effectively assisting the semantic segmentation control model in learning the semantic structure and spatial relationships of interior spaces, enhancing the model's ability to discriminate different spatial regions and its depth of understanding, and providing solid data support for achieving more refined interior scene generation and intelligent editing.

[0053] S804: Pair each second indoor image sample in the second indoor image set with the corresponding image in the corresponding depth image set to obtain multiple second training sample pairs.

[0054] To achieve accurate training of the semantic segmentation control model, each image sample with furniture in the second indoor image set must be accurately paired with the corresponding semantic segmentation map in the corresponding semantic segmentation image set, thus forming multiple second training sample pairs. Through this pairing, the training data not only contains rich visual information (i.e., the real scene of furniture and environment in the indoor space) but also integrates clear spatial semantic annotations, enabling the model to simultaneously capture detailed features in the images and their corresponding semantic categories during the learning process. This multimodal and multidimensional data combination provides sufficient supervision signals for the neural network, enabling it to better understand the functional attributes and boundary relationships of different regions, thereby improving the segmentation and recognition capabilities of various furniture and spatial structures in complex indoor environments. Furthermore, accurate image-to-semantic segmentation map matching ensures the consistency and effectiveness of the training samples, laying a solid foundation for the optimization and generalization capabilities of the ControlNet Segmentation network, ultimately promoting the realization of high-quality indoor space semantic parsing and intelligent generation tasks.

[0055] S805: Input the multiple second training sample pairs into the semantic segmentation conditional control network to train the model and obtain the semantic segmentation control model.

[0056] To enable the semantic segmentation conditional control network to effectively learn the complex correspondence between visual information and semantic labels in interior spaces, multiple second training sample pairs are input into the network for systematic model training. Each training sample pair consists of an interior image with furniture and its corresponding precise semantic segmentation map. This multimodal input provides the network with rich and detailed supervision signals, enabling it to simultaneously capture visual features such as texture and shape in the image, as well as the semantic category information corresponding to each region. During training, the network continuously optimizes the loss function, gradually improving its ability to recognize and segment different room types, furniture categories, and materials, while enhancing its perception and understanding of spatial structural boundaries. After sufficient iteration and parameter tuning, the network finally obtains a highly accurate and robust semantic segmentation control model, capable of accurately dividing interior images into different semantic regions in practical applications.

[0057] In one possible implementation, the training process of the parameter fine-tuning model includes A1-A2: A1: Obtain the third indoor image set.

[0058] To effectively improve the model's understanding and representation of indoor scenes, a third set of indoor images is first required. This image set consists of multiple third-room image samples with furniture. Each sample not only contains rich visual information but is also annotated with detailed auxiliary labels. The auxiliary labels specifically include the room type of each image sample, as well as important attributes such as the category and material of all furniture in the image. This information provides the model with additional semantic descriptions, helping to enhance the model's ability to recognize and distinguish spatial structures and furniture features. Simultaneously, the auxiliary labels also contain specially designed trigger words. These trigger words exist as placeholders, their main function being to prevent the model from directly learning or extracting the actual semantic content from the auxiliary labels during training, thereby preventing the model from over-relying on these labels and causing a decline in generalization performance. Through this design, the auxiliary labels can provide the model with necessary contextual information while effectively avoiding semantic interference and overfitting risks, ensuring that the model maintains good robustness and generalization ability when trained using additional semantic descriptions, thus laying a solid foundation for subsequent parameter fine-tuning based on this data.

[0059] A2: Input the third indoor image set into the LoRA model for model training to obtain a parameter fine-tuning model.

[0060] After acquiring and constructing a third set of indoor images with rich auxiliary labels, the next step is to input this image set into a Low-Rank Adaptation (LoRA) model for systematic training. The LoRA model effectively fine-tunes the parameters of a large pre-trained model by introducing a low-rank matrix, making the training process both efficient and computationally efficient. Utilizing the diverse samples and accompanying auxiliary label information from the third set of indoor images, the LoRA model can further enhance its ability to identify and understand detailed features such as room type, furniture category, and material in indoor scenes while maintaining the original model's capabilities. During training, the model gradually adjusts its internal parameters, minimizing prediction errors by optimizing the objective function, thus better adapting the model to specific task requirements. After sufficient iteration and training, the final parameter-fine-tuned model not only inherits the powerful expressive capabilities of the pre-trained model but also integrates knowledge and adaptation capabilities tailored to indoor image features. This fine-tuned model can more accurately capture and parse complex indoor environmental semantics, providing solid technical support for subsequent automated interior design, intelligent scene reconstruction, and virtual reality applications, while significantly improving the overall system performance and response efficiency.

[0061] See Figure 10 , Figure 10 This is a schematic diagram of an indoor scene image generation device provided in an embodiment of this application. Figure 10As shown, the indoor scene image generation device includes: The acquisition and elimination unit 1001 is used to acquire the image to be processed and user input information, and to perform furniture removal on the image to be processed to obtain an empty room image; the image to be processed is an indoor scene image with furniture; the user input information includes the room type and target decoration style of the image to be processed; The prompt word generation unit 1002 is used to generate image prompt words based on a preset prompt word template and the user input information, and to obtain the depth map and semantic segmentation map of the empty room image; The feature encoding unit 1003 is used to input the depth map and semantic segmentation map of the empty room image into the pre-trained depth map control model and semantic segmentation control model respectively for feature encoding to obtain depth control features and semantic segmentation control features. The parameter adjustment unit 1004 is used to adjust the parameters of the raw image large model using the depth control features and the semantic segmentation control features to obtain the target raw image model; the raw image large model has been fine-tuned by the parameter fine-tuning model. The image generation unit 1005 is used to input the empty room image and the image prompt into the target image model to generate an image and obtain a target image.

[0062] In one possible implementation, the device further includes: The first acquisition unit is used to acquire a first indoor image set; the first indoor image set includes multiple first indoor image samples with furniture, each first indoor image sample is marked with a target label; the target label includes the room type of the first indoor image sample and the furniture category and furniture material of all furniture in the first indoor image sample; The first furniture removal unit is used to remove furniture from each of the first indoor image samples in the first indoor image set to obtain a first empty room image set. The second acquisition unit is used to acquire the depth map of each image in the first empty room image set to obtain a depth image set; The first pairing unit is used to pair each first indoor image sample in the first indoor image set with the corresponding image in the corresponding depth image set to obtain multiple first training sample pairs; The first model training unit is used to input the plurality of first training sample pairs into the depth map conditional control network for model training to obtain the depth map control model.

[0063] In one possible implementation, the device further includes: The third acquisition unit is used to acquire a second indoor image set; the second indoor image set includes multiple second indoor image samples with furniture, each second indoor image sample is marked with a target label; the target label includes the room type of the second indoor image sample and the furniture category and furniture material of all furniture in the second indoor image sample; The second furniture removal unit is used to remove furniture from each of the second indoor image samples in the second indoor image set to obtain a second empty room image set. The fourth acquisition unit is used to acquire the semantic segmentation map of each image in the second empty room image set to obtain a semantic segmentation image set; The second pairing unit is used to pair each second indoor image sample in the second indoor image set with the corresponding image in the corresponding depth image set to obtain multiple second training sample pairs; The second model training unit is used to input the multiple second training sample pairs into the semantic segmentation conditional control network for model training, thereby obtaining the semantic segmentation control model.

[0064] In one possible implementation, the device further includes: The fifth acquisition unit is used to acquire a third indoor image set; the third indoor image set includes multiple third indoor image samples with furniture, and each third indoor image sample is labeled with an auxiliary label; the auxiliary label includes the room type of the third indoor image sample, the furniture category and furniture material of all furniture in the third indoor image sample, and a trigger word; the auxiliary label is used to provide additional semantic description; the trigger word exists as a placeholder to prevent the model from learning the auxiliary label or extracting actual semantics; The third model training unit is used to input the third indoor image set into the LoRA model for model training to obtain a parameter fine-tuning model.

[0065] In addition, this application embodiment also provides an indoor scene image generation device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the indoor scene image generation method as described above.

[0066] In addition, this application embodiment also provides a computer-readable storage medium storing instructions that, when executed on a terminal device, cause the terminal device to perform the indoor scene image generation method described above.

[0067] This application first performs furniture removal on the image to be processed, transforming the non-empty room into an empty room image that retains only the core spatial structure (walls, doors, windows, etc.). This operation fundamentally removes the "constraint of the original furniture on the generation process," avoiding the limitation of existing technologies that "can only skin the original furniture under high ControlNet weights." Subsequently, feature encoding of the empty room image is performed through a depth map control model and a semantic segmentation control model, which can more accurately lock the spatial framework of the empty room (such as the position of doors and windows, the orientation of walls, and spatial depth) without relying on ControlNet weight adjustment to balance "structure" and "furniture," fundamentally eliminating the problem of "abnormal spatial structure caused by low weights." Meanwhile, the raw image model has been pre-optimized through parameter fine-tuning. Based on the stable spatial framework of an empty room, it can generate diverse furniture that conforms to the logic of spatial dimensions by combining raw image prompts (rather than being limited by the original furniture layout). Ultimately, it achieves "both strictly preserving the authenticity of the original room's spatial structure and meeting users' personalized design needs for furniture layout and categories", completely avoiding the two types of defects caused by the imbalance of ControlNet weights in existing technologies.

[0068] The foregoing provides a detailed description of an indoor scene image generation method, apparatus, device, and storage medium provided in this application. The various embodiments are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make various improvements and modifications to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of the claims of this application.

[0069] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A method of generating an indoor scene image, characterized by, The method comprises: obtaining a to-be-processed image and user input information, and performing furniture removal on the to-be-processed image to obtain an empty room image; the to-be-processed image is an indoor scene image with furniture; the user input information comprises a room type and a target decoration style of the to-be-processed image; generating a prompt word for a generated image based on a preset prompt word template in combination with the user input information, and obtaining a depth map and a semantic segmentation map of the empty room image; inputting the depth map and the semantic segmentation map of the empty room image into a pre-trained depth map control model and a semantic segmentation control model respectively for feature encoding to obtain depth control features and semantic segmentation control features; performing parameter adjustment on a large generated image model by using the depth control features and the semantic segmentation control features to obtain a target generated image model; the large generated image model has been subjected to parameter fine-tuning by a parameter fine-tuning model; inputting the empty room image and the prompt word for the generated image into the target generated image model for image generation to obtain a target image.

2. The method of claim 1, wherein, The training process of the depth map control model comprises: obtaining a first indoor image set; the first indoor image set comprises a plurality of first indoor image samples with furniture, and each first indoor image sample is labeled with a target label; the target label comprises a room type of the first indoor image sample and furniture categories and furniture materials of all furniture in the first indoor image sample; performing furniture removal on each first indoor image sample in the first indoor image set to obtain a first empty room image set; obtaining a depth map of each image in the first empty room image set to obtain a depth image set; pairing each first indoor image sample in the first indoor image set with a corresponding image in the depth image set to obtain a plurality of first training sample pairs; inputting the plurality of first training sample pairs into a depth map conditional control network for model training to obtain the depth map control model.

3. The method of claim 1, wherein, The training process of the semantic segmentation control model comprises: obtaining a second indoor image set; the second indoor image set comprises a plurality of second indoor image samples with furniture, and each second indoor image sample is labeled with a target label; the target label comprises a room type of the second indoor image sample and furniture categories and furniture materials of all furniture in the second indoor image sample; performing furniture removal on each second indoor image sample in the second indoor image set to obtain a second empty room image set; obtaining a semantic segmentation map of each image in the second empty room image set to obtain a semantic segmentation image set; pairing each second indoor image sample in the second indoor image set with a corresponding image in the depth image set to obtain a plurality of second training sample pairs; inputting the plurality of second training sample pairs into a semantic segmentation conditional control network for model training to obtain the semantic segmentation control model.

4. The method of claim 1, wherein, The training process of the parameter fine-tuning model comprises: obtain a third indoor image set; the third indoor image set includes a plurality of third indoor image samples with furniture, each third indoor image sample is marked with an auxiliary label; the auxiliary label includes a room type of the third indoor image sample, a furniture category and a furniture material of all furniture in the third indoor image sample, and a trigger word; the auxiliary label is used to provide additional semantic description; the trigger word exists as a placeholder to prevent the model from learning or extracting actual semantics from the auxiliary label; input the third indoor image set into the LoRA model for model training to obtain a parameter fine-tuning model.

5. An indoor scene image generation apparatus characterized by comprising: The device comprises: an acquisition unit configured to acquire a to-be-processed image and user input information, and remove furniture from the to-be-processed image to obtain an empty room image; the to-be-processed image is an indoor scene image with furniture; the user input information includes a room type and a target decoration style of the to-be-processed image; a prompt word generation unit configured to generate a generated image prompt word based on a preset prompt word template in combination with the user input information, and acquire a depth map and a semantic segmentation map of the empty room image; a feature encoding unit configured to input the depth map and the semantic segmentation map of the empty room image into a pre-trained depth map control model and a semantic segmentation control model respectively for feature encoding to obtain depth control features and semantic segmentation control features; a parameter adjustment unit configured to adjust parameters of a generated image large model using the depth control features and the semantic segmentation control features to obtain a target generated image model; the generated image large model has been parameter-adjusted by a parameter fine-tuning model; an image generation unit configured to input the empty room image and the generated image prompt word into the target generated image model for image generation to obtain a target image.

6. The apparatus of claim 5, wherein, The device further comprises: a first acquisition unit configured to acquire a first indoor image set; the first indoor image set includes a plurality of first indoor image samples with furniture, each first indoor image sample is marked with a target label; the target label includes a room type of the first indoor image sample and a furniture category and a furniture material of all furniture in the first indoor image sample; a first furniture removal unit configured to remove furniture from each first indoor image sample in the first indoor image set to obtain a first empty room image set; a second acquisition unit configured to acquire a depth map of each image in the first empty room image set to obtain a depth image set; a first pairing unit configured to pair each first indoor image sample in the first indoor image set with a corresponding image in the depth image set to obtain a plurality of first training sample pairs; a first model training unit configured to input the plurality of first training sample pairs into a depth map conditional control network for model training to obtain the depth map control model.

7. The apparatus of claim 5, wherein, The device further comprises: a third obtaining unit, configured to obtain a second indoor image set; the second indoor image set comprises a plurality of second indoor image samples with furniture, each second indoor image sample is labeled with a target label; the target label comprises a room type of the second indoor image sample and furniture categories and furniture materials of all furniture in the second indoor image sample; a second furniture elimination unit, configured to perform furniture elimination on each second indoor image sample in the second indoor image set respectively, to obtain a second empty room image set; a fourth obtaining unit, configured to obtain semantic segmentation maps of images in the second empty room image set, to obtain a semantic segmentation image set; a second pairing unit, configured to pair each second indoor image sample in the second indoor image set with a corresponding image in the depth image set, to obtain a plurality of second training sample pairs; a second model training unit, configured to input the plurality of second training sample pairs into a semantic segmentation conditional control network for model training, to obtain the semantic segmentation control model.

8. The apparatus of claim 5, wherein, The apparatus further comprises: a fifth obtaining unit, configured to obtain a third indoor image set; the third indoor image set comprises a plurality of third indoor image samples with furniture, each third indoor image sample is labeled with an auxiliary label; the auxiliary label comprises a room type of the third indoor image sample, furniture categories and furniture materials of all furniture in the third indoor image sample, and a trigger word; the auxiliary label is used to provide additional semantic description; the trigger word exists as a placeholder, and is used to prevent the model from learning the auxiliary label or extracting actual semantics; a third model training unit, configured to input the third indoor image set into a LoRA model for model training, to obtain a parameter fine-tuning model.

9. An indoor scene image generation device, characterized by, comprise: a memory, a processor, and a computer program stored in the memory and executable on the processor, when the processor executes the computer program, the indoor scene image generation method in any one of claims 1-4 is implemented.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores instructions, when the instructions run on the terminal equipment, the terminal equipment executes the indoor scene image generation method in any one of claims 1-4.

Images