System and method for removing and rearranging furniture in a space, and for removing objects from a space.

Abstract

An exemplary system may access data in a multidimensional space representing a physical environment and use a first machine learning model to identify interior elements in the multidimensional space. Interior elements may represent furniture in the physical environment. To create the appearance of a space with the furniture removed, the system may mask one or more interior elements with a mask, filling each mask with an image of the physical environment, so that the space with the furniture removed has one or more interior elements that appear to be missing from the multidimensional space representing the physical environment. The system may provide all or part of the space with the furniture removed for display.

Description

Technical Field

[0001] This application generally relates to techniques for modifying digital models, and more specifically, to the removal of interior elements of a digital model generated based on a real-world environment.

Background Art

[0002] Walkthroughs of virtual environments are becoming increasingly common on the Internet. For example, many users have the option to navigate through different houses virtually using a real estate network by doing a walkthrough. The ability to navigate through a house before physically visiting it has revolutionized home buying, allowing many more people to freely visit properties even if they are not geographically convenient to travel to.

[0003] Unfortunately, users are unable to interact with these virtual environments. As a result, users have to guess and rely on their imagination as to what a room, house, or other environment would look like without furnishings or decorations. Similarly, if a user wants to consider alternative users, or furnishings, decorations, or styles, they have to guess dimensions, possible furnishings arrangements, preferred styles, etc.

Summary of the Invention

[0004] In various embodiments, the techniques described herein relate to a non-temporal computer-readable medium including executable instructions, wherein the executable instructions are executable by one or more processors to perform a method, the method comprising: accessing data in a multidimensional space representing a physical environment; identifying interior elements in the multidimensional space using a first machine learning model, wherein the interior elements represent furnishings in the physical environment; masking one or more of the interior elements with a mask; filling each of the masks with an image of the physical environment to create the appearance of a furniture-removed space, wherein the furniture-removed space has one or more interior elements that appear to be missing from the multidimensional space representing the physical environment; and providing all or part of the furniture-removed space for display.

[0005] In some embodiments, the physical environment is a room with furniture arranged in it. The multidimensional space can be a 2D representation of the physical environment. In some embodiments, the 2D representation of the physical environment is used to generate a corresponding 3D representation of the physical environment. In various embodiments, the multidimensional space is a 3D representation of the physical environment.

[0006] The data in the multidimensional space may be a textured 3D mesh. Interior elements in the multidimensional space may further include at least one wall in the physical environment. The first machine learning model may be a semantic segmentation neural network. Filling each of the masks with an image of the physical environment may involve applying modifications (inpainting) to the masks.

[0007] The method may further include receiving a selection of at least one design style type from the user, and selecting previously identified interior elements in a multidimensional space based on the at least one design style type from the user, wherein masking at least a portion of the interior elements includes masking at least a portion of the interior elements that are at least one design style type.

[0008] In some embodiments, the method further includes receiving a selection of at least one design style type from the user, and selecting interior elements in a previously identified multidimensional space based on the at least one design style type from the user, wherein masking at least a portion of the interior elements includes masking at least a portion of the interior elements that do not belong to at least one design style type.

[0009] In some embodiments, the techniques described herein relate to receiving an additional request from a user, the additional request comprising one or more additional interior elements to be added to a furniture-removed space, identifying a representation of the one or more additional interior elements, arranging the one or more additional interior elements in the furniture-removed space, and providing the one or more additional interior elements arranged in the furniture-removed space for display in all or part of the furniture-removed space.

[0010] In some embodiments, receiving an additional request from a user includes receiving a prompt from the user and applying the prompt to a large language model to receive a response from the large language model, the response identifying one or more additional interior elements.

[0011] An exemplary system includes at least one processor and memory containing executable instructions. The executable instructions may be executable by at least one processor to access data in a multidimensional space representing a physical environment and to identify interior elements in the multidimensional space using a first machine learning model, wherein the interior elements represent furnishings in the physical environment; to identify; to mask at least a portion of the interior elements with a mask; and to fill each of the masks with an image of the physical environment to create the appearance of a furniture-free space, wherein the furniture-free space has at least a portion of the interior elements that appear to be missing from the multidimensional space representing the physical environment; and to provide all or part of the furniture-free space for display.

[0012] In some embodiments, the physical environment is a room with furniture arranged in it. The multidimensional space can be a 2D representation of the physical environment. In some embodiments, the 2D representation of the physical environment is used to generate a corresponding 3D representation of the physical environment. In various embodiments, the multidimensional space is a 3D representation of the physical environment.

[0013] The data in the multidimensional space may be a textured 3D mesh. Interior elements in the multidimensional space may further include at least one wall in the physical environment. The first machine learning model may be a semantic segmentation neural network. Filling each of the masks with an image of the physical environment may include applying modifications to the masks.

[0014] An executable instruction may further be executable by at least one processor to receive a selection of at least one design style type from a user and, based on the at least one design style type from the user, select interior elements in a previously identified multidimensional space, wherein masking at least a portion of the interior elements includes at least a portion of the interior elements that are at least one design style type.

[0015] In some embodiments, the executable instructions may be executable by at least one processor to receive a selection of at least one design style type from the user and to select interior elements in a previously identified multidimensional space based on the at least one design style type from the user, and masking at least a portion of the interior elements includes masking at least a portion of the interior elements that are not of the at least one design style type.

[0016] In various embodiments, the executable instructions may be executable by at least one processor to receive an additional request from a user, the additional request comprising one or more additional interior elements to be added to a furniture-removed space; identify a representation of the one or more additional interior elements; place the one or more additional interior elements in the furniture-removed space; and provide, for display purposes, the one or more additional interior elements placed in the furniture-removed space to all or part of the furniture-removed space.

[0017] In some embodiments, an executable instruction that can be executed by at least one processor to receive an additional request from a user includes an executable instruction further configured by at least one processor to receive a prompt from a user and to apply the prompt to a large language model to receive a response from the large language model, the response identifying one or more additional interior elements. [Brief explanation of the drawing]

[0018] [Figure 1] Figure 1 shows an exemplary spatial modification system in several embodiments. [Figure 2]Figure 2 shows a method, in some embodiments, for removing furniture and rearranging one or more parts (e.g., rooms) of a digital model of a real-world space (referred to herein as “space”). [Figure 3A] Figure 3A is an exemplary user interface that may be used for removing furniture from a space in some embodiments. [Figure 3B] Figure 3B shows an exemplary user interface that may be used for removing furniture from a space in some embodiments. [Figure 3C] Figure 3C is an exemplary user interface that may be used for removing furniture from a space in some embodiments. [Figure 4A] Figure 4A is an exemplary user interface that may be used to remove an object from space in some embodiments. [Figure 4B] Figure 4B is an exemplary user interface that may be used to remove an object from space in some embodiments. [Figure 4C] Figure 4C is an exemplary user interface that may be used to remove an object from space in some embodiments. [Figure 5A] Figure 5A is an exemplary user interface that may be used to arrange furniture in a space in several embodiments. [Figure 5B] Figure 5B is an exemplary user interface that may be used to arrange furniture in a space in some embodiments. [Figure 5C] Figure 5C is an exemplary user interface that may be used to arrange furniture in a space in some embodiments. [Figure 5D] Figure 5D is an exemplary user interface that may be used to arrange furniture in a space in several embodiments. [Figure 6]FIG. 6 is a block diagram of an exemplary digital device that can be utilized by the techniques described herein, according to some embodiments.

BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Throughout the drawings, like reference numerals are understood to refer to like parts, components, and structures.

[0020] This description pertains to a system that receives or generates data regarding a space (room, house, other building, outdoor area, etc.), generates a representation of an alternative version of the space using different furniture, materials, etc., and then displays these alternative versions to a user. One embodiment starts with a digital twin of a space, removes all fixtures and other furniture, adds new furniture that matches a style prompt such as "mid-century modern", and displays a new version of the space so that the user can see how it would look with different furniture.

[0021] Many places in this description refer to "internal elements", "furniture", or "furniture and / or materials". These terms are not intended to be limiting and may include any internal elements such as fixtures, wall hangings, exercise equipment, appliances, window treatments, paint, wallpaper, flooring, kitchen and bathroom fixtures and cabinets, lighting, plants, landscaping elements, walls, fireplaces, ceiling fans, handrails and stairs, and other goods and elements that can be modified with respect to the physical space.

[0022] Figure 1 shows an exemplary spatial modification system 102 in several embodiments. The spatial modification system 102 may be run on a digital device. A digital device is any system having memory and a processor. Digital devices are further discussed herein (for example, with respect to Figure 6). The spatial modification system 102 may include a communication module 104, an identification module 106, a deletion module 108, a modification module 110, a furniture placement module 112, a layout module 114, a model training module 116, a rule module 118, a rule data store 120, a digital asset data store 122, and a digital spatial data store 124. It will be recognized that the spatial modification system 102 may include more or fewer modules, as shown in Figure 1. Furthermore, it will be recognized that modules may reside on any number of digital devices communicating with each other (for example, the spatial modification system 102 may include any number of digital devices). Modules may be hardware, software, or a combination of both hardware and software.

[0023] The communication module 104 exchanges information with the spatial modification system 102 and with any number of modules within the spatial modification system 102. In some embodiments, the communication module 104 may receive digital models (e.g., digital twins or digital spaces representing real-world environments) from a source 126. Source 126 may be one or more digital devices and / or data storage capable of providing any number of digital models or parts of digital models. In one example, source 126 is a third party or a user. The user may provide one or more digital models to the communication module 104 by providing commands to one or more sources 126 (e.g., a web server, web platform, data lake, digital device, etc.). Alternatively, the user may provide commands to the spatial modification system 102 to retrieve digital models or parts of digital models from any number of sources 126 (e.g., via one or more networks such as the Internet, by the communication module 104 which may have access to a source API).

[0024] In some embodiments, the communication module 104 may receive 2D data, 3D data, and / or digital models, as discussed herein.

[0025] The identification module 106 can identify one or more internal elements within a digital model, 2D data, and / or 3D data. In various embodiments, the identification module 106 can identify internal elements (e.g., fixtures, equipment, load-bearing structures, non-load-bearing structures, walls, lighting, etc.) by applying a rule-based approach, an AI model approach (e.g., CNN), or a combination of both. Rules and internal elements are further described herein.

[0026] The deletion module 108 can delete one or more internal elements identified by the identification module 106 from the digital model. In some embodiments, the deletion module 108 deletes internal elements identified in 2D data, 3D data, and / or the digital model. In various embodiments, the deletion module 108 can mask and fill in the internal elements to make them appear deleted in the 2D data, 3D data, and / or the digital model.

[0027] The modification module 110 can optionally modify the digital model. In various embodiments, the modification module 110 can apply different colors, textures, styles, etc., to 2D data, 3D data, and / or surface or internal elements within the digital model.

[0028] The furniture placement module 112 may optionally add additional internal elements to the digital model. In various embodiments, the user may manually select any number of internal elements to add to a part of the digital model (e.g., a room) using a user interface (UI). In various embodiments, a catalog may be provided to help the user find internal elements. The catalog may allow searching for internal elements by type, design style, brand, cost, etc. In various embodiments, the furniture placement module 112 may utilize an AI model (e.g., CNN) and / or a rule-based approach to select and add additional internal elements to the digital model, or to recommend additional internal elements to the user. Based on user input, the furniture placement module 112 may use an AI model and / or a rule-based approach to adjust and improve recommendations based on feedback.

[0029] In some embodiments, the furniture placement module 112 utilizes interior element rules and layout rules to help identify the correct interior elements for a particular room (for example, a sofa in the living room but not in the bathroom). Layout rules can be applied to position and orient interior elements. Exemplary layout rules may include not blocking doorways, placing sofas along walls, and leaving gaps between walls and different types of interior elements (for example, ensuring space for sitting in chairs or accessing toilets). In various embodiments, layout rules may help the furniture placement module 112 select the appropriate type of interior element to add to a room (for example, a sofa of a specific length, or removing an end table due to lack of space).

[0030] The layout module 114 can position and orient 2D data, 3D data, and / or internal elements within a digital model. In some embodiments, the layout module 114 can apply layout rules to position and orient internal elements (e.g., light fixtures, fittings, objects, walls, bars, etc.). In some embodiments, the furniture placement module 112 performs the layout operation, and the model training module 116 is not required.

[0031] The model training module 116 can train one or more machine learning (e.g., artificial intelligence) models. A machine learning model may be trained to identify and classify internal elements (e.g., to add metadata to internal elements to assist in identification, selection, and / or placement). While a model may be used to select internal elements to remove, other machine learning models may be trained to provide the user with the option to virtually stage a space using desired internal elements (e.g., fixtures that meet specific requirements such as desired style, shape, and color).

[0032] The rule module 118 can retrieve, modify, add, delete, and / or store rules (for example, in the rule data store 120). The rule module 118 can, for example, apply a rule approach to assist in the selection of internal elements, providing the user with guidance for selecting internal elements (for example, based on design style, type of internal element, spatial allocation), deleting internal elements, etc. These rules can replace or extend the AI ​​model, as discussed herein. In addition to, or instead of, the rule module 118 can retrieve, modify, add, delete, and / or store internal element rules and / or layout rules in the rule data store 120, as discussed herein.

[0033] The rule data store 120 is an arbitrary data structure (e.g., a database, table, file system, etc.) that stores rules enabling the selection of internal elements, provides access to these rules, and provides the user with guidance for selecting internal elements (e.g., based on design style, type of internal element, spatial allocation), deleting internal elements, etc. These rules may replace or extend the AI ​​model as discussed herein. In addition, or instead, the rule data store 120 may store or provide access to internal element rules and / or layout rules as discussed herein. The rule data store 120 may be local or remote to the spatial modification system 102. In various embodiments, the rule data store 120 may include a local portion and a portion that is moved to the spatial modification system 102 (e.g., the rule data store 120 may comprise multiple databases containing similar or different information across different digital devices and networks).

[0034] The digital asset data store 124 is any data structure (e.g., a database, table, file system, etc.) that can store digital assets representing internal elements such as added, identified, deleted, evaluated for guidance, and used for training AI models, and provide access to these digital assets. The digital asset data store 124 can be local or remote to the spatial modification system 102. In various embodiments, the digital asset data store 124 may include a local portion and a portion that is moved to the spatial modification system 102 (for example, the digital asset data store 124 may have multiple databases containing similar or different information across different digital devices and networks).

[0035] In some embodiments, the spatial modification system 102 may include or communicate with a model generation system. In this example, the model generation system may generate a digital model based on 2D and / or 3D data. An example of a model generation system can be found in U.S. Patent No. 11,094,137, titled "Employing three-dimensional (3D) data predicted from two-dimensional (2D) images using neural networks for 3D modeling applications and other applications," which is incorporated herein by reference. The identification module 106 will be recognized to be able to identify internal elements from the 2D and / or 3D data before, after, or during the creation of a 3D model using the 2D and / or 3D data.

[0036] In various embodiments, the spatial modification system 102 (for example, utilizing the module of U.S. Patent No. 11,094,137 discussed herein) uses a 2D image of space to generate a corresponding 3D representation (for example, via photogrammetry, monocular depth estimation, neural radiance field (NeRF), and / or other 2D-to-3D techniques). In this case, the input 2D image can be received or captured with or without auxiliary positioning and orientation data. A 2D image with positioning and orientation data can be considered 3D data.

[0037] In various embodiments, the spatial modification system 102 (for example, utilizing the module of U.S. Patent No. 11,094,137 discussed herein) generates 3D data about the space from a 2D floor plan or other layout information, and then infers the height of the space. Other possible representations of the 3D data about the space include textured or untextured meshes, point clouds, depth maps, and / or implicit or latent representations within NeRF or other neural networks.

[0038] Figure 2 shows a method for removing furniture and rearranging one or more parts (e.g., rooms) of a digital model of a real-world space (referred to herein as the “space”), in several embodiments. In step 202, the communication module 104 receives 2D or 3D data (e.g., a digital model or digital twin) associated with the real-world space. The 2D or 3D data may be received from one or more sources. Sources may include data storage, systems, etc. In one example, an external source may include a third party that generates and / or stores a digital model of the space. In one example, the communication module 104 may use an API to retrieve the digital model from the third party over a network (e.g., from the internet). In another example, the communication module 104 may receive the digital model from the third party over a network (e.g., via a push of the digital model from the third-party source). In some embodiments, the 2D or 3D data is retrieved from a local or remote data structure (e.g., a database). In various embodiments, a digital model is generated, and then the communication module 104 receives the digital model.

[0039] In some embodiments, the communication module 104 may receive a digital twin of space. The digital twin may include a textured 3D mesh. The digital twin is a virtual model or representation of an environment (e.g., a room, building, and / or environment). The digital twin of space may enable interaction within the space. The 3D mesh may be a collection of vertices, edges, and faces that define the shape of objects, structures, surfaces, etc., within the digital twin. The mesh may provide a geometric framework that depicts the dimensions of (one or more) physical properties and the contours of structures. The textured 3D mesh may be generated by texture mapping (e.g., using 2D images and / or point cloud data of a real-world environment).

[0040] Other spatial data may also be received or generated. For example, 2D images may be received or captured directly, or generated from 3D representations. Data concerning the structure, connectivity, and / or use of different parts of the space may also be received or generated (e.g., via metadata) to label a particular part of the space as a “kitchen,” or to provide information about the location of doors and windows within the space. In some embodiments, the spatial modification system 102 utilizes an automated classification system discussed in relation to U.S. Patent No. 11,670,076, titled “Automated Classification Based on Phot-Realistic Image / Model Mappings,” which is incorporated herein by reference. In one example, the spatial modification system 102 uses 2D data, 3D data, and / or the automatic classification of rooms and spaces in the relevant digital model (for example, utilizing semantic segmentation) to identify and classify rooms, spaces, parts of spaces, etc., and generates labels and / or other classifications which can be used for the selection of interior elements and / or the placement of interior elements (for example, utilizing the interior element rules and / or layout rules discussed herein).

[0041] In step 204, the identification module 106 identifies one or more internal elements within the digital model. In some embodiments, the identification module 106 identifies internal elements (e.g., furniture, objects, fixtures, load-bearing and unload-bearing components, properties, etc.) within a 2D image of space (panorama or planar image) using publicly available pre-trained semantic segmentation neural networks such as ViT-Adapter-L+Mask2Former+BEiTv2 pre-trained, CLIPSeg, and / or many other semantic segmentation or other networks capable of identifying and segmenting objects or regions within a 2D image. This identifies parts of the image as belonging to a class such as “wall” or “chair”. In various embodiments, the identification module 106 generates labels that can be associated (e.g., via metatags, tags, metadata, etc.) with relevant structures indicating the class, type of internal element, placement of the internal element, etc.

[0042] In some embodiments, the identification module 106 may utilize multiple neural networks to perform any of the following: classify internal elements from digital models, 2D data, and / or 3D data; aggregate results (e.g., add or overwrite labels and tags); and / or score results for later comparison. In some embodiments, when different neural networks are compared with other neural network classifications for a particular aspect, different classification aspects may be weighted to receive higher scores.

[0043] The identification module 106 can identify internal elements in 3D space using a publicly available, pre-trained semantic segmentation neural network. As discussed above, this allows each part of an image to be identified as belonging to a class such as "wall" or "chair". Similarly, as discussed herein, the identification module 106 can generate labels that can be associated with the relevant internal elements (e.g., via metatags, tags, metadata, etc.) indicating the class, type of internal element, location of the internal element, etc.

[0044] It will be recognized that the identification module 106 can identify one or more internal elements of a digital model, 2D data, and / or 3D data at any time. For example, a digital model may be received, and the identification module 106 can identify the digital model, 2D data, and / or one or more internal elements of the 3D data. The 2D data of the digital model, and / or further internal elements of one or more other models or 3D data from different points in time may be identified. Labels and tags may be created and added to the database or digital model over time.

[0045] In step 206, the delete module 108 deletes one or more internal elements from the digital model (for example, existing internal elements of the digital model identified in step 204). It will be recognized that the delete module 108 may delete any internal elements and / or structures from the digital model. For example, the delete module 108 may delete one or more objects, materials, textures, walls, floors, etc., to create a "de-furnitured" version of the digital model (for example, a "de-furnitured" version of a space). What needs to be deleted depends on the specific application and / or user input, and it may not be necessary to delete anything in this step.

[0046] The deletion module 108 can mask each portion of the image identified as an internal element to be deleted. In some embodiments, the deletion module 108 can mask each portion of the image identified as an internal element to be deleted and fill one or more of these portions with an image of the room or background as it would be without these internal elements.

[0047] In some embodiments, the deletion module 108 may fill in one or more portions of each image using a publicly available, pre-trained modification neural network, such as stable-diffusion-2-inpainting, with a text prompt such as "empty room". Digital modification is the task of generating pixels to cover areas of an image. In one example, to generate a 3D representation of a furniture-removed version of a space, a mask from each of the above 2D images is projected onto a 3D mesh of the space, using the known positioning and orientation of the camera that captured each 2D image, and the mesh faces corresponding to the masked image areas are removed from the 3D mesh. This may leave holes in the 3D mesh, which can be filled using techniques such as planar expansion to fill in walls or floors behind the removed objects. Combining these creates a 3D representation and 2D image of the furniture-removed version of the space.

[0048] In addition to furniture removal, or instead of furniture removal, materials may be replaced rather than removed. In one example, the removal module 108 may identify properties or internal elements to be modified, such as floor tiles, wall sections, or countertops. In an optional step 208, the modification module 110 may utilize one or more of the same semantic segmentation networks as above to assist in identifying and / or replacing internal elements or properties of internal elements in the digital space. For example, the modification module 110 may optionally use a publicly available, pre-trained instance segmentation neural network such as Segment Anything to separate different material sections that may be of the same class, such as two adjacent walls. The modification module 208 may replace or modify the removed sections above to add appearance, such as floor tiles, wall sections, or countertops.

[0049] In some embodiments, the identification module 106 may first identify internal elements and parts of the space in 3D, and then map these 3D identifications to 2D, rather than mapping from 2D to 3D as described above. For example, the identification module 106 may receive or generate a 2D or 3D floor plan, or other simplified representation of the shape of the space, and then use the differences between the complete 3D representation of the space and the simplified shape to identify furniture. In one example, a chair can be identified as an internal element because it exists in the complete 3D representation but not in the floor plan. Surfaces in the simplified representation may also be identified as walls or floors having materials that can be replaced by the modification module 110.

[0050] The identification module 106 may utilize additional or alternative methods to identify internal elements and materials from a 3D representation of space. In one example, the identification module 106 may perform a rule-based analysis of shape, or it may run a neural network that accepts a 3D point cloud or graph-type input and generates an output that describes the space in a way that can be used as input to another identification system, such as feature vectors that can be used as input to another neural network trained to identify furniture.

[0051] It will be recognized that the identification of internal elements, materials, and other parts of space does not need to be done exclusively in 3D or 2D first, but that techniques can be applied to both domains (2D and 3D) and aggregated and projected onto other domains as needed. For example, identification module 106 may identify furniture in a 2D image, these 2D detections are projected into 3D using information about the pose of the 2D image and the shape of the space, and the detections are then improved based on how they overlap in 3D, and the improved 3D detections may optionally be reprojected onto the 2D image.

[0052] Starting from a complete representation of the space, objects can be removed to arrive at a "furniture-free" version, or the furniture-free version of the space can be generated directly. The furniture placement module 112 can adapt a plane to the 3D representation until the main structure of the space (floor, walls, etc.) is reproduced, without adding details that match the details of the furniture placement. In various embodiments, the furniture placement module 112 can utilize a neural network to directly generate data about the furniture-free version of the space, given data about the original space as input. For example, a model that generates images from images can be trained to take an image of a furniture-placed space as input and produce an image of the furniture-free version of that space as output. Similarly, a bird's-eye view of the space can be provided to the network to generate a floor plan (unfurnished) view of the space as output. This can also be done in 3D, where the network takes images and / or 3D data about the space as input and produces a 3D representation of the furniture-free, i.e., simplified space as output. This example involves training and generating NeRF or Gaussian splatting representations of the space and can be combined with other techniques, such as using a diffusion-based prior distribution to fill in areas where furniture has been removed.

[0053] This identification and furniture removal step does not need to be performed on the entire space at once. For example, interior elements can be identified, modified, and / or removed (and new structures added) starting from individual rooms or sub-regions of a space, or it can be done one interior element or room at a time. This can be done in an automated, step-by-step manner (removing furniture from one object, such as a room, and then using the partially removed furniture result as input for removing furniture from the next object, such as a room), and / or it can be a user guide.

[0054] For example, the communication module 104 may provide a user interface (UI) and / or receive input from a UI generated by another device or system. In this example, the user may browse a digital model in the device's UI and interact with it by clicking on objects, deleting objects, or clicking on a room to remove furniture from a particular room, or the process may be guided by text prompts from the user, such as "delete the chairs in the dining room." The text prompts may be generated by or modified by a Large-Scale Language Model (LLM). Other forms of user guidance may include approving or rejecting suggestions made by the system, providing feedback on such suggestions, providing text or drawing prompts to guide the system to a better solution, and even manual editing such as editing 2D images or 3D data. Also, when describing these and other forms of user guidance, the user providing the guidance may not be the intended end user who verifies the final result; for example, a service company may have an operator providing this guidance to achieve better results without manual intervention from the intended end user.

[0055] The pre-trained networks discussed herein, as well as the other approaches described above, are merely examples, and many other networks exist that can perform similar functions. In addition, the system can be made more efficient by training or fine-tuning the network specifically for this task. For example, model training module 116 provides additional training for the stable-diffusion-2-inpainting network using a dataset of images of furniture-removed spaces to improve the generation of that type of image, rather than relying on the ability to generate all kinds of images, and rather relying on text prompts to make the image look like a version of a furniture-removed space. Such additional training can use training samples that match or approximate the input expected when the system is used, such as using input images with artifacts that are not present in the target image. For example, the training input image may contain imperfectly masked objects (or other internal elements) that are not present in the target image, or shadows or reflections of such objects. This makes the trained network more robust to such artifacts, such as the network learning that if an object is partially masked, the entire object and its shadow should be removed from the output image.

[0056] This technique or other techniques can be combined with an algorithm or network that determines how to combine the generated image and the input image to improve the visual quality of the output, for example, by using an input image in a relatively unchanged region and mixing it with the generated image. Other techniques for improving the results of a neural network may include the communication module 104 providing auxiliary data from different domains, such as using ControlNet to provide a simplified or furniture-free version of a depth image, normal vector image, and / or semantic segmentation image of a 3D mesh projected to match the 2D image when generating a 2D modified image of a space that appears to be furniture-free. The auxiliary data may also include the results of previous modifications or furniture removal, such as first removing furniture from one or more viewpoints and then conveying the furniture-removed image or 3D data as input to a later modification step, so that the later step can produce results that match the previous step or benefit from the context that was available in the previous step. In addition to correcting images individually or sequentially, the correction module 110 can also generate or correct images simultaneously using techniques such as correspondence-aware attention, thereby improving the consistency between the generated images.

[0057] Many of the approaches described for this step involve neural networks, but rule-based approaches, algorithmic approaches, or combinations of neural networks, rule-based approaches, and / or algorithmic approaches can be used. For example, geometric analysis can be used to remove everything that is not part of the spatial structure, generating a simplified version of the 3D mesh, and then the simplified mesh can be used to generate a 2D image with the furniture removed, filling in the areas of each 2D image that are projected onto the parts of the mesh that were removed during simplification. To generate the modified image, algorithmic techniques can be used to convey texture along an extended plane to fill the same areas in 3D.

[0058] In some embodiments, new interior elements may be added to a space or a portion of a space (e.g., one or more rooms within a space). New interior elements may be added to an existing space (e.g., one where furniture has not been removed) or to a space where furniture has been removed using the systems and methods described herein. Addition of interior elements may be manual (e.g., interior elements may be added to the space manually via a UI), automatic (e.g., the furniture placement module 112 may automatically add and place interior elements), or a combination of both. Furthermore, it will be recognized that the user or the automated system may be guided by the user's preferred instructions and / or previous selections / placements.

[0059] In step 210, the furniture placement module 112 may generate new interior elements and / or materials within the space in a reasonable manner, taking into account the shape and / or other characteristics of the space. For example, new fixtures must not penetrate walls or other fixtures, not block doorways, and be placed at a height appropriate for placement on the floor. One or more interior element rules and / or layout rules may be created (e.g., by a system user or operator) that define the function for appropriate height, limit penetration of walls and fixtures, and not block doorways. Interior element rules may include one or more rules indicating the types of interior elements that can be deleted or added to the digital space. Interior element rules may include, for example, types of interior elements for room types (e.g., a sofa for a living room, not a bathroom), and the probability of finding a particular type of interior element in a particular room (e.g., if the wall height of a game room is less than 10 feet, there is a 60% chance of finding a sofa in a living room and a 25% chance of finding a sofa in a game room). Layout rules can also restrict the placement and orientation of internal elements, or recommend placement / orientation based on specific parameters. For example, layout rules might require that newly added internal elements not block entrances and / or that certain internal elements be added based on user input, such as specifying an "open" layout style.

[0060] Internal element rules and / or layout rules may be common internal element rules and / or layout rules for similar spaces (for example, a house may have internal element and layout rules similar to other houses, and a warehouse may have internal element and layout rules similar to other warehouses), or may be generated by the user (for example, customized according to the needs of a particular user) based on a space selected by the user to emulate (as further discussed herein). Internal element rules and layout rules may be stored in any location (for example, locally and / or remotely). In one example, all or some of the internal element rules and / or layout rules may be stored in the rule data store 120.

[0061] The furniture placement module 112 may, in part, generate new interior elements and / or place new interior elements into a digital model (for example, a digital model with furniture removed) based on these rules. For example, the furniture placement module 112 or layout module 114 may place new flooring to cover the entire floor of a room, but not extend it to walls or other objects (for example, by obtaining and applying appropriate interior elements and / or layout rules).

[0062] It will be recognized that internal elements and / or layout rules can be generated, stored, and applied by the rule module 118 in many different ways. In some embodiments, a user may generate global rules that apply to any number of models of a specific user, a group or team of that user, or any number of other related users (for example, based on a user ID). In some embodiments, a user may generate internal elements and / or layouts and apply them to only one digital model at a time, a subset of digital models, or all digital models associated with that user. In various embodiments, the UI may provide the user with a list of available digital models, and the user may select one or more (e.g., all) digital models to which the internal elements and / or layouts should be applied. Similarly, the UI may provide the user with options to register or associate themselves with one or more other users or groups of users. The UI may enable a user to apply internal elements and / or layouts to one or more (e.g., all) users when using the system to place furniture in a space. The UI and system will make it clear to users that they can customize rules, create new rules, delete rules, and apply rules in the manner they desire (for example, to any number of digital models and / or other users, depending on their permissions for digital models and / or permissions to apply / provide rules to other users).

[0063] In various embodiments of adding interior elements to a digital model (e.g., placing or rearranging furniture), the furniture placement module 112 may add furniture to the space in the form of textured 3D objects from a library or catalog. Adding 3D objects to the same coordinate system as the 3D mesh from which the furniture in the space was removed can be straightforward, and the combined 3D data (space mesh + object mesh) can then be provided and / or displayed (e.g., in a UI) from any viewpoint to show how the space looks with the furniture placed alongside the new objects.

[0064] In some embodiments, the furniture placement module 112 determines which internal elements to add (e.g., objects and / or walls) and the orientation and orientation that should be used to place these internal elements. In some embodiments, the furniture placement module 112 utilizes a rule-based system, along with a random seed, to select and place objects.

[0065] The furniture placement module 112 can divide the space into sub-regions (rooms) based on a roughly enclosed area of ​​a 3D mesh, acquire images of the rooms, and assign one or more labels, such as "kitchen" or "bedroom," to each room using an image classification neural network that predicts the room type class that matches each image. As discussed herein, in some embodiments, the spatial modification module 102 utilizes an automated classification system discussed in relation to U.S. Patent No. 11,670,076, titled "Automated Classification Based on Phot-Realistic Image / Model Mappings" and incorporated herein by reference. In one example, the spatial modification system 102 utilizes automated classification of rooms and spaces in 2D data, 3D data, and / or in a relevant digital model (e.g., utilizing semantic segmentation) to identify and classify rooms, spaces, parts of spaces, etc., and generates labels and / or other classifications that can be used for the selection and / or placement of interior elements (e.g., utilizing interior element rules and / or layout rules discussed herein).

[0066] In some embodiments, new interior elements may be added to a space or a portion of a space (e.g., one or more rooms within a space). New interior elements may be added to an existing space (e.g., a space that has not been “removed of furniture”) or to a space that has been “removed of furniture” using the systems and methods described herein. The addition of interior elements may be manual (e.g., interior elements may be added to the space manually through a UI), automatic (e.g., the furniture placement module 112 may automatically add and place interior elements), or a combination of both. Furthermore, it will be recognized that the user or the automated system may be guided by the user’s preferred instructions and / or previous selections / placements.

[0067] In various embodiments, the UI may provide the user with options and / or guidance. The UI may enable the user to remove furniture from a space (for example, using the systems and methods described herein) and / or to add interior elements (e.g., one or more rooms) to a digital model. In one example, the UI may depict a specific room (e.g., a living room) and provide a catalog of interior elements to add to that room. The interior elements may be organized by category and may include, for example, specific types of furniture, movable objects, fixtures, walls, fireplaces, lighting (with or without light emitted from one or more lights), etc. The user may select an interior element and then drag it to the desired part of the digital model displayed by the UI, or choose to have the interior element automatically placed within the digital model (e.g., a specific room and / or a desired location).

[0068] In some embodiments, the UI may provide a guided experience. In one example, the UI may provide a list of any number of interior elements (stored, for example, in data store 124). One or more of the interior elements may be associated with metadata (e.g., tags or labels) that identifies the type of interior element (e.g., sofa, chair, table, wall, pendant light fixture, toy, etc.). In some embodiments, one or more of the interior elements may include metadata indicating one or more rooms in which the interior element may be found, and / or the design style of the interior element. The design style may indicate the type of style associated with the interior element. Examples of design styles include, but are not limited to, contemporary, modern, southwestern, vintage, traditional, European, Scandinavian, naval, industrial, rustic, bohemian, and art deco.

[0069] In some embodiments, the user may indicate the type of design style desired for a particular room (for example, by an options menu or by typing the desired style into a field in the UI). The UI or furniture placement module 112 may provide a list of interior elements of that style (for example, by matching labels and / or metadata that fit the desired interior elements). In some embodiments, the user may identify the type (e.g., a lounge chair) and style (e.g., traditional) of an interior element to add to a room (e.g., a living room) in the digital model. The UI may provide a list of lounge chairs of the desired type and style for the user's selection for placement in the room, based on metadata associated with the interior element (for example, the metadata for the interior element may include “Type”, any number of associated “Interior Styles”, and / or the appropriate room (or an unsuitable room, for example, few people would want to put a lounge chair in a bathroom).

[0070] It will be recognized that any number of interior elements can be associated with metadata (e.g., labels) using any number of methods. For example, an interior element may be associated with one or more labels based on manual labeling by a user (e.g., customization) and / or manual labeling by trained personnel (e.g., manual labeling or using the rules discussed herein). In another example, an interior element may be associated with one or more labels by a trained model (e.g., CNN and / or LLM).

[0071] In some embodiments, the furniture placement module 112 may determine one or more interior elements to add to a space or part of a space by generating scores for different interior elements based on user input (e.g., interior element priority), metadata described herein, and / or room type. Furthermore, the interior element rules described herein may assist in the calculation of scores or provide weighting for scores. In one example, the furniture placement module 112 may identify the room type from the room classification (discussed herein), retrieve one or more interior element rules and layout rules associated with the room type, and assist in identifying interior elements and placements appropriate for the room type. For example, to place the first furniture in a “living room” type room, the furniture placement module 112 may have a 60% probability of selecting a sofa as the first object. The furniture placement module 112 may then select a wall in the room where the sofa will be placed, measure the available space (e.g., based on the layout rules discussed herein) that does not obstruct doorways or other openings in that wall, and search a catalog for a “sofa” type object of appropriate dimensions. Subsequently, the furniture placement module 112 may select the placement and orientation of the sofa relative to the selected wall. Then, based on the room type and the presence of the sofa, the furniture placement module 112 may have a 25% probability that the next object is an entertainment center, using the rule that it should attempt to place it facing the wall and opposite the sofa. The system can continue selecting and placing internal elements in the room until the room is sufficiently furnished and no further objects are added, indicated by a metric (which may be randomized), user instructions, or inputs associated with density / sparseness. This process can be repeated for each room until the entire space is furnished. The user may provide additional inputs during or after the process to manage or modify the selection and / or placement of internal elements.

[0072] It will be recognized that each room can be furnished independently (for example, using a greedy algorithm) to select and arrange its interior elements based on the room type, room shape, and / or existing interior elements.

[0073] It will be recognized that the furniture arrangement module 112 can provide guidance or automate the layout of selected interior elements (for example, interior elements selected manually, automatically, or in a combination of manual and automatic selection).

[0074] In some embodiments, the user or layout module 114 may select a specific design style to assist in the layout (e.g., placement and positioning) of interior elements. In one example, style selection may be a basic form of guidance, but many more forms of guidance or user input may be used when generating the layout.

[0075] The user will be aware that options, requirements, or restrictions may be provided to assist in the selection and layout of interior elements. For example, there may be any number of layout styles associated with different interior element layouts. A layout may include specific rules that conform to the layout of interior elements (e.g., walls and / or furniture) in a portion or subspace within the digital model. For example, layout module 114 may receive instructions from the user for a specific room and a specific layout style (e.g., open, enclosed, efficient, etc.). Layout module 114 may retrieve a layout policy or layout rule associated with the specific layout style and use the layout rule associated with the layout style to place the interior elements within the selected room.

[0076] In some embodiments, interior elements may include metadata indicating accessibility (e.g., for people with specific restrictions or preferences), color (e.g., for color schemes), price (e.g., for the product's budget or in total with other interior elements), and relevance to specific events, brands, sources (e.g., local or department store), environmental friendliness, locally sourced materials, preferred materials, or excluded materials. Users may be provided with one or more of these options (and other options that may become apparent) in the UI to assist in selecting the layout and / or interior design elements for one or more rooms. For example, a user might select a specific room or area to place furniture in, change the type or purpose of the room (e.g., furniture this bedroom as a home office), choose a layout style (e.g., open or efficient), add requirements to the layout such as accessibility, select a furniture color scheme, provide a target budget for the total furniture cost, include or exclude specific elements (e.g., ensure there is an exercise bike in the study), design for a specific purpose or event (e.g., an open-plan office or a wedding), prioritize or exclude specific brands or sources of furniture, and prioritize or exclude specific materials such as leather. User guidance for all these purposes can be provided as individual selections and values ​​(e.g., selected from menus in the UI) or as free-form text, voice, and / or image prompts interpreted by a neural network.

[0077] Another option for guidance input is to provide another space (e.g., a link to a Matterport space) as a style reference to emulate. In this example, layout module 114 may retrieve metadata associated with the selected space, identify interior elements, and determine metadata associated with the interior elements (e.g., color, accessibility, price, relevant events, brand, source, eco-friendly, locally sourced materials, recommended materials, and / or excluded materials). Layout module 114 may also retrieve layout styles associated with the selected space. Then, based on the retrieved metadata, layout module 114 may identify one or more interior elements and identify one or more interior elements to include in the user's space. For example, layout module 114 may determine the type of room in which the user wants to place furniture. Based on the layout (for example, by applying applicable rules associated with the type of room), interior elements can be identified and selected based on metadata associated with the room selected for imitation. The layout module 114 can retrieve and imitate the layout associated with the selected room and apply it to the user's room for the placement and orientation of interior elements (for example, based on layout rules). In some embodiments, the layout module 114 can also identify interior elements and determine the degree of density or rarity (e.g., number of interior elements) of the interior elements in the selected room to be imited when placed in the user's room where furniture will be placed.

[0078] It will be recognized that these rules can be generated, stored, and applied in many different ways. In some embodiments, a user may generate global rules that apply to any number of models for a specific user (e.g., based on user ID), a group or team of that user, or any number of other related users. In some embodiments, a user may generate a rule and apply it to only one digital model at a time, a subset of digital models, or all digital models associated with that user. In various embodiments, the UI may provide the user with a list of available digital models, and the user may select one or more (e.g., all) digital models to which the rule applies. Similarly, the UI may provide the user with the option to register or associate themselves with one or more other users or groups of users. The UI may allow the user to apply rules to one or more (e.g., all) users when using the system to place furniture in a space. The UI and system will make it clear to users that they can customize rules, create new rules, delete rules, and apply rules in their desired manner (for example, to any number of digital models and / or other users, depending on their permissions for digital models and / or permissions to apply / provide rules to other users).

[0079] An alternative to rule-based systems for selecting and placing objects is a trained neural network that generates one or more layouts. In some embodiments, the network is trained to represent spatial layout and design element information (e.g., layout and design element data) as multi-channel images, such as a simplified floor plan image, with different colors or other channel values ​​indicating object types and placement metadata. These image encodings of spatial layouts can pass through a variational autoencoder (VAE), and the latent representation of each space can then be used with a latent diffusion model to generate variations in how furniture is placed in the space. The output images can be interpreted according to the image encoding system, and information about the type, dimensions, and placement of objects used to select and place objects from a 3D object catalog can be extracted.

[0080] In various embodiments, spatial layouts, room types, and objects placed within each space are represented as a series of tokens, and a Large-Scale Language Model (LLM) type network (for example, by the model training module 116) is then trained to output a token sequence describing a furniture-placed layout, given as input a token sequence describing a space with furniture removed. Data for training such a network (either of the described types or others) can be obtained from the analysis of layouts in real spaces or from layouts in synthetic 3D spaces. Such data regarding synthetic layouts can be obtained from the system itself after further interaction with the user or other manual or automated evaluation systems. For example, if the furniture-placed module 112 generates a layout and parts of that layout receive user approval or disapproval instructions or are edited by the user, the model training module 116 can be trained to generate layouts similar to the layout that has received user approval or the layout after it has been edited by the user.

[0081] In both rule-based and neural network approaches, there may be additional inputs that guide furniture selection. One such input is a choice of furniture style, such as "contemporary" or "traditional." This can be used when selecting specific 3D objects from a catalog, and each object can have information about how well it matches each possible style choice, with the final object selection being weighted to prioritize objects that match the input style.

[0082] These style associations can be manually entered for each catalog object, or they can be generated by a neural network trained to classify the object's image according to its style.

[0083] Style inputs can be used early in the process and may influence the general type and arrangement of objects. In rule-based systems, each style can have different weights, such as the type of furniture associated with the style or the total amount of furniture used. In systems using trained neural networks, layouts used as training examples for the network can be labeled by style, and the network can then directly learn how to modify its output to better match each style.

[0084] In addition to guidance in generating the initial furniture arrangement, the UI may provide guidance to help users modify the generated furniture. Several possible forms of this modification include repositioning individual pieces of furniture, replacing individual pieces of furniture (either by newly generated replacements or by allowing the user to select a specific replacement from a suggestion list), selecting variations of individual pieces of furniture or materials, such as different colors, or adding specific new pieces of furniture. When modifying furniture, user input may again take the form of free-form text prompts or questions interpreted by a neural network. In one example, the text prompts may be generated by or modified by a Large-Scale Language Model (LLM). These replacements or modifications can also be on a scale of multiple pieces of furniture, all furniture in one or more rooms, or even the entire space. For example, the user could simply request a new variation of the furniture arrangement for the entire space and instruct the system to keep retrying (using a different random seed each time) until a satisfactory result is achieved. The user could also request more specific modifications, such as rearranging the furniture in a room to increase seating capacity, or changing furniture to fit a budget while keeping a similar layout. Furthermore, as mentioned in the furniture removal step above, user guidance can be used to determine which furniture to remove from the original space, and this same modification guidance can also be used to modify or replace the original furniture rather than the generated furniture.

[0085] This system allows you to not only place furniture but also replace materials within a space, such as changing the floor from carpet to hardwood or altering the color of the wall paint. Material options can again be obtained from a catalog, but in this case, the catalog will be a catalog of 2D materials that can be applied to surfaces in the space based on 3D data regarding the shape and position of the surfaces. Even for 3D objects from the catalog, "furniture" is a convenient descriptor for many things that can be added to the space, but you can add other types of objects, such as fireplaces, ceiling beams, and window treatments—anything available in the 3D or 2D catalog of objects and materials. Objects from the catalog can also be modified by changing their dimensions, color, and other properties.

[0086] Placing 3D objects from a catalog is not the only possible approach to adding furniture. Another option is to generate 2D images or 3D data of new furniture or materials directly from a neural network. For example, instead of generating a set of object criteria that can be matched to a catalog, or directly selecting catalog entries, the system could generate text or potential descriptions of objects, layouts, or styles to add to a room, and then use an image generation network to modify that content in the appropriate place in the room, using the content description as guidance to the network. This can also be done on a larger scale, directly generating images or 3D data of the entire room furnished, instead of generating layout metadata (object types and locations) as an intermediate step. One problem with this approach is that, without proper guidance, the generated images are unlikely to be consistent across different viewpoints. To address this, several approaches have been established, including furniture removal approaches such as simultaneous image generation, or those discussed under conveying results from one or more generation steps as input to later generation steps. When directly generating 3D data for new furniture, such as 3D meshes, mesh textures, point clouds, NeRF, or Gaussian splatting representations, consistency between viewpoints is an inherent advantage of 3D data.

[0087] Another option for generating furniture is to use some or all of the furniture detected and removed from the original space. 3D models or other representations of this furniture can be extracted using data from the original space associated with each piece of furniture, and these models of the original furniture can be added to the space in the same manner as furniture selected from a catalog. This would support the rearrangement of furniture in the space, instead of adding new furniture, or in addition to it. Instead of creating 3D models or other representations of these objects, the system could also find matching or similar objects within a catalog of 3D objects, or use the detected furniture as guidance to employ other approaches, such as the latent diffusion modification described above. The original furniture detected and extracted in this manner can also be used during modifications to the generated furniture, such as restoring some of the removed original furniture, regardless of whether it is part of the initial set of generated furniture.

[0088] All of these methods for generating furniture and / or materials can be used individually or in combination to generate one or more variations of a space. Each variation of a space can be generated individually, some variations can be generated based on previously generated variations, or variations can be generated simultaneously to have specific properties, such as intentionally displaying a diverse set of layouts.

[0089] In step 212, the communication module 104 displays or represents, in some embodiments, one or more variations of the modified space from a previous step. One form of display is to generate one or more 2D images of the variations of the space. 2D images can be generated from 3D data. In one example, Figure 5A shows a user interface showing a digital model of a room, and Figure 5B shows a user interface showing a modified variation in which some internal elements of the digital model of the room have been replaced and changed.

[0090] In embodiments of rearranging furniture in Matterport space, variations of the space can be displayed on the user's device (e.g., via the Matterport web viewer or UI) with the same view modes and navigation controls as the original digital model (e.g., the original digital space before any modifications), including the ability to view a textured 3D mesh of the space from various angles and viewpoints ("dollhouse mode"), the ability to view the space from above in a floor plan style view, and the ability to navigate within the space to see how the space looks from many different viewpoints of different rooms and angles. In some embodiments, the space viewed from all these viewpoints in all these modes may not be perfectly consistent (even an unmodified space may have some degree of multiview inconsistency due to conflicting data or imperfect processing), but generally, variations of the space are presented with multiview consistency, and the viewer can navigate around or look around the space to better understand how new furniture and / or materials look, fit in, and look relative to each other, using multiple viewpoints that include new viewpoints different from the viewpoints from which data was captured within the space. In various embodiments, the space viewed from one, multiple, or all of these modes can be consistent.

[0091] In addition to interactive web experiences, there are other methods by which the modified space can be presented in the UI, which may include other interactive experiences such as dedicated apps, VR, or AR, as one or more 2D images and / or videos (panoramic or planar images, or videos including stereo pairs), and / or as one or more different 3D representations, such as implicit or latent representations like point clouds, 3D meshes, light fields, or NeRF. The space can also be presented as text or audio descriptions, including descriptions of added furniture and materials, added locations, or layout characteristics such as "this office layout can accommodate 50 people." Added furniture and materials can also be presented as lists or other types of summaries, including other data or measurements such as quantity, price, and / or the amount of paint or flooring needed to cover the surface. These various forms of presentation can also be combined, such as by labeling or highlighting added furniture in 2D images or interactive 3D web viewers.

[0092] In addition to passive presentations such as 2D images or videos, or interactive presentations intended for navigation or viewpoint selection in UIs such as 3D web viewers, presentations can include other types of interactive elements and extensions. In particular, presentation interfaces may also support all interactions necessary for the user guidance described above, such as menus, buttons, or text inputs that allow the user to modify the generated furniture. This may include an interface for selecting a space to be processed by the system and initiating system operations such as requesting the system to generate variations of the space with new furniture or requesting new variations of the space. The interface may also support searching or filtering generated furniture, saving variations of the space or other information about the generated furniture, undoing and redoing modifications to the generated furniture, switching between representations of different variations of the space or different modifications to the generated furniture, displaying only a portion of the variations of the space, and / or immediately displaying multiple variations side-by-side or split, by changing the amount of each variation displayed using a comparison slider, or by displaying multiple variations superimposed or mixed together.

[0093] The presentation may also include other context and information about the space (original and / or variations of the presentation), such as measurements, dimensions, other attributes, or information about the space itself. In addition, the presentation may be adapted based on user interaction, such as by representing additional information or making suggestions based on the user's current perspective. For example, if the user is looking at a kitchen in a house, the system may suggest that the user try the guidance prompt, "How many chairs can fit on the kitchen table?"

[0094] One way to represent variations in space is through the appearance of added furniture and / or materials. The goal of presentation is to make the added furniture look photorealistic, aesthetically pleasing, to match its appearance to the rest of the space, and / or to represent the furniture in an intentionally unrealistic style for the purpose of aiding understanding of the space or highlighting the added furniture. In an approach where furniture is added as 3D objects from a library, a simple rendering of the objects placed in space may not achieve these appearance goals. One technique to improve the appearance by considering preferred appearance styles (e.g., photorealistic, vintage, or unrealistic styles) is to use more accurate or realistic lighting and shadows during the rendering process, for which information about the 3D space can be used. In particular, the layout module 114 and / or furniture placement module 112 may utilize the shape of the space to calculate where to place shadows based on composite light sources or detected light sources, or using approaches such as ambient occlusion. In addition, 3D objects are illuminated based on the lighting present in the space, including light sources detected in the data about the original space (those not removed during furniture removal) and light sources added by the system (e.g., lamps added as furniture). To detect light sources in the original space, the identification module 106 can treat each image captured in the known space as a light probe, and the same light sources detected by the probes are used when rendering the added 3D objects (e.g., by the furniture placement module 112 and layout module 114), giving these 3D objects an appearance that more closely matches how they would look if they were actually present in the space. Also, if light sources detected from the original space are identified as those that need to be removed along with the furniture, information about the modified light sources can be used when rendering the added 3D objects and can also be used to update the appearance of the furniture-removed version of the space, so that the removal of furniture looks more realistic, including the impact on light and shadow in the space.

[0095] Another technique to improve the appearance of added furniture or the entire space is to use AI-generated images. Information about the original space and the added furniture (e.g., user input and / or metadata) can be fed into the network, and an image of a variation of that space with the newly placed furniture can be output. For example, a latent diffusion network like Stable Diffusion can be trained using a high-quality composite image of the space with the furniture in place as a "ground truth" image, with its training input being a rendering of the same scene, but with reduced rendering quality for some objects in the scene (e.g., missing lighting or shadows). In this way, the network is trained to take low-quality renderings as input and produce higher-quality, more realistic images as output (e.g., with correct lighting and / or shadows). Another option would be to use a latent diffusion modification network to modify the area of ​​the image where the furniture is added, and provide the modification network with a guidance image (using ControlNet, for example) that contains information about the added furniture, such as lines and edges of objects rendered in the image. In this way, the diffusion network can produce a realistic image (as it is generally trained), but guided or constrained to match the shape and other attributes of the added furniture. If the goal for the appearance of added furniture is something other than photorealism, the system can also be configured or trained to generate other appearances, including those selected or adjusted based on individual user preferences or feedback.

[0096] The user interface for presenting variations of a space may also include features for collaboration and / or sharing among multiple users. For example, a user browsing a space may add notes or comments (generally associated with the space, a specific furniture variation or detail, and / or a 3D location within the space), and other users browsing the same space or other variations may see these notes and respond (for example, enabling interaction between any number of users browsing the same or similar digital model in multiple UIs, thereby via a communication module 104 that enables communication via a centralized communication system). The system may also include features for saving and / or sharing variations of the space (whole or in part) with other users, such as on social media. The interface may also support live collaboration among users related to the space, such as text, voice, and / or video chat. Other methods for supporting collaboration include including drawing and sketching tools, or showing how other users are interacting with the space, such as by displaying other users' live mouse cursors or other interactions. Another feature that enables collaboration is the support for simultaneous editing by multiple users using version control, history, and / or change notifications. One form of collaboration is for a designer to create one or more variations of a space and then share these variations with the owner, tenant, or facility manager who will be furnishing the space, or for such users to collaborate in creating one or more variations of the space.

[0097] In some embodiments, the UI may optionally allow or assist the user in retrieving and placing generated furniture and / or materials to physically create variations of the space. For example, once a user uses the system to generate a preferred variation of a house, the system may display an option to purchase some or all of the furniture and / or materials added to that space and have them delivered to the user. For example, Figures 5A–5D show a user interface with an “auto-designer” option to assist the user in retrieving and placing internal elements.

[0098] In some embodiments of rearranging furniture in a digital model (e.g., a space), this can be done by associating a vendor with each item in a furniture catalog, and then, during the presentation of the space (or as another communication), the space modification system 102 can provide an action to purchase furniture (e.g., via a communication module 104 and UI). If the user selects that action (e.g., by clicking a button), the communication module 104 may collect product and vendor information for each added piece of furniture, material, etc., generate web links or app links for each product and / or vendor that can be used to purchase the product from the vendor, and then present these links to the user. The user can click these links (presented via UI, email, text message, secure message, etc.) to complete the purchase and delivery of the product. The communication module 104 may not only generate and provide links to purchase products, but may also manage more of the shopping experience and implement deeper integration with vendors. For example, by incorporating a virtual shopping cart and checkout flow into the user interface (or another interface), the user can purchase products without having to interact with other vendors. In any of these cases, the set of products involved could be a single product, multiple products, all products in a room, all products in an entire space, or a set of these products from multiple variations of a space.

[0099] There are other possible approaches besides maintaining vendor and other purchase information for each item in the furniture catalog. One such approach is for the identification module 106 to attempt to match the generated furniture to actual purchasable items or suitable alternatives, even if there is no prior association with each generated piece of furniture. For example, the identification module 106 can use information about the generated furniture from a search service to find purchasable items with similar characteristics such as appearance, product type, color, size, and function (perhaps using image-based search). The identification module 106 could also use a neural network trained to match information about an item (such as one or more images of the item) to purchasable items. This approach allows the user to retrieve furniture or a suitable close match, even if the generated furniture is not a representation of an actual object (for example, the furniture was generated from an AI-generated image). When an approximate match for an item is found as described above, or when the exact item is known and available from among the generated furniture, the communication module 104 and / or the identification module 106 can provide one or more alternative options for the item from among the generated furniture and / or materials. The communication module 104 and / or the identification module 106 can offer the same or similar products from different vendors to target different budgets or to match other goals or constraints. The communication module 104 and / or the identification module 106 can also export data about the generated furniture, allowing users to perform their own searches or even create their own furniture through 3D printing or other means.

[0100] The communication module 104 can not only assist the user in purchasing or acquiring the generated furniture, but can also provide additional functions that help the user physically create variations of the space. These additional functions may include guides on how to physically place the furniture, such as floor plans, placement guides, or instructions. The communication module 104 can even integrate with services or vendors that place or install furniture and / or materials to match the generated design.

[0101] To further support this step, other steps may include arrangements for ultimately purchasing or acquiring furniture and / or materials. For example, when generating and presenting furniture, communication module 104 may choose to use only furniture known to be available for purchase (or may provide the option to apply such restrictions). This can be achieved by integrating with furniture vendors and by tracking which items are currently available for purchase, including adding new items as they become available. Furthermore, communication module 104 may also integrate with these vendors to see which items are currently in stock and available, and use only such items for the generated furniture and / or materials. Even if communication module 104 does not restrict the generated furniture to only those available for purchase, it may later provide the option to modify the spatial variations to replace items that are not physically available with available / purchasable items, or generally update the variations in a way that allows the user to physically reproduce them. In addition to creating or modifying furniture and / or materials to use only available goods, the communication module 104 can also create or modify generated furniture to prioritize or exclude specific vendors, use goods with specific budget targets or restrictions, or use goods that can be delivered within a specific timeframe.

[0102] In addition to assisting the user in acquiring generated furniture, the communication module 104 can also enable the user to identify and acquire furniture present in the original space. This can be achieved by approximate matching as described above, or by pre-matching items in the space with purchasable items (using automated, manual, or semi-manual approaches). This functionality can also be used independently of the rest of the communication module 104. For example, if a user is browsing a Matterport space in an interactive web viewer, the web viewer can identify purchasable items in the space and enable or invite the user to purchase these items using any of the shopping integrations described above.

[0103] Figures 3A–3C show exemplary user interfaces (UI) 300 that may be used in some embodiments to remove furniture from a space (i.e., a digital model). For example, the exemplary user interface 300 allows the user to select a furniture removal icon and specify the type of room in the digital model (e.g., kitchen, living room, bathroom, and office). In some embodiments, the UI may display the room type based on metadata associated with the space, indicating the most likely room type based on metadata that is part of the CNN, 2D data, 3D data, and / or the digital model. The user may then select a button to remove all objects from the space.

[0104] As discussed herein, the identification module 106 may identify interior elements of a room using a rule-based approach and / or an AI model. In some embodiments, the identification module 106 may utilize metadata to identify interior elements within a room from 2D data, 3D data, and / or a digital model.

[0105] Figure 3B shows a UI that displays options for removing furniture from different rooms in a digital model. The UI will make it clear to the user that they may be able to select internal elements to delete or remove furniture from a portion of a room.

[0106] Figure 3C shows a living room with furniture removed in several embodiments. In this example, the delete module 108 removed all internal elements (e.g., fixtures) from the living room in the digital model. In some embodiments, the delete module 108 masks and embeds the internal elements to make the digital model appear as if these internal elements are absent.

[0107] Figures 4A–4C show exemplary user interfaces 400 that may be used to remove objects from a space in some embodiments. The exemplary user interface 100 allows the user to select an organize icon. The user can then select objects individually or select a button to automatically organize the space. In this example, the identification module 106 may identify small (e.g., within a threshold size), movable internal elements within a room selected by the user.

[0108] Figure 4B shows an exemplary user interface 400 in the process of “erasing” small, movable internal elements (e.g., objects) from a selected room in several embodiments. As discussed herein, the delete module 108 can delete all internal elements (e.g., small, portable objects identified by the identification module 106) that qualify for rules and / or user selection. The delete module 108 can mask and fill in a mask to make it appear as if the object has been removed from the display of the selected room.

[0109] Figure 4C shows an exemplary user interface 400 in which, in some embodiments, small, portable internal elements appear to have been removed from the selected room.

[0110] Figures 5A to 5D show exemplary user interfaces 500 that may be used to arrange furniture in a space in several embodiments. The exemplary user interface 500 allows the user to select furniture placement icons and choose one or more styles and / or features. In this example, the exemplary user interface 500 also allows the user to enter text prompts that may be provided to the LLM. The LLM's response may be used to determine the style to be used to arrange the furniture in the space.

[0111] Examples of this technology's use are as follows:

[0112] Interior Design and Space Utilization: We help homeowners, designers, and property managers rethink physical spaces using automated virtual interior design and staging, including recommendations on how to optimize space usage.

[0113] Design and Architecture: Enabling homeowners, architects, and builders to easily create more efficient, sustainable, and accessible buildings.

[0114] Energy Efficiency: Provides insights into ways to reduce energy consumption within a building, which can lead to lower energy costs and reduced carbon dioxide emissions.

[0115] Maintenance and Repair: Proactively identify the most common maintenance and repair issues in buildings, along with tips on how to prevent them, helping building owners and managers address problems before they become costly.

[0116] Safety and Security: Focus on potential safety and security risks to buildings, such as fire hazards and building code violations, along with recommendations on how to address these risks.

[0117] Other use cases for this technology will also become apparent.

[0118] Figure 5B shows rooms with different designs in several embodiments. In this example, LLM may provide a design style. The delete module 108 may delete internal elements that do not fit the selected design style based on rules associated with the design style (e.g., color palette preference, style type, etc.). The furniture placement module 112, layout module 114, and modification module 110 may modify, position, orient, or modify the textures and colors of internal elements to conform to the design style.

[0119] Figure 5C shows a different room design in several embodiments. Users may be given different options to experiment with different design styles and aesthetics.

[0120] Figure 6 shows a block diagram of an exemplary digital device 600 that may be used by the techniques described herein, according to several embodiments. The digital device 600 is shown in the form of a general-purpose computing device. The digital device 600 includes at least one processor 602, RAM 604, a communication interface 606, an input / output device 608, storage 610, and a system bus 612 that connects various system components, including the storage 610, to at least one processor 602. A system such as a computing system may be one or more of the digital devices 600, or may include one or more of the digital devices 600.

[0121] System Bus 612 represents one or more of several types of bus structures, including a memory bus or memory controller, peripheral buses, accelerated graphics ports, and a processor bus or local bus using one of various bus architectures. Examples, but not limited to, such architectures include the Industry Standard Architecture (ISA) bus, Microchannel Architecture (MCA) bus, Extended ISA (EISA) bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Interconnect (PCI) bus.

[0122] The digital device 600 typically includes various computer system-readable media, such as computer system-readable storage media. Such media may be any available media accessible by any of the systems described herein, and include both volatile and non-volatile media, and removable and non-removable media.

[0123] In some embodiments, at least one processor 602 is configured to execute executable instructions (e.g., programs). In some embodiments, at least one processor 602 comprises a circuit configuration or any processor capable of processing executable instructions.

[0124] In some embodiments, RAM 604 stores programs and / or data. In various embodiments, working data is stored in RAM 604. The data in RAM 604 can be cleared before resetting the digital device 600 and / or before powering it off, or it can eventually be transferred to storage 610.

[0125] In some embodiments, the digital device 600 is connected to a network via a communication interface 606.

[0126] In some embodiments, the input / output device 608 is any device that inputs data (e.g., a mouse, keyboard, stylus, sensor, etc.) or any device that outputs data (e.g., a speaker, display, virtual reality headset).

[0127] In some embodiments, storage 610 may include computer system-readable media in the form of non-volatile memory such as read-only memory (ROM), programmable read-only memory (PROM), solid-state drives (SSDs), flash memory, and / or cache memory. Storage 610 may further include other removable / non-removable, volatile / non-volatile computer system storage media. As just one example, storage 610 may be provided for reading and writing to a non-removable non-volatile magnetic medium. Storage 610 may include non-temporary computer-readable media, or a plurality of non-temporary computer-readable media, for storing programs or applications to perform functions such as those described herein. Although not shown, magnetic disk drives for reading and writing to removable non-volatile magnetic disks (e.g., “floppy disks”) and optical disk drives for reading and writing to removable non-volatile optical disks such as CD-ROMs, DVD-ROMs, or other optical media may also be provided. In such cases, each may be connected to the system bus 612 by one or more data medium interfaces. As further illustrated and described below, the storage 610 may include at least one program product having a set of program modules (e.g., at least one) configured to perform the functions of embodiments of the present invention. In some embodiments, RAM 604 is located within the storage 610.

[0128] A program / utility having a set of program modules (at least one) may be stored in storage 610, as well as, but not limited to, an operating system, one or more application programs, other program modules, and program data. Each of the operating systems, one or more application programs, other program modules, and program data, or a combination thereof, may include implementation of a network environment. The program modules generally perform functions and / or methodologies of embodiments of the present invention described herein.

[0129] It should be understood that other hardware and / or software components, not shown in the illustrations, can be used in conjunction with the digital device 600. Examples include, but are not limited to, microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archive storage systems.

[0130] Exemplary embodiments are described in detail herein with reference to the accompanying drawings. However, since this disclosure can be implemented in various ways, it should not be construed as being limited to the embodiments disclosed herein. Rather, these embodiments are provided to provide a thorough and complete understanding of this disclosure and to fully convey its scope.

[0131] It will be recognized that one or more embodiments may be embodied as a system, method, or computer program product. Thus, embodiments may take the form of entirely hardware embodiments, entirely software embodiments (including firmware, resident software, microcode, etc.), or embodiments combining software and hardware embodiments, all of which may be generally referred to herein as “circuits,” “modules,” or “systems.” Furthermore, embodiments may take the form of a computer program product embodied in one or more computer-readable media having embodied computer-readable program code.

[0132] Any combination of one or more computer-readable media may be used. A computer-readable media may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media may include electrical connections with one or more wires, portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), removable programmable read-only memory (EPROM or flash memory), solid-state drives (SSDs), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In the context of this document, a computer-readable storage medium may be any tangible medium that contains or can store programs or data for use by or associated with an instruction execution system, apparatus, or device.

[0133] A temporary computer-readable signal medium may contain propagated data signals, for example, as part of a baseband or carrier wave, along with embodied computer-readable program code. Such propagated signals may take any of a variety of forms, including but not limited to electromagnetic, optical, or any suitable combination thereof.

[0134] Program code embodied in a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, fiber optic cable, RF, or any suitable combination thereof.

[0135] Computer program code for performing operations according to aspects of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and Python, and conventional procedural programming languages ​​such as the C programming language or similar programming languages. The computer program code may be fully executed on any of the systems described herein, or any combination thereof.

[0136] Aspects of the present invention will be described below with reference to flowchart examples and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in the flowchart examples and / or block diagrams, and combinations of blocks within the flowchart examples and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device, and instructions executed via the processor of the computer or other programmable data processing device may generate a machine such that it creates means for performing the functions / operations specified in the blocks of the flowchart and / or one or more block diagrams.

[0137] These computer program instructions may also be stored in a computer-readable medium that can instruct a computer, other programmable data processing device, or other device to function in a particular manner, and as a result, instructions stored in a computer-readable medium may generate a manufactured article containing instructions that perform functions / operations specified in a flowchart and / or a block of one or more block diagrams.

[0138] Computer program instructions can also be loaded into a computer, other programmable data processing device, or other device, and a series of operational steps can be executed on the computer, other programmable device, or other device to generate a computer execution process, which in turn can provide a process for instructions executed on the computer or other programmable device to perform functions / operations specified in the blocks of a flowchart and / or one or more block diagrams.

[0139] While specific examples are described above for illustrative purposes, various equivalent modifications are possible. For example, a process or block is presented in a given order, but alternative implementations may execute routines with steps or apply systems with blocks in a different order, and some processes or blocks may be deleted, moved, added, subdivided, combined, and / or modified to provide alternatives or partial combinations. Each of these processes or blocks may be implemented in various different ways. Also, while processes or blocks may be shown to be executed sequentially, these processes or blocks may be executed or implemented simultaneously or in parallel, or at different times. Furthermore, the specific numbers mentioned herein are merely examples, and alternative implementations may apply different values ​​or ranges.

[0140] Throughout this specification, multiple examples may implement components, operations, and structures, while internal elements may be described as single examples. Structures, internal elements, and functions presented as individual components in exemplary configurations may be implemented as combined structures or components. Similarly, structures, internal elements, and functions presented as single components may be implemented as individual components. These and other variations, modifications, additions, and improvements fall within the scope of the subject matter of this specification. Furthermore, specific numerical values ​​mentioned herein are merely examples, and alternative implementations may apply different values ​​or ranges.

[0141] Components may be described or illustrated as being included in or connected to other components. Such descriptions or illustrations are merely examples, and the same or similar functionality may be achieved in other configurations. Components may be described or illustrated as being “combined,” “combinable,” “operatably combined,” “communicatively combined,” etc., with other components. Such descriptions or illustrations should be understood as such components being able to cooperate or interact with one another and to be in direct or indirect physical, electrical, or communicative contact with one another.

[0142] Components may be described or illustrated as "configured to," "adapted to," "operate to," "configurable to," "adaptable to," or "operable to." Such descriptions or illustrations should be understood to cover both active and inactive or standby components unless otherwise required by context.

[0143] The use of “or” in this disclosure is not intended to be perceived as an exclusive “or.” Rather, “or” should be understood as including “and / or.” For example, the phrase “to provide products or services” is intended to be understood as having multiple meanings: “to provide products,” “to provide services,” and “to provide products and services.”

[0144] It may become apparent that various modifications can be made and other embodiments can be used without departing from the broader scope of the discussion herein. Therefore, these and other variations of the exemplary embodiments are intended to be covered by the disclosure herein.

Claims

1. A non-temporary computer-readable medium comprising executable instructions, wherein the executable instructions are executable by one or more processors to perform a method, and the method is Accessing multidimensional spatial data representing the physical environment, The first machine learning model is used to identify interior elements in the multidimensional space, wherein the interior elements represent furniture in the physical environment. Masking one or more of the aforementioned interior elements with a mask, To create the appearance of a space from which furniture has been removed, each of the masks is filled with an image of the physical environment, wherein the space from which furniture has been removed has one or more interior elements that appear to be missing from the multidimensional space representing the physical environment. For display purposes, the method includes providing all or part of the space where the furniture has been removed. A non-temporary computer-readable medium.

2. The aforementioned physical environment is a room with furniture arranged in it. A non-temporary computer-readable medium according to claim 1.

3. The aforementioned multidimensional space is a 2D representation of the physical environment. A non-temporary computer-readable medium according to claim 1.

4. Using the 2D representation of the physical environment, a corresponding 3D representation of the physical environment is generated. The non-temporary computer-readable medium according to claim 3.

5. The aforementioned multidimensional space is a 3D representation of the physical environment. A non-temporary computer-readable medium according to claim 1.

6. The data in the aforementioned multidimensional space is a textured 3D mesh. A non-temporary computer-readable medium according to claim 1.

7. The interior elements in the multidimensional space further include at least one wall in the physical environment, A non-temporary computer-readable medium according to claim 1.

8. The first machine learning model is a semantic segmentation neural network. A non-temporary computer-readable medium according to claim 1.

9. Filling each of the aforementioned masks with the aforementioned image of the physical environment comprises applying a modification to the mask. A non-temporary computer-readable medium according to claim 1.

10. The above method further, The system requires receiving a selection of at least one design style type from the user, The process includes selecting an interior element in the multidimensional space previously identified based on the at least one design style type from the user, and masking at least a portion of the interior element, which is the at least one design style type of the interior element. A non-temporary computer-readable medium according to claim 1.

11. The above method further, The system requires receiving a selection of at least one design style type from the user, The process includes selecting interior elements in the multidimensional space previously identified based on the at least one design style type from the user, and masking at least a portion of the interior elements, which includes masking at least a portion of the interior elements that are not of the at least one design style type. A non-temporary computer-readable medium according to claim 1.

12. The above method further, Receiving an additional request from a user, the additional request including one or more additional interior elements to be added to the space where the furniture has been removed, Identifying the representation of one or more additional interior elements, To place one or more additional interior elements in the space where the aforementioned furniture has been removed, To place, for display purposes, one or more of the additional interior elements that were placed in the space where the furniture was removed, in all or part of the space where the furniture was removed, Equipped with, A non-temporary computer-readable medium according to claim 1.

13. Receiving an additional request from the user comprises receiving a prompt from the user and applying the prompt to a large language model to receive a response from the large language model, the response identifying the one or more additional interior elements. The non-temporary computer-readable medium according to claim 12.

14. A system comprising at least one processor and memory containing executable instructions, wherein the executable instructions are Accessing multidimensional spatial data representing the physical environment, The first machine learning model is used to identify interior elements in the multidimensional space, wherein the interior elements represent furniture in the physical environment. Masking at least a portion of the aforementioned interior elements with a mask, To create the appearance of a space from which furniture has been removed, each of the masks is filled with an image of the physical environment, wherein the space from which furniture has been removed has at least some of the interior elements that appear to be missing from the multidimensional space representing the physical environment. To provide all or part of the space where the furniture has been removed for display purposes, the at least one processor can perform the following: system.

15. The aforementioned physical environment is a room with furniture arranged in it. The system according to claim 14.

16. The aforementioned multidimensional space is a 2D representation of the physical environment. The system according to claim 14.

17. Using the 2D representation of the physical environment, a corresponding 3D representation of the physical environment is generated. The system according to claim 16.

18. The aforementioned multidimensional space is a 3D representation of the physical environment. The system according to claim 14.

19. The data in the aforementioned multidimensional space is a textured 3D mesh. The system according to claim 14.

20. The interior elements in the multidimensional space further include at least one wall in the physical environment, The system according to claim 14.

21. The first machine learning model is a semantic segmentation neural network. The system according to claim 14.

22. Filling each of the aforementioned masks with the aforementioned image of the physical environment comprises applying a modification to the mask. The system according to claim 14.

23. The aforementioned executable instruction further states: The system requires receiving a selection of at least one design style type from the user, To select previously identified interior elements in the multidimensional space based on the at least one design style type from the user, It is executable by at least one of the aforementioned processors, Masking at least a portion of the interior elements means masking at least a portion of the interior elements that are of at least one design style type. The system according to claim 14.

24. The aforementioned executable instruction further states: The system requires receiving a selection of at least one design style type from the user, To select previously identified interior elements in the multidimensional space based on the at least one design style type from the user, the at least one processor can perform the following: masking at least a portion of the interior elements includes masking at least a portion of the interior elements that are not of the at least one design style type. The system according to claim 14.

25. The aforementioned executable instruction further states: Receiving an additional request from a user, the additional request including one or more additional interior elements to be added to the space where the furniture has been removed, Identifying the representation of one or more additional interior elements, To place one or more additional interior elements in the space where the aforementioned furniture has been removed, To place, for display purposes, one or more of the additional interior elements that were placed in the space where the furniture was removed, in all or part of the space where the furniture was removed, Executable by at least one of the aforementioned processors, The system according to claim 14.

26. The executable instructions, further configured by the at least one processor to receive additional requests from the user, The executable instructions are further configured by the at least one processor for receiving a prompt from a user and applying the prompt to the large language model in order to receive a response from the large language model, The response identifies the one or more additional interior elements. The system according to claim 25.

Images