Indoor scene reconstruction method and system based on single view, terminal and storage medium

Abstract

The invention discloses an indoor scene reconstruction method and system based on a single view, a terminal and a storage medium. The method comprises the steps that an indoor single-view-angle RGB image of an indoor scene is acquired; generating a plurality of image features according to the indoor single-view RGB image, constructing a feature attention map and target object attributes according to the plurality of image features, and constructing an indoor preliminary model according to the feature attention map and the target object attributes; according to the plurality of image features and the target object attribute, predicting to obtain a background object attribute, and according to the background object attribute, optimizing the indoor preliminary model to obtain a final indoor model; and optimizing the final indoor model based on a differentiatable renderable pipeline, and generating complete shapes, appearances and postures of all objects in the indoor scene. According to the method, the image features are extracted from the single image, the final indoor model is deduced, the indoor scene is divided into foreground and background representation, and the geometric shape and appearance of the scene are restored robustly by optimizing the model.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a method, system, terminal, and storage medium for indoor scene reconstruction based on a single view. Background Technology

[0002] Currently, image-based 3D reconstruction technology has become a highly promising and popular field with important applications in many areas, such as e-commerce, aerospace, remote sensing and mapping, and virtual museums. Compared with traditional modeling methods, such as using modeling software (3D Max, AutoCAD, etc.) and scanning objects with a 3D scanner to obtain a 3D model, image-based 3D reconstruction is less expensive, more realistic, and more automated, thus having broad application prospects.

[0003] Inferring meaningful geometric scenes from a single image is a fundamental problem in computer vision, and various techniques and approaches have addressed it from different angles. The most common method is monocular depth estimation, which predicts depth values ​​per pixel. While achieving remarkable results, these methods reconstruct the scene as a whole and do not incorporate knowledge about scene objects and layout. Alternatively, 3D detection pipelines detect individual objects and recover their masks and 3D bounding boxes, or merge relationships between objects using graphical or physical simulations. While understanding how objects look in 2D and their 3D poses, these methods cannot recover the shapes of objects or the rest of the scene. Other methods aim to reconstruct detected objects by fitting predicted masks to the input image and aligning them with a 3D CAD model, but they do not generalize to unknown instances outside the model database. Although neural radiation field-based methods have achieved better results, they require extensive training for each scene and cannot scale to scenes not seen outside the training set.

[0004] Therefore, existing technologies still need to be improved and developed. Summary of the Invention

[0005] The main objective of this invention is to provide an indoor scene reconstruction method, terminal, and storage medium based on a single view, aiming to solve the problem in the prior art where the incomplete inference of meaningful geometric scene information from a single image leads to poor scene reconstruction results.

[0006] To achieve the above objectives, the present invention provides a method for reconstructing indoor scenes based on a single view, the method comprising the following steps:

[0007] Acquire an indoor single-view RGB image of an indoor scene;

[0008] Multiple image features are generated based on the indoor single-view RGB image, and a feature attention map and target object attributes are constructed based on the multiple image features. An indoor preliminary model is constructed based on the feature attention map and the target object attributes.

[0009] Background object attributes are predicted based on multiple image features and target object attributes. The preliminary indoor model is then optimized based on the background object attributes to obtain the final indoor model.

[0010] The final interior model is optimized based on a differentiable rendering pipeline to generate the complete shape, appearance, and pose of all objects in the interior scene.

[0011] Optionally, the step of generating multiple image features based on the indoor single-view RGB image, constructing a feature attention map and target object attributes based on the multiple image features, and constructing a preliminary indoor model based on the feature attention map and the target object attributes, further includes the following steps before:

[0012] Define a scene decomposition network, which includes a transformer detection module, an object reasoning module, and a 3D reasoning module;

[0013] The transformer detection module includes a transformer encoder and a transformer decoder, and the object reasoning module includes multiple multilayer perceptrons.

[0014] Optionally, the step of generating multiple image features based on the indoor single-view RGB image, constructing a feature attention map and target object attributes based on the multiple image features, and constructing a preliminary indoor model based on the feature attention map and the target object attributes specifically includes:

[0015] The transformer encoder encodes the indoor single-view RGB image to obtain multiple image features;

[0016] The transformer decoder constructs a feature attention map and target object attributes based on multiple image features;

[0017] The transformer decoder constructs a preliminary indoor model based on the feature attention map and the target object attributes.

[0018] Optionally, the step of predicting background object attributes based on multiple image features and target object attributes, and optimizing the preliminary indoor model based on the background object attributes to obtain the final indoor model, further includes:

[0019] The object reasoning module obtains foreground object attributes and background object attributes by regressing and predicting the target object attributes through multiple multilayer perceptrons, wherein the multilayer perceptron is a three-layer perceptron with ReLU activation.

[0020] Optionally, the step of predicting background object attributes based on multiple image features and target object attributes, and optimizing the preliminary indoor model based on the background object attributes to obtain the final indoor model, specifically includes:

[0021] The transformer decoder constructs a foreground model based on the foreground object attributes and a background model based on the background object attributes;

[0022] The 3D reasoning module merges the background model into the preliminary indoor model to obtain the final indoor model.

[0023] Optionally, the step of optimizing the final interior model based on a differentiable rendering pipeline to generate the complete shape, appearance, and pose of all objects in the interior scene further includes:

[0024] The shapes of the foreground model and the background model are encoded into symbolic distance fields with relevant latent feature vectors based on a differentiable prior database.

[0025] The shapes of objects in the final indoor model are distinguished to obtain the maximum likelihood latent vector of the object shape. The object shapes are then matched and optimized based on the shape prior information in the differentiable prior database and the maximum likelihood latent vector.

[0026] Optionally, optimizing the final interior model based on a differentiable rendering pipeline to generate the complete shape, appearance, and pose of all objects in the interior scene specifically includes:

[0027] Define a differentiable rendering pipeline, and use the differentiable rendering pipeline to apply an image loss to render the shape and appearance of the symbolic distance field;

[0028] Based on the differentiable prior database, color values ​​are obtained by reprojection of the indoor single-view RGB image through color sampling.

[0029] The color values ​​are matched with the surface of the final interior model, and an image loss constraint optimization process is applied according to the differentiable rendering pipeline to generate the appearance of the final interior model. The optimized final interior model is used to generate the complete shape, appearance and pose of all objects in the interior scene.

[0030] Furthermore, to achieve the above objectives, the present invention also provides an indoor scene reconstruction system based on a single view, wherein the indoor scene reconstruction system based on a single view includes:

[0031] The image acquisition module is used to acquire indoor single-view RGB images of indoor scenes;

[0032] The indoor preliminary model construction module is used to generate multiple image features based on the indoor single-view RGB image, construct a feature attention map and target object attributes based on the multiple image features, and construct an indoor preliminary model based on the feature attention map and the target object attributes.

[0033] The final indoor model generation module is used to predict background object attributes based on multiple image features and target object attributes, and optimize the preliminary indoor model based on the background object attributes to obtain the final indoor model.

[0034] The indoor scene reconstruction module is used to optimize the final indoor model based on a differentiable rendering pipeline, generating the complete shape, appearance, and pose of all objects in the indoor scene.

[0035] Furthermore, to achieve the above objectives, the present invention also provides a terminal, wherein the terminal includes a processor and a memory connected to the processor, the memory storing a single-view-based indoor scene reconstruction program, wherein the configuration program based on different chip solutions is executed by the processor to perform the single-view-based indoor scene reconstruction method as described above.

[0036] Furthermore, to achieve the above objectives, the present invention also provides a storage medium storing a computer program for single-view-based indoor scene reconstruction, the computer program being executable by a processor to implement the single-view-based indoor scene reconstruction method described above.

[0037] In this invention, an indoor single-view RGB image of an indoor scene is acquired; multiple image features are generated based on the indoor single-view RGB image, and a feature attention map and target object attributes are constructed based on the multiple image features; a preliminary indoor model is constructed based on the feature attention map and the target object attributes; background object attributes are predicted based on the multiple image features and the target object attributes; the preliminary indoor model is optimized based on the background object attributes to obtain a final indoor model; the final indoor model is optimized based on a differentiable rendering pipeline to generate the complete shape, appearance, and pose of all objects in the indoor scene. This invention extracts image features from a single image and infers the final indoor model, dividing the indoor scene into foreground and background representations, and robustly recovers the geometry and appearance of the scene by optimizing the model. Attached Figure Description

[0038] Figure 1 This is a flowchart of a preferred embodiment of the indoor scene reconstruction method based on a single view of the present invention;

[0039] Figure 2This is a schematic diagram illustrating the principle of a preferred embodiment of the indoor scene reconstruction system based on a single view according to the present invention.

[0040] Figure 3 This is a schematic diagram of the operating environment of a preferred embodiment of the terminal of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of this invention clearer and more explicit, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0042] The preferred embodiment of the indoor scene reconstruction method based on a single view described in this invention, such as... Figure 1 As shown, the single-view-based indoor scene reconstruction method includes the following steps:

[0043] Step S10: Obtain an indoor single-view RGB image of the indoor scene.

[0044] Specifically, the indoor single-view RGB image is an RGB image of a typical indoor scene that does not require any processing. It is acquired through a camera or other device, such as by shooting, recording, or reading images from a storage device. The indoor single-view RGB image includes layout information of the indoor scene, such as furniture, home appliances, and information about other objects.

[0045] Step S20: Generate multiple image features based on the indoor single-view RGB image, construct a feature attention map and target object attributes based on the multiple image features, and construct an indoor preliminary model based on the feature attention map and the target object attributes.

[0046] Specifically, image processing and computer vision techniques can be used to extract multiple useful image features from the single-view RGB image. These features can include information such as color, texture, and depth. For example, for a given RGB image of an indoor scene, a deep learning model, such as a convolutional neural network (CNN), can be used to extract image features, which can include color channels, texture features, edge information, etc. Then, using multiple image features, a feature attention map and target object attributes are constructed. The feature attention map displays the feature intensity or density in different regions of the image, and the target object attributes can include information such as the object's category, location, and size. Using the constructed feature attention map and the target object, a preliminary indoor model is built, which can represent the objects in the scene and their relative positions.

[0047] Furthermore, the step of generating multiple image features based on the indoor single-view RGB image, constructing a feature attention map and target object attributes based on the multiple image features, and constructing a preliminary indoor model based on the feature attention map and the target object attributes, further includes the following steps before:

[0048] Define a scene decomposition network, which includes a transformer detection module, an object reasoning module, and a 3D reasoning module;

[0049] The transformer detection module includes a transformer encoder and a transformer decoder, and the object reasoning module includes multiple multilayer perceptrons.

[0050] The Detection Transformer Module: This is one of the core components of the Scene Decomposition Network (SDN), comprising a transformer encoder and a transformer decoder. The transformer is a popular sequence prediction architecture, first applied in computer vision to models such as DETR (Data-efficient Image Transformer). The Detection Transformer Module is responsible for detecting objects in the scene and generating a set of features for each detected object, which can be used for subsequent analysis and inference.

[0051] Object reasoning module: This module receives output features from the detection transformer module and uses multiple multilayer perceptrons (MLPs) to predict the attributes of the target object. These attributes include the object's category (e.g., sofa, chair, table, etc.), 2D bounding box (the object's position and size in the image), signed distance field (SDF) feature vector, etc. Multilayer perceptrons typically consist of multiple fully connected layers, where each layer transforms the input features into the output through a non-linear activation function.

[0052] In this embodiment, based on an RGB image of an indoor scene, SDN is first used to parse the scene. The detection transformer module receives image features and generates a set of features, each associated with a detected object in the image. For example, if the scene includes a sofa, table, and chair, the detection transformer module generates a set of features for each object. Then, the object inference module processes these features to infer the attributes of each object. For example, for a sofa object, the category of the sofa can be predicted as "sofa," and the 2D bounding box information of the sofa and the SDF feature vector associated with the object can be provided. Similarly, for other detected objects, the module will also predict their attributes. These attributes will be used to construct a preliminary indoor model to determine the location, shape, and appearance of each object, thereby achieving a preliminary reconstruction of the indoor scene.

[0053] Further, the step of generating multiple image features based on the indoor single-view RGB image, constructing a feature attention map and target object attributes based on the multiple image features, and constructing a preliminary indoor model based on the feature attention map and the target object attributes specifically includes:

[0054] The transformer encoder encodes the indoor single-view RGB image to obtain multiple image features.

[0055] A transformer encoder is a neural network architecture that transforms an input indoor single-view RGB image into multiple image features. This encoding process can include convolutional neural network (CNN) layers to extract information from the image, such as texture and shape.

[0056] The transformer decoder constructs a feature attention map and target object attributes based on multiple image features.

[0057] The transformer decoder uses image features output by the encoder to construct a feature attention map. A feature attention map is a data structure used to represent the correlation and attention distribution between different regions in an image, helping the system understand the relationships between key features and objects in the image. Simultaneously, the transformer decoder also uses these features to predict the attributes of target objects, such as category, 2D bounding box location, and SDF feature vectors. These attributes describe the various objects detected in the image and provide a basis for modeling.

[0058] The transformer decoder constructs a preliminary indoor model based on the feature attention map and the target object attributes.

[0059] By using feature attention maps and target object attributes, the transformer decoder can build a preliminary indoor model. This preliminary model includes estimates of the position, shape, and appearance of objects in the scene, as well as representations of relationships and layouts between objects.

[0060] In this embodiment, an RGB image of an indoor scene, including a sofa, table, and chairs, is used. First, the transformer encoder receives the image and encodes it into multiple image features, each containing information about the image. Next, the transformer decoder uses these image features to construct a feature attention map, which describes the correlation between different regions in the image. Based on the image features, the attributes of the target objects are predicted. For the sofa object, the sofa's category, 2D bounding box location, and SDF feature vector can be predicted. Finally, based on the feature attention map and the target object attributes, the transformer decoder constructs a preliminary indoor model. This preliminary model describes the position, shape, and appearance of the sofa, table, and chairs. This helps the system understand the scene and provides a foundation for subsequent reconstruction and optimization. The goal of this part is to use the transformer encoder and decoder to construct a preliminary indoor model, representing the estimation of the indoor scene through the feature attention map and target object attributes.

[0061] Step S30: Predict background object attributes based on multiple image features and target object attributes, optimize the preliminary indoor model based on the background object attributes, and obtain the final indoor model.

[0062] Specifically, based on target object attributes and multiple image features, the system uses a machine learning model (such as a neural network) to predict background object attributes. These attributes can include the background object's category, shape, color, and other properties. During training, the machine learning model learns how to infer background object attributes from target object attributes and image features. These attributes are then used to optimize the initial indoor model, resulting in the final indoor model. Because the final indoor model incorporates background prediction, once the background object attributes are predicted, the system uses these attributes to optimize the initial indoor model. This includes adjusting elements such as the shape and position of background objects to ensure consistency with the predicted attributes and harmony with other objects and the background in the scene. It can also restore invisible occluded areas in the image, making it more complete and closer to the real scene than the initial indoor model.

[0063] Further, the step of predicting background object attributes based on multiple image features and target object attributes, and optimizing the preliminary indoor model based on the background object attributes to obtain the final indoor model, includes the following prior steps:

[0064] The object reasoning module obtains foreground object attributes and background object attributes by regressing and predicting the target object attributes through multiple multilayer perceptrons, wherein the multilayer perceptron is a three-layer perceptron with ReLU activation.

[0065] The object inference module receives output features from the detection transformer module, which contains information about each detected object in the image. A multilayer perceptron (MLP) is used to regress (predict) the attributes of the target objects, including foreground and background object attributes. Each attribute, such as object category, 2D bounding box location, and SDF feature vector, corresponds to a MLP responsible for predicting the attribute value. Each MLP can include multiple hidden layers, using ReLU (Modified Linear Unit) activation function. ReLU's advantages include faster network training, increased network non-linearity, and prevention of gradient vanishing, resulting in better performance in predicting attribute values.

[0066] Further, the step of predicting background object attributes based on multiple image features and target object attributes, and optimizing the preliminary indoor model based on the background object attributes to obtain the final indoor model, specifically includes:

[0067] The transformer decoder constructs a foreground model based on the foreground object attributes and a background model based on the background object attributes;

[0068] The 3D reasoning module merges the background model into the preliminary indoor model to obtain the final indoor model.

[0069] Using the predicted background object attributes and the known foreground object attributes, these are input into the transformer decoder to construct a foreground model and a background model. The foreground model includes the geometry and position attributes of the detected target objects, while the background model covers background elements such as invisible occluded areas in the indoor scene.

[0070] In this embodiment, the 3D inference module is used to merge the background model into the preliminary interior model to generate the final interior model. This ensures the consistency of the interior scene by considering the interaction between foreground and background objects.

[0071] For example, given an RGB image of an interior living room, with a sofa as the target object and background elements such as walls, floor, and ceiling, the process begins by extracting multiple image features. Based on these features, the sofa's position and appearance attributes are identified. These features and attributes then predict background object attributes, such as the wall color being white, the floor texture being wood grain, and the ceiling height being 2.5 meters. Next, using these attributes, a foreground model (the sofa's geometry and material properties) and a background model (including white walls, wood-grain flooring, and a 2.5-meter ceiling) are constructed. Finally, a 3D inference module merges the background model into the initial interior model, considering the interaction between the sofa and the walls, floor, and ceiling, resulting in the final interior model. This model encompasses the complete living room scene, including the sofa and environmental elements, and can accurately reconstruct the living room's shape, appearance, and background attributes.

[0072] In another embodiment, the foreground model and background model can be merged into the initial interior model to better consider the interaction between foreground objects and the background. Since foreground objects may occlude background elements, such as a sofa obscuring part of a wall, merging the foreground models can better handle occlusion relationships, ensuring that background elements can be correctly drawn even in occluded areas. This helps ensure consistency between foreground objects and their surrounding environmental elements (such as walls and floors), improving the quality and realism of the scene.

[0073] Step S40: Optimize the final indoor model based on the differentiable rendering pipeline to generate the complete shape, appearance and pose of all objects in the indoor scene.

[0074] Specifically, after obtaining the final interior model, the process of optimizing the final interior model based on a differentiable rendering pipeline aims to adjust the shape, appearance, and pose of all objects in the interior scene to better align them with the actual image data. This is done by rendering the reconstructed 3D model onto a 2D image and optimizing based on the differences between the reconstruction result and the actual image.

[0075] Using a differentiable rendering pipeline, an initial 3D model (including detected objects and background) is rendered onto a 2D image. This rendering process considers the model's geometry, appearance, and pose to generate a simulated 2D image. The rendered simulated image is then compared to the actual input image, and the difference between them is calculated; this is called image loss, which measures the gap between the reconstructed model and the actual image. Utilizing gradient descent or other optimization algorithms, through backpropagation, the system adjusts the parameters of the 3D model (such as object position, shape, and color) to minimize image loss, gradually bringing the model's shape and appearance closer to a state matching the actual image. This process can be applied to all objects in the scene, including their geometric attributes (shape and position), appearance attributes (color and texture), and pose attributes (pose and angle). Finally, through multiple iterations of optimization, the model gradually approaches a state matching the actual indoor scene, generating a final indoor scene model that includes the complete shape, appearance, and pose of all objects, making the reconstruction of the indoor scene more accurate and realistic.

[0076] Furthermore, the optimization of the final interior model based on the differentiable rendering pipeline to generate the complete shape, appearance, and pose of all objects in the interior scene also includes:

[0077] The shapes of the foreground model and the background model are encoded into symbolic distance fields with relevant latent feature vectors based on a differentiable prior database.

[0078] The shapes of objects in the final indoor model are distinguished to obtain the maximum likelihood latent vector of the object shape. The object shapes are then matched and optimized based on the shape prior information in the differentiable prior database and the maximum likelihood latent vector.

[0079] In this embodiment, the shapes of both the foreground and background models are encoded as signed distance fields (SDFs) with associated latent feature vectors. SDF is a method for representing object geometry that encodes the distance from each point of the object to the nearest surface as a positive or negative value, making the shape representation more flexible and differentiable. The shapes of individual objects in the final interior model are then distinguished, and a maximum likelihood latent vector containing information about the object's shape is determined for each object. This step is to further refine and optimize the object shapes.

[0080] Differentiable prior databases are trained by extracting frames or features from pre-collected real image or video data, reflecting realistic scene layouts. Based on shape prior information and maximum likelihood latent vectors in the differentiable prior database, object shapes are matched and optimized, covering details, curvature, and geometric features. The final shape is improved based on shape prior information and observed data. Through these steps, the most likely latent vectors explaining the observed shapes can be found, and shape prior information is used to match and optimize object shapes, thereby refining the object shape and making the reconstructed object shape more accurate.

[0081] For example, in reconstructing an indoor scene containing a table and a chair, the shapes of the table and chair are first encoded as SDFs, where each SDF includes shape information of the object. Then, through shape differentiation, maximum likelihood latent vectors are determined for each table and chair; these vectors contain shape information about them. The shapes of the table and chair are then matched and optimized using shape prior information from a differentiable prior database, along with observed scene information. For instance, based on known general shape characteristics of the table and chair, their curvature and geometric features can be better adjusted to better match the actual scene. Ultimately, through this optimization process, the shapes of the table and chair will more accurately reflect the details of the actual scene, making the indoor model more realistic and accurate. This is crucial for applications in interior design, simulation, virtual reality, and other fields because it provides a better method for scene reconstruction and shape optimization.

[0082] Furthermore, the optimization of the final interior model based on the differentiable rendering pipeline to generate the complete shape, appearance, and pose of all objects in the interior scene specifically includes:

[0083] Define a differentiable rendering pipeline, and use the differentiable rendering pipeline to apply an image loss to render the shape and appearance of the symbolic distance field;

[0084] Based on the differentiable prior database, color values ​​are obtained by reprojection of the indoor single-view RGB image through color sampling.

[0085] The color values ​​are matched with the surface of the final interior model, and an image loss constraint optimization process is applied according to the differentiable rendering pipeline to generate the appearance of the final interior model. The optimized final interior model is used to generate the complete shape, appearance and pose of all objects in the interior scene.

[0086] First, a differentiable rendering pipeline needs to be defined. This is a mathematical model that maps 3D scene information (the shape and appearance of the final interior model) to a 2D image space. This rendering pipeline must be differentiable so that gradients can be computed during optimization, allowing for parameter updates. Within the rendering pipeline, the shape and appearance information of the final interior model is passed, generating a rendered 2D image. Then, the rendered image is compared to the original single-view RGB image of the interior to calculate image loss. Image loss is a metric used to measure the difference between the rendered image and the actual image, such as using mean squared error (MSE) or other suitable loss functions. Next, color values ​​are sampled from the single-view RGB image of the interior using a reprojection technique, leveraging a differentiable prior database. This is to obtain color information for each object in the scene to match the appearance information of the rendered model. The color values ​​sampled from the differentiable prior database are matched against the surfaces of the final interior model to ensure that the color information matches the appearance of the 3D model.

[0087] Image loss and color matching loss are combined as constraints for the entire optimization process. Optimization algorithms (such as gradient descent) are then used to minimize these losses, thereby updating the shape and appearance information of the final interior model.

[0088] Once the optimization process is complete, the final interior model will contain accurate shape, appearance, and pose information of the objects. This model can be used to generate a complete interior scene, including the 3D geometry and appearance of all objects, as well as their position and pose within the scene.

[0089] It should be noted that this invention allows color sampling from color values ​​across different frames. This means that color sampling can be performed not only using the same frame as the input frame, but also using frames different from the input frame, and even obtaining color information from multiple frames with different lighting angles. This provides flexibility for inference about occluded regions during training. This allows the system to better understand objects and occlusion in the scene. Furthermore, although different frames can be used for color sampling, the coordinate positions of objects do not change in the real world. This means that even if the sampling comes from different frames, the position and geometry of objects in an indoor scene remain consistent. Therefore, during inference, there is no need to sample from different frames; all information can be processed based on a single view.

[0090] This invention is fully editable, allowing users to edit directly on the 3D model even when there are minor changes to the interior environment layout. This means that if modifications to the interior scene are needed, there is no need to rescan the entire environment; editing can be done directly on the existing model, providing greater flexibility and efficiency.

[0091] This invention enables self-supervised training from video data. Each time an indoor panoramic scan is performed, the data is fed back and used to update the model. This means the system can continuously improve and optimize the generated indoor scenes, making them more accurate and realistic. This self-supervised training method helps to continuously improve the model's performance and accuracy.

[0092] Furthermore, such as Figure 2 As shown, based on the above-described single-view-based indoor scene reconstruction method, the present invention also provides a single-view-based indoor scene reconstruction system, wherein the single-view-based indoor scene reconstruction system includes:

[0093] Image acquisition module 51 is used to acquire indoor single-view RGB images of indoor scenes;

[0094] The indoor preliminary model construction module 52 is used to generate multiple image features based on the indoor single-view RGB image, construct a feature attention map and target object attributes based on the multiple image features, and construct an indoor preliminary model based on the feature attention map and the target object attributes.

[0095] The final indoor model generation module 53 is used to predict background object attributes based on multiple image features and target object attributes, optimize the preliminary indoor model based on the background object attributes, and obtain the final indoor model.

[0096] The indoor scene reconstruction module 54 is used to optimize the final indoor model based on the differentiable rendering pipeline, and generate the complete shape, appearance and pose of all objects in the indoor scene.

[0097] Furthermore, such as Figure 3 As shown, based on the above-mentioned single-view-based indoor scene reconstruction method and system, the present invention also provides a terminal, which includes a processor 10, a memory 20 and a display 30. Figure 3 Only some of the terminal components are shown; however, it should be understood that it is not required to implement all of the components shown, and more or fewer components may be implemented instead.

[0098] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as a hard disk or memory. In other embodiments, the memory 20 may be an external storage device of the terminal, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc. Further, the memory 20 may include both internal and external storage devices. The memory 20 is used to store application software and various types of data installed on the terminal, such as program code installed on the terminal. The memory 20 can also be used to temporarily store data that has been output or will be output. In one embodiment, the memory 20 stores a single-view-based indoor scene reconstruction program 40, which can be executed by the processor 10 to implement the single-view-based indoor scene reconstruction method of this application.

[0099] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor, or other data processing chip, used to run program code stored in the memory 20 or process data, such as executing the single-view-based indoor scene reconstruction method.

[0100] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch-sensitive liquid crystal display, or an OLED (Organic Light-Emitting Diode) touchscreen. The display 30 is used to display information on the terminal and to display a visual user interface. The components 10-30 of the terminal communicate with each other via a system bus.

[0101] In one embodiment, when the processor 10 executes the single-view-based indoor scene reconstruction program 40 in the memory 20, the following steps are performed:

[0102] Acquire an indoor single-view RGB image of an indoor scene;

[0103] Multiple image features are generated based on the indoor single-view RGB image, and a feature attention map and target object attributes are constructed based on the multiple image features. An indoor preliminary model is constructed based on the feature attention map and the target object attributes.

[0104] Background object attributes are predicted based on multiple image features and target object attributes. The preliminary indoor model is then optimized based on the background object attributes to obtain the final indoor model.

[0105] The final interior model is optimized based on a differentiable rendering pipeline to generate the complete shape, appearance, and pose of all objects in the interior scene.

[0106] The process of generating multiple image features from the indoor single-view RGB image, constructing a feature attention map and target object attributes based on the multiple image features, and constructing a preliminary indoor model based on the feature attention map and the target object attributes, further includes the following steps before:

[0107] Define a scene decomposition network, which includes a transformer detection module, an object reasoning module, and a 3D reasoning module;

[0108] The transformer detection module includes a transformer encoder and a transformer decoder, and the object reasoning module includes multiple multilayer perceptrons.

[0109] Specifically, the step of generating multiple image features based on the indoor single-view RGB image, constructing a feature attention map and target object attributes based on the multiple image features, and constructing a preliminary indoor model based on the feature attention map and the target object attributes specifically includes:

[0110] The transformer encoder encodes the indoor single-view RGB image to obtain multiple image features;

[0111] The transformer decoder constructs a feature attention map and target object attributes based on multiple image features;

[0112] The transformer decoder constructs a preliminary indoor model based on the feature attention map and the target object attributes.

[0113] The step of predicting background object attributes based on multiple image features and target object attributes, and optimizing the preliminary indoor model based on the background object attributes to obtain the final indoor model, further includes:

[0114] The object reasoning module obtains foreground object attributes and background object attributes by regressing and predicting the target object attributes through multiple multilayer perceptrons, wherein the multilayer perceptron is a three-layer perceptron with ReLU activation.

[0115] Specifically, the step of predicting background object attributes based on multiple image features and target object attributes, and optimizing the preliminary indoor model based on the background object attributes to obtain the final indoor model, includes:

[0116] The transformer decoder constructs a foreground model based on the foreground object attributes and a background model based on the background object attributes;

[0117] The 3D reasoning module merges the background model into the preliminary indoor model to obtain the final indoor model.

[0118] The step of optimizing the final indoor model based on a differentiable rendering pipeline to generate the complete shape, appearance, and pose of all objects in the indoor scene also includes:

[0119] The shapes of the foreground model and the background model are encoded into symbolic distance fields with relevant latent feature vectors based on a differentiable prior database.

[0120] The shapes of objects in the final indoor model are distinguished to obtain the maximum likelihood latent vector of the object shape. The object shapes are then matched and optimized based on the shape prior information in the differentiable prior database and the maximum likelihood latent vector.

[0121] Specifically, optimizing the final indoor model based on a differentiable rendering pipeline to generate the complete shape, appearance, and pose of all objects in the indoor scene includes:

[0122] Define a differentiable rendering pipeline, and use the differentiable rendering pipeline to apply an image loss to render the shape and appearance of the symbolic distance field;

[0123] Based on the differentiable prior database, color values ​​are obtained by reprojection of the indoor single-view RGB image through color sampling.

[0124] The color values ​​are matched with the surface of the final interior model, and an image loss constraint optimization process is applied according to the differentiable rendering pipeline to generate the appearance of the final interior model. The optimized final interior model is used to generate the complete shape, appearance and pose of all objects in the interior scene.

[0125] The present invention also provides a storage medium storing a single-view-based indoor scene reconstruction program, which, when executed by a processor, implements the steps in the single-view-based indoor scene reconstruction method described above.

[0126] In summary, this invention provides a method and related equipment for indoor scene reconstruction based on a single view. The method includes: acquiring an indoor single-view RGB image; obtaining scene layout features in the indoor single-view RGB image according to a scene decomposition network, and reconstructing an indoor scene model; and optimizing the reconstructed indoor scene model according to a differentiable rendering pipeline. This invention infers a fully editable and re-renderable model of the indoor scene from a single image. Movable objects are represented separately from the immovable background, and the complete 3D model of each different object in the scene and its spatial relationships are restored. The model is fully editable and re-renderable. When there are minor changes in the indoor environment layout, it can be edited directly on the 3D model without rescanning the entire environment. It can robustly restore the scene geometry and appearance, and reconstruct and re-render the scene from a new viewpoint.

[0127] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes that element.

[0128] Of course, those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware (such as a processor, controller, etc.). The program can be stored in a computer-readable storage medium, and when executed, it can include the processes described in the above method embodiments. The storage medium can be a memory, magnetic disk, optical disk, etc.

[0129] It should be understood that the application of the present invention is not limited to the examples above. Those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A method for reconstructing indoor scenes based on a single view, characterized in that, The single-view-based indoor scene reconstruction method includes: Acquire an indoor single-view RGB image of an indoor scene; Multiple image features are generated based on the indoor single-view RGB image, and a feature attention map and target object attributes are constructed based on the multiple image features. An indoor preliminary model is constructed based on the feature attention map and the target object attributes. Background object attributes are predicted based on multiple image features and target object attributes. The preliminary indoor model is then optimized based on the background object attributes to obtain the final indoor model. The final interior model is optimized based on a differentiable rendering pipeline to generate the complete shape, appearance, and pose of all objects in the interior scene.

2. The indoor scene reconstruction method based on a single view according to claim 1, characterized in that, The process of generating multiple image features based on the indoor single-view RGB image, constructing a feature attention map and target object attributes based on the multiple image features, and constructing a preliminary indoor model based on the feature attention map and the target object attributes, further includes the following steps beforehand: Define a scene decomposition network, which includes a transformer detection module, an object reasoning module, and a 3D reasoning module; The transformer detection module includes a transformer encoder and a transformer decoder, and the object reasoning module includes multiple multilayer perceptrons.

3. The indoor scene reconstruction method based on a single view according to claim 2, characterized in that, The process of generating multiple image features based on the indoor single-view RGB image, constructing a feature attention map and target object attributes based on the multiple image features, and constructing a preliminary indoor model based on the feature attention map and the target object attributes specifically includes: The transformer encoder encodes the indoor single-view RGB image to obtain multiple image features; The transformer decoder constructs a feature attention map and target object attributes based on multiple image features; The transformer decoder constructs a preliminary indoor model based on the feature attention map and the target object attributes.

4. The indoor scene reconstruction method based on a single view according to claim 3, characterized in that, The process of predicting background object attributes based on multiple image features and target object attributes, optimizing the preliminary indoor model based on the background object attributes, and obtaining the final indoor model, further includes: The object reasoning module obtains foreground object attributes and background object attributes by regressing and predicting the target object attributes through multiple multilayer perceptrons, wherein the multilayer perceptron is a three-layer perceptron with ReLU activation.

5. The indoor scene reconstruction method based on a single view according to claim 4, characterized in that, The step of predicting background object attributes based on multiple image features and target object attributes, and optimizing the preliminary indoor model based on the background object attributes to obtain the final indoor model, specifically includes: The transformer decoder constructs a foreground model based on the foreground object attributes and a background model based on the background object attributes; The 3D reasoning module merges the background model into the preliminary indoor model to obtain the final indoor model.

6. The indoor scene reconstruction method based on a single view according to claim 5, characterized in that, The process of optimizing the final interior model based on a differentiable rendering pipeline to generate the complete shape, appearance, and pose of all objects in the interior scene, previously included: The shapes of the foreground model and the background model are encoded into symbolic distance fields with relevant latent feature vectors based on a differentiable prior database. The shapes of objects in the final indoor model are distinguished to obtain the maximum likelihood latent vector of the object shape. The object shapes are then matched and optimized based on the shape prior information in the differentiable prior database and the maximum likelihood latent vector.

7. The indoor scene reconstruction method based on a single view according to claim 6, characterized in that, The optimization of the final indoor model based on a differentiable rendering pipeline to generate the complete shape, appearance, and pose of all objects in the indoor scene specifically includes: Define a differentiable rendering pipeline, and use the differentiable rendering pipeline to apply an image loss to render the shape and appearance of the symbolic distance field; Based on the differentiable prior database, color values ​​are obtained by reprojection of the indoor single-view RGB image through color sampling. The color values ​​are matched with the surface of the final interior model, and an image loss constraint optimization process is applied according to the differentiable rendering pipeline to generate the appearance of the final interior model. The optimized final interior model is used to generate the complete shape, appearance and pose of all objects in the interior scene.

8. A single-view-based indoor scene reconstruction system, characterized in that, The single-view-based indoor scene reconstruction system includes: The image acquisition module is used to acquire indoor single-view RGB images of indoor scenes; The indoor preliminary model construction module is used to generate multiple image features based on the indoor single-view RGB image, construct a feature attention map and target object attributes based on the multiple image features, and construct an indoor preliminary model based on the feature attention map and the target object attributes. The final indoor model generation module is used to predict background object attributes based on multiple image features and target object attributes, and optimize the preliminary indoor model based on the background object attributes to obtain the final indoor model. The indoor scene reconstruction module is used to optimize the final indoor model based on a differentiable rendering pipeline, generating the complete shape, appearance, and pose of all objects in the indoor scene.

9. A terminal, characterized in that, The system includes a processor and a memory connected to the processor, the memory storing a single-view-based indoor scene reconstruction program, wherein the configuration program based on different chip solutions, when executed by the processor, is used to perform the single-view-based indoor scene reconstruction method as described in any one of claims 1-8.

10. A storage medium, characterized in that, The storage medium stores a computer program for single-view-based indoor scene reconstruction, which can be executed by a processor to implement the single-view-based indoor scene reconstruction method as described in any one of claims 1-8.

Images