Method and device for reconstructing three-dimensional indoor scene, computer device and storage medium

Abstract

The application relates to a three-dimensional indoor scene reconstruction method and device, computer equipment, a storage medium and a computer program product. The method comprises the following steps: acquiring an indoor scene image, extracting object pose information and an object bounding box in the indoor scene image; performing feature extraction on the object in the indoor scene image according to the object bounding box, obtaining initial instance features corresponding to the object; filtering mixed features in the initial instance features, obtaining object instance features; generating an instance template according to the object in the indoor scene image, and performing instance feature deformation on the object instance features according to the instance template, obtaining an instance offset; and reconstructing a three-dimensional indoor scene by using a preset pixel alignment implicit function, the instance template, the instance offset, the object pose information and the object instance features. The method can use the pixel alignment implicit function to more accurately reconstruct the invisible area in the image, and can more accurately reconstruct the three-dimensional indoor scene.

Description

Technical Field

[0001] This application relates to the field of three-dimensional scene reconstruction technology, and in particular to a method, apparatus, computer equipment, storage medium and computer program product for reconstructing a three-dimensional indoor scene. Background Technology

[0002] Single-view indoor scene reconstruction is a long-standing problem involving the construction of a complete 3D scene from a single image. It involves various sub-techniques such as room layout estimation, object detection and pose estimation, and 3D shape reconstruction. Traditional solutions have proposed using 3D room bounding boxes to recover the room layout, followed by object pose recovery.

[0003] For example, traditional techniques estimate room layouts and interior objects using 3D bounding boxes to reconstruct 3D interior scenes; however, this method cannot provide detailed information about object shapes. Similarly, mesh-based retrieval methods improve instance shapes, but their accuracy and efficiency depend on the size and diversity of the database. Furthermore, voxel-based methods have limited resolution, and traditional techniques use Total3D and Im3D to reconstruct scenes in a simplified form, lacking detailed shapes and complex backgrounds. Moreover, to address the issue of instance occlusion in a scene, traditional techniques propose using self-attention channel filtering to separate mixed features. In summary, the solutions proposed by traditional techniques cannot effectively reconstruct the topological structure of instances in an interior scene; due to the failure to correctly reconstruct instances within the scene, the accuracy of instance categories and topological relationships is low. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, computer equipment, computer-readable storage medium, and computer program product for reconstructing three-dimensional indoor scenes with higher scene restoration accuracy to address the above-mentioned technical problems.

[0005] Firstly, this application provides a method for reconstructing a three-dimensional indoor scene. The method includes:

[0006] Acquire an indoor scene image, extract object pose information and object bounding boxes from the indoor scene image, the pose information being used to characterize the position and orientation of the object in the indoor scene image relative to the image acquisition device, the image acquisition device being a device for acquiring the indoor scene image;

[0007] Based on the object bounding box, feature extraction is performed on the objects in the indoor scene image to obtain the initial instance features corresponding to the objects;

[0008] Filter the mixed features in the initial instance features to obtain the object instance features;

[0009] Based on the objects in the indoor scene image, an instance template is generated, and the instance features of the objects are deformed according to the instance template to obtain the instance offset;

[0010] A 3D indoor scene is reconstructed using a preset pixel alignment implicit function, the instance template, the instance offset, the object pose information, and the object instance features.

[0011] In one embodiment, the step of extracting features from the objects in the indoor scene image based on the object bounding box to obtain the initial instance features corresponding to the objects includes:

[0012] Construct a watertight mesh corresponding to the indoor scene image;

[0013] Point cloud sampling is performed on the watertight mesh to obtain the point cloud model corresponding to the object in the indoor scene;

[0014] Convert the world coordinate system of the point cloud model corresponding to the object in the indoor scene to the model coordinate system;

[0015] Based on the point cloud model in the model coordinate system, extract the pixel position information corresponding to objects in the indoor scene;

[0016] Based on the pixel position information corresponding to the object, feature extraction is performed to obtain the scene features corresponding to the indoor scene;

[0017] The scene features are identified and filtered by bounding boxes to obtain the initial instance features corresponding to objects in the indoor scene.

[0018] In one embodiment, filtering the mixed features in the initial instance features to obtain object instance features includes:

[0019] The initial instance features are input into a preset feature selector, and the feature weights of the mixed features in the initial instance features are calculated.

[0020] The hybrid features are calculated based on the feature weights of the hybrid features and the initial instance features;

[0021] Filter the mixed features in the initial instance features to obtain intermediate state features;

[0022] Extract local instance features from the hybrid features;

[0023] The local instance features and the intermediate state features are combined to obtain the object instance features.

[0024] In one embodiment, extracting local instance features from the mixed features includes:

[0025] The mixed features are downsampled to obtain downsampled features;

[0026] The downsampled features are restored using a bilinear interpolation algorithm to obtain dimension-restored features.

[0027] The dimensionality recovery feature is convolved with the downsampled feature to obtain local instance features.

[0028] In one embodiment, generating an instance template based on objects in the indoor scene image includes:

[0029] Extract object instance images from indoor scene images;

[0030] The latent feature encoding of the object instance image is performed to obtain the latent variable of the instance category;

[0031] The latent variables of the instance category are input into a preset template feature generator to generate an instance template.

[0032] In one embodiment, the step of deforming the object instance features according to the instance template to obtain the instance offset includes:

[0033] The object instance features are input into a trained self-attention model to predict the initial offset.

[0034] The instance template, the object instance features, and the initial offset are concatenated to form combined data;

[0035] The combined data is input into a preset multilayer perceptron, and the contribution weight is calculated through the activation function in the multilayer perceptron.

[0036] The instance offset is calculated based on the contribution weight, the instance template, and the object instance features.

[0037] In one embodiment, the process of reconstructing a 3D indoor scene using a preset pixel alignment implicit function, the instance template, the instance offset, the object pose information, and the object instance features includes:

[0038] Obtain the depth value of pixels in an indoor scene image;

[0039] Based on the depth value, instance template, and object instance features, a 3D object model is generated using a preset pixel alignment implicit function.

[0040] Based on the depth value, instance template, and instance offset, a background instance is generated using a preset pixel alignment implicit function;

[0041] Based on the object's pose information, the 3D object model is combined with the background instance to obtain a 3D indoor scene.

[0042] In one embodiment, the step of combining the 3D object model with the background instance based on the object pose information to obtain a 3D indoor scene includes:

[0043] The implicit representation information corresponding to the three-dimensional object model is combined with the implicit representation information of the background instance to obtain the implicit representation information of the indoor scene.

[0044] Based on the implicit representation information of the indoor scene, a polygonal mesh of the indoor scene is obtained;

[0045] Based on the object pose information, the position information of the polygons in the polygonal mesh is corrected;

[0046] The corrected polygonal mesh is then converted to obtain a 3D indoor scene.

[0047] Secondly, this application also provides a device for reconstructing a three-dimensional indoor scene. The device includes:

[0048] The information extraction module is used to acquire indoor scene images and extract object pose information and object bounding boxes from the indoor scene images. The pose information is used to characterize the position and posture of the objects in the indoor scene images relative to the image acquisition device. The image acquisition device is a device for acquiring the indoor scene images.

[0049] The feature extraction module is used to extract features of objects in the indoor scene image based on the object bounding box to obtain the initial instance features corresponding to the objects;

[0050] The feature filtering module is used to filter the mixed features in the initial instance features to obtain the object instance features;

[0051] The feature deformation module is used to generate an instance template based on the objects in the indoor scene image, and to perform instance feature deformation on the object instance features based on the instance template to obtain the instance offset;

[0052] The scene reconstruction module is used to reconstruct a three-dimensional indoor scene using a preset pixel alignment implicit function, the instance template, the instance offset, the object pose information, and the object instance features.

[0053] Thirdly, this application also provides a computer device. The computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:

[0054] Acquire an indoor scene image, extract object pose information and object bounding boxes from the indoor scene image, the pose information being used to characterize the position and orientation of the object in the indoor scene image relative to the image acquisition device, the image acquisition device being a device for acquiring the indoor scene image;

[0055] Based on the object bounding box, feature extraction is performed on the objects in the indoor scene image to obtain the initial instance features corresponding to the objects;

[0056] Filter the mixed features in the initial instance features to obtain the object instance features;

[0057] Based on the objects in the indoor scene image, an instance template is generated, and the instance features of the objects are deformed according to the instance template to obtain the instance offset;

[0058] A 3D indoor scene is reconstructed using a preset pixel alignment implicit function, the instance template, the instance offset, the object pose information, and the object instance features.

[0059] Fourthly, this application also provides a computer-readable storage medium. The computer-readable storage medium stores a computer program thereon, which, when executed by a processor, performs the following steps:

[0060] Acquire an indoor scene image, extract object pose information and object bounding boxes from the indoor scene image, the pose information being used to characterize the position and orientation of the object in the indoor scene image relative to the image acquisition device, the image acquisition device being a device for acquiring the indoor scene image;

[0061] Based on the object bounding box, feature extraction is performed on the objects in the indoor scene image to obtain the initial instance features corresponding to the objects;

[0062] Filter the mixed features in the initial instance features to obtain the object instance features;

[0063] Based on the objects in the indoor scene image, an instance template is generated, and the instance features of the objects are deformed according to the instance template to obtain the instance offset;

[0064] A 3D indoor scene is reconstructed using a preset pixel alignment implicit function, the instance template, the instance offset, the object pose information, and the object instance features.

[0065] Fifthly, this application also provides a computer program product. The computer program product includes a computer program that, when executed by a processor, performs the following steps:

[0066] Acquire an indoor scene image, extract object pose information and object bounding boxes from the indoor scene image, the pose information being used to characterize the position and orientation of the object in the indoor scene image relative to the image acquisition device, the image acquisition device being a device for acquiring the indoor scene image;

[0067] Based on the object bounding box, feature extraction is performed on the objects in the indoor scene image to obtain the initial instance features corresponding to the objects;

[0068] Filter the mixed features in the initial instance features to obtain the object instance features;

[0069] Based on the objects in the indoor scene image, an instance template is generated, and the instance features of the objects are deformed according to the instance template to obtain the instance offset;

[0070] A 3D indoor scene is reconstructed using a preset pixel alignment implicit function, the instance template, the instance offset, the object pose information, and the object instance features.

[0071] This application provides a method, apparatus, computer equipment, storage medium, and computer program product for reconstructing a 3D indoor scene. The scheme utilizes object detection technology to obtain the bounding boxes and object pose information of indoor objects from an input indoor scene image. Then, through feature extraction, it extracts the initial instance features of the objects from the image. The scheme further separates the mixed feature portion from the initial instance features to obtain complete object instance features; this enables more accurate and effective separation of object instance features from cluttered scene information. Based on the obtained object instance features, the scheme uses the instance template corresponding to the object to deform the object instance features, obtaining instance offsets. The scheme generates corresponding instance templates and instance offsets for different objects, which not only allows for more accurate extraction of instance features but also enables fitting the topological structure of different object categories. Finally, based on a pixel-aligned implicit function, the scheme reconstructs the 3D indoor scene using the aforementioned obtained instance templates, instance offsets, object instance features, and object pose information. The scheme utilizes the pixel-aligned implicit function for more accurate reconstruction of invisible areas in the image, enabling more accurate reconstruction of the 3D indoor scene. Attached Figure Description

[0072] Figure 1 This is an application environment diagram of a three-dimensional indoor scene reconstruction method in one embodiment;

[0073] Figure 2 This is a flowchart illustrating a method for reconstructing a three-dimensional indoor scene in one embodiment;

[0074] Figure 3 This is a flowchart illustrating the sub-step of obtaining initial instance features in one embodiment;

[0075] Figure 4 This is a flowchart illustrating a method for reconstructing a three-dimensional indoor scene in another embodiment;

[0076] Figure 5 This is a structural block diagram of a three-dimensional indoor scene reconstruction device in one embodiment;

[0077] Figure 6 This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0078] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0079] The three-dimensional indoor scene reconstruction method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, both terminal 102 and image acquisition device 106 communicate with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be integrated onto server 104 or placed on a cloud or other network server. In this application environment, the image acquisition device 106 first acquires images of the indoor scene requiring 3D modeling, and then uploads the acquired indoor scene images to server 104. After receiving the indoor scene images, server 104 uses an object detection algorithm to obtain the pose and bounding boxes of each object in the scene. The pose information describes the position and orientation of the object; in this embodiment, it may include position information and orientation information. After obtaining the bounding boxes, necessary data preprocessing is performed, i.e., preprocessing the indoor scene images for parameters such as brightness and saturation. Then, based on the intrinsic parameters of image acquisition device 106, scene features of the indoor scene image are generated. Further, server 104 extracts features from the scene features of the indoor scene image based on the generated bounding boxes to obtain initial instance features in the indoor scene image. It should be noted that the initial feature extraction in this embodiment, using bounding boxes, results in instance features containing some complex and invalid feature information; that is, the extracted initial instance features are coarse instance features. After obtaining the coarse instance features, the server separates and filters the features of other instances within the coarse instance features to obtain complete object instance features. After obtaining the object instance features, the server uses an instance deformation model to generate a corresponding instance template by inputting the aforementioned object instance features; similarly, the indoor scene image is input into this instance deformation model to generate instance offsets corresponding to the object instance features. Finally, the server 104 captures the obtained instance template, instance offsets, and object instance features using an implicit function reconstruction model (ISDPIFu), reconstructing a high-quality indoor object model through the model's pixel-aligned implicit function; ISDPIFu can reconstruct a high-precision scene background using background templates and instances; then, based on the object pose information in the indoor scene image, the indoor object model is combined with the scene background to reconstruct a 3D indoor scene. In this application environment, terminal 102 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Server 104 can be implemented using a standalone server or a server cluster consisting of multiple servers. Image acquisition device 106 can include, but is not limited to, cameras, webcams, and scanners.

[0080] Indoor scene reconstruction has wide applications in AR and VR. Single-view indoor scene reconstruction requires accurate reconstruction of both visible and invisible areas of the indoor scene, including all objects in the scene and the background of the room. The visible area is the part captured by the camera in the view, while the invisible area is the part missing from the view due to mutual occlusion of objects, as well as the parts of objects not captured by the camera. Objects in indoor scenes often exhibit different topological structures, and mutual occlusion often exists between objects. Traditional deep learning methods alone are often insufficient to reconstruct accurate indoor scenes. This application provides a high-fidelity single-view reconstruction method based on instance separation and deformation. Figure 3 A method for reconstructing indoor scenes. Compared with traditional methods that directly learn individual instance features in a scene, the method provided in this application can separate effective instance features from cluttered scenes and generate corresponding instance templates and instance offsets according to different objects. This allows for more accurate extraction of instance features, fitting of the topological structure of different types of objects, and more accurate reconstruction of invisible areas in the image, enabling the reconstruction of high-fidelity indoor scenes.

[0081] In one embodiment, such as Figure 2 As shown, a method for reconstructing a three-dimensional indoor scene is provided, which can be applied to... Figure 1 Taking server 104 as an example, the following steps are included:

[0082] Step 202: Acquire an indoor scene image and extract the object pose information and object bounding boxes from the indoor scene image. The pose information is used to characterize the position and orientation of objects in the indoor scene image relative to the image acquisition device, which is a device for acquiring indoor scene images.

[0083] Pose information is data describing the position and orientation of an object, and in this embodiment, it may include position information and orientation information. Further, in this embodiment, position information describes the object's position in three-dimensional space and can be represented by coordinates. Orientation information in this embodiment describes the object's orientation or rotation relative to a reference coordinate system; orientation information mainly describes changes in the object's orientation, tilt, and rotation. In this embodiment, since pose information describes the pose and position between the image acquisition device and the objects in the acquired indoor scene, the reference coordinate system can be a spatial coordinate system constructed based on the position set by the image acquisition device. Additionally, in this embodiment, the object bounding box is typically used to determine the location of each target object in a specific image; that is, the bounding box is a rectangular box that exactly contains the object.

[0084] For example, the embodiment can use an object detection algorithm to obtain the pose and bounding box of each object in the input scene image. More specifically, the embodiment can use object recognition algorithms such as YOLO, Faster R-CNN, and SSD to identify and extract objects from the indoor scene image input to the algorithm model. In the embodiment, the objects to be extracted are various pieces of furniture in the scene. If necessary, the embodiment can also use a pre-trained classification model to classify the identified object to determine its category, for example, identifying furniture such as sofas and coffee tables in the indoor scene image. In the embodiment, after determining the category of the object, a corresponding pose estimation algorithm can be used to calculate the object's pose. The pose estimation algorithm in the embodiment can be based on feature matching, deep learning, or other methods. For example, a 3D model can be used to match the feature points of the object in the image to calculate the object's pose information.

[0085] Step 204: Extract features from objects in the indoor scene image based on object bounding boxes to obtain the initial instance features corresponding to the objects.

[0086] The initial instance features are coarse instance features obtained by performing simple preliminary extraction and filtering of scene features in indoor scene images.

[0087] For example, the initial extraction process in this embodiment is entity alignment, or instance alignment. In machine learning and computer vision, instance alignment can align instances in different domains, making them more consistent in feature representation. This embodiment employs Domain Adaptation, training a model that can share knowledge between the source and target domains to reduce the differences between the two domains and achieve feature alignment. Specifically, in this embodiment, a convolutional module, such as the Region Alignment (RoI) module, is used to extract features from the indoor scene image to obtain scene features. Then, based on the bounding boxes obtained in the preceding steps, the scene features are divided into coarse instance features of different objects.

[0088] Step 206: Filter the mixed features in the initial instance features to obtain the object instance features.

[0089] Among them, mixed features refer to the fact that, due to the complex occlusion relationship between a specific object instance and other object instances in an indoor scene image, its initial instance features, i.e., coarse instance features, are partially mixed with the feature information of other instances.

[0090] For example, since the coarse instance features of a single instance obtained after instance alignment in the embodiment are only locally mixed with features of other instances, it is not necessary to separate the entire instance features. It is only necessary to filter out the locally mixed features and perform feature separation. Specifically, in the embodiment, a feature separation model can be pre-built and trained using a large number of historical instance features to obtain a trained feature separation model. The coarse instance features obtained after instance alignment are input into the model to identify and extract the mixed features in the coarse instance features. Furthermore, some effective local instance features in the mixed features are filtered and retained, and combined with the coarse instance features after separating the mixed features to obtain the complete object instance features.

[0091] Step 208: Generate instance templates based on objects in the indoor scene image, and perform instance feature deformation on the object instance features based on the instance templates to obtain instance offsets.

[0092] The instance template is an abstract topological structure that reflects an instance, providing an accurate topological template for reconstructing the instance model. Instance feature deformation in this embodiment refers to the process of encoding instance features using latent variables to obtain a latent variable representation. The instance offset in this embodiment represents the latent variable representation of environmental background features in an indoor scene image. Furthermore, the latent encoding used in this embodiment is a form of dimensionality reduction or compression, aiming to express the essence of the data with less information.

[0093] Specifically, in this embodiment, after obtaining the instance features of objects, to more accurately separate or classify specific furniture, background, and other objects in the indoor scene image, the embodiment uses a latent space method for template learning. By learning in the latent space, interference from object details can be filtered out, focusing on the abstract representation of the object and reducing interference from mixed features, thereby better classifying the target object. Therefore, in this embodiment, the corresponding instance image is first obtained from the input scene image using bounding boxes. An instance latent variable encoder constructed using convolutional blocks and a multilayer perceptron (MLP) encodes the instance image into a 1024-dimensional category latent variable. Based on the obtained category latent variable, a corresponding instance template is generated using a pre-built template feature generator. Since multiple pieces of furniture are placed in the space described by the indoor scene image in this embodiment, the background in the scene will inevitably be occluded when forming the indoor scene image. Therefore, in this embodiment, it is necessary to restore and reconstruct the occluded background content based on the extracted object instances. Thus, instance offsets are used to describe the background information that needs to be restored and reconstructed. In this embodiment, the complete object instance features obtained in step S206 are first encoded as latent variables to obtain 1024-dimensional latent variables. Then, these 1024-dimensional latent variables are input into a trained instance offset deformation model for preliminary prediction to obtain an initial offset. Further, the instance offset deformation model concatenates the initial offset, the instance template, and the complete object instance features, calculating the instance offset using the weights formed during the training phase. It should be noted that, during the concatenation phase, the instance template and the complete object instance features involved in the concatenation are also latent variable representations.

[0094] Step 210: The three-dimensional indoor scene is reconstructed using the preset pixel alignment implicit function, instance template, instance offset, object pose information, and object instance features.

[0095] Among them, the pixel-aligned implicit function (PIFu) is an implicit representation that locally aligns the pixels of a two-dimensional image with the context of the corresponding three-dimensional object model.

[0096] For example, in this embodiment, an implicit function reconstruction model (ISDPIFu) is constructed based on PIFu. Specifically, in this embodiment, the latent vector representations of the object instance features and instance templates obtained in the aforementioned steps are input into the implicit function reconstruction model (ISDPIFu) to reconstruct a high-quality indoor object model. For the background of the indoor scene, in this embodiment, the entire indoor scene image is separated into background instances through feature separation. Then, based on the indoor scene image and instance offsets, a corresponding background template is generated. Finally, based on the object pose information, the indoor object model and the background template are combined to reconstruct a three-dimensional indoor scene.

[0097] The aforementioned method for reconstructing 3D indoor scenes utilizes object detection technology to obtain the bounding boxes and object pose information of indoor objects from the input indoor scene image. Then, through feature extraction, it extracts the initial instance features of the objects from the image. The method further separates the mixed feature portion from the initial instance features to obtain complete object instance features; this allows for more accurate and effective separation of object instance features from cluttered scene information. Based on the obtained object instance features, the method uses the corresponding instance template to deform the object instance features, obtaining instance offsets. The method generates corresponding instance templates and instance offsets for different objects, which not only allows for more accurate extraction of instance features but also enables fitting of the topological structure of different object categories. Finally, based on the pixel-aligned implicit function, the method reconstructs the 3D indoor scene using the aforementioned instance templates, instance offsets, object instance features, and object pose information. The method utilizes pixel alignment implicit functions for more accurate reconstruction of invisible areas in the image, resulting in a more accurate reconstruction of the 3D indoor scene.

[0098] In one embodiment, such as Figure 3 As shown, the method involves extracting features from objects in an indoor scene image based on their bounding boxes to obtain the initial instance features corresponding to the objects. This process may include the following steps:

[0099] Step 302: Construct a watertight mesh corresponding to the indoor scene image.

[0100] A watertight mesh can be defined as a mesh that is an edge manifold and a vertex manifold but does not self-intersect.

[0101] For example, in this embodiment, necessary data preprocessing is first required for the indoor scene image. The scene image is preprocessed using an image enhancement algorithm to adjust parameters such as brightness and saturation. Then, the real-world model corresponding to the objects in the scene is converted into a watertight mesh. Further, the process of converting the real-world model into a watertight mesh in this embodiment first defines the geometry of the real-world model. Then, the geometry is discretized into a finite number of mesh cells. Vertices are created for each mesh cell, and edges are constructed based on the relationships between vertices. By connecting adjacent vertices, the topology of the entire model can be generated. Material attributes, such as color and texture coordinates, are assigned to each vertex. The mesh is then optimized, and duplicate model vertices are removed, resulting in the final watertight mesh model.

[0102] Step 304: Perform point cloud sampling on the watertight mesh to obtain the point cloud model corresponding to the object in the indoor scene.

[0103] In this context, a point cloud refers to a dataset of points in a specific coordinate system. These points contain rich information, including 3D coordinates (X, Y, Z), color, classification value, intensity value, and so on. The point cloud model in this embodiment is constructed using points from this dataset. Furthermore, the point cloud sampling methods used in this embodiment include, but are not limited to, uniform sampling, geometric sampling, random sampling, and grid sampling.

[0104] For example, after obtaining the watertight mesh model of each object in the indoor scene image based on step 302, a point cloud model corresponding to the watertight mesh model can be generated by uniform sampling. More specifically, in this embodiment, the initial point cloud is first denoted as C, and the sampling point set is denoted as S, which is initialized as an empty set. A seed point Seed is randomly sampled and placed into S. Each time, a point is sampled and placed into S. The sampling rule is to find a point in set CS that is farthest from set S. The distance from the point to the set is the distance from the point to all points in the set that is the smallest distance. Finally, the point cloud model corresponding to the watertight mesh is obtained.

[0105] Step 306: Convert the world coordinate system of the point cloud model corresponding to the object in the indoor scene into the model coordinate system.

[0106] The world coordinate system refers to the coordinate system constructed in the real-world scene where the object is placed, used to describe the object's position information in the real world. The model coordinate system in the embodiment, also known as the camera coordinate system or the canonical coordinate system, is the coordinate system used to represent the pixel positions of points in the point cloud model in the image.

[0107] For example, the embodiment converts a realistic model of objects in the scene into a watertight mesh, performs point cloud sampling on it, and uses camera extrinsic parameters to transform the point cloud from the world coordinate system to the normalized coordinate system. The camera extrinsic parameters may include a rotation matrix and a translation matrix. Further, the rotation matrix describes the orientation of the coordinate axes of the world coordinate system relative to the camera, i.e., the coordinate axes of the image acquisition device. The translation matrix describes the position of the spatial origin in the camera coordinate system of the image acquisition device. In the embodiment, points in the world coordinate system can be mapped to the camera coordinate system based on both the rotation matrix and the translation matrix.

[0108] Step 308: Extract the pixel position information of objects in the indoor scene based on the point cloud model in the model coordinate system.

[0109] For example, in this embodiment, camera extrinsic parameters are used to transform the point cloud from the world coordinate system to the canonical coordinate system, and the pixel positions of the point cloud in the image are obtained by combining them with camera intrinsic parameters. The camera intrinsic parameters consist of focal length, pixels, and other parameters. Directional width, How many pixels are there within 1 millimeter in a direction? Described using pixels. The length of the focal length along the axis, described in pixels. The intrinsic parameter matrix is ​​composed of parameters such as the focal length along the axial direction and the actual position of the principal point. This intrinsic parameter matrix reflects the camera's own properties and is different for each camera; these parameters need to be obtained through calibration. In this embodiment, world coordinates are converted to camera coordinates, then camera coordinates are converted to image plane / image physical coordinates, and finally, pixel coordinates are obtained based on the intrinsic parameter matrix.

[0110] Step 310: Extract features based on the pixel position information of the object to obtain the scene features corresponding to the indoor scene.

[0111] For example, in this embodiment, scene features of scene images can be learned using a convolutional model. First, the constructed convolutional model is trained using a large number of historical scene images. Then, the pixel coordinates obtained from processing the indoor scene image using steps 302 to 308, along with the pixel information of the objects, are input into the trained convolutional model to obtain the scene features of the scene image. It should be noted that in this embodiment, the historical scene images will also undergo the same processing as the indoor scene images during model training to obtain the corresponding pixel coordinates and pixel information.

[0112] Step 312: Identify and filter scene features using bounding boxes to obtain initial instance features corresponding to objects in the indoor scene.

[0113] For example, in this embodiment, the bounding box is used to divide scene features into coarse instance features of different objects. The method provided in this embodiment can obtain more comprehensive and accurate initial instance features, providing reliable data support for subsequent processing steps such as extracting instance features and fitting the topological structure of different types of objects.

[0114] In one embodiment, the process of filtering mixed features from initial instance features to obtain object instance features includes the following steps:

[0115] Step 1: Input the initial instance features into the preset feature selector and calculate the feature weights of the mixed features in the initial instance features.

[0116] Step 2: Calculate the mixed features based on the feature weights of the mixed features and the initial instance features.

[0117] Step 3: Filter the mixed features in the initial instance features to obtain intermediate features.

[0118] Step 4: Extract local instance features from the mixed features.

[0119] Step 5: Combine the local instance features and intermediate state features to obtain the object instance features.

[0120] The feature selector is based on an embedded feature selection model. In this embodiment, a specific machine learning model is used for training to obtain the weight coefficients of each feature, and features are selected according to the coefficients from largest to smallest. The blended features in this embodiment are feature information formed by the interlacing occlusion relationships between objects in an indoor scene image. Further, intermediate features refer to the remaining part after the coarse instance features separate the blended features. Local instance features in this embodiment refer to the effective feature content included in the blended features for describing a specific object instance.

[0121] For example, in this embodiment, a convolutional model is used to learn scene features from scene images. And using the learned bounding box, scene features are... Rough instance features divided into different objects In the embodiments, coarse instance features It only locally mixes features from other instances; there's no need to separate the features of the entire instance. We only need to filter out the locally mixed features and perform feature separation. In this example, a feature selector is learned using convolutional blocks. Feature weights and through Obtaining blended features .for Further feature separation is performed to obtain local instance features. and through Obtain instance features .

[0122] In one embodiment, the process of extracting local instance features from the blended features may include the following steps:

[0123] Step 1: Downsample the mixed features to obtain the downsampled features.

[0124] Step 2: Use bilinear interpolation algorithm to restore the feature dimension of the downsampled features to obtain the dimension-restored features.

[0125] Step 3: Perform a convolution operation between the dimension recovery features and the downsampled features to obtain local instance features.

[0126] For example, in an embodiment, a sampling module based on a Unet structure can be used to perform feature separation to obtain local instance features. Furthermore, for the Unet structure, a convolutional layer with a 3x3 kernel, a stride of 2, and padding of 1 is used instead of a max-pooling layer for downsampling. After downsampling, a convolutional layer is used to optimize the sampling result. For upsampling, bilinear interpolation is used to restore the dimension of the features, and the restored features are concatenated with the corresponding downsampled features and then processed by a convolutional layer. Specifically, in the embodiment, a single feature separation cannot completely filter out mixed features, so multiple separations are required. In this invention, the number of separations is set to 4.

[0127] In one embodiment, the process of generating an instance template based on objects in an indoor scene image may include the following steps:

[0128] Step 1: Extract object instance images from indoor scene images.

[0129] Step 2: Encode the latent features of the object instance images to obtain the latent variables of the instance categories.

[0130] Step 3: Input the latent variables of the instance category into the preset template feature generator to generate the instance template.

[0131] For example, in this embodiment, bounding boxes are used to obtain corresponding instance images from the input scene image, and an 18-layer ResNet network is used to encode the instance images into 1024-dimensional class latent variables. More specifically, in this embodiment, class latent variables can be learned directly from the scene image using a ResNet network to guide the module in learning the correct instance template. Based on the instance and class latent variables, a template feature generator is implemented using convolutional blocks and a multi-layer MLP. Learning Example Templates This template reflects the abstract topology of the instance, and the accurate topology of the instance can be recovered from this template.

[0132] In one embodiment, the process of deforming the object instance features based on the instance template to obtain the instance offset includes:

[0133] Step 1: Input the object instance features into the trained self-attention model to predict the initial offset.

[0134] Step 2: Combine the instance template, object instance features, and initial offset to form combined data.

[0135] Step 3: Input the combined data into a preset multilayer perceptron, and calculate the contribution weight through the activation function in the multilayer perceptron.

[0136] Step 4: Calculate the instance offset based on the contribution weight, instance template, and object instance characteristics.

[0137] For example, regarding instance deviation The implementation uses an instance offset deformation module for learning. More specifically, firstly, the implementation uses a self-attention mechanism, that is, it learns from instance features using a trained self-attention model. Instance offsets are obtained from learning. Then, by using the instance template Instance characteristics and instance offset The system connects and uses convolutional blocks and a multi-layer MLP to implement a weight learning module that learns contribution weights. ,pass Optimize instance deviation Among them, for Values ​​close to 1 indicate that they come from The strong contribution, while values ​​close to 0 indicate that they come from This demonstrates a significant contribution. To further refine instance offsets, the implementation can utilize a Unet structure similar to that used in instance feature separation to optimize instance offsets. .

[0138] In one embodiment, the process of deforming the object instance features based on the instance template to obtain the instance offset may include the following steps:

[0139] Step 1: Obtain the depth value of the pixels in the indoor scene image.

[0140] Step 2: Based on the depth value, instance template, and object instance features, a 3D object model is generated using a preset pixel alignment implicit function.

[0141] Step 3: Based on the depth value, instance template, and instance offset, generate a background instance using a preset pixel alignment implicit function.

[0142] Step 4: Based on the object pose information, combine the 3D object model with the background instance to obtain a 3D indoor scene.

[0143] For example, in the specific embodiment, the process of implicit scene reconstruction using ISDPIFu first involves the embodiment using an instance template. Instance characteristics Instance deviation Aligning implicit functions with pixels, a 3D object model is reconstructed. and background examples And finally combined to form a three-dimensional indoor scene. The pixel alignment implicit function specifically constructed in this embodiment is as follows:

[0144]

[0145]

[0146]

[0147] in, For a point in space, Give Two-dimensional image projection points, It is a fully convolutional image encoder exist Local image features extracted at the location, It is the depth value in weak perspective camera coordinates. express Global features that encode specific features , and It defines a three-dimensional surface as a function. An implicit representation of the level set. Furthermore, a loss function is used in the embodiments. , Network training is conducted, among which It is a three-dimensional point in the normed space.

[0148] In one embodiment, the process of combining a 3D object model with a background instance based on object pose information to obtain a 3D indoor scene may include the following steps:

[0149] Step 1: Combine the implicit representation information corresponding to the 3D object model with the implicit representation information of the background instance to obtain the implicit representation information of the indoor scene.

[0150] Step 2: Based on the implicit information of the indoor scene, convert it into a polygonal mesh of the indoor scene.

[0151] Step 3: Correct the position information of polygons in the polygon mesh based on the object pose information.

[0152] Step four: Convert the corrected polygonal mesh to obtain a 3D indoor scene.

[0153] For example, in this embodiment, a polygonal mesh is used to represent the scene for display. The embodiment uses the Marching Cubes algorithm from computer graphics to transform the scene stored implicitly in the ISDPIFu representation into a mesh. Further, the embodiment corrects the vertex positions of the mesh based on the pose information of objects extracted from the indoor scene image to achieve the transformation of object poses in the scene. Furthermore, since a normalized spatial coordinate system is used during reconstruction, after transformation into a mesh, the mesh coordinates need to be transformed from the normalized spatial coordinate system to the world coordinate system. By combining the technical features of the extracted object instances, the size of the objects in the world coordinate system can be obtained. Using this size to transform the vertex coordinates of the mesh, the size of the mesh in the world coordinate system can be recovered.

[0154] Refer to the instruction manual. Figure 4 The specific implementation process of single-view indoor scene reconstruction is described in complete and detailed steps as follows:

[0155] Step 1: Use an object detection algorithm to obtain the pose and bounding box of each object in the scene from the input scene image.

[0156] Step 2, Data Preprocessing. The scene images are preprocessed using image enhancement algorithms to adjust parameters such as brightness and saturation. The realistic models of objects in the scene are converted into a watertight mesh, and point cloud sampling is performed on it. Camera extrinsic parameters are used to transform the point cloud from the world coordinate system to the normalized coordinate system, and combined with camera intrinsic parameters, the corresponding pixel positions of the point cloud in the image are obtained.

[0157] Step 3, Instance Alignment. The convolutional module is used to learn scene features from the scene image, and the bounding boxes learned in Step 1 are used to divide the scene features into coarse instance features of different objects.

[0158] Step 4, Instance Feature Separation. A convolutional block is used to implement feature weights learned by a feature selector, and further, mixed features are obtained. For the mixed features, a sampling module based on the UNet structure is used to separate the features into local instance features. These local instance features are then combined with the coarse instance features obtained after separating the mixed features to obtain the object instance features.

[0159] Step 5, Instance Feature Deformation. The corresponding instance image is obtained from the input scene image using the bounding box from Step 1, and encoded into a 1024-dimensional categorical latent variable using an 18-layer ResNet network. ResNet directly learns the categorical latent variable from the scene image, guiding the module to learn the correct instance template. Based on the instance and categorical latent variable, a template feature generator implemented using convolutional blocks and a multi-layer MLP learns the instance template. For instance offset, an instance offset deformation module is used for learning: First, the instance offset is learned from the instance features using a self-attention mechanism. Then, by concatenating the instance template, instance features, and instance offset, and using a weight learning module implemented using convolutional blocks and a multi-layer MLP, contribution weights are learned, optimizing the instance offset through these contribution weights.

[0160] Step Six: Implicit Reconstruction of the PIFU Scene. Using the instance template, instance feature Fi, instance offset, and PIFU implicit representation from Step Five, reconstruct the objects and background in the scene.

[0161] Step 7: Extract the Mesh from the Implicit Representation. Using the Marching Cubes algorithm from computer graphics, the scene stored in the ISDPIFu implicit representation is transformed into a Mesh. Using the object pose information learned in Step 1, the object pose in the scene is transformed by correcting the vertex positions of the Mesh. Furthermore, since the normalized coordinate system is used during reconstruction, the Mesh coordinates need to be transformed from the normalized coordinate system to the world coordinate system after transformation. By combining the object data processing in Steps 1 and 2, the size of the object in the world coordinate system can be obtained. Using this size to transform the Mesh vertex coordinates, the size of the Mesh in the world coordinate system can be recovered.

[0162] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0163] Based on the same inventive concept, this application also provides a three-dimensional indoor scene reconstruction apparatus for implementing the three-dimensional indoor scene reconstruction method described above. The solution provided by this apparatus is similar to the implementation scheme described in the above method; therefore, the specific limitations in one or more embodiments of the three-dimensional indoor scene reconstruction apparatus provided below can be found in the limitations of the three-dimensional indoor scene reconstruction method described above, and will not be repeated here.

[0164] In one embodiment, such as Figure 5 As shown, a three-dimensional indoor scene reconstruction device 500 is provided, including: an information extraction module 501, a feature extraction module 502, a feature filtering module 503, a feature deformation module 504, and a scene reconstruction module 505, wherein:

[0165] The information extraction module 501 is used to acquire indoor scene images, extract object pose information and object bounding boxes in the indoor scene images, and the pose information is used to characterize the position and posture of objects in the indoor scene images relative to the image acquisition device. The image acquisition device is a device for acquiring indoor scene images.

[0166] The feature extraction module 502 is used to extract features of objects in an indoor scene image based on the object bounding box to obtain the initial instance features corresponding to the object.

[0167] The feature filtering module 503 is used to filter the mixed features in the initial instance features to obtain the object instance features.

[0168] The feature deformation module 504 is used to generate an instance template based on the objects in the indoor scene image, and to perform instance feature deformation on the object instance features based on the instance template to obtain the instance offset.

[0169] The scene reconstruction module 505 is used to reconstruct a 3D indoor scene using preset pixel alignment implicit functions, instance templates, instance offsets, object pose information, and object instance features.

[0170] In one embodiment, the feature extraction module 502 is further configured to construct a watertight mesh corresponding to the indoor scene image; perform point cloud sampling on the watertight mesh to obtain a point cloud model corresponding to the object in the indoor scene; convert the world coordinate system of the point cloud model corresponding to the object in the indoor scene into a model coordinate system; extract the pixel position information corresponding to the object in the indoor scene based on the point cloud model in the model coordinate system; perform feature extraction based on the pixel position information corresponding to the object to obtain the scene features corresponding to the indoor scene; and identify and filter the scene features through bounding boxes to obtain the initial instance features corresponding to the object in the indoor scene.

[0171] In one embodiment, the feature filtering module 503 is further configured to input the initial instance features to a preset feature selector, calculate the feature weights of the mixed features in the initial instance features; calculate the mixed features based on the feature weights of the mixed features and the initial instance features; filter the mixed features in the initial instance features to obtain intermediate features; extract the local instance features in the mixed features; and combine the local instance features and the intermediate features to obtain the object instance features.

[0172] In one embodiment, the feature filtering module 503 is further configured to downsample the mixed features to obtain downsampled features; restore the feature dimension of the downsampled features using a bilinear interpolation algorithm to obtain dimension-restored features; and perform a convolution operation between the dimension-restored features and the downsampled features to obtain local instance features.

[0173] In one embodiment, the feature deformation module 504 is further configured to extract object instance images from indoor scene images; encode latent features in the object instance images to obtain latent variables of instance categories; and input the latent variables of instance categories into a preset template feature generator to generate instance templates.

[0174] In one embodiment, the feature deformation module 504 is further configured to input object instance features into a trained self-attention model to predict an initial offset; concatenate the instance template, object instance features, and initial offset to form combined data; input the combined data into a preset multilayer perceptron and calculate the contribution weight through the activation function in the multilayer perceptron; and calculate the instance offset based on the contribution weight, instance template, and object instance features.

[0175] In one embodiment, the scene reconstruction module 505 is further configured to obtain the depth values ​​of pixels in an indoor scene image; generate a three-dimensional object model based on the depth values, instance template, and object instance features using a preset pixel alignment implicit function; generate a background instance based on the depth values, instance template, and instance offset using a preset pixel alignment implicit function; and combine the three-dimensional object model with the background instance based on the object pose information to obtain a three-dimensional indoor scene.

[0176] In one embodiment, the scene reconstruction module 505 is further configured to combine the implicit representation information corresponding to the three-dimensional object model with the implicit representation information of the background instance to obtain the implicit representation information of the indoor scene; convert the implicit representation information of the indoor scene to obtain a polygonal mesh of the indoor scene; correct the position information of the polygons in the polygonal mesh according to the object pose information; and convert the corrected polygonal mesh to obtain a three-dimensional indoor scene.

[0177] The modules in the aforementioned 3D indoor scene reconstruction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in the processor of a computer device in hardware form or independent of it, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0178] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 6 As shown, this computer device includes a processor, memory, input / output (I / O) interfaces, and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides the environment for the operating system and computer programs stored in the non-volatile storage media. The database stores indoor scene images and related data. The I / O interfaces are used for information exchange between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for reconstructing a three-dimensional indoor scene.

[0179] Those skilled in the art will understand that Figure 6The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0180] In one embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0181] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0182] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0183] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0184] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0185] The above embodiments are merely illustrative of several implementation methods of this application, and their descriptions are relatively specific and detailed. However, they should not be construed as limiting the scope of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for reconstructing a three-dimensional indoor scene, characterized in that, The method includes: Acquire an indoor scene image, extract object pose information and object bounding boxes from the indoor scene image, the pose information being used to characterize the position and orientation of the object in the indoor scene image relative to the image acquisition device, the image acquisition device being a device for acquiring the indoor scene image; Based on the object bounding box, feature extraction is performed on the objects in the indoor scene image to obtain the initial instance features corresponding to the objects; Filter the mixed features in the initial instance features to obtain the object instance features; Based on the objects in the indoor scene image, an instance template is generated, and the instance features of the objects are deformed according to the instance template to obtain the instance offset; A 3D indoor scene is reconstructed using a preset pixel alignment implicit function, the instance template, the instance offset, the object pose information, and the object instance features.

2. The method according to claim 1, characterized in that, The step of extracting features from objects in the indoor scene image based on the object bounding box to obtain the initial instance features corresponding to the objects includes: Construct a watertight mesh corresponding to the indoor scene image; Point cloud sampling is performed on the watertight mesh to obtain the point cloud model corresponding to the object in the indoor scene; Convert the world coordinate system of the point cloud model corresponding to the object in the indoor scene to the model coordinate system; Based on the point cloud model in the model coordinate system, extract the pixel position information corresponding to the objects in the indoor scene; Based on the pixel position information corresponding to the object, feature extraction is performed to obtain the scene features corresponding to the indoor scene; The scene features are identified and filtered by bounding boxes to obtain the initial instance features corresponding to objects in the indoor scene.

3. The method according to claim 1, characterized in that, The process of filtering the mixed features from the initial instance features to obtain object instance features includes: Obtain the feature weights of the mixed features in the initial instance features; The hybrid features are calculated based on the feature weights of the hybrid features and the initial instance features; Filter the mixed features in the initial instance features to obtain intermediate state features; Extract local instance features from the hybrid features; The local instance features and the intermediate state features are combined to obtain the object instance features.

4. The method according to claim 3, characterized in that, The extraction of local instance features from the mixed features includes: The mixed features are downsampled to obtain downsampled features; The downsampled features are restored by performing feature dimension recovery using a bilinear interpolation algorithm to obtain dimension-restored features; The dimensionality recovery feature and the downsampled feature are convolved to obtain local instance features.

5. The method according to claim 1, characterized in that, The step of generating an instance template based on objects in the indoor scene image includes: Extract object instance images from indoor scene images; The latent feature encoding of the object instance image is performed to obtain the latent variable of the instance category; The latent variables of the instance category are input into a preset template feature generator to generate an instance template.

6. The method according to claim 1, characterized in that, The step of deforming the object instance features according to the instance template to obtain the instance offset includes: The object instance features are input into a trained self-attention model to predict the initial offset. The instance template, the object instance features, and the initial offset are concatenated to form combined data; The combined data is input into a preset multilayer perceptron, and the contribution weight is calculated through the activation function in the multilayer perceptron. The instance offset is calculated based on the contribution weight, the instance template, and the object instance features.

7. The method according to any one of claims 1-6, characterized in that, The process of reconstructing a 3D indoor scene using a preset pixel alignment implicit function, the instance template, the instance offset, the object pose information, and the object instance features includes: Obtain the depth value of pixels in an indoor scene image; Based on the depth value, instance template, and object instance features, a 3D object model is generated using a preset pixel alignment implicit function. Based on the depth value, instance template, and instance offset, a background instance is generated using a preset pixel alignment implicit function; Based on the object's pose information, the 3D object model is combined with the background instance to obtain a 3D indoor scene.

8. The method according to claim 7, characterized in that, The step of combining the 3D object model with the background instance based on the object pose information to obtain a 3D indoor scene includes: The implicit representation information corresponding to the three-dimensional object model is combined with the implicit representation information of the background instance to obtain the implicit representation information of the indoor scene. Based on the implicit representation information of the indoor scene, a polygonal mesh of the indoor scene is obtained; Based on the object pose information, the position information of the polygons in the polygonal mesh is corrected; The corrected polygonal mesh is then converted to obtain a 3D indoor scene.

9. A device for reconstructing a three-dimensional indoor scene, characterized in that, The device includes: The information extraction module is used to acquire indoor scene images and extract object pose information and object bounding boxes from the indoor scene images. The pose information is used to characterize the position and posture of the objects in the indoor scene images relative to the image acquisition device. The image acquisition device is a device for acquiring the indoor scene images. The feature extraction module is used to extract features of objects in the indoor scene image based on the object bounding box to obtain the initial instance features corresponding to the objects; The feature filtering module is used to filter the mixed features in the initial instance features to obtain the object instance features; The feature deformation module is used to generate an instance template based on the objects in the indoor scene image, and to perform instance feature deformation on the object instance features based on the instance template to obtain the instance offset; The scene reconstruction module is used to reconstruct a three-dimensional indoor scene using a preset pixel alignment implicit function, the instance template, the instance offset, the object pose information, and the object instance features.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 8.

Images