A monocular image-based open world three-dimensional scene reconstruction and perception method
By using a pre-trained model for instance segmentation and occlusion filling, combined with a monocular reconstruction model, the problems of multi-instance occlusion and lack of multi-view information in open-world scenes are solved, realizing 3D reconstruction and perception based on monocular images, and improving the restoration capability and generalization performance of complex scenes.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- XIAMEN UNIV
- Filing Date
- 2023-05-17
- Publication Date
- 2026-08-04
AI Technical Summary
Existing technologies struggle to perform effective 3D reconstruction and perception based on monocular images in open-world scenarios, especially in multi-instance scenarios where there are issues of occlusion and lack of multi-view information. Furthermore, existing methods lack the ability to generalize to different scenarios.
A pre-trained scene perception model is used for instance segmentation and perception. The instance segmentation model and generative adversarial network are used to fill in occluded parts. A monocular reconstruction model is combined for 3D reconstruction. The 3D position is calculated by scene depth image and instance depth image. Finally, the 3D perception information is fused.
It realizes multi-instance 3D reconstruction and perception based on monocular images in open-world scenes, improves robustness and scene restoration ability, can handle instance occlusion problems in complex scenes, and shows good generalization ability in different scenes.
Smart Images
Figure CN117475105B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of computer vision technology, and in particular relates to a method for open-world 3D scene reconstruction and perception based on monocular images. Background Technology
[0002] 3D scene reconstruction and perception is an interdisciplinary field of computer vision and computer graphics. Its purpose is to reconstruct the original 3D scene and its corresponding 3D semantics based on sensor information such as images. In the field of 3D reconstruction, from traditional RGBD-based KinectFusion and BundleFusion, to MVS (Multi-View Stereo)-based COLMAP, MVE, PMVS, SMVS, and OpenMVS, to voxel, point cloud, and surface-based Pix2Mesh, Pix2vox, GANverse3D, and MeshMVS, and more recently, NeRF (Neural Radiance Field) based on implicit representation functions, most works rely on relatively dense multi-view information for reconstruction, making it impossible to complete reconstruction based on monocular images. In the field of 3D perception, from traditional 3D object detection methods such as PointSIFT, PointNet, PointNet++, Frustum PointNet, VoxelNet, SECOND, and PointRCNN, to recent occupancy prediction methods, most works require accurate multi-view 3D annotation information. This results in high costs, and the quantity and diversity of samples are insufficient to handle complex open-world scenes. Compared to closed-world scenes, open-world scenes contain more unknowns and uncertainties; therefore, open-world perception models need to handle situations with greater uncertainty and complexity.
[0003] Chinese patent application CN115375844A proposes a monocular 3D reconstruction method that integrates an attention mechanism. The method is based on dense point clouds for 3D reconstruction. However, the method has poor scene generalization ability, does not decouple multiple instance objects in open worlds, cannot perform instance perception, and does not handle possible occlusion problems. Therefore, it is not suitable for monocular 3D reconstruction and perception in open worlds. Summary of the Invention
[0004] The purpose of this invention is to address the limitations of existing monocular imaging, the complexity of open worlds, and the problems of numerous instances, occlusion between instances, and lack of multi-view information in monocular imaging. It provides a monocular image-based method for open-world 3D scene reconstruction and perception, specifically for multi-instance open-world scenes, without relying on complex 3D annotation information. First, the scene is decoupled from instances; each instance is perceived, processed, and reconstructed separately before being reassembled. To address the occlusion problem, a filling network is used to fill in the occluded portions of instances. Finally, to address the lack of multi-view information in monocular imaging, additional pre-trained perception and generative models are used to assist the reconstruction process.
[0005] This invention proposes an open-world 3D scene reconstruction and perception method based on monocular images, comprising the following steps:
[0006] 1) Scene perception steps: Given a monocular image, use a pre-trained scene perception model to perceive the scene in the image and obtain two-dimensional scene perception information, including depth images.
[0007] 2) Scene decoupling steps: Use the instance segmentation model to segment the image scene into instances, and obtain the perceptual information of each instance based on the visible information of each instance in the current image scene and other instance perception models.
[0008] 3) Instance processing steps: By performing some additional centering, content filling and other preprocessing, we avoid the reconstruction difficulties caused by mutual occlusion and inconsistent positions between instances. We then perform perception again on each instance image after processing to obtain the semantic information of the instance, including the depth image.
[0009] 4) Instance reconstruction steps: Using the instance's image and other semantic information, perform 3D reconstruction on each instance through a monocular reconstruction model, and extract the 3D model of each instance after reconstruction.
[0010] 5) Scene reconstruction step: The 3D position of each instance's 3D model in the scene is calculated using the scene depth image and instance depth image, thereby recombining the instances into the 3D scene corresponding to the original image;
[0011] 6) Scene perception information fusion step: Using the 3D model of each instance reconstructed in step 4), calculate the 3D perception information of each instance in the reconstructed scene.
[0012] In step 2), the specific steps of the scene decoupling step include:
[0013] 2.1) Use an instance segmentation model to perform instance perception on the image scene and obtain a segmentation map for each instance in the scene;
[0014] 2.2) Use other instance-aware models to acquire instance-level perceptual information for each instance, such as obtaining a natural language description of each instance through a pre-trained image description model;
[0015] In step 3), the specific method of the instance processing step includes:
[0016] 3.1) To avoid reconstruction difficulties caused by occlusion between instances, each instance is preprocessed to aid subsequent 3D reconstruction; preprocessing includes, but is not limited to, filling the content of the occluded part of the image, placing the instance in the center of the image, adding extra edges, etc.; the method of filling the content of the occluded part of the image includes, but is not limited to, generative adversarial networks, diffusion denoising models, etc.
[0017] 3.2) For the preprocessed instance images, perform instance perception again to obtain the instance semantic information of each instance; the instance semantic information includes the image, mask, and depth.
[0018] In step 4), the three-dimensional reconstruction uses a three-dimensional reconstruction method based on monocular images. The reconstructed three-dimensional representation includes, but is not limited to, triangular mesh, point cloud, neural radiation field (NeRF), and symbolic distance function (SDF).
[0019] In step 6), the three-dimensional perception information includes three-dimensional bounding boxes, three-dimensional semantics with instance annotations, or occupancy grids, etc.
[0020] In steps 1), 2), and 3), the present invention may use pre-trained scene and instance-aware networks, including semantic segmentation networks, object detection networks, image description networks, and depth estimation networks;
[0021] This invention proposes a method for open-world 3D scene reconstruction and perception based on monocular images, which can achieve 3D scene reconstruction and perception given a single open-world monocular image. It effectively handles problems such as varying instance counts and instance occlusion in open-world scenes, improving the robustness of open-world 3D reconstruction and perception.
[0022] This invention proposes an open-world 3D scene reconstruction and perception method based on monocular images. By decoupling, perceiving, and reconstructing instances in an open-world scene, it can effectively restore the 3D shape of each instance and the state of unknown viewpoints in complex scene images, and finally reassemble them into the original 3D scene, obtaining the corresponding 3D perception information. This method combines multiple pre-trained open-world or single-object reconstruction and perception models to complete the entire reconstruction and perception process, fully leveraging the advantages of each perception model, instance segmentation model, content filling model, and 3D reconstruction model to achieve better results in reconstructing complex scenes. Furthermore, each submodule in this method is independent and can be customized and continuously improved according to user needs. Attached Figure Description
[0023] Figure 1 This is a framework diagram for open-world 3D scene reconstruction and perception methods based on monocular images;
[0024] Figure 2 This is a user interface reference diagram for the scene perception and decoupling steps of the present invention;
[0025] Figure 3 This is a user interface reference diagram of the processing steps in an example of the present invention;
[0026] Figure 4 This is a user interface reference diagram for the reconstruction steps of an example of the present invention;
[0027] Figure 5 This is a user interface reference diagram for the scene reconstruction and perception information fusion steps of the present invention;
[0028] Figure 6 This invention provides an observation of the final 3D scene and 3D bounding box from different perspectives.
[0029] Figure 7 This is an example of the three-dimensional reconstruction result of a monocular food image according to the present invention;
[0030] Figure 8 This is an example of the three-dimensional perception results of monocular animal images according to the present invention. Detailed Implementation
[0031] To make the objectives, technical solutions, and advantages of this invention clearer, the following embodiments will be used in conjunction with the accompanying drawings to further illustrate the invention. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
[0032] like Figure 1 As shown in the figure, this invention proposes an open-world 3D scene reconstruction and perception method based on monocular images, which specifically includes the following steps:
[0033] 1) Scene perception steps: Given a monocular image X, in this embodiment, a three-fruit image with occlusion is used. The pre-trained depth estimation model DPT (Vision Transformers for Dense Prediction, https: / / github.com / isl-org / DPT) is used to perceive the scene and obtain the scene depth D.
[0034] 2) Scene decoupling steps, user interface reference Figure 2 As shown:
[0035] 2.1) The SegmentAnythingModel (https: / / github.com / facebookresearch / segment-anything) is used to perform instance awareness on the image scene. Users interact by specifying positive and negative sample points of a single instance, and the segmentation map m of each instance in the scene is obtained sequentially. i Each instance is denoted as instance i∈I, and in this embodiment, the segmentation diagrams of the three fruits can be obtained accordingly.
[0036] 2.2) Use the dense image description model BLIP (Bootstrapping Language-Image Pre-training, https: / / github.com / salesforce / BLIP) to obtain the natural language description c for each instance. i Users can modify this description to make it more accurate. In this embodiment, a description for each fruit image can be obtained.
[0037] 3) Instance processing steps, user interface reference Figure 3 As shown:
[0038] 3.1) For each instance, use the segmentation map m generated in step 2.1). i Calculate its 2D bounding box, fill it with a certain edge distance, obtain the original image and mask input of the instance, and describe c in the language of the instance. i As a guide, the pre-trained diffusion denoising model StableDiffusion (https: / / github.com / huggingface / diffusers) is used to fill in the content outside the mask. This step can be omitted if the instance itself is not occluded. In this example, the apple image is incomplete because it is occluded by foreground fruit; after filling, it is restored to a complete apple.
[0039] 3.2) For the inflated instance images, use the instance segmentation model SAM and the depth estimation model DPT for instance perception to re-obtain the image x of each instance. i , mask m i and depth d i In this embodiment, taking an apple as an example, the mask of the apple image and the depth estimate of a single apple object were obtained.
[0040] 4) Instance reconstruction steps, user interface reference Figure 4 As shown: The image x after processing each instance is generated using the Stable-Dreamfusion method (https: / / github.com / ashawkey / stable-dreamfusion / tree / main / nerf) for monocular reconstruction. i , mask m i and depth d i A 3D reconstruction is performed, and its geometric and material information is extracted. Finally, a triangular mesh is exported as the 3D model representation of this instance, denoted as Mesh. i This includes vertex and UV. RGB Information such as bananas are used in this embodiment to obtain a three-dimensional mesh representation of bananas.
[0041] 5) Scene reconstruction steps:
[0042] 5.1) Based on the position and size of the two-dimensional bounding box in step 3) and the set edge distance, obtain the position and scale of each instance i on the camera plane.
[0043] 5.2) Based on the depth D of the entire scene and the depth d of each instance. i Calculate the position of each instance i in the camera depth direction.
[0044] 6) Scene-aware information fusion step, user interface reference Figure 5 As shown:
[0045] 6.1) Obtain the 3D bounding box of each instance in the scene by the position of the triangular mesh grid points, including AABB (Axis-Aligned Bounding Box), OBB (Oriented Bounding Box), and convex hull, etc. In this embodiment, the AABB bounding box of each object is drawn.
[0046] 6.2) By using the triangular mesh grid positions of each instance and the semantic information of that instance, its 3D instance segmentation texture map UV in the scene can be obtained. SegFor each vertex of the instance triangular mesh i The .vertex variable is specifically operated on as follows:
[0047] UV Seg (Mesh i .vertex)=i(1)
[0048] 6.3) By calculating the triangular mesh occupancy information of each instance in a certain voxel space v∈V, the 3D grid occupancy of the instance in the scene is obtained. The specific calculation is as follows:
[0049] Occ(v) = iif any(Mesh) i .vertexinv)else Empty(2)
[0050] Figure 6 The invention presents observations of the final 3D scene obtained from different viewpoints. It can be seen that the invention achieves open-world 3D scene reconstruction and perception based on monocular images.
[0051] The interface in this embodiment of the invention is written using the Grado library, and the user interface provided in the embodiment is for reference only.
[0052] This invention enables the reconstruction and perception of multiple object instances in an open world using only monocular information, significantly expanding the utilization rate of monocular information. Furthermore, due to the decoupling of multiple object instances, it can effectively reconstruct the 3D shapes of individual objects and their states from unknown perspectives even in complex scene images, ultimately restoring them to the original scene. The entire process is completed by reconstructing and combining multiple modules, leveraging the advantages of each module—perception, filling, and reconstruction—to achieve the best reconstruction results. Simultaneously, this method has excellent generalization capabilities, easily replacing any single module, allowing for personalized customization at any step of the model's process.
[0053] Experiments show that this invention has extremely high generalization ability to various open scenes and can be applied to different images with huge distribution differences, such as... Figure 7 For the 3D scene reconstruction of monocular images of hamburgers, fries, and cola, and for observation from a new perspective, Figure 8 3D instance segmentation and AABB bounding box extraction for puppy images.
[0054] The above embodiments are merely preferred embodiments of the present invention and should not be considered as limiting the scope of the present invention. All equivalent variations and improvements made within the scope of the present invention should still fall within the patent coverage of the present invention.
Claims
1. A method for open-world 3D scene reconstruction and perception based on monocular images, characterized in that... Includes the following steps: 1) Scene perception steps: Given a monocular image, use a pre-trained scene perception model to perceive the scene in the image and obtain two-dimensional scene perception information, including depth images. 2) Scene decoupling steps: Use the instance segmentation model to segment the image scene into instances, and obtain the perceptual information of each instance based on the visible information of each instance in the current image scene and other instance perception models. 3) Instance processing steps: Preprocessing avoids reconstruction difficulties caused by mutual occlusion and inconsistent positions between instances. The processed instance images are then perceptualized again to obtain semantic information of the instances, including depth images. 4) Instance reconstruction steps: Using the instance's image and other semantic information, perform 3D reconstruction on each instance through a monocular reconstruction model, and extract the 3D model of each instance after reconstruction. 5) Scene reconstruction step: The 3D position of each instance's 3D model in the scene is calculated using the scene depth image and instance depth image, thereby recombining the instances into the 3D scene corresponding to the original image; 6) Scene perception information fusion step: Using the 3D model of each instance reconstructed in step 4), calculate the 3D perception information of each instance in the reconstructed scene.
2. The method for open-world 3D scene reconstruction and perception based on monocular images as described in claim 1, characterized in that... In step 2), the scene decoupling step specifically includes: 2.1) Use an instance segmentation model to perform instance perception on the image scene and obtain a segmentation map for each instance in the scene; 2.2) Use other instance-aware models to acquire instance-level perception information for each instance, and obtain a natural language description of each instance through a pre-trained image description model.
3. The method for open-world 3D scene reconstruction and perception based on monocular images as described in claim 1, characterized in that... In step 3), the instance processing steps specifically include: 3.1) To avoid reconstruction difficulties caused by occlusion between instances, each instance is preprocessed to aid subsequent 3D reconstruction; the preprocessing includes content filling of the occluded part of the image, placing the instance in the center of the image, and adding extra edges; the method for content filling of the occluded part of the image includes generative adversarial networks and diffusion denoising models. 3.2) For the preprocessed instance images, perform instance perception again to obtain the instance semantic information of each instance; the instance semantic information includes the image, mask, and depth.
4. The method for open-world 3D scene reconstruction and perception based on monocular images as described in claim 1, characterized in that... In step 4), the 3D reconstruction uses a monocular image-based 3D reconstruction method, and the reconstructed 3D representation includes a triangular mesh, a point cloud, a neural radiation field (NeRF), and a symbolic distance function (SDF).
5. The method for open-world 3D scene reconstruction and perception based on monocular images as described in claim 1, characterized in that... In step 6), the three-dimensional perception information includes a three-dimensional bounding box, three-dimensional semantics with instance annotations, or occupancy grid.
6. The method for open-world 3D scene reconstruction and perception based on monocular images as described in claim 1, characterized in that... It uses pre-trained scene and instance-aware networks, including semantic segmentation networks, object detection networks, image description networks, and depth estimation networks.