An Adaptive 3D Scene Reconstruction Method and System Based on a Single Panoramic Image

Abstract

This invention discloses an adaptive 3D scene reconstruction method and system based on a single panoramic image, belonging to the field of computer vision processing. The method first generates a depth map and constructs an initial 3D mesh from an indoor scene panoramic image; then, it generates multi-view images and their occlusion masks based on viewpoint transformation, forming training sample pairs for fine-tuning the diffusion completion model and injecting prior scene information into the model; next, it extracts the mesh boundary contours and calculates the central axis, adaptively constructing a camera pose set; after iteratively completing and optimizing the mesh integrity, it is converted into a 3D Gaussian sputtering field, and the Gaussian point parameters are optimized using multi-view data; during the optimization process, rendering error feedback and edge Gaussian point upsampling strategies are adopted, finally outputting a scene model with complete geometric texture. This invention reconstructs high-quality 3D scenes from a single panoramic image through structural adaptive completion and Gaussian refinement techniques, and is particularly suitable for immersive roaming reconstruction of indoor scenes.

Description

Technical Field

[0001] This invention belongs to the field of computer vision processing, and in particular relates to an adaptive 3D scene reconstruction method and system based on a single panoramic image. Background Technology

[0002] In the digital economy era, the demand for digital replication of the physical world is growing, especially in the field of 3D reconstruction of indoor scenes, where its application has moved from cutting-edge exploration to large-scale commercialization. Starting with a single 360-degree panoramic image for 3D reconstruction has become a key technological approach to address this need due to its convenient data acquisition and low cost. However, a single panoramic image is essentially a two-dimensional projection, and its inherent depth ambiguity and severe occlusion problems make directly recovering a complete 3D space extremely challenging. The core challenge in this field is how to not only reconstruct the visible parts but also reasonably infer and complete the occluded areas.

[0003] To address this challenge, existing technologies have evolved from implicit to explicit representations. Neural Radiance Fields (NeRF), as an implicit representation method, learns continuous volumetric representations of scenes through deep networks, enabling the synthesis of photorealistic new perspective images. However, its reliance on volume rendering, which involves dense sampling along each ray and network inference, results in extremely high computational costs for training and rendering, making real-time interaction difficult. To solve the rendering efficiency bottleneck, 3D Gaussian Splatting (3DGS) has been proposed as an explicit scene representation method. 3DGS represents a scene as a collection of millions of 3D Gaussian ellipsoids, each defined by position, covariance, color, and opacity. During rendering, these 3D Gaussians are directly projected onto a 2D image plane and rasterized using an efficient alpha blending algorithm.

[0004] However, neither NeRF nor 3DGS can solve the fundamental problem of "information loss" caused by single-viewpoint input. Relying solely on visible information results in an incomplete "empty shell" model. To address this, researchers have introduced generative techniques, such as diffusion models, to inpaint unobserved regions. By setting a new virtual viewpoint in the scene, an image containing the missing regions is rendered, then filled in by the diffusion model, and finally, the incomplete information is used to optimize the 3D representation.

[0005] Although combining 3DGS with generative models is the current mainstream approach, existing technologies still suffer from the following three core shortcomings in practical implementation, resulting in deficiencies in the completeness, consistency, and accuracy of the reconstruction results:

[0006] First, the blind sampling of viewpoints leads to incomplete scene completion. Existing methods typically employ a naive strategy of uniform or random sampling within the scene when selecting new viewpoints to complete it. This strategy ignores the scene's own geometry. When faced with scenes with complex layouts (such as L-shaped rooms or large occlusions), this strategy cannot guarantee the effective exploration of all occluded key areas, and the virtual camera may never be positioned to observe the area behind structural occlusions. This directly results in the final reconstructed model still containing large areas of structural voids, severely lacking in completeness, and also wastes computational resources.

[0007] Second: General generative models lead to inconsistencies in the completed content. Existing methods typically use diffusion models pre-trained on general large-scale datasets for content inpainting. These models lack prior knowledge of the style of the specific scene (such as unique wallpaper textures, furniture materials, and lighting conditions). Therefore, while the generated completed content may be locally realistic, it often exhibits significant visual inconsistencies with the actually observed parts globally. For example, it may generate textures that do not match the surroundings or shadows that contradict global lighting, creating "illusion" artifacts that undermine the overall realism of the scene. Existing technologies generally lack an effective scene prior injection mechanism, such as using low-rank adaptation (LoRA) techniques to quickly fine-tune the model and ensure the consistency of the generated content.

[0008] Third: The 3D representation optimization process leads to the loss of geometric details. When using a view containing generated content to optimize the final 3DGS representation, the standard optimization process does not specifically address the boundary between "real data" and "generated data." Standard adaptive densification strategies (such as split or cloned Gaussian) are general mechanisms that may not be able to add enough new Gaussian points to accurately depict details in these critical stitching areas. This results in problems such as blurred object edges, geometric breaks, or texture distortion at the boundary, causing the clear geometric details in the original scene to suffer a loss of accuracy after reconstruction.

[0009] Therefore, there is an urgent need for a method to generate a 3D model that combines completeness, consistency, and high fidelity from a single panoramic image. Summary of the Invention

[0010] The purpose of this invention is to address the following technical problems existing in the reconstruction of 3D scenes from a single panoramic image: 1) how to adaptively plan viewpoints according to the scene structure to ensure the integrity of scene completion; 2) how to enable the generated model to perceive prior information of a specific scene to ensure the consistency of the completed content; 3) how to optimize the update process of 3D representation to preserve and enhance the geometric details of the scene. This invention provides an adaptive 3D scene reconstruction method and system based on a single panoramic image. Its core is to use 3D Mesh adaptive completion and 3D Gaussian upsampling optimization to generate a geometrically and texturally consistent static 3D Gaussian sputtering field, reconstructing a 3D indoor scene with fine texture, accurate geometry, and coherence, enabling multi-view indoor roaming and new perspective generation tasks.

[0011] The core technical concept of this invention includes: firstly, extracting structural information such as scene boundaries and central axis from the input panoramic image of an indoor scene to guide adaptive synthesis of the camera viewpoint; secondly, using a diffusion completion model to perform semantically guided image inpainting of occluded areas from the synthesized viewpoint, combining camera pose information and semantic cues; and finally, converting the completed 3D mesh into a 3D Gaussian sputtering representation and using a proposed Gaussian upsampling strategy to subdivide and optimize edge regions to improve the geometric and texture accuracy of the model. This invention, through pixel-level alignment-guided 3D scene generation, preserves the fine details of the original panoramic image while automatically completing unobserved areas, thus achieving high-quality free-viewpoint 3D reconstruction.

[0012] To achieve the above-mentioned objectives, the present invention specifically adopts the following technical solution:

[0013] In a first aspect, the present invention provides an adaptive 3D scene reconstruction method based on a single panoramic image, which includes the following steps:

[0014] S1: Acquire a panoramic color image of the indoor scene and estimate its depth information to generate a depth map. Perform edge estimation and back projection operations on the depth map to generate an initial 3D mesh of the indoor scene.

[0015] S2: Based on the initial 3D mesh, six original single-view images from a cube perspective are generated through viewpoint transformation; then the projection camera pose is transformed, and a projection image of each original single-view image after the camera pose transformation and its occlusion mask are generated through projection transformation; then the projection image and its occlusion mask are back-projected to form training sample pairs and the diffusion completion model is fine-tuned; each training sample pair contains four parts: original single-view image, projection image and occlusion mask, and camera pose text description.

[0016] S3: Extract the boundary contour of the initial 3D mesh, generate the boundary contour curve, and smooth it using Gaussian filtering to generate a smooth closed boundary curve; calculate the central axis of the interior contour skeleton of the indoor scene using the central axis transformation algorithm based on the boundary points uniformly sampled from the boundary curve; sample along the central axis at a preset step size, use the coordinates of the sampled points as the camera pose coordinates, and use the vector direction from the origin of the indoor scene to the sampled points as the camera pose direction to form a camera pose set; perform multiple iterations to complete the initial 3D mesh based on the camera pose set, optimize the integrity of the initial 3D mesh through conflict detection and merging, form the completed 3D mesh, and output a multi-viewpoint dataset;

[0017] S4: Convert the completed 3D mesh into a 3D Gaussian sputtering field and initialize the Gaussian point parameters. Optimize the Gaussian point parameters based on the multi-view dataset. During the optimization process, adjust the consistency of the 3D Gaussian sputtering field through rendering error feedback loss. When the number of optimization rounds exceeds the preset round threshold, use the Gaussian upsampling strategy to split and refine the Gaussian points located at the edges, and finally output a complete 3D scene model.

[0018] Based on the above scheme, each step can be implemented in the following preferred manner.

[0019] As a preferred embodiment of the first aspect mentioned above, the specific process of step S1 is as follows: acquire a two-dimensional panoramic color image, predict its depth value through a panoramic depth prediction model, form a depth map from the predicted depth value, extract the edge mask of the depth map through Canny edge detection and dilation operation, map the color and depth information of the panoramic color image to the point and surface information of the three-dimensional mesh based on back projection operation, segment the triangular mesh facets connecting different objects using the edge mask, and generate the initial three-dimensional mesh of the indoor scene at the origin of the indoor scene.

[0020] As a preferred embodiment of the first aspect mentioned above, the specific process of step S2 is as follows:

[0021] S21: In the initial 3D mesh scene, keep the camera pose coordinates at the origin of the indoor scene, adjust the camera pose orientation to the six faces of the cube, and generate 6 original single-view images of the indoor scene and their corresponding camera pose text descriptions.

[0022] S22: Perform the following operations on each original single-view image: backproject the original single-view image to 3D space to generate a corresponding single-view mesh and use it as the first single-view mesh; perform up, down, left, and right deflection transformations on each camera pose according to a preset deflection angle; and generate multiple different rendering images through the first single-view mesh. Then, project the rendering images back to 3D space to generate corresponding single-view meshes and use them as the second single-view mesh. Backproject the original viewpoint corresponding to the origin of the indoor scene using the second single-view mesh and calculate the mask to obtain the occlusion area mask. Add the occlusion area mask to the original single-view image to form a masked single-view image as the projection image. Finally, combine an original single-view image, a projection image, an occlusion area mask, and a camera pose text description into a set of training sample pairs to form a training dataset. On this training dataset, use the LoRA model to fine-tune the diffusion completion model.

[0023] As a preferred embodiment of the first aspect mentioned above, in step S22, the fine-tuning objects include the self-attention layer and cross-attention layer of the diffusion completion model U-Net, and the self-attention layer of the Text Encoder network.

[0024] As a preferred embodiment of the first aspect mentioned above, the specific process of step S3 is as follows:

[0025] S31: Based on the geometric structure information of the indoor scene in the initial 3D mesh, extract the origin boundary points of the indoor scene to generate the boundary contour curve of the indoor scene; smooth the boundary contour curve by Gaussian filtering to generate a smooth closed boundary curve.

[0026] S32: Uniformly sample boundary points from the boundary curve, and calculate the central axis of the interior contour skeleton of the indoor scene based on the sampled boundary points using the central axis transformation algorithm;

[0027] S33: Sample a preset number of points on the central axis at a preset step size, and use the coordinates of the sampled points as camera pose coordinates. The camera pose direction is the three-dimensional vector direction from the origin of the indoor scene to the sampled points. The camera pose coordinates and the camera pose direction vector form a set of camera poses.

[0028] S34: Perform multiple iterations to complete the initial 3D mesh. In each iteration, the mesh updated in the previous iteration is used as the current iteration mesh. Render the scene projection map and its mask for each camera pose in the camera pose set using isometric spherical projection. Calculate the completeness of each camera pose using the mask of the scene projection map and sort them in ascending order. Use the scene projection map corresponding to the camera pose with the highest completeness as the panoramic image to be completed. Project the panoramic image to be completed onto an icosahedral viewpoint using isometric spherical projection to generate 20 single-view images to be repaired and their masks. Then, combine the single-view images to be repaired, the masks, and the camera pose text... This description is input into the fine-tuned diffusion completion model to form a repaired single-view image, which is then back-projected to form a completed panoramic image, which is then projected onto a cube viewpoint to generate a new completed viewpoint image as a multi-view image. The multi-view image is combined with the camera pose text description to form 6 sets of multi-view data pairs, which are then added to the multi-view dataset generated in the previous iteration to form an updated multi-view dataset. The depth map corresponding to the completed panoramic image is obtained and a candidate mesh is generated. The conflict-free region is selected from the candidate mesh using a pre-constructed conflict mask, and the conflict-free region is merged into the current iteration mesh to complete one completion cycle.

[0029] As a preferred embodiment of the first aspect, in step S34, the specific process of generating the collision mask is as follows: In the i-th iteration, the panoramic image completed in the i-th iteration is processed by a pre-trained monocular depth estimator to generate a corresponding depth map, which is then used as the first reference depth map. The panoramic image completed in the (i-1)-th iteration is processed by a pre-trained monocular depth estimator to generate a corresponding depth map, which is then used as the second reference depth map. The process is then repeated for each iteration. The completed panoramic image and the first reference depth map are used to generate candidate meshes. The candidate meshes are then projected onto the camera pose used in the (i-1)th iteration to generate the first predicted depth map. The depth difference between the second reference depth map and the first predicted depth map is compared pixel by pixel. When the depth difference corresponding to a pixel coordinate is greater than the preset depth conflict threshold, the value of the conflict mask at that pixel coordinate is 1 to indicate a conflict; otherwise, it is 0 to indicate no conflict. After all pixels have been compared, the conflict-free region is formed from the coordinates of all conflict-free pixels, and a complete conflict mask is obtained.

[0030] Further, in step S34, the collision mask The calculation process is as follows:

[0031]

[0032] in, Indicates at pixel coordinates The collision mask value at the location; This indicates that the second reference depth map is at pixel coordinates. The depth value at that location; This indicates that the first predicted depth map is at pixel coordinates of The depth value at that location; Represents the L2 norm; This indicates the depth of the conflict threshold.

[0033] As a preferred embodiment of the first aspect, the specific process of initializing the Gaussian point parameters in step S4 is as follows: the three-dimensional mesh completed in step S3 is converted into a 3D Gaussian sputtering field representation; the center position coordinates of the 3D Gaussian ellipsoid are initialized from the coordinates of the midpoints of the completed three-dimensional mesh; the color attributes of the 3D Gaussian ellipsoid are initialized from the color information of the completed three-dimensional mesh; and parameters such as the covariance matrix and opacity of the 3D Gaussian ellipsoid are initialized according to preset rules.

[0034] As a preferred embodiment of the first aspect above, in step S4, the rendering error feedback loss is composed of the original loss function used during the training of the 3D Gaussian sputtering field and the depth similarity loss. The depth similarity loss measures the structural similarity between the single-view depth map rendered by the 3D Gaussian sputtering field and the real depth map in the multi-view dataset through the Pearson correlation coefficient.

[0035] Furthermore, in step S4, the rendering error feedback loss... The function form is as follows:

[0036]

[0037]

[0038]

[0039]

[0040] in, This represents the original loss function used during training of the 3D Gaussian sputtering field; Represents depth similarity loss; All represent hyperparameters; This represents a single-view image rendered by a 3D Gaussian sputtering field during the optimization process. This represents multiview images in a multiview dataset; express The corresponding single-view depth map; express The corresponding true depth map; Indicates SSIM loss; This indicates the calculation of Pearson correlation. This indicates the calculation of covariance; This indicates the calculation of variance; This represents the L1 norm.

[0041] As a preferred embodiment of the first aspect, in step S4, the process of optimizing the 3D Gaussian sputtering field using a Gaussian upsampling strategy is as follows: Within one optimization round, the K-nearest neighbor algorithm is used to calculate the scale properties of the 3D Gaussian ellipsoid. Starting from the origin Gaussian ellipsoid, each Gaussian in the 3D Gaussian ellipsoid is traversed, and the following process is performed on each Gaussian: a Gaussian in the 3D Gaussian ellipsoid is taken as the source Gaussian, the K nearest neighbors of the source Gaussian are marked as target Gaussians, the distance between a source Gaussian and each of its corresponding target Gaussians is calculated, and the average of the distance calculation results is taken as the proximity score of the source Gaussian. The connected graph formed by the source Gaussian and its corresponding target Gaussians is taken as... The nearest neighbor graph is used. Based on the nearest neighbor graph and the proximity score of each source Gaussian, if the proximity score of a source Gaussian exceeds a preset score threshold, the straight line connecting the source Gaussian and its corresponding target Gaussian is used as an edge. A new Gaussian is created at one-third and two-thirds of each edge. When the newly created Gaussian is close to the source Gaussian, its scale and opacity properties are consistent with the source Gaussian. When the newly created Gaussian is close to the target Gaussian, its scale and opacity properties are consistent with the target Gaussian, while other properties are initialized to zero. After processing all Gaussians, a 3D Gaussian sputtering field optimized in this optimization round is formed. Here, K represents the number of nearest neighbors selected.

[0042] Secondly, the present invention provides an adaptive 3D scene reconstruction system based on a single panoramic image, comprising:

[0043] The mesh generation module is used to acquire panoramic color images of indoor scenes and estimate their depth information, generate depth maps, perform edge estimation and back projection operations on the depth maps, and generate the initial 3D mesh of the indoor scene.

[0044] The data generation and model fine-tuning module is used to generate six original single-view images from a cubic perspective based on an initial 3D mesh through viewpoint transformation; then, the projection camera pose is transformed, and a projection image of each original single-view image after the camera pose transformation and its occlusion mask are generated through projection transformation; then, the projection image and its occlusion mask are back-projected to form training sample pairs and fine-tuning the diffusion completion model; each training sample pair contains four parts: the original single-view image, the projection image and the occlusion mask, and the camera pose text description.

[0045] The mesh completion module is used to extract the boundary contour of the initial 3D mesh, generate boundary contour curves, and smooth them using Gaussian filtering to generate smooth, closed boundary curves. Based on the boundary points uniformly sampled from the boundary curves, the centerline transformation algorithm is used to calculate the centerline of the internal contour skeleton of the indoor scene. Sampling is performed on the centerline at a preset step size, and the coordinates of the sampled points are used as camera pose coordinates. The vector direction from the origin of the indoor scene to the sampled points is used as the camera pose direction, forming a camera pose set. Based on the camera pose set, the initial 3D mesh is iteratively completed multiple times. The integrity of the initial 3D mesh is optimized through conflict detection and merging to form a completed 3D mesh and output a multi-viewpoint dataset.

[0046] The Gaussian field optimization module is used to convert the completed 3D mesh into a 3D Gaussian sputtering field and initialize the Gaussian point parameters. It optimizes the Gaussian point parameters based on a multi-view dataset. During the optimization process, the consistency of the 3D Gaussian sputtering field is adjusted by the rendering error feedback loss. When the number of optimization rounds exceeds the preset round threshold, a Gaussian upsampling strategy is used to split and refine the Gaussian points located at the edges, and finally outputs a complete 3D scene model.

[0047] Compared with the prior art, the present invention has the following advantages:

[0048] This invention combines deep learning, computer vision, and 3D reconstruction techniques to propose a novel technical approach: an adaptive 3D scene reconstruction method based on a single panoramic image. This method enhances completion capabilities by injecting prior scene information into the LoRA-fine-tuned diffusion completion model. The adaptive camera pose generation technology covers all key regions, ensuring accurate capture and generation of occluded areas, a unique innovation compared to using predefined camera poses or randomly sampling poses from the scene. During Gaussian field optimization, the Gaussian upsampling strategy improves geometric continuity and texture detail while maintaining overall structural consistency. These technological breakthroughs significantly differentiate this method from traditional approaches. Through adaptive structural completion and Gaussian refinement, it reconstructs high-quality 3D scenes from a single panoramic image, significantly outperforming existing methods in accuracy and completeness, making it particularly suitable for immersive scene navigation and indoor scene reconstruction. In terms of accuracy, completeness, and detail preservation, this invention significantly surpasses existing methods. Attached Figure Description

[0049] Figure 1 This is a flowchart of the method of the present invention;

[0050] Figure 2 This is a schematic diagram of scene mesh initialization in the method of the present invention;

[0051] Figure 3 This is a schematic diagram of the fine-tuning diffusion completion model of the method of the present invention;

[0052] Figure 4 This is a schematic diagram of scene mesh adaptive completion using the method of the present invention;

[0053] Figure 5 This is a schematic diagram of the 3D Gaussian sputtering field optimization method of the present invention;

[0054] Figure 6 This is a schematic diagram of the Gaussian upsampling strategy designed for this invention;

[0055] Figure 7 This invention provides a schematic diagram of a three-dimensional scene reconstruction result;

[0056] Figure 8 This is a schematic diagram of another three-dimensional scene reconstruction result provided by the present invention;

[0057] Figure 9 This is a system block diagram of the present invention. Detailed Implementation

[0058] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Many specific details are set forth in the following description to provide a thorough understanding of the present invention. However, the present invention can be practiced in many other ways different from those described herein, and those skilled in the art can make similar modifications without departing from the spirit of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. Technical features in the various embodiments of the present invention can be combined accordingly without mutual conflict.

[0059] In the description of this invention, it should be understood that the terms "first" and "second" are used only for descriptive purposes and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features.

[0060] To address the shortcomings of existing single-view scene reconstruction technologies, such as occlusion omissions, detail loss, and poor consistency, this invention provides an adaptive 3D scene reconstruction method based on a single panoramic image. It is implemented using LoRA model fine-tuning, Mesh rendering, 3D Gaussian sputtering, and Gaussian upsampling. It should be noted that 3D Gaussian sputtering and LoRA model fine-tuning are existing technologies. Gaussian sputtering is a core method for 3D scene rendering, mapping point cloud or volume data in 3D space to a Gaussian distribution set, achieving efficient and realistic real-time rendering. LoRA model fine-tuning injects existing scene geometry and texture information into a pre-trained diffusion completion model, extracting and optimizing 3D content. The specific principles of this technology are not elaborated here. Furthermore, pre-trained diffusion completion models already exist in existing technologies; this invention can also further fine-tune these pre-trained diffusion models to meet the generation tasks of this invention.

[0061] like Figure 1 As shown, in a preferred embodiment of the present invention, the adaptive 3D scene reconstruction method based on a single panoramic image includes the following steps S1 to S4. The specific implementation process of each step will be described in detail below.

[0062] S1: Acquire a panoramic color image of the indoor scene and estimate its depth information to generate a depth map. Perform edge estimation and back projection operations on the depth map to generate an initial 3D mesh of the indoor scene.

[0063] It should be noted that, as Figure 2 As shown, the specific process of step S1 in this invention is as follows: A two-dimensional panoramic color image (resolution W×H is 2048×1024 or similar, where W is the width of the panoramic color image and H is the height of the panoramic color image) is acquired. Its depth value is predicted using the open-source panoramic depth prediction model PanoFusionDistancePredictor, and a depth map is formed from the predicted depth values. Figure 2 The panoramic depth map is used to extract edge masks through Canny edge detection and dilation operations. Based on back projection, the color and depth information of the panoramic color image is mapped to the point and face information of the 3D mesh. The edge masks are used to segment the triangular mesh patches connecting different objects, generating the initial 3D mesh of the indoor scene at the origin. .

[0064] In this embodiment, S1 is the initialization of the scene mesh. In step S1, edge estimation is performed on the depth map to generate an edge mask, and triangular mesh patches between different objects are segmented to improve mesh smoothness. Back projection operation is used to enhance the color and depth information of the panoramic color image to the point and surface information of the 3D mesh. Here, a point refers to the vertex of the 3D mesh, and a surface refers to the patch of the 3D mesh. Finally, an initial 3D mesh is generated in the 3D coordinate system of the indoor scene with the origin coordinates (0, 0, 0). This mesh represents the scene texture and geometric information from the perspective of the origin of the indoor scene, serving as the initial representation of the indoor scene and providing texture and geometric priors for subsequent model fine-tuning.

[0065] S2: Based on the initial 3D mesh, generate 6 original single-view images from a cube perspective through viewpoint transformation; then transform the projection camera pose, and generate the projection image of each original single-view image after the camera pose transformation and its occlusion mask through projection transformation; then backproject the projection image and its occlusion mask to form training sample pairs and fine-tune the diffusion completion model; each training sample pair contains four parts: original single-view image, projection image and occlusion mask, and camera pose text description.

[0066] It should be noted that, in this invention, as Figure 3 As shown, the specific process of step S2 is as follows:

[0067] S21: In the initial 3D mesh scene, keep the camera pose coordinates at the origin of the indoor scene, adjust the camera pose orientation to the six faces of the cube, and generate 6 original single-view images of the indoor scene and their corresponding camera pose text descriptions.

[0068] It should be noted that in this embodiment S21, since the initial three-dimensional mesh is generated... The viewpoint is fixed at the origin of the indoor scene. Changing the camera pose keeps the camera pose coordinates at the origin constant, and the camera pose orientation is the direction of the cube. The six original single-view images of the indoor scene were obtained. and 6 camera pose text descriptions .

[0069] S22: Perform the following operations on each original single-view image: backproject the original single-view image into 3D space to generate a corresponding single-view mesh and use it as the first single-view mesh; perform up, down, left, and right deflection transformations on each camera pose according to a preset deflection angle, and generate multiple different rendering images through the first single-view mesh; then project the rendering images back into 3D space to generate corresponding single-view meshes and use them as the second single-view mesh; backproject the original viewpoint corresponding to the origin of the indoor scene using the second single-view mesh and calculate the mask to obtain the occlusion area mask. The occluded area mask is then added to the original single-view image to form a masked single-view image. As a projected image; ultimately composed of a single-view image. Projected images , occlusion area mask and camera pose text description The samples are combined into a set of training sample pairs to form a training dataset, and the LoRA model is used to fine-tune the diffusion completion model on the training dataset.

[0070] In this embodiment, random sampling is performed within the range of [0, 45°] to generate the aforementioned deflection angle. Four different rendering images are then generated for each original single-view image using a single-view mesh rendering process. An occlusion region mask is generated based on this rendering image, designed to simulate object occlusion in an indoor scene due to viewpoint changes. Following this process, each original single-view image corresponds to a camera pose text description, and each original single-view image corresponds to four masked single-view images. Each masked single-view image corresponds to a unique occlusion region mask. Finally, the original single-view images, the masked single-view images (projected images), the occlusion region masks, and the camera pose text descriptions are combined into 24 training sample pairs for fine-tuning the diffusion completion model.

[0071] In this embodiment, the diffusion completion model employs Stable Diffusion 2-Inpainting and is fine-tuned using a low-rank adapter (LoRA model) to learn the texture and geometric priors of the indoor scene, ensuring that the completed content remains consistent with the overall scene under camera pose and semantic cue constraints. Specifically, the fine-tuning targets include the self-attention and cross-attention layers of the diffusion completion model U-Net and the self-attention layer of the Text Encoder network. The training data used for fine-tuning is the training sample pairs generated in step S22. The fine-tuning process consists of 200 rounds, with the LoRA parameters set to: rank 16, scaling factor α 32, and a learning rate of [missing information - likely a specific value]. The objective function used for fine-tuning is the same as the loss function used during pre-training of the diffusion completion model, as follows:

[0072]

[0073] in, Indicates the input image; This represents the added Gaussian noise randomly sampled from a normal distribution; This represents the normal distribution. Represents the identity matrix; Indicates the time step of the diffusion process; A set representing time steps; It is a weighted function that depends on the time step; This represents the noise value predicted by the U-Net network in the diffusion completion model; It is the latent vector of the input image. The noise version, This indicates the masking conditions for the input image. Textual descriptions of camera pose conditions; This represents the square of the L2 norm.

[0074] S3: Extract the boundary contour of the initial 3D mesh, generate the boundary contour curve, and smooth it using Gaussian filtering to generate a smooth closed boundary curve; calculate the central axis of the interior contour skeleton of the indoor scene using the central axis transformation algorithm based on the boundary points uniformly sampled from the boundary curve; sample on the central axis at a preset step size, use the coordinates of the sampled points as the camera pose coordinates, and use the vector direction from the origin of the indoor scene to the sampled points as the camera pose direction to form a camera pose set; perform multiple iterations to complete the initial 3D mesh based on the camera pose set, optimize the integrity of the initial 3D mesh through conflict detection and merging, form the completed 3D mesh, and output a multi-viewpoint dataset.

[0075] It should be noted that, in this invention, as Figure 4 As shown, the specific process of step S3 is as follows:

[0076] S31: Based on the geometric structure information of the indoor scene in the initial 3D mesh, extract the origin boundary points of the indoor scene to generate the boundary contour curve of the indoor scene; smooth the boundary contour curve by Gaussian filtering to generate a smooth closed boundary curve.

[0077] In this embodiment S31, considering that the values ​​of the origin boundary points may change abruptly between adjacent vertices, Gaussian filtering interpolation is used to expand the boundary contour curve point set to generate a smooth and closed boundary curve to address this abrupt change phenomenon between adjacent vertices. .

[0078] S32: Uniformly sample boundary points from the boundary curve, and calculate the central axis of the interior contour skeleton of the indoor scene based on the sampled boundary points using the central axis transformation algorithm.

[0079] In this embodiment S32, the coordinates of the boundary points on the boundary curve are stored in a list. A boundary point is sampled from this list at regular intervals, and then the centerline can be calculated using the centerline transformation algorithm. This centerline describes the internal contour structure skeleton of the indoor scene. Furthermore, the centerline transformation algorithm is a relatively mature method in the prior art; therefore, its specific implementation process will not be described in detail in this embodiment.

[0080] S33: Sample a preset number of points evenly on the central axis with a preset step size, and use the coordinates of the sampled points as camera pose coordinates. The camera pose direction is the three-dimensional vector direction from the origin of the indoor scene to the sampled points. The camera pose coordinates and the camera pose direction vector form a set of camera poses.

[0081] In this embodiment S33, to construct a camera pose set to complete the initial 3D mesh, 15 points are uniformly sampled from the central axis at a preset step size, and the coordinates of these 15 points are used as camera pose coordinates. The camera pose direction is the 3D vector direction from the origin of the indoor scene to the sampling point. Then, the camera pose set is constructed from the camera pose coordinates and the camera pose direction vector. This ensures that the generated camera poses cover all key areas. The camera pose coordinates are a 3×3 matrix, the camera pose direction vector is a 3×1 vector, and the camera pose set is a 3×4 matrix.

[0082] S34: Perform multiple iterations to complete the initial 3D mesh. In each iteration, the mesh updated in the previous iteration is used as the current iteration mesh. The scene projection map and its mask under each camera pose in the camera pose set are rendered by isometric spherical projection. The completeness of each camera pose is calculated from the mask of the scene projection map and sorted in ascending order. The scene projection map corresponding to the camera pose with the highest completeness is used as the panoramic image to be completed. The panoramic image to be completed is then projected onto an icosahedral viewpoint using an equidistant spherical projection, generating 20 single-view images to be repaired along with their masks. These single-view images, masks, and camera pose descriptions are then input into a finely tuned diffusion completion model to create the repaired single-view images. These repaired single-view images are then back-projected to form the completed panoramic image. The image is then projected onto a cube viewpoint to generate a new, completed viewpoint image, which is then used as a multi-viewpoint image. This multi-viewpoint image is combined with the camera pose text description to form six sets of multi-viewpoint data pairs, which are then added to the multi-viewpoint dataset generated in the previous iteration, forming an updated multi-viewpoint dataset. The depth map corresponding to the completed panoramic image is obtained, and a candidate mesh is generated. The conflict-free regions are then selected from the candidate mesh using a pre-constructed conflict mask, and these conflict-free regions are merged into the current iteration mesh, completing one completion cycle.

[0083] In this embodiment S34, the initial three-dimensional mesh is... The initial 3D mesh, used as the mesh in the first iteration of the completion process, was completed iteratively 15 times. The scene projection map and mask of the initial 3D mesh were rendered using isometric spherical projection, and the scene completeness corresponding to each camera pose was calculated and sorted. Here, the completeness of each camera pose was calculated using the mask of the scene projection map; a mask value of 1 indicates occlusion, and 0 indicates no occlusion. Therefore, completeness = 1 - number of occluded pixels / total number of pixels. Then, the camera pose with the highest completeness was selected, and a panoramic image was rendered using isometric spherical projection. The images are then projected onto an icosahedral viewpoint, generating 20 single-view images to be repaired, along with their corresponding masks and camera pose text descriptions. The generated single-view images are then repaired using the diffusion completion model fine-tuned in step S2, completing unobserved areas and ensuring geometric and semantic consistency. Finally, the repaired single-view images are back-projected to generate a completed panoramic image. The completed panoramic image is projected onto a hexahedral viewpoint to generate a new completed viewpoint image. Each new completed viewpoint image is combined with its corresponding mask and camera pose text description to form a multi-view data pair. After projection onto the hexahedral viewpoint, six new completed viewpoint images are generated. Accordingly, each iteration generates six sets of multi-view data pairs. Before iterative completion, an empty multi-view dataset is constructed. The multi-view data pairs generated in each iteration are added to the multi-view dataset to update it. After iterative completion, a complete multi-view dataset is obtained, containing 90 sets of 15×6 multi-view data pairs, which are used for optimizing the Gaussian field in step S4. The conflict mask between the newly added region and the original region is calculated, non-conflicting parts are filtered out, and the effective completion content is merged into the mesh updated in the previous iteration. Through multiple iterations, the scene integrity, accuracy, and geometric precision are improved.

[0084] In step S34 of this invention, the specific process of generating the collision mask is as follows: in the first step... In the iteration, the th The panoramic image, after being completed in the second iteration, is processed by a pre-trained monocular depth estimator to generate a corresponding depth map, which is then used as the first reference depth map. The panoramic image after the second iteration is completed. The pre-trained monocular depth estimator generates a corresponding depth map, which is then used as a second reference depth map. , by the The completed panoramic image and the first reference depth map are used to generate candidate meshes in the second iteration. The next iteration uses projection based on the camera pose to generate the first predicted depth map. The depth difference between the second reference depth map and the first predicted depth map is compared pixel by pixel. When the depth difference corresponding to a pixel coordinate is greater than the preset depth conflict threshold, the value of the conflict mask at that pixel coordinate is 1 to indicate that there is a conflict, otherwise it is 0 to indicate that there is no conflict. After all pixels have been compared, the conflict-free region is formed by the coordinates of all conflict-free pixels, and the complete conflict mask is obtained.

[0085] In this embodiment, the collision mask The calculation process is as follows:

[0086]

[0087] in, Indicates at pixel coordinates The collision mask value at the location; This indicates that the second reference depth map is at pixel coordinates. The depth value at that location; This indicates that the first predicted depth map is at pixel coordinates of The depth value at that location; Represents the L2 norm; This indicates the depth of the conflict threshold.

[0088] S4: Convert the completed 3D mesh into a 3D Gaussian Splatting (3DGS) field and initialize the Gaussian point parameters. Optimize the Gaussian point parameters based on a multi-view dataset. During the optimization process, adjust the consistency of the 3D Gaussian Splatting field through rendering error feedback loss. When the number of optimization rounds exceeds the preset round threshold, use a Gaussian upsampling strategy to split and refine the Gaussian points located at the edges. Finally, output a geometrically continuous and texture-rich complete 3D scene model.

[0089] It should be noted that, in this invention, the specific process of initializing the Gaussian point parameters in step S4 is as follows: the three-dimensional mesh completed in step S3 is converted into a 3D Gaussian sputtering field (3D Gaussian Splatting, 3DGS) representation; the center position coordinates of the 3D Gaussian ellipsoid are initialized by the coordinates of the midpoints of the completed three-dimensional mesh; the color attributes of the 3D Gaussian ellipsoid are initialized by the color information of the completed three-dimensional mesh; and the parameters such as the covariance matrix and opacity of the 3D Gaussian ellipsoid are initialized according to preset rules.

[0090] It should be noted that in step S4 of the present invention, the rendering error feedback loss is composed of the original loss function used during the training of the 3D Gaussian sputtering field and the depth similarity loss. The depth similarity loss is measured by the Pearson correlation coefficient between the single-view depth map rendered by the 3D Gaussian sputtering field and the real depth map in the multi-view dataset.

[0091] In step S4 of this embodiment, as Figure 5 As shown, the coordinates and colors of the points in the completed 3D mesh are extracted to initialize the center coordinates and color of the 3D Gaussian ellipsoid. Other properties of the 3D Gaussian ellipsoid, such as covariance and opacity, are also initialized. During the optimization of the 3D Gaussian sputtering field, the multi-view image generated in step S3 is used as the observation basis. To inject more regularization into the optimization, this invention proposes an additional depth adjustment method and applies pixel-level geometric correspondences during training. Furthermore, this embodiment uses a pre-trained monocular depth estimator, DensePrediction Transformer, to predict multi-view images. Depth map To address the scale ambiguity issue in depth estimation, this invention employs a depth similarity loss based on Pearson correlation, which forces distribution consistency between the estimated depth map and the rendered depth map. This loss, combined with the original loss function used during 3D Gaussian sputtering field training, forms a rendering error feedback loss. The 3DGS parameters (including the position, covariance, color, and opacity of the 3D Gaussian points) are iteratively optimized through backpropagation of the loss gradient. (Rendering error feedback loss) The function form is as follows:

[0092]

[0093]

[0094]

[0095]

[0096] in, This represents the original loss function used during training of the 3D Gaussian sputtering field; Represents depth similarity loss; All of these represent hyperparameters. In this embodiment, Set to 0.8, Set to 0.2, Set to 0.05; This represents a single-view image rendered by a 3D Gaussian sputtering field during the optimization process. This represents multiview images in a multiview dataset; express The depth map, i.e., the single-view depth map; express The depth map, i.e., the true depth map; This represents the SSIM loss (Structural Similarity Index Measure Loss, SSIM Loss). This indicates the calculation of the Pearson Correlation Coefficient. This indicates the calculation of covariance; This indicates the calculation of variance; This represents the L1 norm.

[0097] It should be noted that in step S4 of this invention, the specific process of optimizing the 3D Gaussian sputtering field using the Gaussian upsampling strategy is as follows: Within one optimization round, the K-Nearest Neighbors (KNN) algorithm is used to calculate the scale properties of the 3D Gaussian ellipsoid. Starting from the origin Gaussian ellipsoid, each Gaussian in the 3D Gaussian ellipsoid is traversed, and the following process is performed on each Gaussian: a Gaussian in the 3D Gaussian ellipsoid is taken as the source Gaussian, the K nearest neighbors of the source Gaussian are marked as target Gaussians, the distance between a source Gaussian and each of its corresponding target Gaussians is calculated, and the average of the distance calculation results is taken as the proximity score of the source Gaussian. The connected graph formed by the source Gaussian and its corresponding target Gaussians is taken as the neighborhood graph. Based on the neighborhood graph and the proximity score of each source Gaussian, If the proximity score of a source Gaussian exceeds a preset score threshold, the straight line connecting the source Gaussian and its corresponding target Gaussian is used as an edge. A new Gaussian is created at each of the one-third and two-thirds points of this edge. When the newly created Gaussian is close to the source Gaussian, its scale and opacity properties are consistent with the source Gaussian; when it is close to the target Gaussian, its scale and opacity properties are consistent with the target Gaussian. Meanwhile, other properties (such as rotation and SH coefficients) are initialized to zero. After processing all Gaussians, the optimized 3D Gaussian sputtering field for this optimization round is formed. Here, K represents the number of nearest neighbors selected. In this embodiment, K=3.

[0098] In step S4 of this embodiment, after scene completion and preliminary optimization in S3, the sparse distribution of Gaussian ellipsoids still exists in the edge regions of objects within the scene. Therefore, for edge regions with significant geometric errors, when the number of optimization rounds exceeds a preset round threshold of 2000, a Gaussian upsampling strategy is adopted to split and refine each edge Gaussian point, generating dense sub-Gaussian clusters to improve the geometric accuracy of edge details, such as... Figure 6 As shown, this Gaussian upsampling strategy distributes new Gaussians around the edge regions of the scene, systematically filling in the observation gaps during the optimization process.

[0099] The adaptive 3D scene reconstruction method based on a single panoramic image described in the above embodiments will be applied to specific text prompts for comparative experiments. The specific steps are as described in S1~S4, and will not be repeated here. The main focus is on demonstrating the specific parameters and technical effects.

[0100] Example

[0101] To verify the effectiveness of this invention, a comprehensive user study was conducted on eight scenes (room_0, room_1, room_2, office_0, office_1, office_2, office_3, office_4) in the Replica dataset. The method of this invention was compared with the two best-performing models currently available: PERF (using Neural Radiation Field (NeRF) as a 3D representation) and Pano2Room (using 3D GS as a 3D representation). The input conditions, training epochs, and other experimental conditions were kept consistent across all three methods. Forty-five new perspective images and four predefined camera trajectory videos were rendered from the final reconstruction results for comparison. Users were asked to vote based on visual perception in four aspects: texture, geometry, scene integrity, and consistency of video frame rate (rendering speed) to objectively compare the reconstruction performance of different models on the panoramic images of the input scene. The comparison results are shown in Table 1. This embodiment also compared the differences in quantitative metrics (PSNR, SSIM, LPIPS, NIQE) and time costs between the new perspective images generated by the three methods and the original dataset. The comparison results are shown in Table 2. It is evident that the present invention is significantly superior to existing methods in terms of accuracy and completeness. Furthermore, to enable those skilled in the art to intuitively perceive the reconstruction effect of the present invention, this embodiment also provides schematic diagrams of two 3D scene reconstruction results using the method of the present invention, as follows: Figure 7 and Figure 8 As shown.

[0102] Table 1. Evaluation Table for Texture, Geometry, Scene Integrity, and Video Frame Rate

[0103]

[0104] Table 2. Generation Efficiency Evaluation Table

[0105]

[0106] It should also be noted that the adaptive 3D scene reconstruction method based on a single panoramic image in the above embodiments can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides an adaptive 3D scene reconstruction system based on a single panoramic image, corresponding to the adaptive 3D scene reconstruction method based on a single panoramic image provided in the above embodiments, such as... Figure 9 As shown, it includes:

[0107] The mesh generation module is used to acquire panoramic color images of indoor scenes and estimate their depth information, generate depth maps, perform edge estimation and back projection operations on the depth maps, and generate the initial 3D mesh of the indoor scene.

[0108] The data generation and model fine-tuning module is used to generate six original single-view images from a cubic perspective based on an initial 3D mesh through viewpoint transformation; then, the projection camera pose is transformed, and a projection image of each original single-view image after the camera pose transformation and its occlusion mask are generated through projection transformation; then, the projection image and its occlusion mask are back-projected to form training sample pairs and fine-tuning the diffusion completion model; each training sample pair contains four parts: the original single-view image, the projection image and the occlusion mask, and the camera pose text description.

[0109] The mesh completion module is used to extract the boundary contour of the initial 3D mesh, generate boundary contour curves, and smooth them using Gaussian filtering to generate smooth, closed boundary curves. Based on the boundary points uniformly sampled from the boundary curves, the centerline transformation algorithm is used to calculate the centerline of the internal contour skeleton of the indoor scene. Sampling is performed on the centerline at a preset step size, and the coordinates of the sampled points are used as camera pose coordinates. The vector direction from the origin of the indoor scene to the sampled points is used as the camera pose direction, forming a camera pose set. Based on the camera pose set, the initial 3D mesh is iteratively completed multiple times. The integrity of the initial 3D mesh is optimized through conflict detection and merging to form a completed 3D mesh and output a multi-viewpoint dataset.

[0110] The Gaussian field optimization module is used to convert the completed 3D mesh into a 3D Gaussian sputtering field and initialize the Gaussian point parameters. It optimizes the Gaussian point parameters based on a multi-view dataset. During the optimization process, the consistency of the 3D Gaussian sputtering field is adjusted by the rendering error feedback loss. When the number of optimization rounds exceeds the preset round threshold, a Gaussian upsampling strategy is used to split and refine the Gaussian points located at the edges, and finally outputs a complete 3D scene model.

[0111] It should also be noted that those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. In the embodiments provided in this application, the division of steps or modules in the system and method is merely a logical functional division, and there may be other division methods in actual implementation. For example, multiple modules or steps may be combined or integrated together, and a module or step may also be split.

[0112] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the invention. Therefore, all technical solutions obtained through equivalent substitution or transformation fall within the protection scope of the present invention.

Claims

1. An adaptive 3D scene reconstruction method based on a single panoramic image, characterized in that, Includes the following steps: S1: Acquire a panoramic color image of the indoor scene and estimate its depth information to generate a depth map. Perform edge estimation and back projection operations on the depth map to generate an initial 3D mesh of the indoor scene. S2: Based on the initial 3D mesh, generate 6 original single-view images from a cube perspective through viewpoint transformation; Then, the projection camera pose is transformed, and the projection image of each original single-view image after the camera pose transformation and its occlusion area mask are generated through projection transformation. Then, the projection image and its occlusion area mask are back-projected to form training sample pairs and fine-tuning the diffusion completion model. Each training sample pair contains four parts: original single-view image, projection image and occlusion area mask, and camera pose text description. S3: Extract the boundary contour of the initial 3D mesh, generate the boundary contour curve, and smooth it using Gaussian filtering to generate a smooth closed boundary curve; calculate the central axis of the interior contour skeleton of the indoor scene using the central axis transformation algorithm based on the boundary points uniformly sampled from the boundary curve; sample along the central axis at a preset step size, use the coordinates of the sampled points as the camera pose coordinates, and use the vector direction from the origin of the indoor scene to the sampled points as the camera pose direction to form a camera pose set; perform multiple iterations to complete the initial 3D mesh based on the camera pose set, optimize the integrity of the initial 3D mesh through conflict detection and merging, form the completed 3D mesh, and output a multi-viewpoint dataset; S4: Convert the completed 3D mesh into a 3D Gaussian sputtering field and initialize the Gaussian point parameters. Optimize the Gaussian point parameters based on the multi-view dataset. During the optimization process, adjust the consistency of the 3D Gaussian sputtering field through rendering error feedback loss. When the number of optimization rounds exceeds the preset round threshold, use the Gaussian upsampling strategy to split and refine the Gaussian points located at the edges, and finally output a complete 3D scene model.

2. The adaptive 3D scene reconstruction method based on a single panoramic image as described in claim 1, characterized in that, The specific process of step S1 is as follows: acquire a two-dimensional panoramic color image, predict its depth value through a panoramic depth prediction model, form a depth map from the predicted depth value, extract the edge mask of the depth map through Canny edge detection and dilation operation, map the color and depth information of the panoramic color image to the point and surface information of the three-dimensional mesh based on back projection operation, segment the triangular mesh facets connecting different objects using the edge mask, and generate the initial three-dimensional mesh of the indoor scene at the origin of the indoor scene.

3. The adaptive 3D scene reconstruction method based on a single panoramic image as described in claim 1, characterized in that, The specific process of step S2 is as follows: S21: In the initial 3D mesh scene, keep the camera pose coordinates at the origin of the indoor scene, adjust the camera pose orientation to the six faces of the cube, and generate 6 original single-view images of the indoor scene and their corresponding camera pose text descriptions. S22: Perform the following operations on each original single-view image: backproject the original single-view image to 3D space to generate a corresponding single-view mesh and use it as the first single-view mesh; perform up, down, left, and right deflection transformations on each camera pose according to a preset deflection angle; and generate multiple different rendering images through the first single-view mesh. Then, project the rendering images back to 3D space to generate corresponding single-view meshes and use them as the second single-view mesh. Backproject the original viewpoint corresponding to the origin of the indoor scene using the second single-view mesh and calculate the mask to obtain the occlusion area mask. Add the occlusion area mask to the original single-view image to form a masked single-view image as the projection image. Finally, combine an original single-view image, a projection image, an occlusion area mask, and a camera pose text description into a set of training sample pairs to form a training dataset. On this training dataset, use the LoRA model to fine-tune the diffusion completion model.

4. The adaptive 3D scene reconstruction method based on a single panoramic image as described in claim 3, characterized in that, In step S22, the fine-tuning objects include the self-attention layer and cross-attention layer of the diffusion completion model U-Net, and the self-attention layer of the Text Encoder network.

5. The adaptive 3D scene reconstruction method based on a single panoramic image as described in claim 1, characterized in that, The specific process of step S3 is as follows: S31: Based on the geometric structure information of the indoor scene in the initial 3D mesh, extract the origin boundary points of the indoor scene to generate the boundary contour curve of the indoor scene; smooth the boundary contour curve by Gaussian filtering to generate a smooth closed boundary curve. S32: Uniformly sample boundary points from the boundary curve, and calculate the central axis of the interior contour skeleton of the indoor scene based on the sampled boundary points using the central axis transformation algorithm; S33: Sample a preset number of points on the central axis at a preset step size, and use the coordinates of the sampled points as camera pose coordinates. The camera pose direction is the three-dimensional vector direction from the origin of the indoor scene to the sampled points. The camera pose coordinates and the camera pose direction vector form a set of camera poses. S34: Perform multiple iterations to complete the initial 3D mesh. In each iteration, the mesh updated in the previous iteration is used as the current iteration mesh. Render the scene projection map and its mask for each camera pose in the camera pose set using isometric spherical projection. Calculate the completeness of each camera pose using the mask of the scene projection map and sort them in ascending order. Use the scene projection map corresponding to the camera pose with the highest completeness as the panoramic image to be completed. Project the panoramic image to be completed onto an icosahedral viewpoint using isometric spherical projection to generate 20 single-view images to be repaired and their masks. Then, combine the single-view images to be repaired, the masks, and the camera pose text... This description is input into the fine-tuned diffusion completion model to form a repaired single-view image, which is then back-projected to form a completed panoramic image, which is then projected onto a cube viewpoint to generate a new completed viewpoint image as a multi-view image. The multi-view image is combined with the camera pose text description to form 6 sets of multi-view data pairs, which are then added to the multi-view dataset generated in the previous iteration to form an updated multi-view dataset. The depth map corresponding to the completed panoramic image is obtained and a candidate mesh is generated. The conflict-free region is selected from the candidate mesh using a pre-constructed conflict mask, and the conflict-free region is merged into the current iteration mesh to complete one completion cycle.

6. The adaptive 3D scene reconstruction method based on a single panoramic image as described in claim 5, characterized in that, In step S34, the specific process of generating the collision mask is as follows: In the i-th iteration, the panoramic image completed in the i-th iteration is processed by a pre-trained monocular depth estimator to generate a corresponding depth map, which is then used as the first reference depth map. The panoramic image completed in the (i-1)-th iteration is processed by a pre-trained monocular depth estimator to generate a corresponding depth map, which is then used as the second reference depth map. The completed panoramic image and the first reference depth map are used to generate candidate meshes. The candidate meshes are then projected onto the camera pose used in the (i-1)th iteration to generate the first predicted depth map. The depth difference between the second reference depth map and the first predicted depth map is compared pixel by pixel. When the depth difference corresponding to a pixel coordinate is greater than the preset depth conflict threshold, the value of the conflict mask at that pixel coordinate is 1 to indicate a conflict; otherwise, it is 0 to indicate no conflict. After all pixels have been compared, the conflict-free region is formed from the coordinates of all conflict-free pixels, and a complete conflict mask is obtained.

7. The adaptive 3D scene reconstruction method based on a single panoramic image as described in claim 1, characterized in that, In step S4, the specific process of initializing the Gaussian point parameters is as follows: the three-dimensional mesh completed in step S3 is converted into a 3D Gaussian sputtering field representation; the center position coordinates of the 3D Gaussian ellipsoid are initialized from the coordinates of the midpoints of the completed three-dimensional mesh; the color attributes of the 3D Gaussian ellipsoid are initialized from the color information of the completed three-dimensional mesh; and parameters such as the covariance matrix and opacity of the 3D Gaussian ellipsoid are initialized according to preset rules.

8. The adaptive 3D scene reconstruction method based on a single panoramic image as described in claim 1, characterized in that, In step S4, the rendering error feedback loss is composed of the original loss function used during the training of the 3D Gaussian sputtering field and the depth similarity loss. The depth similarity loss is measured by the Pearson correlation coefficient between the single-view depth map rendered by the 3D Gaussian sputtering field and the real depth map in the multi-view dataset.

9. The adaptive 3D scene reconstruction method based on a single panoramic image as described in claim 1, characterized in that, In step S4, the specific process of optimizing the 3D Gaussian sputtering field using the Gaussian upsampling strategy is as follows: Within one optimization round, the K-nearest neighbor algorithm is used to calculate the scale properties of the 3D Gaussian ellipsoid. Starting from the origin Gaussian ellipsoid, each Gaussian in the 3D Gaussian ellipsoid is traversed, and the following process is performed on each Gaussian: a Gaussian in the 3D Gaussian ellipsoid is taken as the source Gaussian, the K nearest neighbors of the source Gaussian are marked as target Gaussians, the distance between a source Gaussian and each of its corresponding target Gaussians is calculated, and the average of the distance calculation results is taken as the proximity score of the source Gaussian. The connected graph formed by the source Gaussian and its corresponding target Gaussians is taken as the neighborhood graph. Based on the proximity graph and the proximity score of each source Gaussian, if the proximity score of a source Gaussian exceeds a preset score threshold, the straight line connecting the source Gaussian and its corresponding target Gaussian is taken as an edge, and a new Gaussian is created at one-third and two-thirds of each edge. When the newly created Gaussian is close to the source Gaussian, its scale and opacity properties are consistent with the source Gaussian; when the newly created Gaussian is close to the target Gaussian, its scale and opacity properties are consistent with the target Gaussian, while other properties are initialized to zero. After processing all Gaussians, a 3D Gaussian sputtering field optimized in this optimization round is formed; where K represents the number of nearest neighbors selected.

10. An adaptive 3D scene reconstruction system based on a single panoramic image, characterized in that, include: The mesh generation module is used to acquire panoramic color images of indoor scenes and estimate their depth information, generate depth maps, perform edge estimation and back projection operations on the depth maps, and generate the initial 3D mesh of the indoor scene. The data generation and model fine-tuning module is used to generate six original single-view images from a cubic perspective based on an initial 3D mesh through perspective transformation. Then, the projection camera pose is transformed, and the projection image of each original single-view image after the camera pose transformation and its occlusion area mask are generated through projection transformation. Then, the projection image and its occlusion area mask are back-projected to form training sample pairs and fine-tuning the diffusion completion model. Each training sample pair contains four parts: original single-view image, projection image and occlusion area mask, and camera pose text description. The mesh completion module is used to extract the boundary contour of the initial 3D mesh, generate boundary contour curves, and smooth them using Gaussian filtering to generate smooth, closed boundary curves. Based on the boundary points uniformly sampled from the boundary curves, the centerline transformation algorithm is used to calculate the centerline of the internal contour skeleton of the indoor scene. Sampling is performed on the centerline at a preset step size, and the coordinates of the sampled points are used as camera pose coordinates. The vector direction from the origin of the indoor scene to the sampled points is used as the camera pose direction, forming a camera pose set. Based on the camera pose set, the initial 3D mesh is iteratively completed multiple times. The integrity of the initial 3D mesh is optimized through conflict detection and merging to form a completed 3D mesh and output a multi-viewpoint dataset. The Gaussian field optimization module is used to convert the completed 3D mesh into a 3D Gaussian sputtering field and initialize the Gaussian point parameters. It optimizes the Gaussian point parameters based on a multi-view dataset. During the optimization process, the consistency of the 3D Gaussian sputtering field is adjusted by the rendering error feedback loss. When the number of optimization rounds exceeds the preset round threshold, a Gaussian upsampling strategy is used to split and refine the Gaussian points located at the edges, and finally outputs a complete 3D scene model.

Images