Self-adaptive three-dimensional scene reconstruction method and system based on single panorama

Abstract

The invention discloses a self-adaptive three-dimensional scene reconstruction method and system based on a single panorama, and belongs to the field of computer vision processing. The method comprises the following steps: firstly, generating a depth map through an indoor scene panorama and constructing an initial three-dimensional grid; then generating a multi-view image and a shielding mask thereof based on view conversion, forming a training sample pair for fine tuning of the diffusion completion model, and injecting scene prior information for the model; then extracting a grid boundary contour and calculating a central axis, and adaptively constructing a camera pose set; after the integrity of the grid is optimized through iteration completion, the grid is converted into a 3D Gaussian sputtering field, and Gaussian point parameters are optimized through multi-view data; in the optimization process, an up-sampling strategy of rendering error feedback and edge Gaussian points is adopted, and finally a scene model with complete geometric textures is output. According to the method, a high-quality three-dimensional scene is reconstructed from a single panorama through a structure self-adaptive completion and Gaussian refining technology, and the method is particularly suitable for immersive roaming reconstruction of indoor scenes.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision processing, and in particular relates to a method and system for adaptively reconstructing a three-dimensional scene based on a single panoramic image. Background Art

[0002] In the digital economy era, the demand for digital reproduction of the physical world is growing, especially in the field of three-dimensional reconstruction of indoor scenes, where its application has moved from cutting-edge exploration to large-scale commercialization. Three-dimensional reconstruction from a single 360-degree panorama has become a key technical approach to addressing this demand due to its convenient data acquisition and low cost. However, a single panorama is essentially a two-dimensional projection, and its inherent depth ambiguity and severe occlusion problems make it extremely challenging to directly restore the complete three-dimensional space from it. How to not only reconstruct the visible part, but also reasonably infer and complete the occluded areas is a core problem in this field.

[0003] To address this challenge, existing technologies have evolved from implicit to explicit representation. Neural Radiance Fields (NeRF), as an implicit representation method, uses a deep network to learn a continuous volumetric representation of the scene, enabling the synthesis of photo-realistic images from new perspectives. However, it relies on a volume rendering process that densely samples along each ray and performs network inference, resulting in extremely high computational costs for training and rendering, making real-time interaction difficult. To address the rendering efficiency bottleneck, 3D Gaussian Splatting (3DGS) was proposed as an explicit scene representation method. 3DGS represents the scene as a collection of millions of three-dimensional Gaussian ellipsoids, each defined by position, covariance, color, and opacity. During rendering, these three-dimensional Gaussians are directly projected onto a two-dimensional image plane and rasterized using an efficient alpha blending algorithm.

[0004] However, neither NeRF nor 3DGS, by themselves, can address the fundamental issue of "missing information" caused by single-view input. Relying solely on visible information, the reconstructed model will be an incomplete "shell." To address this, researchers have introduced generative techniques, such as diffusion models, to perform image inpainting on unobserved areas. By setting a new virtual viewpoint in the scene, an image containing the missing regions is rendered, which is then filled in using the diffusion model. Finally, the completed information is used to optimize the 3D representation.

[0005] Although combining 3DGS with generative models is currently the mainstream approach, existing technologies still suffer from the following three core flaws in their implementation, resulting in deficiencies in the completeness, consistency, and precision of reconstruction results:

[0006] First, the blindness of viewpoint sampling leads to incomplete scene completion. Existing methods typically adopt a naive strategy of uniform or random sampling within the scene when selecting new viewpoints to complete the scene. This strategy ignores the scene's inherent geometric structure. When faced with scenes with complex layouts (such as L-shaped rooms or large occluders), this strategy cannot guarantee the effective exploration of all key occluded areas. The virtual camera may never be placed in a position where it can observe behind structural occlusions. This directly results in the final reconstructed model still having large structural holes, a serious lack of completeness, and a waste of computational resources.

[0007] Second, general generative models lead to inconsistent completed content. Existing methods typically directly use diffusion models pre-trained on general large-scale datasets for content restoration. Such models lack prior knowledge of the style of the current specific scene (such as unique wallpaper textures, furniture materials, and lighting conditions). Therefore, although the generated completed content is realistic in some areas, it often produces obvious visual inconsistencies with the actual observed parts globally. For example, it generates textures that do not match the surroundings or shadows that contradict global lighting, forming "hallucination" artifacts that undermine the overall realism of the scene. Existing technologies generally lack an effective scene prior injection mechanism, such as using techniques such as low-rank adaptation (LoRA) to quickly fine-tune the model to ensure the consistency of the generated content.

[0008] Third: The 3D representation optimization process leads to loss of geometric details. When using views containing generated content to optimize the final 3DGS representation, the standard optimization process does not perform special treatment on the boundary area between "real data" and "generated data." Standard adaptive densification strategies (such as splitting or cloning Gaussians) are general mechanisms that may not be able to invest enough new Gaussian points in these critical stitching areas to accurately depict details. This leads to problems such as blurred object edges, geometric breaks, or texture distortion at the boundary, causing the clear geometric details in the original scene to lose accuracy after reconstruction.

[0009] Therefore, there is an urgent need for a method that can generate a complete, consistent and high-fidelity 3D model from a single panoramic image. Summary of the Invention

[0010] The present invention aims to address the following technical challenges in reconstructing a 3D scene from a single panoramic image: 1) how to adaptively plan viewpoints based on the scene structure to ensure the completeness of the scene completion; 2) how to enable the generative model to perceive prior information about the specific scene to ensure the consistency of the completed content; and 3) how to optimize the updating process of the 3D representation to preserve and enhance the scene's geometric details. This invention also provides a method and system for adaptive 3D scene reconstruction from a single panoramic image. The core of the method is to use 3D mesh adaptive completion and 3D Gaussian upsampling to optimize the generation of a geometrically and texture-consistent static 3D Gaussian splatter field. This method reconstructs a 3D indoor scene with fine texture, precise geometry, and coherence, enabling multi-view indoor roaming and novel viewpoint generation.

[0011] The core technical concept of this invention includes: first, extracting structural information such as scene boundaries and central axis based on the input indoor scene panorama to guide the adaptive synthesis of camera viewpoints; second, using the diffusion completion model to perform semantically guided image repair of occluded areas under the synthesized viewpoint, combined with camera pose information and semantic cues; finally, converting the completed 3D mesh into a 3D Gaussian sputtering representation, and using the proposed Gaussian upsampling strategy to perform subdivision optimization in the edge areas to improve the model's geometric and texture accuracy. Through pixel-level alignment-guided 3D scene generation, this invention not only retains the fine content of the original panorama but also automatically completes unobserved areas, thereby achieving high-quality free-viewpoint 3D reconstruction.

[0012] In order to achieve the above-mentioned object of the invention, the present invention specifically adopts the following technical solutions:

[0013] In a first aspect, the present invention provides a method for adaptively reconstructing a three-dimensional scene based on a single panoramic image, comprising the following steps:

[0014] S1: Acquire a panoramic color image of the indoor scene and estimate its depth information to generate a depth map. Perform edge estimation and back-projection operations on the depth map to generate an initial 3D mesh of the indoor scene.

[0015] S2: Based on the initial 3D mesh, six original monoscopic images are generated from the perspective of the cube through perspective transformation. The projection camera pose is then transformed, and the projection image and occlusion area mask of each original monoscopic image after the camera pose transformation are generated through projection transformation. The projection image and its occlusion area mask are then back-projected to form training sample pairs and fine-tune the diffusion completion model. Each training sample pair consists of four parts: the original monoscopic image, the projection image and occlusion area mask, and a text description of the camera pose.

[0016] S3: Extract the boundary contour of the initial 3D mesh, generate a boundary contour curve, and smooth it through Gaussian filtering to generate a smooth and closed boundary curve; use the central axis transformation algorithm to calculate the central axis of the internal contour skeleton of the indoor scene based on the boundary points uniformly sampled from the boundary curve; sample on the central axis according to the preset step size, use the coordinates of the sampling points as the camera pose coordinates, and use the vector direction from the origin of the indoor scene to the sampling points as the camera pose direction to form a camera pose set; based on the camera pose set, perform multiple iterative completions on the initial 3D mesh, optimize the integrity of the initial 3D mesh through conflict detection and merging, form a completed 3D mesh, and output a multi-viewpoint dataset;

[0017] S4: Convert the completed 3D mesh into a 3D Gaussian sputtering field and initialize the Gaussian point parameters. Optimize the Gaussian point parameters based on a multi-viewpoint dataset. During the optimization process, adjust the consistency of the 3D Gaussian sputtering field through the rendering error feedback loss. When the number of optimization rounds is greater than the preset round threshold, a Gaussian upsampling strategy is used to split and refine the Gaussian points at the edge, and finally output a complete 3D scene model.

[0018] Based on the above solution, each step can be implemented in the following preferred specific manner.

[0019] As a preferred embodiment of the above-mentioned first aspect, the specific process of step S1 is as follows: obtain a two-dimensional panoramic color image, predict its depth value through a panoramic depth prediction model, form a depth map by the predicted depth value, extract the edge mask of the depth map through Canny edge detection and dilation operation, map the color and depth information of the panoramic color image to the point and surface information of the three-dimensional grid based on the back projection operation, use the edge mask to segment the triangular mesh patches connecting different objects, and generate an initial three-dimensional grid of the indoor scene at the origin of the indoor scene.

[0020] As a preferred embodiment of the first aspect, the specific process of step S2 is as follows:

[0021] S21: In the initial 3D mesh scene, keep the camera pose coordinates at the origin of the indoor scene, adjust the camera pose orientation to the six faces of the cube, and generate 6 original single-view images of the indoor scene and their corresponding camera pose text descriptions;

[0022] S22: Perform the following operations on each original monoscopic image: back-project the original monoscopic image into three-dimensional space to generate a corresponding monoscopic grid and use it as the first monoscopic grid, perform up, down, left, and right deflection transformations on each camera pose according to a preset deflection angle, and generate multiple different renderings through the first monoscopic grid rendering, and then project the renderings back into three-dimensional space to generate corresponding monoscopic grids and use them as the second monoscopic grid, back-project the second monoscopic grid based on the original perspective corresponding to the origin of the indoor scene and calculate the mask to obtain the occlusion area mask, and add the occlusion area mask to the original monoscopic image to form a masked monoscopic image as the projected image; finally, an original monoscopic image, a projected image, an occlusion area mask, and a text description of the camera pose are combined into a group of training sample pairs to form a training data set, and the LoRA model is used to fine-tune the diffusion completion model on the training data set.

[0023] As a preferred embodiment of the first aspect, in step S22, the fine-tuning objects include the self-attention layer, the cross-attention layer of the diffusion completion model U-Net and the self-attention layer of the Text Encoder network.

[0024] As a preferred embodiment of the first aspect, the specific process of step S3 is as follows:

[0025] S31: Based on the geometric structure information of the indoor scene in the initial three-dimensional grid, extract the origin boundary points of the indoor scene to generate a boundary contour curve of the indoor scene; smooth the boundary contour curve through Gaussian filtering to generate a smooth and closed boundary curve;

[0026] S32: uniformly sampling boundary points from the boundary curve, and calculating the central axis of the internal contour skeleton of the indoor scene based on the sampled boundary points based on the central axis transformation algorithm;

[0027] S33: uniformly sampling a preset number of points on the central axis with a preset step size, and using the coordinates of the sampled points as camera pose coordinates. The camera pose direction is the direction of a three-dimensional vector from the origin of the indoor scene to the sampling points. A camera pose set is formed by the camera pose coordinates and the camera pose direction vector.

[0028] S34: Perform multiple iterations to complete the initial 3D mesh. In each iteration, the mesh updated in the previous iteration is used as the current iteration mesh. The scene projection map and its mask under each camera pose in the camera pose set are rendered by equidistant spherical projection. The completeness of each camera pose is calculated by the mask of the scene projection map and sorted in ascending order. The scene projection map corresponding to the camera pose with the highest completeness is used as the panorama to be completed. The panorama to be completed is projected to the icosahedron perspective through the equidistant spherical surface to generate 20 single-view images to be repaired and their masks. The single-view images to be repaired, the masks and the camera pose text are then combined into a complete image. This description is input into the fine-tuned diffusion completion model to form a repaired single-view image and back-project it to form a completed panorama and project it into the cubic perspective. The completed new-view image is generated and used as a multi-view image. The multi-view image is combined with the text description of the camera pose to form 6 sets of multi-view data pairs and added to the multi-view dataset generated in the previous iteration to form an updated multi-view dataset. The depth map corresponding to the completed panorama is obtained and a candidate mesh is generated. The conflict-free area is filtered from the candidate mesh by the constructed conflict mask and merged into the current iteration mesh to complete a completion.

[0029] As a preferred embodiment of the first aspect, in step S34, the specific process of generating the conflict mask is as follows: in the i-th iteration, the panoramic image completed in the i-th iteration is passed through the pre-trained monocular depth estimator to generate a corresponding depth map and use it as the first reference depth map, the panoramic image completed in the (i-1)-th iteration is passed through the pre-trained monocular depth estimator to generate a corresponding depth map and use it as the second reference depth map, and the first iteration is passed through the pre-trained monocular depth estimator to generate a corresponding depth map and use it as the second reference depth map. The panoramic image completed in the first iteration and the first reference depth map are used to generate a candidate grid, and the candidate grid is used to be projected under the camera pose adopted in the (i-1)th iteration to generate a first predicted depth map. The depth difference between the second reference depth map and the first predicted depth map is compared pixel by pixel. When the depth difference corresponding to a pixel coordinate is greater than the preset depth conflict threshold, the value of the conflict mask at the pixel coordinate is 1, indicating a conflict, otherwise it is 0, indicating no conflict. When all pixels are compared, a conflict-free area is formed by all conflict-free pixel coordinates, and a complete conflict mask is obtained.

[0030] Furthermore, in step S34, the conflict mask The calculation process is as follows:

[0031]

[0032] in, Indicates that the pixel coordinates are The conflict mask value at ; Indicates that the second reference depth map is at pixel coordinates The depth value at Indicates that the first predicted depth map is at pixel coordinates The depth value at represents the L2 norm; Indicates the depth collision threshold.

[0033] As a preference of the above-mentioned first aspect, in step S4, the specific process of initializing the Gaussian point parameters is: converting the three-dimensional grid completed in step S3 into a 3D Gaussian sputtering field representation, initializing the center position coordinates of the 3D Gaussian ellipsoid from the midpoint coordinates of the completed three-dimensional grid, initializing the color attributes of the 3D Gaussian ellipsoid from the color information of the completed three-dimensional grid, and initializing the covariance matrix, opacity and other parameters of the 3D Gaussian ellipsoid according to preset rules.

[0034] As a preferred embodiment of the above-mentioned first aspect, in step S4, the rendering error feedback loss is composed of the original loss function used in 3D Gaussian sputtering field training and the depth similarity loss. The depth similarity loss measures the structural similarity between the single-view depth map rendered by the 3D Gaussian sputtering field and the real depth map in the multi-view dataset through the Pearson correlation coefficient between the two.

[0035] Furthermore, in step S4, the rendering error feedback loss The functional form is as follows:

[0036]

[0037]

[0038]

[0039]

[0040] in, represents the original loss function used in 3D Gaussian sputtering field training; represents the deep similarity loss; All represent hyperparameters; Represents a single-view image rendered by a 3D Gaussian sputtering field during the optimization process; Represents multi-view images in a multi-view dataset; express The corresponding single-view depth map; express The corresponding real depth map; represents SSIM loss; Indicates calculation of Pearson correlation; Indicates the calculation of covariance; Indicates the calculation of variance; represents the L1 norm.

[0041] As the first aspect mentioned above, in step S4, the processing process of optimizing the 3D Gaussian sputtering field by using the Gaussian upsampling strategy is as follows: within one optimization round, the K nearest neighbor algorithm is used to calculate the scale properties of the 3D Gaussian ellipsoid, starting from the origin Gaussian, traversing each Gaussian in the 3D Gaussian ellipsoid, and performing the following process on each Gaussian: a Gaussian in the 3D Gaussian ellipsoid is used as the source Gaussian, the K neighbors closest to the source Gaussian are marked as target Gaussians, the distance between a source Gaussian and each corresponding target Gaussian is calculated, and the average value of the distance calculation results is used as the proximity score of the source Gaussian, and the connectivity graph formed by the source Gaussian and its corresponding target Gaussian is used as the distance between the source Gaussian and the target Gaussian. Proximity graph, based on the proximity graph and the proximity score of each source Gaussian, if the proximity score of a source Gaussian exceeds the preset score threshold, the straight line formed by connecting the source Gaussian and its corresponding target Gaussian is used as an edge, and a new Gaussian is created at one-third and two-thirds of each edge. When the newly created Gaussian is close to the source Gaussian, its scale and opacity attributes are consistent with the source Gaussian. When the newly created Gaussian is close to the target Gaussian, its scale and opacity attributes are consistent with the target Gaussian, and other attributes are initialized to zero; after processing all Gaussians, the optimized 3D Gaussian sputtering field of this optimization round is formed; where K represents the number of selected nearest neighbors.

[0042] In a second aspect, the present invention provides an adaptive three-dimensional scene reconstruction system based on a single panoramic image, comprising:

[0043] The mesh generation module is used to obtain a panoramic color image of the indoor scene and estimate its depth information, generate a depth map, perform edge estimation and back-projection operations on the depth map, and generate an initial 3D mesh of the indoor scene;

[0044] The data generation and model fine-tuning module is used to generate six original monoscopic images from the perspective of a cube based on the initial 3D mesh through perspective transformation. The projected camera pose is then transformed to generate the projected image and occlusion area mask of each original monoscopic image after the camera pose transformation. The projected image and its occlusion area mask are then back-projected to form training sample pairs and fine-tune the diffusion completion model. Each training sample pair consists of four parts: the original monoscopic image, the projected image and occlusion area mask, and a text description of the camera pose.

[0045] The mesh completion module is used to extract the boundary contour of the initial 3D mesh, generate a boundary contour curve, and smooth it through Gaussian filtering to generate a smooth and closed boundary curve. Based on the boundary points uniformly sampled from the boundary curve, the central axis of the internal contour skeleton of the indoor scene is calculated using the central axis transformation algorithm. Sampling is performed on the central axis according to a preset step size, and the coordinates of the sampling points are used as the camera pose coordinates. The vector direction from the origin of the indoor scene to the sampling points is used as the camera pose direction to form a camera pose set. Based on the camera pose set, the initial 3D mesh is iteratively completed multiple times, and the integrity of the initial 3D mesh is optimized through conflict detection and merging to form a completed 3D mesh and output a multi-viewpoint dataset.

[0046] The Gaussian field optimization module is used to convert the completed three-dimensional mesh into a 3D Gaussian sputtering field and initialize the Gaussian point parameters. The Gaussian point parameters are optimized based on a multi-viewpoint dataset. During the optimization process, the consistency of the 3D Gaussian sputtering field is adjusted through the rendering error feedback loss. When the optimization round is greater than the preset round threshold, the Gaussian upsampling strategy is used to split and refine the Gaussian points at the edge, and finally output a complete three-dimensional scene model.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] The present invention combines deep learning, computer vision and 3D reconstruction technology, and proposes a novel technical path, namely an adaptive 3D scene reconstruction method based on a single panoramic image. This method injects scene prior information through LoRA fine-tuning diffusion completion model, thereby improving the completion capability; the adaptive camera pose generation technology can cover all key areas, ensuring the accurate capture and generation of occluded areas, which is different from using predefined camera poses or randomly sampling poses from the scene, showing unique innovation; during Gaussian field optimization, the Gaussian upsampling strategy improves geometric continuity and texture details while maintaining overall structural consistency. These technological breakthroughs are significantly different from traditional methods. Through structural adaptive completion and Gaussian refinement technology, high-quality 3D scenes are reconstructed from a single panoramic image, which is significantly better than existing methods in terms of accuracy and completeness, and is particularly suitable for immersive scene roaming and indoor scene reconstruction. In terms of accuracy, completeness and detail retention, the present invention is significantly better than existing methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 is a flow chart of the method of the present invention;

[0050] Figure 2 This is a schematic diagram of scene grid initialization in the method of the present invention;

[0051] Figure 3 Schematic diagram of fine-tuning the diffusion completion model of the present invention;

[0052] Figure 4 Schematic diagram of scene grid adaptive completion according to the method of the present invention;

[0053] Figure 5 Schematic diagram of 3D Gaussian sputtering field optimization of the method of the present invention;

[0054] Figure 6 Schematic diagram of the Gaussian upsampling strategy designed for the present invention;

[0055] Figure 7 A schematic diagram of a three-dimensional scene reconstruction result provided by the present invention;

[0056] Figure 8 This is another schematic diagram of a three-dimensional scene reconstruction result provided by the present invention;

[0057] Figure 9 This is a system block diagram of the present invention. DETAILED DESCRIPTION

[0058] In order to make the above-mentioned objects, features and advantages of the present invention more clearly understood, the specific embodiments of the present invention are described in detail below with reference to the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the connotation of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed below. The technical features in the various embodiments of the present invention can be combined accordingly without conflicting with each other.

[0059] In the description of the present invention, it should be understood that the terms "first" and "second" are used solely for descriptive purposes and are not to be construed as indicating or implying relative importance or implicitly specifying the number of technical features being described. Therefore, features defined as "first" or "second" may explicitly or implicitly include at least one of such features.

[0060] To address the shortcomings of single-view scene reconstruction in the prior art, such as occlusion omission, detail loss, and poor consistency, the present invention provides an adaptive three-dimensional scene reconstruction method based on a single panoramic image. It is implemented based on LoRA model fine-tuning, Mesh rendering, 3D Gaussian sputtering, and Gaussian upsampling. It should be noted that the 3D Gaussian sputtering and LoRA model fine-tuning in the present invention belong to the prior art. Among them, Gaussian sputtering technology is the core method for three-dimensional scene rendering. It achieves efficient and realistic real-time rendering effects by mapping point clouds or volume data in three-dimensional space into a set of Gaussian distributions. LoRA model fine-tuning extracts and optimizes three-dimensional content by injecting existing scene geometry and texture information into pre-training, and using the pre-trained diffusion completion model. The specific principles of this technology will not be repeated here. In addition, pre-trained diffusion completion models already exist in the prior art. The present invention can also be further fine-tuned based on the pre-trained diffusion model to meet the generation task of the present invention.

[0061] like Figure 1 As shown, in a preferred implementation of the present invention, the above-mentioned adaptive 3D scene reconstruction method based on a single panoramic image includes the following steps S1 to S4. The specific implementation process is described in detail below.

[0062] S1: Obtain a panoramic color image of the indoor scene and estimate its depth information to generate a depth map. Perform edge estimation and back-projection operations on the depth map to generate an initial 3D mesh of the indoor scene.

[0063] It should be noted that if Figure 2 As shown, the specific process of step S1 of the present invention is as follows: a two-dimensional panoramic color image (resolution W×H is 2048×1024 or similar specifications, W is the width of the panoramic color image; H is the height of the panoramic color image) is obtained, its depth value is predicted by the open source panoramic depth prediction model PanoFusionDistancePredictor, and a depth map is formed by the predicted depth value ( Figure 2 The edge mask of the depth map is extracted through Canny edge detection and dilation operation. The color and depth information of the panoramic color image are mapped to the point and surface information of the 3D mesh based on the back projection operation. The edge mask is used to segment the triangular mesh patches connecting different objects, and the initial 3D mesh of the indoor scene is generated at the origin of the indoor scene. .

[0064] In this embodiment, S1 is the scene mesh initialization. In step S1, the depth map is edge estimated, an edge mask is generated, and the triangular mesh patches between different objects are segmented to improve the mesh smoothness. The back projection operation is used to upgrade the color and depth information of the panoramic color image to the point and surface information of the three-dimensional mesh, where the points refer to the vertices of the three-dimensional mesh and the surfaces refer to the patches of the three-dimensional mesh. Finally, an initial three-dimensional mesh is generated in the three-dimensional coordinate system of the indoor scene with the origin coordinates (0, 0, 0). The mesh represents the scene texture and geometric information from the perspective of the origin of the indoor scene, serves as the initialization representation of the indoor scene, and provides texture and geometric priors for subsequent model fine-tuning.

[0065] S2: Based on the initial 3D mesh, six original monoscopic images from the perspective of a cube are generated through perspective transformation. The projection camera pose is then transformed, and the projection image and occlusion area mask of each original monoscopic image after the camera pose transformation are generated through projection transformation. The projection image and its occlusion area mask are then back-projected to form training sample pairs and fine-tune the diffusion completion model. Each training sample pair consists of four parts: the original monoscopic image, the projection image and occlusion area mask, and a text description of the camera pose.

[0066] It should be noted that, in the present invention, Figure 3 As shown, the specific process of step S2 is as follows:

[0067] S21: In the initial 3D mesh scene, keep the camera pose coordinates at the origin of the indoor scene, adjust the camera pose orientation to the six faces of the cube, and generate 6 original single-view images of the indoor scene and their corresponding camera pose text descriptions.

[0068] It should be noted that in this embodiment S21, since the initial three-dimensional grid is generated The perspective is fixed to the origin of the indoor scene, the camera pose is transformed, the camera pose coordinates are located at the origin and remain unchanged, and the camera pose is oriented in the direction of the cube , the obtained 6 original single-view images of indoor scenes and 6 camera pose text descriptions .

[0069] S22: Perform the following operations on each original single-view image: back-project the original single-view image into three-dimensional space to generate a corresponding single-view grid and use it as the first single-view grid, perform up, down, left, and right deflection transformations on each camera pose according to a preset deflection angle, and generate multiple different renderings through the first single-view grid rendering, and then project the renderings back into three-dimensional space to generate corresponding single-view grids and use them as the second single-view grid, and back-project the second single-view grid based on the original view corresponding to the origin of the indoor scene and calculate the mask to obtain the occlusion area mask , and add the occlusion area mask to the original monoscopic image to form a masked monoscopic image As a projected image; ultimately, an original single-view image , projected image , occlusion area mask and text description of camera pose The samples are combined into a set of training sample pairs to form a training dataset, and the LoRA model is used to fine-tune the diffusion completion model on the training dataset.

[0070] In this embodiment, random sampling is performed in the interval [0,45°] to generate the above-mentioned deflection angles, and four different renderings are generated for each original monoscopic image through single-view grid rendering. Based on the rendering, an occlusion region mask is generated, which is intended to simulate the occlusion of scene objects caused by perspective transformation in indoor scenes. According to the above process, each original monoscopic image will correspond to a camera pose text description, and each original monoscopic image will correspond to four masked monoscopic images, and each masked monoscopic image will correspond to a unique occlusion region mask. Finally, the original monoscopic image, the masked monoscopic image (projected image), the occlusion region mask and the camera pose text description are combined into 24 sets of training sample pairs for fine-tuning the diffusion completion model.

[0071] In this embodiment, the diffusion completion model adopts Stable Diffusion 2-Inpainting, and is fine-tuned using a low-rank-based adapter (LoRA model) to learn the texture prior and geometric prior of the indoor scene, ensuring that the completion content is consistent with the overall scene under the constraints of the camera pose and semantic cues. Specifically, the fine-tuning objects include the self-attention layer (self-attention), cross-attention layer (self-attention) of the diffusion completion model U-Net and the self-attention layer of the Text Encoder network, and the training data used for fine-tuning is the training sample pair generated in step S22. There are 200 rounds of fine-tuning, and the LoRA parameters are set as follows: rank is 16, scaling factor α is 32, and the learning rate of fine-tuning training is The objective function used for fine-tuning is consistent with the loss function used in the pre-training of the diffusion completion model, as follows:

[0072]

[0073] in, represents the input image; represents the added Gaussian noise randomly sampled from a normal distribution; represents the normal distribution, represents the identity matrix; represents the time step of the diffusion process; represents a collection of time steps; is a weighting function that depends on the time step; Represents the noise value predicted by the diffusion completion model U-Net network; is the latent vector of the input image The noisy version of represents the mask condition of the input image, A text description condition representing the camera pose; Represents the square of the L2 norm.

[0074] S3: Extract the boundary contour of the initial three-dimensional mesh, generate a boundary contour curve and smooth it through Gaussian filtering to generate a smooth and closed boundary curve; use the central axis transformation algorithm to calculate the central axis of the internal contour skeleton of the indoor scene based on the boundary points uniformly sampled from the boundary curve; sample on the central axis according to the preset step size, use the coordinates of the sampling points as the camera pose coordinates, and use the vector direction from the origin of the indoor scene to the sampling points as the camera pose direction to form a camera pose set; based on the camera pose set, perform multiple iterative completions on the initial three-dimensional mesh, optimize the integrity of the initial three-dimensional mesh through conflict detection and merging, form a completed three-dimensional mesh and output a multi-viewpoint dataset.

[0075] It should be noted that, in the present invention, Figure 4 As shown, the specific process of step S3 is as follows:

[0076] S31: Based on the geometric structure information of the indoor scene in the initial three-dimensional grid, the origin boundary points of the indoor scene are extracted to generate a boundary contour curve of the indoor scene; the boundary contour curve is smoothed by Gaussian filtering to generate a smooth and closed boundary curve.

[0077] In this embodiment S31, considering that the value of the origin boundary point may change suddenly between adjacent vertices, Gaussian filtering interpolation is used to expand the boundary contour curve point set to generate a smooth and closed boundary curve. .

[0078] S32: uniformly sampling boundary points from the boundary curve, and calculating the central axis of the internal contour skeleton of the indoor scene based on the sampled boundary points based on the central axis transformation algorithm.

[0079] In this embodiment, S32, the coordinates of the boundary points on the boundary curve are stored in a list. A boundary point is sampled from this list at regular intervals. The central axis is then calculated using the central axis transformation algorithm. This central axis describes the internal structural skeleton of the indoor scene. The central axis transformation algorithm is a relatively mature method in the prior art, so the specific implementation process will not be detailed in this embodiment.

[0080] S33: uniformly sample a preset number of points on the central axis with a preset step size, and use the coordinates of the sampled points as the camera pose coordinates. The camera pose direction is the direction of the three-dimensional vector from the origin of the indoor scene to the sampling point. The camera pose coordinates and the camera pose direction vector form a camera pose set.

[0081] In this embodiment S33, in order to construct a camera pose set that completes the initial three-dimensional grid, 15 points are uniformly sampled from the central axis according to a preset step size, and the coordinates of these 15 points are used as camera pose coordinates. The camera pose direction is the direction of the three-dimensional vector from the origin of the indoor scene to the sampling point. Then, the camera pose coordinates and the camera pose direction vector constitute a camera pose set. , ensuring that the generated camera pose can cover all key areas. Among them, the camera pose coordinates are a 3×3 matrix, the camera pose direction vector is a 3×1 vector, and the camera pose set is a 3×4 matrix.

[0082] S34: Perform multiple iterations to complete the initial 3D mesh. In each iteration, the mesh updated in the previous iteration is used as the current iteration mesh. The scene projection map and its mask under each camera pose in the camera pose set are rendered by equidistant spherical projection. The completeness of each camera pose is calculated by the mask of the scene projection map and sorted in ascending order. The scene projection map corresponding to the camera pose with the highest completeness is used as the panorama to be completed. The panorama to be completed is projected onto the icosahedron through an equidistant sphere to generate 20 single-view images to be repaired and their masks. The single-view images to be repaired, masks, and camera pose text descriptions are then input into the fine-tuned diffusion completion model to form the repaired single-view image and back-project it to form the completed panorama. And project it to the cube perspective to generate a completed new perspective image and use it as a multi-view image. The multi-view image is combined with the camera pose text description to form 6 groups of multi-view data pairs and added to the multi-view data set generated by the previous iteration to form an updated multi-view data set; obtain the depth map corresponding to the completed panorama and generate a candidate grid. The conflict-free area is filtered from the candidate grid by the constructed conflict mask and merged into the current iteration grid to complete a completion.

[0083] In this embodiment S34, the initial three-dimensional grid As the mesh used in the first iteration of the completion process, a total of 15 iterations of completion are performed on the initial 3D mesh. The scene projection map and mask of the initial 3D mesh are rendered by equidistant spherical projection, and the scene completeness corresponding to each camera pose is calculated and sorted. Here, the completeness of each camera pose is calculated by the mask of the scene projection map. The mask is 1 for occluded and 0 for unoccluded, so the completeness = 1-number of occluded pixels / total number of pixels. Then the camera pose with the highest completeness is selected and the panorama is rendered using equidistant spherical projection. And project it to the icosahedron perspective to generate 20 single-view images to be repaired and their corresponding masks and camera pose text descriptions; repair the generated single-view images through the diffusion completion model fine-tuned in step S2 to complete the unobserved areas and ensure geometric semantic consistency; then back-project the repaired single-view images to generate the completed panorama , and project the completed panorama onto a regular hexahedron perspective to generate a completed new perspective image. Each completed new perspective image is combined with its corresponding mask and camera pose text description to form a multi-view data pair. After projection onto the regular hexahedron perspective, 6 completed new perspective images are formed. Accordingly, each iteration generates 6 sets of multi-view data pairs. Before iterative completion, an empty multi-view dataset is constructed. The multi-view data pairs generated in each iteration are added to the multi-view dataset to update it. After the iterative completion is completed, a complete multi-view dataset is obtained, which contains 90 sets of multi-view data pairs (15×6) for use in step S4 to optimize the Gaussian field. Calculate the conflict mask between the newly added area and the original area, filter the non-conflicting parts, and merge the valid completion content into the mesh updated in the previous iteration. Through multiple iterations, the scene integrity, accuracy, and geometric precision are improved.

[0084] In step S34 of the present invention, the specific process of generating the conflict mask is as follows: In the first iteration, The panorama after the iteration is completed is passed through the pre-trained monocular depth estimator to generate the corresponding depth map and use it as the first reference depth map. The panorama after iteration completion A pre-trained monocular depth estimator generates a corresponding depth map and uses it as the second reference depth map , by The completed panorama and the first reference depth map after the first iteration generate candidate meshes, and the candidate meshes are used in the The first predicted depth map is generated by projecting the camera pose used in the first iteration , compare the depth difference between the second reference depth map and the first predicted depth map pixel by pixel. When the depth difference corresponding to a pixel coordinate is greater than the preset depth conflict threshold, the value of the conflict mask at the pixel coordinate is 1, indicating a conflict, otherwise it is 0, indicating no conflict. When all pixels are compared, a conflict-free area is formed by all conflict-free pixel coordinates, and a complete conflict mask is obtained.

[0085] In this embodiment, the collision mask The calculation process is as follows:

[0086]

[0087] in, Indicates that the pixel coordinates are The conflict mask value at ; Indicates that the second reference depth map is at pixel coordinates The depth value at Indicates that the first predicted depth map is at pixel coordinates The depth value at represents the L2 norm; Indicates the depth collision threshold.

[0088] S4: Convert the completed 3D mesh into a 3D Gaussian Splatting field (3DGS) and initialize the Gaussian point parameters. Optimize the Gaussian point parameters based on a multi-view dataset. During the optimization process, adjust the consistency of the 3D Gaussian splatting field using the rendering error feedback loss. When the number of optimization rounds exceeds the preset round threshold, a Gaussian upsampling strategy is used to split and refine the Gaussian points at the edge, ultimately outputting a complete 3D scene model with geometric continuity and rich textures.

[0089] It should be noted that, in the present invention, in step S4, the specific process of initializing the Gaussian point parameters is: converting the three-dimensional grid completed in step S3 into a 3D Gaussian sputtering field (3D Gaussian Splatting, 3DGS) representation, initializing the center position coordinates of the 3D Gaussian ellipsoid from the midpoint coordinates of the completed three-dimensional grid, initializing the color attributes of the 3D Gaussian ellipsoid from the color information of the completed three-dimensional grid, and initializing the covariance matrix, opacity and other parameters of the 3D Gaussian ellipsoid according to preset rules.

[0090] It should be noted that in step S4 of the present invention, the rendering error feedback loss is composed of the original loss function used in 3D Gaussian sputtering field training and the depth similarity loss. The depth similarity loss measures the structural similarity between the single-view depth map rendered by the 3D Gaussian sputtering field and the true depth map in the multi-view dataset through the Pearson correlation coefficient between the two.

[0091] In step S4 of this embodiment, Figure 5 As shown, the coordinates and colors of the midpoints of the completed three-dimensional grid are extracted to initialize the center position coordinates and color of the 3D Gaussian ellipsoid. Other properties of the 3D Gaussian ellipsoid, such as covariance and opacity, are initialized. In the three-dimensional Gaussian sputtering field optimization process, the multi-view image generated in step S3 is used as the observation basis. At the same time, in order to inject more regularization into the optimization, the present invention proposes an additional depth adjustment method and applies pixel-level geometric correspondence in training. Furthermore, the embodiment of the present invention uses a pre-trained monocular depth estimator DensePrediction Transformer to predict multi-view images. Depth map To address the scale ambiguity problem in depth estimation, this paper adopts a depth similarity loss based on Pearson correlation, which enforces the distribution consistency between the estimated depth map and the rendered depth map. This loss is combined with the original loss function used in 3D Gaussian sputtering field training to form a rendering error feedback loss. The 3DGS parameters (including the position, covariance, color, and opacity of the three-dimensional Gaussian points) are iteratively optimized through backpropagation of the loss gradient. The functional form is as follows:

[0092]

[0093]

[0094]

[0095]

[0096] in, represents the original loss function used in 3D Gaussian sputtering field training; represents the deep similarity loss; All represent hyperparameters. In this embodiment, Set to 0.8, Set to 0.2, Set to 0.05; Represents a single-view image rendered by a 3D Gaussian sputtering field during the optimization process; Represents multi-view images in a multi-view dataset; express The depth map of , that is, the single-view depth map; express The depth map of , that is, the real depth map; Represents SSIM loss (Structural Similarity Index Measure Loss, SSIM Loss); Indicates calculation of Pearson Correlation Coefficient; Indicates the calculation of covariance; Indicates the calculation of variance; represents the L1 norm.

[0097] It should be noted that, in step S4 of the present invention, the processing process of optimizing the 3D Gaussian sputtering field by using the Gaussian upsampling strategy is specifically as follows: within one optimization round, the K-nearest neighbor algorithm (KNN algorithm) is used to calculate the scale properties of the 3D Gaussian ellipsoid, starting from the origin Gaussian, traversing each Gaussian in the 3D Gaussian ellipsoid, and performing the following process on each Gaussian: a Gaussian in the 3D Gaussian ellipsoid is used as the source Gaussian, and the K neighbors closest to the source Gaussian are marked as target Gaussians, and the distance between a source Gaussian and each corresponding target Gaussian is calculated, and the average value of the distance calculation results is used as the proximity score of the source Gaussian, and the connected graph formed by the source Gaussian and its corresponding target Gaussian is used as the proximity graph, based on the proximity graph and the proximity score of each source Gaussian, If the proximity score of a source Gaussian exceeds a preset score threshold, the straight line connecting the source Gaussian and its corresponding target Gaussian is used as an edge, and a new Gaussian is created at one-third and two-thirds of each edge. When the newly created Gaussian is close to the source Gaussian, its scale and opacity properties are consistent with the source Gaussian. When the newly created Gaussian is close to the target Gaussian, its scale and opacity properties are consistent with the target Gaussian. At the same time, other properties (such as rotation and SH coefficient) are initialized to zero. After processing all Gaussians, the optimized 3D Gaussian sputtering field of this optimization round is formed. Where K represents the number of selected nearest neighbors. In this embodiment, K=3.

[0098] In step S4 of this embodiment, after the scene completion and preliminary optimization in S3, there is still a problem of sparse distribution of Gaussian ellipsoids in the edge areas of objects in the scene. Therefore, for the edge areas with significant geometric errors, when the optimization rounds are greater than the preset round threshold of 2000, the Gaussian upsampling strategy is adopted to split and refine each edge Gaussian point to generate dense sub-Gaussian clusters to improve the geometric accuracy of edge details. Figure 6 This Gaussian upsampling strategy distributes new Gaussians around the edge regions of the scene, systematically filling in the observation gaps during the optimization process.

[0099] The following is a comparative experiment on applying the adaptive 3D scene reconstruction method based on a single panoramic image in the above embodiment to a specific text prompt. The specific steps are as described in S1 to S4 and will not be repeated here. The specific parameters and technical effects are mainly demonstrated.

[0100] Example

[0101] To validate the effectiveness of the present invention, a comprehensive user study was conducted on eight scenes from the Replica dataset (room_0, room_1, room_2, office_0, office_1, office_2, office_3, and office_4). The proposed method was compared with two currently leading models: PERF (using Neural Radiance Fields (NeRFs) as the 3D representation) and Pano2Room (using 3D GS as the 3D representation). Experimental conditions, including input conditions and training rounds, were kept consistent across the three methods. Forty-five new-view images and four videos with predefined camera trajectories were rendered from the final reconstruction results for comparison. Users were asked to vote based on their visual perception in four areas: texture, geometry, scene completeness, and video frame rate (rendering speed). This objectively compared the reconstruction performance of the different models for the input scene panoramas. The results are shown in Table 1. This example also compared the quantitative metrics (PSNR, SSIM, LPIPS, and NIQE) and time overhead between the new-view images generated by the three methods and the original dataset. The results are shown in Table 2. It can be seen that the present invention is significantly superior to existing methods in terms of accuracy and completeness. In addition, in order to enable those skilled in the art to intuitively experience the reconstruction effect of the present invention, this embodiment also provides two schematic diagrams of the three-dimensional scene reconstruction results using the method of the present invention, as shown in FIG. Figure 7 and Figure 8 shown.

[0102] Table 1. Texture, geometry, scene completeness, and video frame rate evaluation table

[0103] Table 2. Generation efficiency evaluation table

[0104] It should also be noted that the adaptive 3D scene reconstruction method based on a single panoramic image in the above embodiment can essentially be executed by a computer program or module. Therefore, similarly, based on the same inventive concept, another preferred embodiment of the present invention also provides an adaptive 3D scene reconstruction system based on a single panoramic image corresponding to the adaptive 3D scene reconstruction method based on a single panoramic image provided in the above embodiment, such as Figure 9 As shown, it includes:

[0105] The mesh generation module is used to obtain a panoramic color image of the indoor scene and estimate its depth information, generate a depth map, perform edge estimation and back-projection operations on the depth map, and generate an initial 3D mesh of the indoor scene;

[0106] The data generation and model fine-tuning module is used to generate six original monoscopic images from the perspective of a cube based on the initial 3D mesh through perspective transformation. The projected camera pose is then transformed to generate the projected image and occlusion area mask of each original monoscopic image after the camera pose transformation. The projected image and its occlusion area mask are then back-projected to form training sample pairs and fine-tune the diffusion completion model. Each training sample pair consists of four parts: the original monoscopic image, the projected image and occlusion area mask, and a text description of the camera pose.

[0107] The mesh completion module is used to extract the boundary contour of the initial 3D mesh, generate a boundary contour curve, and smooth it through Gaussian filtering to generate a smooth and closed boundary curve. Based on the boundary points uniformly sampled from the boundary curve, the central axis of the internal contour skeleton of the indoor scene is calculated using the central axis transformation algorithm. Sampling is performed on the central axis according to a preset step size, and the coordinates of the sampling points are used as the camera pose coordinates. The vector direction from the origin of the indoor scene to the sampling points is used as the camera pose direction to form a camera pose set. Based on the camera pose set, the initial 3D mesh is iteratively completed multiple times, and the integrity of the initial 3D mesh is optimized through conflict detection and merging to form a completed 3D mesh and output a multi-viewpoint dataset.

[0108] The Gaussian field optimization module is used to convert the completed three-dimensional mesh into a 3D Gaussian sputtering field and initialize the Gaussian point parameters. The Gaussian point parameters are optimized based on a multi-viewpoint dataset. During the optimization process, the consistency of the 3D Gaussian sputtering field is adjusted through the rendering error feedback loss. When the optimization round is greater than the preset round threshold, the Gaussian upsampling strategy is used to split and refine the Gaussian points at the edge, and finally output a complete three-dimensional scene model.

[0109] It should also be noted that those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the system described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here. In the various embodiments provided in this application, the division of steps or modules in the system and method is only a logical function division. In actual implementation, there may be other division methods, for example, multiple modules or steps can be combined or integrated together, and a module or step can also be split.

[0110] The embodiment described above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Persons skilled in the art may make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, any technical solution obtained by equivalent substitution or equivalent transformation falls within the scope of protection of the present invention.

Claims

1. A method for adaptive 3D scene reconstruction based on a single panoramic image, characterized in that: The following steps are involved: S1: Acquire a panoramic color image of the indoor scene and estimate its depth information to generate a depth map. Perform edge estimation and back-projection operations on the depth map to generate an initial 3D mesh of the indoor scene. S2: Based on the initial 3D mesh, generate 6 original single-view images under the perspective of the cube through perspective transformation; The projection camera pose is then transformed, and the projection image and occlusion area mask of each original single-view image after the camera pose transformation are generated through projection transformation. The projection image and its occlusion area mask are then back-projected to form training sample pairs and fine-tune the diffusion completion model. Each training sample pair consists of four parts: the original single-view image, the projection image and occlusion area mask, and a text description of the camera pose. S3: Extract the boundary contour of the initial 3D mesh, generate a boundary contour curve, and smooth it through Gaussian filtering to generate a smooth and closed boundary curve; use the central axis transformation algorithm to calculate the central axis of the internal contour skeleton of the indoor scene based on the boundary points uniformly sampled from the boundary curve; sample on the central axis according to the preset step size, use the coordinates of the sampling points as the camera pose coordinates, and use the vector direction from the origin of the indoor scene to the sampling points as the camera pose direction to form a camera pose set; based on the camera pose set, perform multiple iterative completions on the initial 3D mesh, optimize the integrity of the initial 3D mesh through conflict detection and merging, form a completed 3D mesh, and output a multi-viewpoint dataset; S4: Convert the completed 3D mesh into a 3D Gaussian sputtering field and initialize the Gaussian point parameters. Optimize the Gaussian point parameters based on a multi-viewpoint dataset. During the optimization process, adjust the consistency of the 3D Gaussian sputtering field through the rendering error feedback loss. When the number of optimization rounds is greater than the preset round threshold, a Gaussian upsampling strategy is used to split and refine the Gaussian points at the edge, and finally output a complete 3D scene model.

2. The method for adaptive 3D scene reconstruction based on a single panoramic image according to claim 1, wherein: The specific process of step S1 is as follows: obtain a two-dimensional panoramic color image, predict its depth value through the panoramic depth prediction model, form a depth map from the predicted depth values, extract the edge mask of the depth map through Canny edge detection and dilation operation, map the color and depth information of the panoramic color image to the point and surface information of the three-dimensional grid based on the back projection operation, use the edge mask to segment the triangular mesh patches connecting different objects, and generate the initial three-dimensional grid of the indoor scene at the origin of the indoor scene.

3. The method for adaptive 3D scene reconstruction based on a single panoramic image according to claim 1, wherein: The specific process of step S2 is as follows: S21: In the initial 3D mesh scene, keep the camera pose coordinates at the origin of the indoor scene, adjust the camera pose orientation to the six faces of the cube, and generate 6 original single-view images of the indoor scene and their corresponding camera pose text descriptions; S22: Perform the following operations on each original monoscopic image: back-project the original monoscopic image into three-dimensional space to generate a corresponding monoscopic grid and use it as the first monoscopic grid, perform up, down, left, and right deflection transformations on each camera pose according to a preset deflection angle, and generate multiple different renderings through the first monoscopic grid rendering, and then project the renderings back into three-dimensional space to generate corresponding monoscopic grids and use them as the second monoscopic grid, back-project the second monoscopic grid based on the original perspective corresponding to the origin of the indoor scene and calculate the mask to obtain the occlusion area mask, and add the occlusion area mask to the original monoscopic image to form a masked monoscopic image as the projected image; finally, an original monoscopic image, a projected image, an occlusion area mask, and a text description of the camera pose are combined into a group of training sample pairs to form a training data set, and the LoRA model is used to fine-tune the diffusion completion model on the training data set.

4. The method for adaptive 3D scene reconstruction based on a single panoramic image according to claim 3, wherein: In step S22, the fine-tuning objects include the self-attention layer, cross-attention layer of the diffusion completion model U-Net and the self-attention layer of the Text Encoder network.

5. The method for adaptive 3D scene reconstruction based on a single panoramic image according to claim 1, wherein: The specific process of step S3 is as follows: S31: Based on the geometric structure information of the indoor scene in the initial three-dimensional grid, extract the origin boundary points of the indoor scene to generate a boundary contour curve of the indoor scene; smooth the boundary contour curve through Gaussian filtering to generate a smooth and closed boundary curve; S32: uniformly sampling boundary points from the boundary curve, and calculating the central axis of the internal contour skeleton of the indoor scene based on the sampled boundary points based on the central axis transformation algorithm; S33: uniformly sampling a preset number of points on the central axis with a preset step size, and using the coordinates of the sampled points as camera pose coordinates. The camera pose direction is the direction of a three-dimensional vector from the origin of the indoor scene to the sampling points. A camera pose set is formed by the camera pose coordinates and the camera pose direction vector. S34: Perform multiple iterations to complete the initial 3D mesh. In each iteration, the mesh updated in the previous iteration is used as the current iteration mesh. The scene projection map and its mask under each camera pose in the camera pose set are rendered by equidistant spherical projection. The completeness of each camera pose is calculated by the mask of the scene projection map and sorted in ascending order. The scene projection map corresponding to the camera pose with the highest completeness is used as the panorama to be completed. The panorama to be completed is projected to the icosahedron perspective through the equidistant spherical surface to generate 20 single-view images to be repaired and their masks. The single-view images to be repaired, the masks and the camera pose text are then combined into a complete image. This description is input into the fine-tuned diffusion completion model to form a repaired single-view image and back-project it to form a completed panorama and project it into the cubic perspective. The completed new-view image is generated and used as a multi-view image. The multi-view image is combined with the text description of the camera pose to form 6 sets of multi-view data pairs and added to the multi-view dataset generated in the previous iteration to form an updated multi-view dataset. The depth map corresponding to the completed panorama is obtained and a candidate mesh is generated. The conflict-free area is filtered from the candidate mesh by the constructed conflict mask and merged into the current iteration mesh to complete a completion.

6. The method for adaptive 3D scene reconstruction based on a single panoramic image according to claim 5, wherein: In step S34, the specific process of generating the collision mask is as follows: in the i-th iteration, the panoramic image completed in the i-th iteration is passed through the pre-trained monocular depth estimator to generate a corresponding depth map and use it as the first reference depth map; the panoramic image completed in the (i-1)-th iteration is passed through the pre-trained monocular depth estimator to generate a corresponding depth map and use it as the second reference depth map; The panoramic image completed in the first iteration and the first reference depth map are used to generate a candidate grid, and the candidate grid is used to be projected under the camera pose adopted in the (i-1)th iteration to generate a first predicted depth map. The depth difference between the second reference depth map and the first predicted depth map is compared pixel by pixel. When the depth difference corresponding to a pixel coordinate is greater than the preset depth conflict threshold, the value of the conflict mask at the pixel coordinate is 1, indicating a conflict, otherwise it is 0, indicating no conflict. When all pixels are compared, a conflict-free area is formed by all conflict-free pixel coordinates, and a complete conflict mask is obtained.

7. The method for adaptive 3D scene reconstruction based on a single panoramic image according to claim 1, wherein: In step S4, the specific process of initializing the Gaussian point parameters is as follows: converting the three-dimensional grid completed in step S3 into a 3D Gaussian sputtering field representation, initializing the center position coordinates of the 3D Gaussian ellipsoid from the midpoint coordinates of the completed three-dimensional grid, initializing the color attributes of the 3D Gaussian ellipsoid from the color information of the completed three-dimensional grid, and initializing the covariance matrix, opacity and other parameters of the 3D Gaussian ellipsoid according to preset rules.

8. The method for adaptive 3D scene reconstruction based on a single panoramic image according to claim 1, wherein: In step S4, the rendering error feedback loss is composed of the original loss function used in 3D Gaussian sputtering field training and the depth similarity loss. The depth similarity loss measures the structural similarity between the single-view depth map rendered by the 3D Gaussian sputtering field and the real depth map in the multi-view dataset through the Pearson correlation coefficient between the two.

9. The method for adaptive 3D scene reconstruction based on a single panoramic image according to claim 1, wherein: In step S4, the processing process of optimizing the 3D Gaussian sputtering field by using the Gaussian upsampling strategy is as follows: within one optimization round, the K nearest neighbor algorithm is used to calculate the scale properties of the 3D Gaussian ellipsoid, starting from the origin Gaussian, traversing each Gaussian in the 3D Gaussian ellipsoid, and performing the following process on each Gaussian: a Gaussian in the 3D Gaussian ellipsoid is used as the source Gaussian, and the K neighbors closest to the source Gaussian are marked as target Gaussians, and the distance between a source Gaussian and its corresponding target Gaussians is calculated, and the average value of the distance calculation results is used as the proximity score of the source Gaussian, and the connected graph formed by the source Gaussian and its corresponding target Gaussian is used as the proximity graph, based on Based on the proximity graph and the proximity score of each source Gaussian, if the proximity score of a source Gaussian exceeds the preset score threshold, the straight line formed by connecting the source Gaussian and its corresponding target Gaussian is used as an edge, and a new Gaussian is created at one-third and two-thirds of each edge. When the newly created Gaussian is close to the source Gaussian, its scale and opacity attributes are consistent with the source Gaussian. When the newly created Gaussian is close to the target Gaussian, its scale and opacity attributes are consistent with the target Gaussian, and other attributes are initialized to zero. After processing all Gaussians, the optimized 3D Gaussian sputtering field of this optimization round is formed. K represents the number of selected nearest neighbors.

10. An adaptive 3D scene reconstruction system based on a single panoramic image, characterized in that: include: The mesh generation module is used to obtain a panoramic color image of the indoor scene and estimate its depth information, generate a depth map, perform edge estimation and back-projection operations on the depth map, and generate an initial 3D mesh of the indoor scene; The data generation and model fine-tuning module is used to generate six original single-view images from the perspective of a cube based on the initial 3D mesh through perspective transformation; The projection camera pose is then transformed, and the projection image and occlusion area mask of each original single-view image after the camera pose transformation are generated through projection transformation. The projection image and its occlusion area mask are then back-projected to form training sample pairs and fine-tune the diffusion completion model. Each training sample pair consists of four parts: the original single-view image, the projection image and occlusion area mask, and a text description of the camera pose. The mesh completion module is used to extract the boundary contour of the initial 3D mesh, generate a boundary contour curve, and smooth it through Gaussian filtering to generate a smooth and closed boundary curve. Based on the boundary points uniformly sampled from the boundary curve, the central axis of the internal contour skeleton of the indoor scene is calculated using the central axis transformation algorithm. Sampling is performed on the central axis according to a preset step size, and the coordinates of the sampling points are used as the camera pose coordinates. The vector direction from the origin of the indoor scene to the sampling points is used as the camera pose direction to form a camera pose set. Based on the camera pose set, the initial 3D mesh is iteratively completed multiple times, and the integrity of the initial 3D mesh is optimized through conflict detection and merging to form a completed 3D mesh and output a multi-viewpoint dataset. The Gaussian field optimization module is used to convert the completed three-dimensional mesh into a 3D Gaussian sputtering field and initialize the Gaussian point parameters. The Gaussian point parameters are optimized based on a multi-viewpoint dataset. During the optimization process, the consistency of the 3D Gaussian sputtering field is adjusted through the rendering error feedback loss. When the optimization round is greater than the preset round threshold, the Gaussian upsampling strategy is used to split and refine the Gaussian points at the edge, and finally output a complete three-dimensional scene model.

Images