New view angle synthesis three-dimensional reconstruction method and system based on single panoramic image

Abstract

The invention discloses a new view angle synthesis three-dimensional reconstruction method and system based on a single panoramic image. The system comprises a panoramic image conversion grid module, a grid completion module and a grid conversion three-dimensional Gaussian field module which are connected in series. A panoramic image is generated from panoramic images captured by any camera at a single location. Comprising the following steps: firstly, constructing a preliminary grid from an input panoramic image, then iteratively optimizing the grid through a panoramic complementing device, and collecting a three-dimensional consistent pseudo new view; and finally, the optimized grid is converted into a three-dimensional Gaussian field, and training is carried out by using the collected pseudo new view. According to the method, the three-dimensional scene of the real world can be reconstructed only according to input of one panorama, the data acquisition cost of three-dimensional reconstruction is remarkably reduced, the situation of large-area shielding can be effectively processed, details of fine textures can be reserved, and a new realistic view with a detailed geometric structure can be synthesized.

Description

Technical Field

[0001] The present invention belongs to the field of computer graphics and image processing technology and artificial intelligence technology, and relates to three-dimensional scene reconstruction technology, and in particular to a high-quality three-dimensional scene automatic reconstruction method and system based on a single-view panoramic image, which can generate a three-dimensional scene and synthesize a new three-dimensional view using only a single panoramic image. Background Art

[0002] High-quality 3D scene generation is a very important task in the field of computer graphics and artificial intelligence, and is widely used in the fields of metaverse, film and television special effects, augmented reality, virtual reality, digital twins, etc. Although multi-view 3D reconstruction methods such as 3D Gaussian fields (Kerbl, Bernhard, et al. "3D Gaussian Splatting for Real-Time Radiance Field Rendering." ACM Trans. Graph. 42.4 (2023): 139-1.) have made significant progress in creating high-quality real-world scenes, existing technologies usually require a large number of input images to generate accurate 3D representations of complex indoor scenes. This method that relies on a large number of images is expensive and time-consuming, which limits its widespread practical application. On the other hand, existing single-view 3D reconstruction methods are not suitable for real-world scenes. Figure 3 Generative 3D methods have made significant progress in synthesizing 3D objects by learning 3D object priors from large-scale 3D object datasets, but single-view Figure 3 The task of 3D indoor scene synthesis remains a daunting challenge, mainly due to the complexity of real-world environments (containing open vocabulary objects, i.e., undefined or unrecognized objects) and the limited availability of high-quality a priori scene-level resources.

[0003] Reconstructing a complete 3D scene from a single panoramic image is very challenging, mainly because it requires mining the complex and complete information of the scene with very little input 3D conditions. These challenges also include the need to maintain the fidelity of the user input image content and accurately reconstruct the geometry, as well as reasoning about large areas of occluded content with detailed 3D consistent textures and geometry and integrating it consistently with the existing scene. Due to these challenges, existing single-view scene reconstruction methods such as Text2Room ( Lukas, et al. "Text2room: Extracting textured 3d meshes from 2dtext-to-image models." Proceedings of the IEEE / CVF International Conference on Computer Vision. 2023.), PERF (Wang, Guangcong, et al. "Perf: Panoramic neural radiance field from a single panorama." IEEE Transactions on Pattern Analysis and Machine Intelligence (2024).), LucidDreamer (Chung, Jaeyoung, et al. "Luciddreamer: Domain-free generation of 3d gaussian splatting scenes." arXiv preprint arXiv: 2311.13384 (2023).), etc. are not only difficult to create 3D scenes that accurately retain the content captured by the user, but also prone to failure when dealing with large-area occlusion scenes. Summary of the invention

[0004] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a new perspective synthetic 3D reconstruction method and system based on a single panoramic image, which can significantly reduce the data acquisition cost of 3D reconstruction, effectively handle large-area occlusion, retain the details of fine textures, and synthesize a new photo-realistic view with detailed geometric structure.

[0005] The technical solution provided by the present invention is as follows:

[0006] A new perspective synthesis 3D reconstruction method and system based on a single panoramic image, with a single panoramic image as input, the present invention first uses the panoramic image and its distance map (i.e., the panoramic distance map, which is the geometric image of the panoramic image) joint completion module to predict the geometric structure of the panoramic image, which is represented by the panoramic distance map (the first step). Then, the present invention uses the panoramic image conversion grid module to convert the input panoramic image and its distance map into an initial grid (the second step). Next, the present invention uses the grid completion module to complete the initial grid, and the grid completion module includes three main submodules: a camera search module, a panoramic image and its distance map joint completion module, and a geometric conflict detection module; wherein the camera search module is used to find the best completion camera position and render the grid under the camera position, and the grid completion module uses the panoramic image and its distance map joint completion module to complete the panoramic image and its distance map. Subsequently, the grid completion module uses the geometric conflict detection module to filter the geometric conflict, and the new texture and geometric structure that does not generate geometric conflict will be merged into the initial grid through the grid completion module. The initial grid is iteratively completed by the grid completion module to obtain the completed grid (the third step). Finally, the present invention uses a mesh-to-three-dimensional Gaussian field module to convert the completed mesh into a three-dimensional Gaussian field, and trains the three-dimensional Gaussian field through the collected pseudo-new views. The three-dimensional Gaussian field after training is the final three-dimensional scene representation of the present invention, which can achieve high-quality new perspective rendering with detailed geometric structure (the fourth step).

[0007] The steps are as follows:

[0008] The first step is to input a panoramic image and predict a panoramic distance map. The input panoramic image can be taken by any monocular camera or generated by an image generation model. The panoramic distance map can be captured by a depth structured light camera or lidar, or predicted by a joint completion module of the panoramic image and its distance map.

[0009] In the second step, the present invention uses a panorama conversion mesh module to convert the input panorama and its distance map into an initial mesh. The present invention uses panorama spherical projection, point cloud triangulation and edge map-based mesh filtering to convert the input panorama I into an initial mesh Mesh = {V, C, F} according to its distance map, where V, C, F represent the vertices, colors and faces of the mesh respectively; including:

[0010] 21) Initialize V and C using the screen coordinates (u, v) and the corresponding colors in the input panorama, respectively. We triangulate the vertex V of the mesh according to the screen coordinates (u, v), connecting every four adjacent vertices in the mesh to create two faces. This triangulation process generates a mesh connecting all adjacent points. The present invention proposes a deep edge filter that uses an edge detector (Canny, John. "A computational approach to edge detection." IEEE Transactions on pattern analysis and machine intelligence 6(1986):679-698.) to extract a deep edge mask M from the panoramic distance map D, and delete the vertices in M ​​and the mesh containing these vertices, thereby filtering the initial mesh.

[0011] 22) The present invention uses spherical projection to convert the vertices in the screen coordinates (u, v) into 3D space (x, y, z) according to the panoramic distance map D. The projection process is expressed by the following formula:

[0012]

[0013] x=sin(φ)·cos(θ)·D(φ,θ),

[0014] y=cos(φ)·D(φ,θ),

[0015] z=sin(φ)·sin(θ)·D(φ,θ), (1)

[0016] Where W and H represent the width and height of I, respectively, and (φ, θ) represent the polar coordinates of (u, v). Through the above projection, an initial grid is obtained.

[0017] In the third step, the present invention uses a mesh completion module to complete the initial mesh to obtain a completed mesh; the mesh completion module uses a camera search module to find the best completion camera position (i.e., the viewpoint with the lowest view integrity), and renders the mesh at the camera position, and uses a panorama and its distance map joint completion module to complete the panorama and its distance map, and collect them as a pseudo-new view. Subsequently, the mesh completion module uses a geometric conflict detection module to filter geometric conflicts, and new textures / geometric structures that do not generate geometric conflicts will be merged into the initial mesh through the mesh completion module. After iteratively completing the initial mesh, a completed mesh is obtained. Including:

[0018] 31) Camera search module. Performing repair under any inappropriate camera viewpoint (with a small angle between the camera direction and the surface normal), or performing multiple completion steps on any occluded area, will result in low-quality completion results. The present invention proposes a camera search module, the goal of which is to minimize the completion steps required to complete the scene. In each scene, first, the camera search module determines the scene boundaries based on the panoramic distance map. The horizontal midline of the panoramic distance map represents the horizontal boundary of the scene, and the vertical midline of the panoramic distance map represents the vertical boundary of the scene. Next, the camera search module searches for the viewpoint with the lowest view integrity from a set of predefined camera poses Cams in the scene covering most of the scene as the best completion camera position:

[0019]

[0020] The above formula indicates that the search goal is to find a function ∑ (u,v)∈(H,W) M render (k)(u,v) reaches the minimum variable value {k∈K}. Where (u,v) represents the screen coordinates, H, W represent the height and width of the mask M. k represents the search camera pose index, there are K groups of search cameras, M render Represents the rendering mask; arg min represents the variable value for finding the minimum value of the function.

[0021] 32) Panorama and its distance map joint completion module. In order to complete the initial mesh, the present invention proposes a panorama and its distance map joint completion module, including a panoramic image completer for generating high-quality textures consistent with the input image, and a panoramic distance completer for predicting new detailed geometry aligned with the existing surface. Specifically:

[0022] 321) The panoramic image completer of the grid completion module is based on the stable diffusion image generation model (Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition. 2022.). Since the stable diffusion image generation model only accepts perspective projection image input instead of panoramic images as input, we iteratively sample the perspective projection image from the panorama and use the stable diffusion image generation model to generate the complement perspective projection image, and then project the complement perspective projection image into the panorama. Since the stable diffusion image generation model cannot obtain a complete perception of the scene when using the perspective projection image as input, resulting in inconsistent generated new textures, in order to fully perceive the panorama, the panoramic image completer of the present invention is fine-tuned for each panorama. Through the low-rank adaptive fine-tuning method (refer to: Hu, Edward J., et al. "Lora: Low-rank adaptation of large language models." arXiv preprint arXiv:2106.09685(2021).) of the completion model of each individual scene, the panoramic image completer of the present invention is able to learn the complete style and features of the input panorama and generate new textures consistent with the scene.

[0023] 322) Panoramic distance completer. The panoramic distance map includes visible surfaces in the scene, which contain complex textures and geometries with complex details. The panoramic distance completer of the grid completion module first projects the panoramic distance map into multiple overlapping perspective projection depth maps, and then uses the monocular depth estimation model DPT (Ranftl, René, Alexey Bochkovskiy, and Vladlen Koltun. "Vision transformers for dense prediction." Proceedings of the IEEE / CVF international conference on computer vision. 2021.) to estimate the depth map of each perspective projection, and then projects the depth map of each perspective projection into the panoramic distance map and performs iterative optimization. In order to iteratively fuse the depth maps projected from each perspective into the panoramic distance map, the present invention optimizes the scale factor s and offset o of the depth map D of each perspective projection to obtain a complete and smooth deep panoramic distance map with detailed geometry. Specifically, the optimization objective is defined as:

[0024]

[0025] Where E fix Refers to the fixed depth error, keeping the existing depth unchanged, E align Refers to the alignment energy loss to ensure the consistency between the depth maps projected from different perspectives, E normal Refers to the surface normal loss, ensuring the rationality of the surface normal, E smooth Refers to the smoothness loss, which ensures the smoothness of the panoramic distance map. After multiple optimizations, the depth maps projected from each perspective are seamlessly merged into the rendered panoramic distance map, resulting in a completed panoramic distance map, which contains detailed new geometry aligned with the existing geometry and surface normals.

[0026] 33) Geometric conflict detection module. When generating occluded content, the newly added content may cause geometric conflicts with the previously rendered camera pose. In order to avoid geometric conflicts, the present invention proposes a geometric conflict detection module based on mesh rendering. After merging the new mesh with the existing mesh, the geometric conflict detection module renders the mesh from the completed camera pose and detects geometric conflicts. The generated content that introduces geometric conflicts will be deleted by the geometric conflict detection module. Specifically, in order to ensure that the newly merged mesh does not occlude the content generated by the completed camera pose, the geometric conflict detection module uses a mask M conflict To identify conflicting completions, the specific calculation formula is as follows:

[0027] M conflict (k)(u,v)=|Iinp (k)(u,v)-R mesh (k)(i,v)|>ε, (4)

[0028] Where (u, v) represents the pixel coordinates, I inp represents the completed panorama, R mesh represents the grid rendering function, k represents the camera pose index, there are K groups of search cameras in total, and ε is the detection threshold.

[0029] In the fourth step, the present invention uses a mesh-to-three-dimensional Gaussian field module to convert the completed mesh into a three-dimensional Gaussian field, and trains the three-dimensional Gaussian field through the collected pseudo-new views. The three-dimensional Gaussian field after training is the final three-dimensional scene representation of the present invention, which can achieve high-quality new perspective rendering with detailed geometric structure. The completed mesh generated by the third step above often contains over-smoothed textures and geometric shapes, thereby reducing the realism and clarity of the rendered new view. In order to solve this problem, the present invention proposes a mesh-to-three-dimensional Gaussian field module. Specifically, the mesh-to-three-dimensional Gaussian field module first uses the V and C of the mesh as the initial point cloud to construct an initial three-dimensional Gaussian field. Then, the mesh-to-three-dimensional Gaussian field module uses the collected pseudo-new views to train the three-dimensional Gaussian field. The three-dimensional Gaussian field is rendered using pixel-based transparency blending, which calculates the color C of the pixel by blending N ordered pixels that overlap with the pixel, as expressed by the following formula:

[0030]

[0031] Where c represents the color of each point and α is the opacity of each point. In order to train the three-dimensional Gaussian field, the present invention uses photometric loss and collected pseudo new views for optimization, including image contrast loss L1 and image structure similarity loss L D-SSIM , the specific calculation is as follows:

[0032] L=L1(R GS ,I inp )+λ LD-SSIM (R GS ,I inp ), (6)

[0033] Where R GS represents the three-dimensional Gaussian field rendering function, I inp represents the collected pseudo new views and λ represents the loss coefficient.

[0034] The iterative mesh completion method with geometric conflict avoidance strategy proposed in the present invention is used to train the collected pseudo new views of the three-dimensional Gaussian field to have 3D consistency and photorealism. By using the collected pseudo new views to train the three-dimensional Gaussian field, it is ensured that the rendering quality of the three-dimensional Gaussian field matches the realism of the image generated by the panoramic image completer of the present invention. Finally, a high-quality trained three-dimensional Gaussian field is obtained, which significantly improves the quality of new view synthesis.

[0035] Through the above steps, new perspective synthesis based on a single panoramic image is achieved.

[0036] In specific implementation, the present invention realizes a new perspective synthesis 3D reconstruction system based on a single panoramic image, and the system includes a panoramic image conversion grid module, a grid completion module, and a grid conversion 3D Gaussian field module. In this system, the panoramic image conversion grid module, the grid completion module, and the grid conversion 3D Gaussian field module are serially connected. The input, output and connection relationship of each part are as follows:

[0037] The panorama-to-grid module is used to perform style rendering on the target semantic map to obtain the initial grid. The input is the panorama and its distance map, and the output is the initial grid.

[0038] The grid completion module is used to complete the grid to obtain the completed grid; the input is the initial grid, and the output is the completed grid. The grid completion module includes three main submodules: camera search module, panorama and distance map joint completion module, and geometric conflict detection module;

[0039] The mesh to 3D Gaussian field module is used to convert the completed mesh into a 3D Gaussian field. The input is the completed mesh and the output is a 3D Gaussian field.

[0040] Compared with the prior art, the beneficial effects of the present invention include:

[0041] The present invention proposes a new perspective synthesis method and system based on a single panoramic image, which generates a panoramic image from a panoramic image captured by an arbitrary camera at a single position using a joint completion module of a panoramic image and its distance map. First, a preliminary grid is constructed from the input panoramic image, and then the grid is iteratively optimized by the joint completion module of the panoramic image and its distance map, while collecting realistic three-dimensional consistent pseudo new views. Finally, the optimized grid is converted into a three-dimensional Gaussian field and trained using the collected pseudo new views.

[0042] The present invention only requires a single panoramic image as input, and reconstructs the three-dimensional scene of the real world based on a panoramic image input, which greatly reduces the time and cost of data acquisition. Through the camera search module and the joint completion module of the panorama and its distance map, the present invention can effectively handle large-area occlusions and generate new textures and geometric structures consistent with the existing scene. The panoramic image completer of the present invention can generate high-quality texture details through low-rank adaptive fine-tuning to ensure that the synthesized new view has a realistic visual effect. The final generated three-dimensional Gaussian field not only has a detailed geometric structure, but also can achieve photo-realistic new perspective rendering, reducing the data acquisition cost of three-dimensional reconstruction, and can effectively handle large-area occlusions. It can also retain the details of fine textures and synthesize realistic new views with detailed geometric structures, significantly improving the quality and robustness of the synthesized view. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 It is an input and output display diagram of the three-dimensional reconstruction and new view method provided by the present invention.

[0044] Figure 2 It is a method flow chart of the three-dimensional reconstruction and new view method provided by the present invention.

[0045] Figure 3 Schematic diagram of the panorama-to-mesh module in the present invention. It shows how to convert a panorama into an initial mesh. The initial mesh vertices and colors are derived from the pixels of the input panorama, and the depth edge map is used to disconnect the faces representing different objects to ensure accurate mesh generation.

[0046] Figure 4 It is a schematic diagram of the joint completion module of the panoramic image and its distance map in the present invention. First, the rendered panoramic image is completed using the stable diffusion image generation model, and then the depth of the completed content is estimated using the monocular depth estimation model, and it is seamlessly integrated into the rendered panoramic depth map.

[0047] Figure 5 It is a schematic diagram of the camera search module in the present invention. It shows how to minimize the completion steps required to complete the scene by searching the viewpoint, avoiding multiple completions and ambiguous results caused by the sequential camera trajectory strategy.

[0048] Figure 6 Schematic diagram of the mesh conversion 3D Gaussian field module in the present invention. The optimized 3D Gaussian field renders the new view with high quality and detailed geometric structure, effectively solving the over-smoothing artifacts introduced by Poisson surface reconstruction.

[0049] Figure 7This is a comparison chart of 3D reconstruction and new view synthesis of the present invention and other existing methods, showing the advantages of the present invention in new view synthesis and depth map generation, especially in terms of texture quality, geometric structure and structural rationality. DETAILED DESCRIPTION

[0050] The present invention will be further described below by way of embodiments in conjunction with the accompanying drawings, but the scope of the present invention is not limited in any way.

[0051] The present invention provides a new perspective synthesis method based on a single panoramic image. First, the input panoramic image is converted into a grid, and the grid is iteratively completed by using the panoramic image and its distance map joint completion module to generate occluded content and geometric structure, while gradually integrating new content into the completed grid. Finally, the completed grid is converted into a three-dimensional Gaussian field and trained using the collected three-dimensional consistent pseudo new views. The optimized three-dimensional Gaussian field can synthesize photorealistic new views with detailed geometric structures.

[0052] The method of the present invention inputs and outputs Figure 1 As shown in the figure, a panoramic image and its distance map are input. The panoramic image can come from a monocular camera or an image generation model, and the distance map can come from a deep structured light camera, a lidar, or be predicted by a joint completion module for the panoramic image and its distance map.

[0053] The process of the method of the present invention is shown in the attached Figure 2 As shown, in the specific implementation, an automatic three-dimensional reconstruction and new perspective synthesis system based on a single panorama is constructed, including: a panorama conversion grid module, a panorama and its distance map joint completion module, and a grid conversion three-dimensional Gaussian field module; wherein, the panorama conversion grid module is used to generate an initial grid, the panorama and its distance map joint completion module is used to complete the grid, and the grid conversion three-dimensional Gaussian field module is used to convert the completed grid into a three-dimensional Gaussian field.

[0054] Attached Figure 3 Shows how to convert a panorama to a mesh. The initial mesh vertices and colors are derived from the pixels of the input panorama (represented by black dots). Triangulation connects adjacent vertices to form faces (represented by black lines). A depth edge map is used to break up faces representing different objects, ensuring accurate mesh generation.

[0055] Attached Figure 4 The module for joint completion of a panorama and its range map is presented. The rendered panorama is first completed using a stable diffusion image generation model, and then the depth of the completed content is estimated using a monocular depth estimation model and seamlessly merged into the rendered panorama depth map.

[0056] Attached Figure 5We show how to minimize the number of completion steps required to complete a scene by searching for viewpoints, avoiding multiple completions and ambiguous results caused by sequential camera trajectory strategies, which lead to multiple completion steps in occluded space, thus producing ambiguous completion results. By utilizing the searched viewpoints, we significantly improve the generation of plausible new textures and geometries.

[0057] Attached Figure 6 Schematic diagram of the Mesh to 3D Gaussian Field module. The optimized 3D Gaussian Field renders new views with high quality and detailed geometry, effectively resolving over-smoothing artifacts introduced by Poisson surface reconstruction.

[0058] Attached Figure 7 The new view generation comparison diagram of the present invention and the existing methods. The new view synthesis and depth map of each method are compared with the corresponding ground truth. Lukas,et al."Text2room:Extracting textured 3d meshes from 2d text-to-image models."Proceedings ofthe IEEE / CVF International Conference on Computer Vision.2023.) and LucidDreamer(Chung,Jaeyoung,et al."Luciddreamer:Domain-free generation of 3d gaussiansplatting scenes."arXiv preprint arXiv:2311.13384(2023).) The new views synthesized are prone to strong artifacts. PERF(Wang,Guangcong,et al."Perf:Panoramic neural radiance fieldfrom a single panorama."IEEE Transactions on Pattern Analysis and Machine Intelligence(2024).) often generates over-smoothed occluded areas with interpolated geometry. Taking into account texture, geometry and structure, the new views of the present invention are more realistic. The advantages of the present invention in novel view synthesis and depth map generation are obvious, especially in terms of texture quality, geometry and structural rationality.

[0059] The above is a clear and complete description of the technical solutions in the embodiments of the present invention in combination with the drawings in the embodiments of the present invention. It is understandable that the examples described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present invention.

Claims

1. A new perspective synthesis 3D reconstruction method based on a single panoramic image, characterized in that: The following steps are involved: The first step is to input a panoramic image and predict its distance map; In the second step, the panorama and its distance map are converted into an initial grid; The third step is to complete the initial grid to obtain the completed grid; include: 31) Find the best camera position to complete the task; 32) Rendering the initial grid under the optimal completion camera position, and collecting the rendered views as pseudo new views; 33) completing the panoramic image and its distance map based on the pseudo new view; 34) removing geometric conflicts generated during the completion process, and integrating new textures and geometric structures without geometric conflicts into the initial mesh; After iteratively completing the initial grid, a completed grid is obtained; The fourth step is to convert the completed mesh into a 3D Gaussian field and train the 3D Gaussian field with the collected pseudo new views, thereby achieving photo-realistic new view rendering with detailed geometric structures. Through the above steps, new perspective synthetic 3D reconstruction based on a single panoramic image is achieved.

2. The new perspective synthesis 3D reconstruction method based on a single panoramic image according to claim 1, characterized in that: In the first step, the input panoramic image is specifically obtained by taking pictures with any monocular camera, or generated by an image generation model; the distance map is specifically obtained by a depth structured light camera or a laser radar.

3. The new perspective synthesis 3D reconstruction method based on a single panoramic image according to claim 1, characterized in that: In the first step, the distance map is predicted by a panoramic distance completer of a panoramic image and its distance map joint completion module, specifically, a depth panoramic distance map is obtained by optimizing the scale factor and offset of the depth map projected at each viewing angle, and the depth panoramic distance map has a geometric shape.

4. The new perspective synthesis 3D reconstruction method based on a single panoramic image according to claim 1, characterized in that: The second step is to use panorama spherical projection, point cloud triangulation and edge map-based mesh filtering to convert the input panorama into an initial mesh according to its distance map; it includes the following processes: 21) constructing a mesh Mesh = {V, C, F}, where V, C, and F represent the vertices, colors, and faces of the mesh, respectively; initializing the vertices and colors of the mesh using the screen coordinates (u, v) and the corresponding colors in the input panorama, respectively; triangulating the vertices of the mesh according to the screen coordinates (u, v), connecting every four adjacent vertices in the mesh to create two faces; thereby generating a mesh connecting all adjacent points; 22) Using the spherical projection method, according to the panoramic distance map, the vertices in the screen coordinates (u, v) are converted into three-dimensional space to obtain the initial mesh; 23) Extract the depth edge mask from the panoramic distance map and delete the vertices in the depth edge mask and the meshes containing these vertices.

5. The new perspective synthesis 3D reconstruction method based on a single panoramic image according to claim 1, characterized in that: In the third step, Determine the scene boundary according to the panoramic distance map; the horizontal center line of the panoramic distance map represents the horizontal boundary of the scene, and the vertical center line of the panoramic distance map represents the vertical boundary of the scene; From a set of predefined camera poses in the scene that cover most of the scene area, search for the viewpoint with the lowest view completeness.

6. The new perspective synthesis 3D reconstruction method based on a single panoramic image as claimed in claim 5, characterized in that: Furthermore, the perspective projection image is iteratively sampled from the panoramic image, a complementary perspective projection image is generated through a stable diffusion image generation model, and then the complementary perspective projection image is projected into the panoramic image; Using a monocular depth estimation model, the panoramic distance map is projected onto multiple overlapping perspective projection depth maps, the depth maps of each perspective projection are estimated, and iterative optimization is performed; by optimizing the scale factor and offset of the depth map of each perspective projection, a deep panoramic distance map with a geometric shape is obtained; the optimization objective is expressed as: Among them, arg min represents the variable value for finding the minimum value of the function; E fix Refers to the fixed depth error, keeping the existing depth unchanged; E align Refers to the alignment energy loss to ensure the consistency between the depth maps projected from different perspectives; E normal Refers to the surface normal loss, ensuring the rationality of the surface normal; E smooth Refers to the smoothing loss, which ensures the smoothness of the panoramic distance map.

7. The new perspective synthesis 3D reconstruction method based on a single panoramic image as claimed in claim 6, characterized in that: In step 34), a geometric conflict detection method based on mesh rendering is designed. After the new mesh is merged with the existing mesh, the mesh is rendered from the completed camera pose to detect geometric conflicts, so that the newly merged mesh does not occlude the content generated by the completed camera pose; Specifically, a mask is used to identify conflicting completions, expressed as: M conflict (k)(u,v)=|I inp (k)(u,v)-R mesh (k)(u,v)|>ε, Among them, M conflict is the mask; (u, v) represents the pixel coordinates, I inp represents the completed panorama, R mesh represents the grid rendering function, k represents the camera pose index, there are K groups of search cameras in total, and ε is the detection threshold.

8. The new perspective synthesis 3D reconstruction method based on a single panoramic image as claimed in claim 7, characterized in that: The fourth step specifically includes the following processes: 41) Use the vertices and colors of the mesh as the initial point cloud to construct an initial three-dimensional Gaussian field; 42) The 3D Gaussian field is trained by the collected pseudo-new views. The 3D Gaussian field is rendered using point-based transparency blending, that is, the color C of the pixel is calculated by blending N ordered points overlapping the pixel, expressed as: Among them, C represents the color of each pixel, and α is the opacity of each point; 43) The training optimization objective function includes photometric loss, which is optimized using photometric loss and the collected pseudo-new views, including image contrast loss L1 and image structure similarity loss L D-SSIM , expressed as: L=L1(R GS ,I inp )+λL D-SSIM (R GS ,I inp ), Among them, R GS represents the three-dimensional Gaussian field rendering function, I inp represents the collected pseudo new views and λ represents the loss coefficient.

9. The new perspective synthesis 3D reconstruction method based on a single panoramic image according to claim 1, characterized in that: According to the method, a new perspective synthesis 3D reconstruction system based on a single panoramic image is realized, comprising: a panoramic image conversion grid module, a grid completion module, and a grid conversion 3D Gaussian field module; the panoramic image conversion grid module, the grid completion module, and the grid conversion 3D Gaussian field module are in a serial connection relationship; The panorama-to-grid module is used to render the target semantic map in a style and obtain the initial grid. The input is the panorama and its distance map, and the output is the initial grid. The grid completion module is used to complete the grid to obtain the completed grid; the input is the initial grid, and the output is the completed grid; the grid completion module includes three submodules: camera search module, panoramic image and distance map joint completion module, and geometric conflict detection module; The mesh conversion to three-dimensional Gaussian field module is used to convert the completed mesh into a three-dimensional Gaussian field; the input is the completed mesh and the output is a three-dimensional Gaussian field.

10. The new perspective synthesis 3D reconstruction method based on a single panoramic image according to claim 9, characterized in that: The module for jointly completing a panorama and its distance map includes a panorama image completer and a panorama distance completer; the panorama image completer is used to generate high-quality textures consistent with the input image, and the panorama distance completer is used to predict new detailed geometry that aligns with the existing surface of the image.

Images