Indoor orthographic image automatic generation method and system based on gaussian splash and diffusion model

Abstract

This invention relates to the fields of computer vision and 3D reconstruction technology, and discloses an automatic method and system for generating indoor orthophotos based on Gaussian splash and diffusion models. The method includes: SfM reconstruction, SVD fitting of camera position reference normals, RANSAC iterative elimination of walls to extract the ground, Rodrigues rotation plus Hough line segment alignment of the coordinate system, 5×5 virtual orthophoto camera generation, perspective projection training + orthographic projection rendering dual-module switching, diffusion model repair, SIFT feature extraction, and image stitching. This invention improves the clarity and usability of the final orthophoto map, achieving fully automatic end-to-end processing from multi-view photographs to indoor orthophoto maps.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D reconstruction technology, and more specifically, to a method and system for automatically generating indoor orthophotos based on Gaussian splash and diffusion models. Background Technology

[0002] Indoor orthophotos (i.e., room floor plans) have significant applications in interior design, property display, property rentals, and building documentation. Traditional methods of obtaining indoor floor plans primarily rely on manual on-site measurements and CAD drafting, which are inefficient, costly, and difficult to scale. Automated indoor orthophoto generation technology based on computer vision can directly generate high-precision room floor plans from multi-view images, which is crucial for lowering the barriers to digitizing interior spaces, improving property information transparency, and enhancing interior design efficiency.

[0003] In recent years, with the development of 3D reconstruction and neural rendering technologies, the complete technological chain from multi-view images to 3D scene representation and then to new view image generation has become increasingly mature and has been widely used in many fields, including: 1. Interior Design and Property Showcase: In scenarios such as interior design previews, property viewings, and property floor plan recording, it is necessary to perform 3D reconstruction of real rooms and generate accurate orthographic overhead floor plans to provide users with intuitive spatial layout information.

[0004] 2. Building Information Modeling (BIM): In building construction and renovation projects, it is necessary to quickly obtain floor plans and dimensional information of the building's internal structure from photos taken on site to assist in design decisions and construction management.

[0005] 3. Digital preservation of cultural heritage: In the digital archiving of ancient buildings and historical sites, it is necessary to reconstruct the interior space with high precision and generate orthophotos to record the spatial layout and decorative details of the building.

[0006] 4. Robot Navigation and Scene Understanding: In indoor service robots and autonomous navigation systems, accurate room floor plans are the foundation for path planning, scene understanding, and semantic map construction.

[0007] Structure from Motion (SfM) technology jointly recovers camera pose and sparse 3D point clouds from multi-view images, providing fundamental geometric information for indoor scene processing. Its technical approach is as follows: extracting and matching local image features → incrementally or globally solving for camera pose and 3D point positions → optimizing the reconstruction results through bundle adjustment. COLMAP, as a representative open-source SfM system, has been widely used in the front-end preprocessing of various 3D reconstruction tasks. However, SfM reconstruction has the following inherent limitations: First, the coordinate axes of the reconstructed coordinate system do not have a definite alignment relationship with the gravity direction and wall direction in the real world; it is an arbitrarily rotated coordinate system. Second, the sparse point cloud contains only a few 3D points with significant features in the scene and usually occupies a very small proportion of the scene surface, making it unsuitable for directly generating dense orthophotos. Third, SfM itself does not involve rendering and cannot independently complete end-to-end generation from a photograph to a planar image. Therefore, SfM is usually only used as initial input for subsequent 3D reconstruction steps.

[0008] 3D Gaussian Splatting (3DGS) explicitly represents a scene as a large number of 3D Gaussian primitives with parameters such as position, covariance, spherical harmonics, and opacity. It achieves efficient projection and real-time rendering from a 3D Gaussian ellipsoid to a 2D pixel plane through GPU differentiable rasterization. Its basic components include: initializing Gaussian primitive positions with SfM sparse point clouds → differentiable sputtering rendering based on perspective projection → optimizing Gaussian primitive parameters through gradient backpropagation and dynamically controlling densification and pruning. However, existing 3DGS methods face the following key technical bottlenecks in automatically generating scenes from indoor orthophotos: 1. Inconsistent Projection Modes: The training process of 3DGS is based on a pinhole camera model (perspective projection) of the input image. Its rasterizer's built-in projection Jacobian matrix and covariance projection formula are derived for perspective projection. However, orthographic top views require orthographic projection (parallel projection) to eliminate the perspective effect that causes objects to appear smaller in the distance. If orthographic projection is used during training, the optimizer will receive incorrect gradient signals because the input image itself is a perspective projection, leading to geometric distortion in the reconstruction. If the orthographic renderer is simply replaced during training, the rendering results will produce severe shape distortion and positional deviations because the Jacobian matrix of the Gaussian covariance under the two projections has completely different mathematical forms from the 3D to the 2D screen space.

[0009] 2. Rendering quality degradation: In orthographic projection rendering scenarios, 3DGS, limited by the limited viewpoint coverage of training data and insufficient optimization process, often suffers from degradation phenomena such as blurred edges, loss of details, and faint floating artifacts. Traditional image enhancement methods (such as sharpening filtering, deblurring algorithms, and super-resolution reconstruction) have limited effectiveness in repairing such structural degradation and cannot effectively restore lost texture details.

[0010] 3. Automatic Coordinate System Alignment Issue: The coordinate system obtained from SfM reconstruction is arbitrary. Traditional Random Sample Consensus (RANSAC) plane fitting methods typically extract the largest plane in the point cloud as the ground. However, in indoor scenes with large areas of blank walls, the extracted largest plane may be a wall rather than the ground, leading to completely incorrect virtual camera pose calculations. Furthermore, even if the ground is correctly identified, the wall orientation (i.e., the principal axis of the room) still needs to be determined separately; otherwise, the generated orthophoto will be rotated, which is not conducive to practical use.

[0011] 4. End-to-end pipeline integration issues: Integrating multiple technical modules such as sparse reconstruction, coordinate system alignment, scene representation training, orthophoto rendering, quality restoration, and image stitching into a complete, automated processing pipeline that requires no manual intervention involves data format connection and parameter transfer between multiple stages, and currently there is a lack of a systematic solution.

[0012] To alleviate these problems, existing work has explored various approaches. Regarding orthophoto generation, traditional photogrammetry methods rely on drones or specialized equipment to capture images from high altitudes outdoors, which is unsuitable for indoor scenes. DIFIX3D+ proposed a method to improve 3D reconstruction quality by using a single-step diffusion model to repair new rendering perspectives; however, this method is still based on a perspective projection framework and does not address the issue of switching from perspective to orthogonal projection. In terms of depth estimation-assisted reconstruction, DepthAnythingV2 provides high-quality dense monocular depth estimation, which can be used for depth supervision during 3DGS training, but it does not involve projection mode switching or orthophoto generation itself.

[0013] Therefore, a new method for automatically generating indoor orthophotos is needed to simultaneously solve key technical problems such as inconsistent projection modes, rendering quality degradation, and automatic coordinate system alignment, so as to achieve end-to-end high-quality automatic generation from multi-view images to indoor orthophoto plans. Summary of the Invention

[0014] To address the shortcomings of existing technologies, the present invention aims to solve the following technical problems: how to coordinate perspective projection training and orthographic projection rendering within a unified 3DGS framework, enabling the training phase to correctly fit the geometric structure of the input perspective photograph, while generating orthographic images without perspective distortion during the rendering phase; how to effectively repair degradation phenomena such as edge blurring, detail loss, and floating artifacts caused by orthographic rendering, thereby improving the clarity and usability of orthographic images; how to automatically identify the main directions of the ground plane and walls in any coordinate system reconstructed by SfM, achieving alignment between the scene coordinate system and the real-world directions; and how to integrate heterogeneous modules such as sparse reconstruction, coordinate alignment, scene training, orthographic rendering, quality restoration, and image stitching into an automated pipeline without human intervention. This invention provides an automatic indoor orthographic image generation method and system based on Gaussian splash and diffusion models, which can improve the clarity and usability of the final orthographic plan and achieve fully automated end-to-end processing from multi-view photographs to indoor orthographic plans.

[0015] To achieve the above objectives, in a first aspect, the present invention provides a method for automatically generating indoor orthophotos based on a Gaussian splash and diffusion model, comprising the following steps: S100: Based on multi-view images or videos of indoor scenes, extract sparse 3D point clouds and camera poses using the structure-reconstruction-motion method, including camera intrinsic and extrinsic parameters, sparse 3D point coordinates and their 2D observation relationships for each image. S200: Based on the sparse 3D point cloud and camera pose, perform iterative ground extraction and coordinate system alignment driven by camera position prior to obtain the aligned sparse 3D point cloud and camera pose. S300: Calculate the percentile bounding box of the aligned sparse 3D point cloud, and uniformly sample the pose of the virtual camera in a grid-like pattern within the percentile bounding box to obtain a virtual orthophoto camera. S400: Perform monocular depth estimation and scale calibration on the multi-view images, calculate the depth scale and offset of each image, and obtain the calibrated inverse depth map; S500: The three-dimensional Gaussian splash model is trained based on the calibrated inverse depth map, sparse three-dimensional point cloud and camera pose to obtain a trained three-dimensional Gaussian splash model. S600: Based on the three-dimensional Gaussian splash model, the virtual orthophoto camera is orthophoto-rendered using an orthogonal renderer to obtain an orthophoto image. S700: The orthophoto image is restored by performing quality restoration on the orthophoto image using a pre-trained single-step diffusion model to obtain the restored orthophoto image; S800: The repaired orthophoto image is stitched together using the SIFT feature matching algorithm to obtain the final complete indoor orthophoto plan.

[0016] As a further preferred option, step S2 specifically includes: S210: Extract the world coordinates of the camera center from the camera pose, and fit the camera position plane using the singular value decomposition method based on the world coordinates to obtain the camera plane normal. S220: Based on the camera plane normal and sparse 3D point cloud, a random sampling consensus algorithm is used to extract the plane to obtain the ground normal and ground reference points; S230: Based on the world coordinates of the camera center, the ground normal, and the ground reference point, obtain the aligned sparse 3D point cloud and the camera pose; As a further preferred option, step S23 specifically includes: S231: Rotate the ground normal to the Z-axis of the world coordinate system to obtain the ground normal alignment matrix; S232: The sparse 3D point cloud is transformed using the ground normal alignment matrix to obtain the transformed point cloud. The transformed point cloud is projected onto the XY plane and subjected to density outlier filtering. The Canny edge detection method and probabilistic Hough transform are used to extract a set of line segments. The line segments in the set that are close to the outer contour of the point cloud are selected as edge line segments. Based on the edge line segments, the total support score of the line segments that are simultaneously aligned to the candidate angle direction and the orthogonal direction of the candidate angle direction is calculated. The angle with the highest score is selected as the main direction of the wall. S233: Obtain the wall orientation alignment matrix based on the main wall orientation. Obtain the final coordinate system transformation based on the wall orientation alignment matrix and the ground normal alignment matrix. Use the final coordinate system transformation to transform each 3D point and camera pose in the sparse 3D point cloud to obtain the aligned sparse 3D point cloud and camera pose.

[0017] As a further preferred embodiment, the formula for calculating the virtual camera pose is:

[0018]

[0019]

[0020] in, Represents the virtual orthophoto camera in the i-th row and j-th column; and These represent the minimum and maximum values ​​of the scene along the X-axis; The grid size is preferably 5; and These represent the minimum and maximum values ​​of the scene along the Y-axis; The height ratio coefficient is preferably set to 0.8; This represents the maximum value of the scene along the Z-axis; The extrinsic translation vector; It is a vertically downward rotation matrix; It is a rotation quaternion.

[0021] As a further preferred embodiment, the formula for calculating the inverse depth map is:

[0022]

[0023]

[0024] in, This is an inverse depth map; For depth scale; This represents the depth value estimated by a single-view depth sensor. For offset; This represents the mean absolute deviation of the monocular depth. This represents the mean absolute deviation of the monocular depth. It is a median function; This represents the average absolute deviation of the COLMAP depth.

[0025] As a further preferred embodiment, the formula for projecting the three-dimensional Gaussian covariance of the three-dimensional Gaussian splash model onto the two-dimensional screen space is:

[0026] in, The Gaussian covariance matrix in two-dimensional screen space; For perspective projection Jacobian matrix; This represents the rotated portion of the view matrix; The Gaussian covariance matrix in three-dimensional space; superscript Represents the transpose of a matrix; The Jacobian matrix of the perspective projection of the three-dimensional Gaussian splash model is:

[0027] in, For perspective projection, the Jacobian matrix is ​​used. and The pixel focal length of the camera in the X and Y directions; The depth coordinates of the center of the Gaussian ellipsoid in the camera coordinate system; and The x and y coordinates of the center of the Gaussian ellipsoid in the camera coordinate system; The Taylor expansion correction term in the X direction; This is the Taylor expansion correction term in the Y direction; 0 indicates no projection contribution. The perspective projection matrix of the three-dimensional Gaussian splash model is:

[0028] in, For perspective projection matrix; Near-plane distance; and These are the right and left boundaries near the plane; and These are the upper and lower boundaries near the plane; The distance is the distance to the far plane; P[3,2]=1 indicates the existence of perspective division; As a further preferred embodiment, the training loss function of the three-dimensional Gaussian splash model is:

[0029] in, For training the loss function; For pixel-by-pixel L1 loss; These are the weighting coefficients for the SSIM loss; For structural similarity loss; For depth regularization loss weights; This represents the loss for deep regularization.

[0030] As a further preferred embodiment, the orthogonal projection matrix of the orthogonal renderer is:

[0031] in, It is an orthogonal projection matrix; Near-plane distance; and These are the right and left boundaries near the plane; and These are the upper and lower boundaries near the plane; The distance is the distance to the far plane; P[3,2]=0 indicates no perspective division.

[0032] As a further preferred embodiment, the covariance Jacobian matrix of the orthogonal renderer is:

[0033] in, The covariance Jacobian matrix; and The pixel focal length of the camera in the X and Y directions; 0 indicates no depth dependency.

[0034] In a second aspect, the present invention provides an automatic indoor orthophoto generation system based on a Gaussian splash and diffusion model, for implementing the method described in any one of the above, comprising: The image acquisition and sparse reconstruction module is used to extract sparse 3D point clouds and camera poses from multi-view images or videos of indoor scenes using the structure-reconstruction-motion method, including the camera intrinsic and extrinsic parameters, sparse 3D point coordinates and their two-dimensional observation relationships for each image. The camera position plane estimation, iterative ground extraction and coordinate system normalization module is used to perform iterative ground extraction and coordinate system alignment driven by the camera position prior based on the sparse 3D point cloud and camera pose, so as to obtain the aligned sparse 3D point cloud and camera pose. The virtual camera generation module is used to calculate the percentile bounding box of the aligned sparse 3D point cloud, and to uniformly sample the pose of the virtual camera in a grid-like pattern within the percentile bounding box to obtain a virtual orthophoto camera. The depth estimation and calibration module is used to perform monocular depth estimation and scale calibration on the multi-view images, calculate the depth scale and offset of each image, and obtain the calibrated inverse depth map. The perspective projection training module is used to train the three-dimensional Gaussian splash model based on the calibrated inverse depth map, sparse three-dimensional point cloud and camera pose, so as to obtain the trained three-dimensional Gaussian splash model. The orthogonal projection rendering module is used to perform orthogonal projection orthogonal rendering on the virtual orthogonal camera based on the three-dimensional Gaussian splash model using an orthogonal renderer to obtain an orthogonal image. The diffusion model repair module is used to perform quality repair on the orthophoto image based on a pre-trained single-step diffusion model to obtain a repaired orthophoto image. The stitching module is used to stitch the repaired orthophotos using the SIFT feature matching algorithm to obtain the final complete indoor orthophoto plan.

[0035] The method and system for automatically generating indoor orthophotos based on Gaussian splash and diffusion models provided by this invention have the following beneficial effects: This invention addresses shortcomings in the field of 3D reconstruction technology, such as inconsistencies between 3DGS perspective projection training and orthographic projection rendering, the lack of physical meaning in the coordinate system of sparse point clouds generated by SfM, and insufficient automatic coordinate system alignment and end-to-end automated integration. It achieves a fully automated process from multi-view indoor images to an orthographic floor plan by decoupling perspective and orthographic dual projection, using a camera position plane to guide RANSAC algorithm iteration to exclude walls and extract the ground, coordinate system standardization, and automatic generation of a virtual orthographic camera. Specifically, it includes: SfM reconstruction, SVD fitting of camera position reference normals, RANSAC iteration to exclude walls and extract the ground, Rodrigues rotation plus Hough line segment alignment of the coordinate system, 5×5 virtual orthographic camera generation, switching between perspective projection training and orthographic projection rendering modules, diffusion model repair, and SIFT feature extraction and image stitching.

[0036] Specifically, this invention resolves the core contradiction of inconsistent projection modes in the 3DGS framework through a dual-projection decoupling mechanism of perspective projection training and orthographic projection rendering; the perspective projection Jacobian matrix... Include Scaling and Taylor correction term ensures accurate fitting of the geometry of the input perspective photograph during training; orthogonal projection Jacobian matrix. It degenerates into a constant matrix, enabling the rendering stage to generate orthophotos without perspective distortion.

[0037] This invention employs an iterative ground extraction method driven by camera position priors, utilizing the natural constraints of the photographer's movement on the ground and SVD camera plane fitting, combined with the normal dot product criterion in RANSAC iteration. It reliably distinguishes between the ground and walls, and automatically differentiates between the ground and ceiling by comparing the vertical distribution span of point clouds at low and high locations, thus solving the problem that traditional methods easily misidentify large areas of walls as the ground in indoor scenes.

[0038] This invention searches for the scoring function score in orthogonal directions. Maximize the total support of line segments for orthogonal directions and automatically determine the main direction angle of the room. This aligns the wall orientation with the coordinate axes.

[0039] This invention utilizes a single-step diffusion model to perform quality restoration on orthogonally rendered orthophotos in a single denoising step, restoring lost texture details and significantly improving the clarity and usability of the final orthophoto plan. Furthermore, this invention achieves fully automated end-to-end processing from multi-view photographs to indoor orthophoto plans, eliminating the need for manual selection of viewpoints, manual ground labeling, or parameter adjustments, thus lowering the technical barrier to digitizing indoor spaces.

[0040] In summary, this invention solves the technical problems of inconsistent projection modes, rendering quality degradation, and automatic coordinate system alignment in existing methods by using a dual projection decoupling mechanism of perspective projection training and orthographic projection rendering, combined with automatic coordinate system alignment based on camera position prior and single-step diffusion model quality repair. It achieves end-to-end automatic generation from multi-view images to high-quality indoor orthographic floor plans, which can serve scenarios such as interior design and real estate display. Attached Figure Description

[0041] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings: Figure 1 This is a flowchart of the automatic generation method for indoor orthophotos based on Gaussian splash and diffusion models provided by the present invention; Figure 2 This is a flowchart of the automatic coordinate system alignment provided by the present invention; Figure 3 This is an orthogonal projection rendering image provided by the present invention; Figure 4 This is the image repaired using the single-step diffusion model provided by this invention; Figure 5 This is the spliced ​​orthophoto plane diagram provided by the present invention. Detailed Implementation

[0042] To provide a clearer understanding of the technical features, objectives, and effects of the present invention, specific embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0043] Example 1: This embodiment provides an automatic method for generating indoor orthophotos based on a Gaussian splash and diffusion model, including the following steps: S100: Based on multi-view images or videos of indoor scenes, extract sparse 3D point clouds and camera poses using the structure-reconstruction-motion method, including camera intrinsic and extrinsic parameters, sparse 3D point coordinates, and their 2D observation relationships for each image.

[0044] S200: Based on the sparse 3D point cloud and camera pose, perform iterative ground extraction and coordinate system alignment driven by camera position prior to obtain the aligned sparse 3D point cloud and camera pose.

[0045] Specifically, step S200 includes: S210: Extract the world coordinates of the camera center from the camera pose, and fit the camera position plane using the singular value decomposition method based on the world coordinates to obtain the camera plane normal. S220: Based on the camera plane normal and sparse 3D point cloud, a random sampling consensus algorithm is used to extract the plane to obtain the ground normal and ground reference points; S230: Based on the world coordinates of the camera center, the ground normal, and the ground reference point, obtain the aligned sparse 3D point cloud and the camera pose.

[0046] Specifically, step S23 includes: S231: Rotate the ground normal to the Z-axis of the world coordinate system to obtain the ground normal alignment matrix; S232: The sparse 3D point cloud is transformed using the ground normal alignment matrix to obtain the transformed point cloud. The transformed point cloud is projected onto the XY plane and subjected to density outlier filtering. The Canny edge detection method and probabilistic Hough transform are used to extract a set of line segments. The line segments in the set that are close to the outer contour of the point cloud are selected as edge line segments. Based on the edge line segments, the total support score of the line segments that are simultaneously aligned to the candidate angle direction and the orthogonal direction of the candidate angle direction is calculated. The angle with the highest score is selected as the main direction of the wall. S233: Obtain the wall orientation alignment matrix based on the main wall orientation. Obtain the final coordinate system transformation based on the wall orientation alignment matrix and the ground normal alignment matrix. Use the final coordinate system transformation to transform each 3D point and camera pose in the sparse 3D point cloud to obtain the aligned sparse 3D point cloud and camera pose.

[0047] Specifically, the formula for calculating the ground normal alignment matrix is ​​as follows:

[0048]

[0049]

[0050]

[0051] in, This is the ground normal alignment matrix; It is a 3×3 identity matrix; Let n_g be the axis of rotation [0,0,1]. It is the antisymmetric matrix of the cross product of the rotation axes; It is the dot product of the ground normal n_g and the real-world vertical upward unit vector [0,0,1]. Let ||v|| be the modulus of rotation. This is the ground normal.

[0052] Specifically, the formula for calculating the wall orientation alignment matrix is ​​as follows:

[0053] in, This is the wall orientation alignment matrix; The angle is the main direction of the wall.

[0054] Specifically, the formula for transforming the position of each 3D point in the sparse 3D point cloud with the camera pose is as follows:

[0055]

[0056]

[0057]

[0058] in, The aligned sparse 3D point cloud; The final coordinate system transformation is A2·A1; For 3D points in a sparse 3D point cloud; Used as a ground reference point; The alignment matrix is ​​the camera rotation matrix. This is the camera rotation matrix in the original SfM coordinate system; This is the transpose of the final coordinate system transformation; These are the transformed camera center coordinates; These are the camera center coordinates in the original SfM coordinate system.

[0059] S300: Calculate the percentile bounding box of the aligned sparse 3D point cloud, and uniformly sample the pose of the virtual camera in a grid-like pattern within the percentile bounding box to obtain a virtual orthophoto camera.

[0060] Specifically, the formula for calculating the virtual camera pose is:

[0061]

[0062]

[0063] in, Represents the virtual orthophoto camera in the i-th row and j-th column; and These represent the minimum and maximum values ​​of the scene along the X-axis; The grid size is preferably 5; and These represent the minimum and maximum values ​​of the scene along the Y-axis; The height ratio coefficient is preferably set to 0.8; This represents the maximum value of the scene along the Z-axis; The extrinsic translation vector; It is a vertically downward rotation matrix; The rotation quaternion is [0,1,0,0].

[0064] S400: Perform monocular depth estimation and scale calibration on the multi-view images, calculate the depth scale and offset of each image, and obtain the calibrated inverse depth map.

[0065] Specifically, the formula for calculating the inverse depth map is:

[0066]

[0067]

[0068] in, This is an inverse depth map; For depth scale; This is the depth value estimated by a single camera. For offset; This represents the mean absolute deviation of the monocular depth. This represents the mean absolute deviation of the monocular depth. It is a median function; This represents the mean absolute deviation of the COLMAP depth.

[0069] S500: The 3D Gaussian splash model is trained based on the calibrated inverse depth map, sparse 3D point cloud and camera pose to obtain a trained 3D Gaussian splash model.

[0070] Specifically, the formula for projecting the three-dimensional Gaussian covariance of the three-dimensional Gaussian splash model onto the two-dimensional screen space is:

[0071] in, The Gaussian covariance matrix in two-dimensional screen space; For perspective projection Jacobian matrix; This represents the rotated portion of the view matrix; The Gaussian covariance matrix in three-dimensional space; superscript This represents the transpose of a matrix.

[0072] Specifically, the Jacobian matrix of the perspective projection of the three-dimensional Gaussian splash model is:

[0073] in, For perspective projection, the Jacobian matrix is ​​used. and The pixel focal length of the camera in the X and Y directions; The depth coordinates of the center of the Gaussian ellipsoid in the camera coordinate system; and The x and y coordinates of the center of the Gaussian ellipsoid in the camera coordinate system; The Taylor expansion correction term in the X direction; This is the Taylor expansion correction term in the Y direction; 0 indicates no projection contribution.

[0074] Specifically, the perspective projection matrix of the three-dimensional Gaussian splash model is:

[0075] in, For perspective projection matrix; Near-plane distance; and These are the right and left boundaries near the plane; and These are the upper and lower boundaries near the plane; The distance is the distance to the far plane; P[3,2]=1 indicates the existence of perspective division.

[0076] Specifically, the training loss function of the three-dimensional Gaussian splash model is:

[0077] in, For pixel-by-pixel L1 loss; These are the weighting coefficients for the SSIM loss; For structural similarity loss; For depth regularization loss weights; This represents the loss for deep regularization.

[0078] S600: Based on the three-dimensional Gaussian splash model, the virtual orthophoto camera is orthogonally projected and orthophoto rendered using an orthogonal renderer to obtain an orthophoto image.

[0079] Specifically, the orthogonal projection matrix of the orthogonal renderer is:

[0080] in, It is an orthogonal projection matrix; Near-plane distance; and These are the right and left boundaries near the plane; and These are the upper and lower boundaries near the plane; The distance is the distance to the far plane; P[3,2]=0 indicates no perspective division.

[0081] Specifically, the covariance Jacobian matrix of the orthogonal renderer is:

[0082] in, The covariance Jacobian matrix; and The pixel focal length of the camera in the X and Y directions; 0 indicates no depth dependency.

[0083] S700: The orthophoto image is restored by performing quality restoration on the orthophoto image using a pre-trained single-step diffusion model to obtain the restored orthophoto image; S800: The repaired orthophoto image is stitched together using the SIFT feature matching algorithm to obtain the final complete indoor orthophoto plan.

[0084] Example 2: This embodiment provides an automatic indoor orthophoto generation system based on a Gaussian splash and diffusion model, used to implement the method described in any one of the above embodiments, including: The image acquisition and sparse reconstruction module is used to extract sparse 3D point clouds and camera poses from multi-view images or videos of indoor scenes using the structure-reconstruction-motion method, including the camera intrinsic and extrinsic parameters, sparse 3D point coordinates and their two-dimensional observation relationships for each image. The camera position plane estimation, iterative ground extraction and coordinate system normalization module is used to perform iterative ground extraction and coordinate system alignment driven by the camera position prior based on the sparse 3D point cloud and camera pose, so as to obtain the aligned sparse 3D point cloud and camera pose. The virtual camera generation module is used to calculate the percentile bounding box of the aligned sparse 3D point cloud, and to uniformly sample the pose of the virtual camera in a grid-like pattern within the percentile bounding box to obtain a virtual orthophoto camera. The depth estimation and calibration module is used to perform monocular depth estimation and scale calibration on the multi-view images, calculate the depth scale and offset of each image, and obtain the calibrated inverse depth map. The perspective projection training module is used to train the three-dimensional Gaussian splash model based on the calibrated inverse depth map, sparse three-dimensional point cloud and camera pose, so as to obtain the trained three-dimensional Gaussian splash model. The orthogonal projection rendering module is used to perform orthogonal projection orthogonal rendering on the virtual orthogonal camera based on the three-dimensional Gaussian splash model using an orthogonal renderer to obtain an orthogonal image. The diffusion model repair module is used to perform quality repair on the orthophoto image based on a pre-trained single-step diffusion model to obtain a repaired orthophoto image. The stitching module is used to stitch the repaired orthophotos using the SIFT feature matching algorithm to obtain the final complete indoor orthophoto plan.

[0085] Example 3: like Figure 1 As shown, this embodiment provides an automatic generation method for indoor orthophotos based on a Gaussian splash and diffusion model, including the following steps: Step 1: Image Acquisition and Sparse Reconstruction Multi-view images or videos of an indoor scene are captured using a mobile device. Sparse 3D point clouds and camera poses are obtained through Structure from Motion (SfM) technology, serving as initial input for subsequent steps. SfM reconstruction provides camera intrinsic and extrinsic parameters, sparse 3D point coordinates, and their 2D observation relationships for each image, stored in the sparse reconstruction directory.

[0086] Step 2: Iterative Ground Extraction and Coordinate System Alignment Driven by Camera Position Prior like Figure 2 The diagram shown is a flowchart of the automatic coordinate system alignment process. Step 2-1: Extract the world coordinates of the camera center from all SfM camera poses. For the i-th camera, its world coordinates are... Through the external parameter rotation matrix Translation vector calculate:

[0087] Step 2-2: Use the RANSAC algorithm to perform iterative plane extraction on the sparse point cloud.

[0088] Let the current point cloud set be P, the RANSAC iteration count r be adaptively set to r = max(200, 0.05 × |P|), and the distance threshold τ = 0.02. In each iteration, three points are randomly sampled, and their corresponding plane normals are calculated. and plane reference point Take the plane with the most interior points and calculate its normal. Normal to the camera plane dot product:

[0089] like A value >0.7 indicates that the plane's normal is parallel to the camera plane's normal, classifying it as either ground or ceiling, and stopping the iteration; otherwise, it's classified as a wall, and the interior points of this plane are removed from the point cloud before continuing the search, with a maximum of 5 iterations. For identified ground / ceiling candidate planes, the point cloud is aligned to the Z-axis using a temporary rotation matrix, and the vertical height distribution percentiles are calculated: .like This is the ceiling; flip the normal. for Otherwise, it is the ground. Obtain the ground normal. and ground reference points .

[0090] Steps 2-3: Apply Rodrigues' rotation formula to the ground normal. Rotate to the world coordinate system Z-axis [0, 0, 1]. Let the axis of rotation be... , , Then the ground normal alignment matrix for:

[0091] in Let I be the antisymmetric cross product matrix of vector v, and let I be the 3×3 identity matrix.

[0092] The transformed point cloud is projected onto the XY plane, and after density outlier filtering, Canny edge detection and probabilistic Hough transform are applied to extract a set of line segments. Line segments close to the outer contour of the point cloud are selected as edge segments.

[0093] For each candidate angle Calculate and align to simultaneously direction and The total support score for line segments in orthogonal directions (score) Select the angle with the highest score. As the primary orientation of the wall. Wall orientation alignment matrix. Rotation about the Z-axis:

[0094] Final coordinate system transformation Apply transformations to each 3D point and camera pose: , ,

[0095] Step 3: Virtual Orthophoto Camera Generation Calculate the percentile bounding box of the scene point cloud in a normalized coordinate system. Uniformly sample the poses of N×N grid-like virtual cameras (N=5) within the bounding box. For the virtual camera in the i-th row and j-th column:

[0096] Where α=0.8 is the height ratio coefficient.

[0097] Each virtual camera faces vertically downwards, rotated by a quaternion. extrinsic translation vector Write the virtual camera pose to the sparse reconstruction output directory.

[0098] Step 4: Monocular Depth Estimation and Scale Calibration Dense depth inference is performed on each input image using a deep learning monocular depth estimation model. The depth scale and offset for each image are calculated by comparing the statistical distributions of the SfM sparse depth and the monocular dense depth at corresponding pixel locations.

[0099] Calibrated inverse depth map This provides in-depth supervision signals for subsequent training.

[0100] Step 5: Perspective Projection 3DGS Training Scene training is performed using the 3DGS framework. The scene is represented as a set of 3D Gaussian ellipsoids, and the renderer uses perspective projection to maintain consistency with the pinhole camera model of the input photograph. The formula for projecting the 3D Gaussian covariance onto the 2D screen space is:

[0101] Where W is the rotation part of the view matrix, and J is the perspective projection Jacobian matrix, which includes depth-dependent scaling terms and Taylor expansion correction terms:

[0102] The perspective projection matrix P_persp is:

[0103] The training loss function is Among them, depth loss weight The Gaussian primitives decay exponentially from the initial value to the final value. During training, Gaussian primitives are dynamically densified and pruned based on an opacity threshold.

[0104] Step 6: Orthographic Projection and Rendering Load the trained 3DGS model and switch to the independent orthographic projection CUDA rasterization module for rendering. Orthographic projection matrix. Covariance Jacobian Matrix They are respectively:

[0105]

[0106] and The key difference is: No deep dependencies Scaling and The Taylor correction term degenerates into a constant matrix containing only the focal length parameter. All Gaussian elements of all depths are scaled proportionally to the screen to achieve a true parallel projection effect. Using this orthogonal renderer, all virtual cameras are rendered, resulting in 25 orthophoto images.

[0107] Step 7: Single-step diffusion model quality repair Each orthophoto rendered orthophoto is input into a pre-trained single-step diffusion model. The input image is encoded as a latent variable. Denoising process after a single time step t ,in To denoise the noise predicted by UNet, c represents the text prompt embedding. Inference parameters are set as follows: number of inference steps = 1, time step t =

[199] , and guiding coefficient = 0.0. The denoised latent variables are reconstructed into the output image using a VAE decoder. The single-step diffusion model effectively removes edge blur and floating artifacts in orthogonally rendered orthophotos.

[0108] Step 8: Orthophoto stitching The SIFT feature matching algorithm was used to stitch together all 25 restored orthophotos. First, five orthophotos per row were horizontally stitched together to form horizontal strips, generating five strips. Then, all horizontal strips were rotated 90 degrees and vertically stitched together. Finally, the images were rotated 90 degrees in the opposite direction to obtain the final complete indoor orthophoto plan. .

[0109] An automatic indoor orthophoto generation system based on Gaussian splash and diffusion models includes: a sparse reconstruction module, a coordinate alignment and virtual camera generation module, a depth estimation and calibration module, and a perspective training module (projection Jacobian matrix). Including deep dependencies Scaling and Taylor expansion correction term), orthogonal rendering module (using independent CUDA rasterizer, projection Jacobian matrix) It consists of a constant matrix containing only focal length parameters, a single-step diffusion model for artifact repair, and a SIFT feature matching image stitching module.

[0110] Example 4: Experiment on generating orthophotos of indoor scenes Table 1 shows the main experimental setup for this embodiment; Table 1: Experimental Relevant Parameters

[0111] Step 1: Take approximately 200-800 photos in the room using a mobile phone or other camera, ensuring sufficient overlap between adjacent photos. Run COLMAP for SfM reconstruction to obtain sparse point clouds and camera pose, storing them in the sparse / 0 directory.

[0112] Step 2: Extract the center position of all cameras The decentralized camera position matrix is ​​decomposed using SVD, and the eigenvector corresponding to the smallest eigenvalue is taken as the camera plane normal. RANSAC iterative plane extraction is performed on the point cloud, with the number of iterations r = max(200, 0.05 × |P|) and a distance threshold τ = 0.02. The extracted plane normal and... Comparison: If Determined to be either the ground or the ceiling, through Compare and differentiate; otherwise, exclude interior points and continue the search (maximum 5 times). Align the ground normal to the Z-axis using Rodrigues rotation, and then determine the main direction of the wall using Hough segment orthogonal search. Final transformation It applies to all points and camera pose.

[0113] Step 3: Calculate the percentile bounding boxes of the aligned point cloud 25 virtual cameras are uniformly sampled within a 5×5 bounding box, with a height of [missing information]. 0.8 times, Oriented vertically downwards, external reference Write the virtual camera to the sparse / 1 directory.

[0114] Step 4: Perform depth inference on each input image using the ViT-Large encoder model of DepthAnythingV2 and save it as a 16-bit TIFF file. Calculate the depth scale and offset for each image by comparing the statistical distributions of COLMAP sparse depth and monocular dense depth.

[0115] Step 5: Train for 30,000 iterations using the 3DGS framework. Use the SparseAdam optimizer with an initial learning rate of 1.6e-4, SSIM weights of 0.2, and depth loss with initial weights of 1.0 that decay exponentially to 0.01. Limit the image resolution to 1600 pixels wide. The renderer uses a perspective projection Jacobian matrix.

[0116] Step 6: Load the trained 3DGS model and switch to the orthographic projection CUDA module. Orthographic projection matrix. Using a fixed viewport top=5, the orthogonal projection Jacobian matrix is:

[0117] like Figure 3 The image shown is a rendering of an orthographic projection. Step 7: Use the Difix single-step diffusion model to repair artifacts in the rendered image. Set the parameters to inference steps = 1, time step t =

[199] , and guiding coefficient = 0.0. VAE encoding → single-step denoising → VAE decoding, and output the deblurred image.

[0118] like Figure 4 The image shown is the result of single-step diffusion model repair. Step 8: Perform SIFT feature matching and stitching on the 25 restored images using a 5×5 grid. In the first stage, horizontal stitching is performed row-by-row to generate 5 horizontal strips. In the second stage, the images are rotated 90 degrees and then vertically stitched. The final orthophoto image is obtained by restoring the orientation by 90 degrees.

[0119] like Figure 5 The image shown is a composite orthophoto plane.

[0120] The entire process, from raw photos to the final orthophoto, takes approximately 30 minutes to process 200-800 photos of a typical indoor scene on an RTX 4090 GPU.

[0121] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of the present invention without departing from the spirit and scope of the claims. All of these forms are within the protection scope of the present invention.

Claims

1. A method for automatically generating indoor orthophotos based on a Gaussian splash and diffusion model, characterized in that, Includes the following steps: S100: Based on multi-view images or videos of indoor scenes, extract sparse 3D point clouds and camera poses using the structure-reconstruction-motion method, including camera intrinsic and extrinsic parameters, sparse 3D point coordinates and their 2D observation relationships for each image. S200: Based on the sparse 3D point cloud and camera pose, perform iterative ground extraction and coordinate system alignment driven by camera position prior to obtain the aligned sparse 3D point cloud and camera pose. S300: Calculate the percentile bounding box of the aligned sparse 3D point cloud, and uniformly sample the pose of the virtual camera in a grid-like pattern within the percentile bounding box to obtain a virtual orthophoto camera. S400: Perform monocular depth estimation and scale calibration on the multi-view images, calculate the depth scale and offset of each image, and obtain the calibrated inverse depth map; S500: The three-dimensional Gaussian splash model is trained based on the calibrated inverse depth map, sparse three-dimensional point cloud and camera pose to obtain a trained three-dimensional Gaussian splash model. S600: Based on the three-dimensional Gaussian splash model, the virtual orthophoto camera is orthophoto-rendered using an orthogonal renderer to obtain an orthophoto image. S700: The orthophoto image is restored by performing quality restoration on the orthophoto image using a pre-trained single-step diffusion model to obtain the restored orthophoto image; S800: The repaired orthophoto image is stitched together using the SIFT feature matching algorithm to obtain the final complete indoor orthophoto plan.

2. The method according to claim 1, characterized in that, Step S2 specifically includes: S210: Extract the world coordinates of the camera center from the camera pose, and fit the camera position plane using the singular value decomposition method based on the world coordinates to obtain the camera plane normal. S220: Based on the camera plane normal and sparse 3D point cloud, a random sampling consensus algorithm is used to extract the plane to obtain the ground normal and ground reference points; S230: Based on the world coordinates of the camera center, the ground normal, and the ground reference point, obtain the aligned sparse 3D point cloud and the camera pose.

3. The method according to claim 2, characterized in that, Step S23 specifically includes: S231: Rotate the ground normal to the Z-axis of the world coordinate system to obtain the ground normal alignment matrix; S232: The sparse 3D point cloud is transformed using the ground normal alignment matrix to obtain the transformed point cloud. The transformed point cloud is projected onto the XY plane and subjected to density outlier filtering. The Canny edge detection method and probabilistic Hough transform are used to extract a set of line segments. The line segments in the set that are close to the outer contour of the point cloud are selected as edge line segments. Based on the edge line segments, the total support score of the line segments that are simultaneously aligned to the candidate angle direction and the orthogonal direction of the candidate angle direction is calculated. The angle with the highest score is selected as the main direction of the wall. S233: Obtain the wall orientation alignment matrix based on the main wall orientation. Obtain the final coordinate system transformation based on the wall orientation alignment matrix and the ground normal alignment matrix. Use the final coordinate system transformation to transform each 3D point and camera pose in the sparse 3D point cloud to obtain the aligned sparse 3D point cloud and camera pose.

4. The method according to claim 1, characterized in that, The formula for calculating the virtual camera pose is: in, Represents the virtual orthophoto camera in the i-th row and j-th column; and These represent the minimum and maximum values ​​of the scene along the X-axis; This refers to the grid size; and These represent the minimum and maximum values ​​of the scene along the Y-axis; This is the height scaling factor; This represents the maximum value of the scene along the Z-axis; The extrinsic translation vector; It is a vertically downward rotation matrix; For rotational quaternions; superscript This represents the transpose of a matrix.

5. The method according to claim 1, characterized in that, The formula for calculating the inverse depth map is: in, This is an inverse depth map; For depth scale; This is the depth value estimated by a single camera. For offset; This represents the mean absolute deviation of the monocular depth. This represents the mean absolute deviation of the monocular depth. It is a median function; This represents the mean absolute deviation of the COLMAP depth.

6. The method according to claim 1, characterized in that, The formula for projecting the three-dimensional Gaussian covariance of the three-dimensional Gaussian splash model onto the two-dimensional screen space is: in, The Gaussian covariance matrix in two-dimensional screen space; For perspective projection Jacobian matrix; This represents the rotated portion of the view matrix; The Gaussian covariance matrix in three-dimensional space; superscript Represents the transpose of a matrix; The Jacobian matrix of the perspective projection of the three-dimensional Gaussian splash model is: in, For perspective projection, the Jacobian matrix is ​​used. and The pixel focal length of the camera in the X and Y directions; The depth coordinates of the center of the Gaussian ellipsoid in the camera coordinate system; and The x and y coordinates of the center of the Gaussian ellipsoid in the camera coordinate system; The Taylor expansion correction term in the X direction; This is the Taylor expansion correction term in the Y direction; 0 indicates no projection contribution. The perspective projection matrix of the three-dimensional Gaussian splash model is: in, For perspective projection matrix; Near-plane distance; and These are the right and left boundaries near the plane; and These are the upper and lower boundaries near the plane; This represents the distance to the far plane.

7. The method according to claim 1, characterized in that, The training loss function of the three-dimensional Gaussian splash model is: in, For training the loss function; For pixel-by-pixel L1 loss; These are the weighting coefficients for the SSIM loss; For structural similarity loss; For depth regularization loss weights; This represents the loss for deep regularization.

8. The method according to claim 1, characterized in that, The orthogonal projection matrix of the orthogonal renderer is: in, It is an orthogonal projection matrix; Near-plane distance; and These are the right and left boundaries near the plane; and These are the upper and lower boundaries near the plane; This represents the distance to the far plane.

9. The method according to claim 1, characterized in that, The covariance Jacobian matrix of the orthogonal renderer is: in, The covariance Jacobian matrix; and The pixel focal length of the camera in the X and Y directions; 0 indicates no depth dependency.

10. An automatic indoor orthophoto generation system based on a Gaussian splash and diffusion model, characterized in that, The system includes: The image acquisition and sparse reconstruction module is used to extract sparse 3D point clouds and camera poses from multi-view images or videos of indoor scenes using the structure-reconstruction-motion method, including the camera intrinsic and extrinsic parameters, sparse 3D point coordinates and their two-dimensional observation relationships for each image. The camera position plane estimation, iterative ground extraction and coordinate system normalization module is used to perform iterative ground extraction and coordinate system alignment driven by the camera position prior based on the sparse 3D point cloud and camera pose, so as to obtain the aligned sparse 3D point cloud and camera pose. The virtual camera generation module is used to calculate the percentile bounding box of the aligned sparse 3D point cloud, and to uniformly sample the pose of the virtual camera in a grid-like pattern within the percentile bounding box to obtain a virtual orthophoto camera. The depth estimation and calibration module is used to perform monocular depth estimation and scale calibration on the multi-view images, calculate the depth scale and offset of each image, and obtain the calibrated inverse depth map. The perspective projection training module is used to train the three-dimensional Gaussian splash model based on the calibrated inverse depth map, sparse three-dimensional point cloud and camera pose, so as to obtain the trained three-dimensional Gaussian splash model. The orthogonal projection rendering module is used to perform orthogonal projection orthogonal rendering on the virtual orthogonal camera based on the three-dimensional Gaussian splash model using an orthogonal renderer to obtain an orthogonal image. The diffusion model repair module is used to perform quality repair on the orthophoto image based on a pre-trained single-step diffusion model to obtain a repaired orthophoto image. The stitching module is used to stitch the repaired orthophotos using the SIFT feature matching algorithm to obtain the final complete indoor orthophoto plan.

Images