A method and device for rendering a three-dimensional Gaussian sputtering enhanced oblique photography scene

Abstract

The present invention provides a method and apparatus for rendering an oblique photography scene enhanced by three-dimensional Gaussian sputtering. The method includes: a preliminary calculation step of oblique photogrammetry; a classification and extraction step of point clouds; a step of forming image recombination and classification data packets; a data merging and separation rendering step; and a step of constructing and implementing a free view. The method for rendering an oblique photography scene enhanced by three-dimensional Gaussian sputtering according to the present invention is mainly based on traditional oblique production technology and point cloud classification and segmentation technology, uses 3D Gaussian Splatting to perform Gaussian modeling on the resulting data, and visually renders the modeled data. Compared with traditional oblique data production, the visualization rendering link is improved, situations such as model adhesion and image distortion are reduced, and the data is optimized and rendered in terms of visualization effect.

Description

Technical Field

[0001] The present invention relates to the field of real - scene three - dimensional technology, and particularly to a method and device for enhancing the rendering of an oblique photography scene based on three - dimensional Gaussian sputtering. Background Art

[0002] Oblique photography is a technology for photographing ground scenes at an oblique angle, which is widely used in fields such as geographic information systems (GIS), urban modeling, and three - dimensional reconstruction. In the visualization of real - scene three - dimensional, Mesh data is usually used as the rendering object. However, in the actual process of generating Mesh data, due to reasons such as fewer image angles and insufficient coverage, the model may have problems such as missing parts and adhesion.

[0003] Chinese patent document CN118172459A discloses an oblique photography model rendering method and a rendering system. Based on the parsed geometric information, texture information, and coordinate information, through diverse technical processes, geometric mapping and texture mapping of the model are realized. Here, the Monte Carlo path tracing algorithm is introduced to realize the natural lighting simulation of the oblique photography model.

[0004] Chinese patent document CN117876552A discloses an oblique photography three - dimensional model rendering method, a computing device, and a storage medium. The client pre - downloads a WASM library file containing an OSGB file parser. When it is necessary to render a target OSGB model, the client sends an OSGB rendering request to the server. The server directly sends the tile file of the target OSGB model to the client in the OSGB format according to the OSGB rendering request, and the client uses the OSGB file parser to parse the tile file to realize rendering.

[0005] Traditional oblique photography rendering methods often have problems such as image distortion, missing details, and insufficient sense of scene depth, resulting in poor visualization effects; moreover, some of the solutions are rather cumbersome and costly. Summary of the Invention

[0006] To solve the above problems, the object of the present invention is to provide a method for enhancing the rendering of an oblique photography scene based on three - dimensional Gaussian sputtering. By utilizing the characteristics of the Gaussian distribution, the oblique photography scene is optimized to solve problems such as adhesion, distortion, and single - layer skin of Mesh data in the oblique photography scene, improve the rendering quality and visual effect, and enhance the application of data in real - scene three - dimensional.

[0007] The present invention provides a method for enhancing the rendering of an oblique photography scene based on three - dimensional Gaussian sputtering, and the method includes:

[0008] Initial calculation steps of oblique photogrammetry: Using oblique photogrammetry software, perform aerial triangulation calculation on the input oblique photographic images, associate the images with their position information, and output image data containing interior and exterior orientation elements and their corresponding point cloud data;

[0009] Classification and extraction steps of point cloud: By analyzing point cloud features and applying feature extraction and semantic understanding, classify different elements in the point cloud data;

[0010] Steps of image recombination and formation of classification data packets: Use the bundle adjustment method to process the point cloud data, and obtain the related images and their ranges;

[0011] Gaussian modeling and data output steps: Package and output the point cloud and radiation energy field to form a Gaussian model;

[0012] Data merging and separated rendering steps: Perform merging processing on the packaged and output data to form separated rendering of individual objects;

[0013] Steps of construction and implementation of free view: Construct a free view for the entire 3D scene to form a state of free viewing.

[0014] Optionally, the classification and extraction steps of the point cloud include:

[0015] Data preprocessing: Include steps of denoising, downsampling, alignment, and normalization to improve calculation efficiency and classification accuracy;

[0016] Feature extraction: Extract features from the preprocessed point cloud;

[0017] Model training: Select a suitable machine learning or deep learning model and use labeled point cloud data for training;

[0018] Model evaluation and optimization: Use the validation set to evaluate the model performance and adjust the model parameters and features as needed;

[0019] Classification and post-processing: Apply the trained model to new data for classification.

[0020] Optionally, the steps of image recombination and formation of classification data packets include:

[0021] Based on the classified point cloud data, use the bundle adjustment method and the idea of inverse projection to project the points in the 3D world onto the 2D image plane;

[0022] Based on the point cloud range and the involved image range, merge and organize the data, and output images and point cloud data of a fixed size.

[0023] Optionally, the Gaussian modeling and data output steps include:

[0024] Perform local three-dimensional reconstruction on the photos in the classified data, calculate the pose of the reconstructed photos and output them;

[0025] Perform Gaussian rendering calculation based on the photo pose and point cloud data, and package and name the output model.

[0026] The present invention also provides an apparatus for enhancing the rendering of an oblique photography scene based on three-dimensional Gaussian sputtering, including one or more processors and a non-transitory computer-readable storage medium storing program instructions. When the one or more processors execute the program instructions, the one or more processors are used to implement the method.

[0027] The present invention also provides a computing device, characterized in that the computing device includes the apparatus.

[0028] The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium is used to store program code, and the program code is used to execute the method.

[0029] The present invention designs a method and apparatus for enhancing the rendering of an oblique photography scene based on three-dimensional Gaussian sputtering. The method of the present invention uses 3D gaussian Splatting to perform visual rendering on traditional oblique results, mainly based on traditional oblique production technology and point cloud classification and segmentation technology. It uses 3Dgaussian Splatting to perform Gaussian modeling on the result data and perform visual rendering on the modeled data. By using the characteristics of the Gaussian distribution, the derived points and radiation energy fields can be well rendered into a high-quality scene, improving the visual rendering link compared with traditional oblique data production, reducing situations such as model adhesion and image distortion, and optimizing the rendering of data in terms of visual effects. Compared with 3Dgaussian Splatting, which mainly solves the input of image data with any number of pixels, mainly using point cloud data, combining with the image pose to obtain the pixels related to the image, reducing the data input volume of 3D gaussian Splatting and reducing the consumption of hardware. It solves the problems of the implicit expression of Nerf being not intuitive, having low controllability, low training efficiency, and being difficult to perform high-resolution real-time rendering. It is beneficial to the better application of data in the construction of real scene three dimensions. By first segmenting and then modeling and rendering, it solves the problem that the model cannot be analyzed, which is beneficial to the in-depth application of data in actual business.

[0030] From the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will become more clear about the above and other objects, advantages and features of the present invention. Description of the Drawings

[0031] By reading the following detailed description of the preferred embodiments, various other advantages and benefits will become clear to those of ordinary skill in the art. The drawings are only for the purpose of showing the preferred embodiments and are not considered to be a limitation of the present invention. Also, throughout the drawings, the same reference numerals are used to denote the same components. In the drawings:

[0032] Figure 1 is a schematic flowchart of the method for rendering a three-dimensional Gaussian sputtering enhanced oblique photography scene according to an embodiment of the present invention;

[0033] Figure 2 is a schematic diagram of the specific process of rendering a three-dimensional Gaussian sputtering enhanced oblique photography scene according to an embodiment of the present invention;

[0034] Figure 3 is a schematic diagram of a sample of the final results included in a folder according to an embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of rendering a complex scene by mixing multiple Gaussian points according to an embodiment of the present invention;

[0036] Figure 5a 、 Figure 5b are respectively the effect diagram and the traditional mesh close-up effect diagram using an embodiment of the present invention;

[0037] Figure 6a 、 Figure 6b are respectively the effect diagram and the traditional mesh close-up effect diagram using another embodiment of the present invention. Detailed Embodiments

[0038] The embodiments of the present invention will be described below with reference to the drawings. Those skilled in the art should understand that these embodiments are only used to explain the present invention and are not restrictive.

[0039] The present invention provides a method for rendering a three-dimensional Gaussian sputtering enhanced oblique photography scene. Combining Figures 1 to 2 it can be seen that the specific process is recorded as follows.

[0040] Step 1: Preliminary calculations for oblique photogrammetry. First, use professional oblique photogrammetry software, such as ContextCapture or PhotoMesh, to perform aerotriangulation calculations on the input images. Optionally, first, import the data into the oblique photogrammetry software and use the software to perform relative orientation on the oblique images; second, manually mark points on the oblique images after relative orientation, and perform absolute orientation on the images after point marking; then, perform overall area adjustment calculations based on the image data after absolute orientation, and perform position and attitude correction and generation of point cloud data based on the adjusted results; finally, output the corrected photos, position and attitude, and point cloud data. Among them, relative orientation can be performed based on the matching of homologous feature pixels of the images, and absolute orientation can be performed by manually adding control points.

[0041] This process associates the images with their position information, thereby outputting image data containing internal and external orientation elements and their corresponding point cloud data. The goal of this stage is to generate high-quality basic data to lay a solid foundation for subsequent processing.

[0042] Step 2: Classification and extraction of point clouds. After obtaining the preliminary point cloud data, the next step is to perform classification processing. By analyzing the point cloud features and using feature extraction and semantic understanding, such as machine learning methods like support vector machine (SVM) and multi-layer perceptron, different elements in the point cloud data are classified, such as buildings, trees, roads, etc.

[0043] In an alternative embodiment of the present invention, the classification and extraction of point clouds can be achieved based on geometric features such as surface normal vectors, curvatures, point densities, and height changes of points. The above features can help identify different types of objects. For example, trees usually have a higher curvature, while building surfaces are usually flat. Use shape descriptors such as voxel grids and concavity to perform local or global shape analysis on the point cloud. In point clouds with RGB information or reflection intensity information, color and intensity features can also provide valuable information. Then, point cloud classification is achieved based on the extracted features.

[0044] The point cloud classification process can be as follows: Data preprocessing: including steps such as denoising, downsampling, alignment, and normalization to improve calculation efficiency and classification accuracy. Feature extraction: Extract geometric, statistical, and other features from the preprocessed point cloud. Model training: Select a suitable machine learning or deep learning model and use labeled point cloud data for training. Model evaluation and optimization: Evaluate the model performance using a validation set and adjust the model parameters and features as needed. Classification and post-processing: Apply the trained model to new data for classification, and further post-processing steps may be required, such as smoothing and filling holes of misclassified areas.

[0045] In practical applications, when performing feature extraction, shape features, histogram features, amplitude features, and other features can be extracted through image feature algorithms such as HOG features, LBP features, and Haar features. Then, based on the features, the data can be automatically classified with labels.

[0046] This step not only improves the accuracy of data processing but also provides necessary classification information for subsequent image recombination.

[0047] Step 3: Image recombination and formation of classification data packets. After the point cloud classification is completed, the beam method is used to process the point cloud data to obtain the related images and their ranges. Based on the classified point cloud data, using the idea of reverse projection through the beam method, the points in the three-dimensional world are projected onto the two-dimensional image plane. According to the point cloud range and the image range involved, the data is merged and sorted, and images and point cloud data of a fixed size are output. Through this method, the images can be recombined to form a complete data packet containing classification information. This process ensures the accuracy and consistency of the images and lays a foundation for efficient data packaging.

[0048] Step 4: Gaussian modeling and data output. Based on the generated classification data packet above, Gaussian modeling work is carried out. Optionally, the specific process of Gaussian modeling and data output can include: First, perform local three-dimensional reconstruction on the photos in the classification data, then calculate and output the poses of the reconstructed photos. Secondly, perform Gaussian rendering calculation based on the photo poses and point cloud data. Finally, package and name the output model. In practical applications, local modeling output and related pose calculations can be performed based on the modeling method in Step 1. Then, Gaussian rendering output is performed to generate a label folder data packet for visual rendering. It should be noted that the finally generated classification data packet is named with a label + serial number, and the final result samples included in the folder are as Figure 3 shown.

[0049] In this step, the point cloud and the radiation energy field are packaged and output to form a Gaussian model. This modeling process is a key link for achieving fine rendering, which can effectively capture the lighting characteristics and spatial relationships of the three-dimensional scene and provide important support for subsequent rendering.

[0050] Step 5: Data merging and separated rendering. After the Gaussian modeling is completed, the packaged and output data is then merged. By parsing the packaged file in Step 4, a rendering package corresponding to the name and point cloud is constructed, and the constructed data is object-labeled to facilitate separated rendering of individual objects in the entire scene. This process can ensure the independence of each object during rendering, making the finally presented three-dimensional scene more realistic and intuitive.

[0051] Step 6: Construction and implementation of free viewpoints. Finally, construct free viewpoints for the entire 3D scene to form a state where it can be freely viewed. Use the Gaussian function to simulate the propagation and reflection of light. Configure the parameters of the virtual camera, including the camera axis R, field of view Fov, camera position T, direction, focal length, etc. Through the viewpoints of observers at different angles, render the image state, thereby constructing a Gaussian rendering image that meets the free angle.

[0052] When using the Gaussian function to simulate the propagation and reflection of light, each point in the point cloud is usually parameterized into only one Gaussian kernel, described by the mean (position), covariance (shape), color, etc., for Gaussian parameterization of the point cloud. Project each Gaussian point onto a 2D image to obtain the color and transparency values of each pixel. Complex scenes can be rendered by mixing multiple Gaussian points, as shown in Figure 4 shown.

[0053] This step allows users to observe the 3D scene from multiple angles through dynamic viewpoint switching, enhancing the interactive experience and visual effects. The implementation of this technology not only improves the user's immersion but also greatly enriches the display effect of the scene. See the comparison diagrams of the effects in Figure 5a , Figure 5b and Figure 6a and Figure 6b respectively.

[0054] The embodiments of the present invention mainly utilize traditional oblique data production techniques. By acquiring calibration images and generating sparse point clouds, classifying the point clouds, and rendering and merging the classified data, an innovative processing and rendering method is formed. This method involves multiple key steps, including classification and segmentation of point clouds, image recombination, data packaging, and finally Gaussian modeling and rendering. Through the following flowcharts, efficient and accurate 3D scene reconstruction and display can be achieved.

[0055] The embodiments of the present invention also provide an apparatus for enhancing the rendering of an oblique photography scene based on three-dimensional Gaussian sputtering, including one or more processors and a non-transitory computer-readable storage medium storing program instructions. When the one or more processors execute the program instructions, the one or more processors are used to implement the method described above.

[0056] The embodiments of the present invention also provide a computing device, characterized in that the computing device includes the apparatus.

[0057] The embodiments of the present invention also provide a computer-readable storage medium, characterized in that the computer-readable storage medium is used to store program code, and the program code is used to execute the method described above.

[0058] The above are only the preferred embodiments of the present invention. The protection scope of the present invention is not limited to the above embodiments. All technical solutions falling within the concept of the present invention belong to the protection scope of the present invention. It should be noted that for those of ordinary skill in the art, several improvements and refinements made without departing from the principle of the present invention should also be regarded as within the protection scope of the present invention.

Claims

1. A method for rendering a scene based on three-dimensional Gaussian sputtering enhancement oblique photography, characterized in that: The method comprises: The preliminary calculation steps of oblique photogrammetry are as follows: using oblique photogrammetry software, perform spatial triangulation calculation on the input oblique photogrammetric image, associate the image with its position information, and output image data containing internal and external orientation elements and their corresponding point cloud data; Point cloud classification and extraction steps: By analyzing point cloud features, using feature extraction and semantic understanding, different elements in the point cloud data are classified; Image Reconstruction and Classification Data Packet Formation Steps: Process the point cloud data using the beam method to obtain the images related to it and their ranges; Gaussian modeling and data output steps: Pack the point cloud and radiation energy field and output them to form a Gaussian model; Data merging and separate rendering steps: merging the packaged output data to form separate rendering of individual objects; The steps of constructing and realizing free perspective are as follows: construct free perspective for the whole three-dimensional scene to form a free viewing state.

2. The method according to claim 1, characterized in that The point cloud classification and extraction step comprises: Data preprocessing: including denoising, downsampling, alignment, and normalization steps to improve computational efficiency and classification accuracy; Feature extraction: extract features from the preprocessed point cloud; Model training: Select an appropriate machine learning or deep learning model and train it using labeled point cloud data; Model evaluation and optimization: Use the validation set to evaluate model performance and adjust model parameters and features as needed; Classification and post-processing: Apply the trained model to new data for classification.

3. The method according to claim 1, characterized in that The steps of forming the image reorganization and classification data packets include: Based on the classified point cloud data, the beam method uses the idea of ​​reverse projection to project the points in the three-dimensional world onto the two-dimensional image plane. According to the point cloud range and the image range involved, the data is merged and sorted, and fixed-size images and point cloud data are output.

4. The method according to claim 1, characterized in that: The Gaussian modeling and data output steps include: Perform local 3D reconstruction on the photos in the classified data, and calculate and output the posture of the reconstructed photos; Gaussian rendering calculation is performed based on the photo pose and point cloud data, and the output model is packaged and named.

5. A device for rendering a scene based on three-dimensional Gaussian sputtering enhanced oblique photography, comprising one or more processors and a non-temporary computer-readable storage medium storing program instructions. When the one or more processors execute the program instructions, the one or more processors are used to implement the method according to any one of claims 1-4.

6. A computing device, characterized in that The computing device comprises the apparatus of claim 5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the method according to any one of claims 1 to 4.

Images