Method and device for enhancing oblique photography scene rendering based on three-dimensional Gaussian sputtering

Abstract

The invention provides a method and a device for enhancing oblique photography scene rendering based on three-dimensional Gaussian sputtering. The method comprises the following steps: a preliminary calculation step of oblique photography measurement; a point cloud classification extraction step; an image recombination and classification data packet forming step; a data merging and separating rendering step; and constructing and realizing a free view angle. The method for enhancing oblique photography scene rendering based on three-dimensional Gaussian sputtering is mainly based on a traditional oblique production technology and a point cloud classification segmentation technology, Gaussian modeling is carried out on result data by utilizing 3D Gaussian Splitting, and visual rendering is carried out on modeling data. Compared with traditional skew data production, the method has the advantages that a visual rendering link is improved, the conditions of model adhesion, image distortion and the like are reduced, and data are optimized and rendered on the visual effect.

Description

Technical Field

[0001] The present invention relates to the field of real-scene three-dimensional technology, and in particular to a method and a device for rendering a scene based on three-dimensional Gaussian sputtering enhancement oblique photography. Background Art

[0002] Oblique photography is a technique for photographing ground scenes at an oblique angle, and is widely used in the fields of geographic information systems (GIS), urban modeling, and 3D reconstruction. In real-life 3D visualization, Mesh data is usually used as the rendering object. However, in the actual production of Mesh data, due to the limited number of image angles and insufficient coverage, the model may be missing or stuck.

[0003] Chinese patent document CN118172459A discloses a method and system for rendering an oblique photography model. Based on the analyzed geometric information, texture information, and coordinate information, a variety of technical processes are used to achieve model geometry mapping and texture mapping. The Monte Carlo path tracing algorithm is introduced to achieve natural lighting simulation of the oblique photography model.

[0004] Chinese patent document CN117876552A discloses an oblique photography three-dimensional model rendering method, computing device and storage medium, which pre-downloads a WASM library file containing an OSGB file parser through a client. When the target OSGB model needs to be rendered, the client sends an OSGB rendering request to the server. The server directly sends the tile file of the target OSGB model in OSGB format to the client according to the OSGB rendering request, and the client uses the OSGB file parser to parse the tile file to achieve rendering.

[0005] Traditional oblique photography rendering methods often have problems such as image distortion, lack of details and insufficient scene depth, resulting in poor visualization effects; moreover, some solutions are more cumbersome and costly. Summary of the invention

[0006] In order to solve the above problems, the purpose of the present invention is to provide a method for rendering oblique photography scenes based on three-dimensional Gaussian sputtering enhancement. By utilizing the characteristics of Gaussian distribution, the oblique photography scene is optimized, and the problems of adhesion, distortion and skinning of Mesh data in the oblique photography scene are solved, thereby improving the rendering quality and visual effects, and enhancing the application of data in real-life three-dimensional.

[0007] The present invention provides a method for rendering a scene based on three-dimensional Gaussian sputtering enhanced oblique photography, the method comprising:

[0008] The preliminary calculation steps of oblique photogrammetry are as follows: using oblique photogrammetry software, perform spatial triangulation calculation on the input oblique photogrammetric image, associate the image with its position information, and output image data containing internal and external orientation elements and their corresponding point cloud data;

[0009] Point cloud classification and extraction steps: By analyzing point cloud features, using feature extraction and semantic understanding, different elements in the point cloud data are classified;

[0010] Image Reconstruction and Classification Data Packet Formation Steps: Process the point cloud data using the beam method to obtain the images related to it and their ranges;

[0011] Gaussian modeling and data output steps: Pack the point cloud and radiation energy field and output them to form a Gaussian model;

[0012] Data merging and separate rendering steps: merging the packaged output data to form separate rendering of individual objects;

[0013] The steps of constructing and realizing free perspective are as follows: construct free perspective for the whole three-dimensional scene to form a free viewing state.

[0014] Optionally, the point cloud classification and extraction step includes:

[0015] Data preprocessing: including denoising, downsampling, alignment, and normalization steps to improve computational efficiency and classification accuracy;

[0016] Feature extraction: extract features from the preprocessed point cloud;

[0017] Model training: Select an appropriate machine learning or deep learning model and train it using labeled point cloud data;

[0018] Model evaluation and optimization: Use the validation set to evaluate model performance and adjust model parameters and features as needed;

[0019] Classification and post-processing: Apply the trained model to new data for classification.

[0020] Optionally, the step of forming the image reorganization and classification data packet includes:

[0021] Based on the classified point cloud data, the beam method uses the idea of ​​reverse projection to project the points in the three-dimensional world onto the two-dimensional image plane.

[0022] According to the point cloud range and the image range involved, the data is merged and sorted, and fixed-size images and point cloud data are output.

[0023] Optionally, the Gaussian modeling and data output steps include:

[0024] Perform local 3D reconstruction on the photos in the classified data, and calculate and output the posture of the reconstructed photos;

[0025] Gaussian rendering calculation is performed based on the photo pose and point cloud data, and the output model is packaged and named.

[0026] The present invention also provides a device for rendering a scene based on three-dimensional Gaussian sputtering enhanced oblique photography, comprising one or more processors and a non-temporary computer-readable storage medium storing program instructions. When the one or more processors execute the program instructions, the one or more processors are used to implement the described method.

[0027] The present invention also provides a computing device, characterized in that the computing device comprises the apparatus.

[0028] The present invention also provides a computer-readable storage medium, characterized in that the computer-readable storage medium is used to store program code, and the program code is used to execute the described method.

[0029] The present invention designs a method and device for rendering scenes based on three-dimensional Gaussian sputtering enhanced oblique photography. The method of the present invention uses 3D gaussian Splatting to visualize the traditional oblique results. It is mainly based on traditional oblique production technology and point cloud classification and segmentation technology. 3D gaussian Splatting is used to perform Gaussian modeling on the results data, and the modeling data is visualized. Utilizing the characteristics of Gaussian distribution, the derived points and radiation energy fields can be well rendered into high-quality scenes. Compared with traditional oblique data production, the visualization rendering link is improved, the model adhesion and image distortion are reduced, and the data is optimized in terms of visualization effect. Compared with 3D gaussian Splatting, it mainly solves the input of image data with any number of pixels, mainly based on point cloud data, combined with image posture, to obtain image-related pixels, reduce the data input amount of 3D gaussian Splatting, and reduce the consumption of hardware. Solve the problems of Nerf implicit expression being not intuitive, low controllability, low training efficiency, and difficulty in high-resolution real-time rendering. It is conducive to better application of data in real-life three-dimensional construction. By first segmenting and then modeling and rendering, it solves the problem of unanalyzable models, which is conducive to the in-depth application of data in actual business.

[0030] Based on the following detailed description of specific embodiments of the present invention in conjunction with the accompanying drawings, those skilled in the art will become more aware of the above and other objects, advantages and features of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Moreover, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0032] Figure 1 is a schematic flow chart of a method for rendering a scene based on three-dimensional Gaussian sputtering enhanced oblique photography according to an embodiment of the present invention;

[0033] Figure 2 3. It is a schematic diagram of a specific process of rendering a scene based on three-dimensional Gaussian sputtering enhanced oblique photography according to an embodiment of the present invention;

[0034] Figure 3 It is a schematic diagram of the sample results finally contained in the folder of the embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of rendering a complex scene by mixing multiple Gaussian points according to an embodiment of the present invention;

[0036] Figure 5a , Figure 5b They are respectively the effect diagram using an embodiment of the present invention and the traditional mesh close-up effect diagram;

[0037] Figure 6a , Figure 6b They are respectively a rendering of another embodiment of the present invention and a traditional mesh close-up rendering. DETAILED DESCRIPTION

[0038] The embodiments of the present invention are described below in conjunction with the accompanying drawings. Those skilled in the art should understand that these embodiments are only used to explain the present invention and are not limiting.

[0039] The present invention provides a method for rendering a scene based on three-dimensional Gaussian sputtering enhanced oblique photography, combining Figure 1-2 It can be seen that the specific records are as follows.

[0040] Step 1: Preliminary calculation of oblique photogrammetry. First, use professional oblique photogrammetry software, such as ContextCapture or PhotoMesh, to perform empty triangulation calculation on the input image. Optionally, first, import the data into the oblique photogrammetry software, and use the oblique photogrammetry software to relatively orient the oblique image; secondly, manually puncture the oblique image after relative orientation, and absolutely orient the image after puncturing; then, perform overall regional adjustment calculation based on the image data after absolute orientation, and perform position and posture correction and point cloud data generation based on the adjustment results; finally, output the corrected photos, position and posture, and point cloud data. Among them, relative orientation can be performed based on the matching of the same-name feature pixels of the image, and absolute orientation can be performed by manually adding control points.

[0041] This process associates the image with its location information, thereby outputting image data containing internal and external orientation elements and their corresponding point cloud data. The goal of this stage is to generate high-quality basic data to lay a solid foundation for subsequent processing.

[0042] Step 2: Classification and extraction of point cloud. After obtaining the preliminary point cloud data, the next step is classification processing. By analyzing the point cloud features, using feature extraction and semantic understanding, such as support vector machine (SVM), multi-layer perceptron and other machine learning methods, different elements in the point cloud data are classified, such as buildings, trees, roads, etc.

[0043] In an optional embodiment of the present invention, classification and extraction of point clouds can be achieved based on geometric features such as surface normal vectors, curvature, point density, point height changes, etc. The above features can help identify different types of objects. For example, trees usually have higher curvatures, while building surfaces are usually flat. Shape descriptors such as voxel grids, convexity, etc. are used to perform local or overall shape analysis on point clouds. In point clouds with RGB information or reflection intensity information, color and intensity features can also provide valuable information. Point cloud classification is then achieved based on the extracted features.

[0044] The point cloud classification process can be as follows: Data preprocessing: including steps such as denoising, downsampling, alignment and normalization to improve computational efficiency and classification accuracy. Feature extraction: extract geometric, statistical and other features from the preprocessed point cloud. Model training: select a suitable machine learning or deep learning model and train it using labeled point cloud data. Model evaluation and optimization: evaluate model performance using a validation set and adjust model parameters and features as needed. Classification and post-processing: apply the trained model to new data for classification, which may require further post-processing steps such as smoothing, filling in holes of misclassified objects, etc.

[0045] In practical applications, when performing feature extraction, image feature algorithms such as HOG features, LBP features, Haar features, etc. can be used to extract other features such as shape features, histogram features, amplitude features, etc., and then the data can be automatically labeled based on the features.

[0046] This step not only improves the accuracy of data processing, but also provides necessary classification information for subsequent image reconstruction.

[0047] Step 3: Image reorganization and formation of classified data packets. After the point cloud classification is completed, the beam method is used to process the point cloud data to obtain the images and their ranges related to it. Based on the classified point cloud data, the beam method uses the idea of ​​reverse projection to project the points in the three-dimensional world onto the two-dimensional image plane. According to the point cloud range and the image range involved, the data is merged and sorted, and the images and point cloud data of fixed size are output. Through this method, the images can be reorganized to form a complete data packet containing classification information. This process ensures the accuracy and consistency of the images and lays the foundation for efficient data packaging.

[0048] Step 4: Gaussian modeling and data output. According to the classification data package generated above, Gaussian modeling work is carried out. Optionally, the specific process of Gaussian modeling and data output may include: first, local three-dimensional reconstruction of the photos in the classification data, then posture calculation and output of the reconstructed photos, secondly, Gaussian rendering calculation based on the photo posture and point cloud data, and finally packaging and naming the output model. In practical applications, posture calculation related to local modeling output can be performed based on the modeling method of step one. Then Gaussian rendering is performed to output the label folder data package for visual rendering. It should be noted that the finally generated classification data package is named with label + serial number, and the final sample results contained in the folder are as follows. Figure 3 shown.

[0049] In this step, the point cloud and the radiation energy field are packaged and output to form a Gaussian model. This modeling process is a key step in achieving fine rendering, which can effectively capture the lighting characteristics and spatial relationships of the three-dimensional scene and provide important support for subsequent rendering.

[0050] Step 5: Data merging and separate rendering. After completing Gaussian modeling, the packaged output data is merged. By parsing the packaged file in step 4, a rendering package with a name corresponding to the point cloud is constructed, and the constructed data is labeled with objects to form separate renderings of individual objects in the entire scene. This process can ensure the independence of each object during rendering, making the final 3D scene more realistic and intuitive.

[0051] Step 6: Construction and implementation of free perspective. Finally, construct a free perspective for the entire 3D scene to form a state that can be viewed freely. Use Gaussian function to simulate the propagation and reflection of light. Configure the parameters of the virtual camera, including the camera axis R, field of view Fov, camera position T, direction, focal length, etc. Render the image state through the viewpoints of observers at different angles, and thus construct a Gaussian rendering that satisfies the free perspective.

[0052] When using Gaussian functions to simulate the propagation and reflection of light, each point in the point cloud is usually parameterized as a Gaussian kernel, described by mean (position), covariance (shape) and color, etc., to perform Gaussian parameterization of the point cloud. Each Gaussian point is projected onto a 2D image to obtain the color and transparency value of each pixel. Complex scenes can be rendered by mixing multiple Gaussian points, combined with Figure 4 shown.

[0053] This step allows users to observe the three-dimensional scene from multiple angles through dynamic perspective switching, enhancing the interactive experience and visual effects. The implementation of this technology not only improves the user's sense of immersion, but also greatly enriches the display effect of the scene. Figure 5a , Figure 5b and Figure 6a and Figure 6b Effect comparison chart.

[0054] The embodiment of the present invention mainly utilizes the traditional tilt data production technology, obtains the calibration image and generates a sparse point cloud, classifies the point cloud data, and renders and merges the classified data, forming an innovative processing and rendering method. The method involves multiple key steps, including point cloud classification and segmentation, image reconstruction, data packaging, and final Gaussian modeling and rendering. Through the following flowchart, efficient and accurate three-dimensional scene reconstruction and display can be achieved.

[0055] An embodiment of the present invention also provides a device for rendering a scene based on three-dimensional Gaussian sputtering enhanced oblique photography, comprising one or more processors and a non-temporary computer-readable storage medium storing program instructions. When the one or more processors execute the program instructions, the one or more processors are used to implement the described method.

[0056] An embodiment of the present invention further provides a computing device, characterized in that the computing device includes the apparatus.

[0057] An embodiment of the present invention further provides a computer-readable storage medium, characterized in that the computer-readable storage medium is used to store program code, and the program code is used to execute the method described.

[0058] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A method for rendering a scene based on three-dimensional Gaussian sputtering enhancement oblique photography, characterized in that: The method comprises: The preliminary calculation steps of oblique photogrammetry are as follows: using oblique photogrammetry software, perform spatial triangulation calculation on the input oblique photogrammetric image, associate the image with its position information, and output image data containing internal and external orientation elements and their corresponding point cloud data; Point cloud classification and extraction steps: By analyzing point cloud features, using feature extraction and semantic understanding, different elements in the point cloud data are classified; Image Reconstruction and Classification Data Packet Formation Steps: Process the point cloud data using the beam method to obtain the images related to it and their ranges; Gaussian modeling and data output steps: Pack the point cloud and radiation energy field and output them to form a Gaussian model; Data merging and separate rendering steps: merging the packaged output data to form separate rendering of individual objects; The steps of constructing and realizing free perspective are as follows: construct free perspective for the whole three-dimensional scene to form a free viewing state.

2. The method according to claim 1, characterized in that The point cloud classification and extraction step comprises: Data preprocessing: including denoising, downsampling, alignment, and normalization steps to improve computational efficiency and classification accuracy; Feature extraction: extract features from the preprocessed point cloud; Model training: Select an appropriate machine learning or deep learning model and train it using labeled point cloud data; Model evaluation and optimization: Use the validation set to evaluate model performance and adjust model parameters and features as needed; Classification and post-processing: Apply the trained model to new data for classification.

3. The method according to claim 1, characterized in that: The steps of forming the image reorganization and classification data packets include: Based on the classified point cloud data, the beam method uses the idea of ​​reverse projection to project the points in the three-dimensional world onto the two-dimensional image plane. According to the point cloud range and the image range involved, the data is merged and sorted, and fixed-size images and point cloud data are output.

4. The method according to claim 1, characterized in that The Gaussian modeling and data output steps include: Perform local 3D reconstruction on the photos in the classified data, and calculate and output the posture of the reconstructed photos; Gaussian rendering calculation is performed based on the photo pose and point cloud data, and the output model is packaged and named.

5. A device for rendering a scene based on three-dimensional Gaussian sputtering enhanced oblique photography, comprising one or more processors and a non-temporary computer-readable storage medium storing program instructions. When the one or more processors execute the program instructions, the one or more processors are used to implement the method according to any one of claims 1-4.

6. A computing device, characterized in that The computing device comprises the apparatus of claim 5.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program code, and the program code is used to execute the method according to any one of claims 1 to 4.

Images