Large-scene lightweight three-dimensional reconstruction method based on NeRF and 3DGS mixed representation

Abstract

The invention discloses a large-scene lightweight three-dimensional reconstruction method based on NeRF and 3DGS mixed representation, and belongs to the technical field of three-dimensional reconstruction, and the method comprises the steps: collecting a multi-view image of a scene and an object in the scene, and carrying out the preprocessing; constructing a hybrid model, wherein the hybrid model comprises a simplified 3DGS model, a simplified NeRF model and a hybrid unit; constructing the total loss of the hybrid model; and training the hybrid model to obtain a lightweight three-dimensional reconstruction model which is used for new view angle image reconstruction in a scene. According to the method, colors are decomposed into independent colors expressed by Gaussian sputtering and observation colors expressed by NeRF, the advantages of efficient storage and complex illumination and reflection effects of NeRF are reserved, the rapid rendering capacity of 3DGS is utilized, storage occupation is remarkably reduced, and meanwhile the rendering quality is improved. The depth map generated by the 3DGS model is used for guiding NeRF dynamic sampling, and the training and rendering efficiency can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and particularly to a lightweight three-dimensional reconstruction method for large scenes based on a hybrid representation of NeRF and 3DGS. Background Art

[0002] In computer vision, three-dimensional reconstruction refers to the process of reconstructing three-dimensional information (such as shape, color) from single-view or multi-view images. Traditional three-dimensional reconstruction methods rely on discrete representation methods such as voxels and point clouds, which approximate the real scene by dividing the continuous three-dimensional space into discontinuous small units. However, these methods have inherent defects. For example, voxel models lose details due to the limitation of grid resolution, and the sparsity of point cloud models leads to inaccurate reconstruction in some areas.

[0003] In recent years, deep learning has made remarkable progress in the field of three-dimensional reconstruction. Neural Radiance Fields (NeRF) is a representative method that implicitly represents the light distribution and transmission characteristics in a scene by constructing a neural network. Its input is a continuous 5D vector, including any spatial position (x, y, z) in the scene and the viewing direction at that position (spherical coordinate representation method, including azimuth angle θ and elevation angle φ), and the output is the volume density at that position and the color (r, g, b) along the viewing direction (θ, φ) at that position. Generally, the spatial position is represented by sampling points, which are sampled from the ray from the camera to a certain pixel on the imaging plane. Assuming there are 1024 pixels on the imaging plane, there are 1024 rays. NeRF can render high-quality real scenes, but its rendering speed is slow, and it requires a large number of neural network operations and iteration times.

[0004] Three-dimensional Gaussian Splatting (3DGS), as a representation method that combines explicit and implicit, balances the contradiction between rendering quality and training time. 3DGS uses a large number of three-dimensional Gaussian distributions to represent the scene, and each three-dimensional Gaussian distribution (also called a 3D Gaussian sphere) has parameters such as position, scaling scale, rotation quaternion, opacity, and spherical harmonic coefficients. However, this method is difficult to handle complex light reflection, transparency, and shadow effects, and requires 48 spherical harmonic coefficients to describe the scene, resulting in complex model optimization and high storage occupancy.

[0005] Glossary of Terms: OpenCV is a cross-platform computer vision and machine learning software library distributed under the Apache 2.0 license (open source) and can run on Linux, Windows, Android, and Mac OS operating systems.

[0006] COLMAP (COLLISION-MAPpping) is a powerful open-source 3D reconstruction tool that supports automated multi-view geometry reconstruction, including core functions such as feature extraction, camera pose estimation, sparse point cloud reconstruction, and dense point cloud generation.

[0007] SSIM (Structural Similarity) is a commonly used metric in image quality assessment to measure the similarity between a reconstructed image and a reference image (usually the original image), and 1 - SSIM is used to represent the structural similarity loss.

[0008] L1 (Mean Absolute Error, MAE) loss calculates the sum of the absolute errors between the predicted values and the true values.

[0009] L2 (Mean Squared Error, MSE) loss calculates the sum of the squared errors between the predicted values and the true values. Summary of the Invention

[0010] The object of the present invention is to provide a lightweight large-scale 3D reconstruction method based on the hybrid representation of NeRF and 3DGS, which not only retains the advantages of NeRF in efficient storage and complex lighting and reflection effects, but also can significantly reduce storage occupancy and improve rendering quality.

[0011] To achieve the above object, the technical solution adopted by the present invention is as follows: A lightweight large-scale 3D reconstruction method based on the hybrid representation of NeRF and 3DGS, comprising the following steps; S1, collect multi-view images of the scene and the objects in the scene; S2, after preprocessing the images, obtain N images, generate a sparse point cloud of the scene and the camera pose of each image, and store the images in the image set in the acquisition order, where the nth image is P n , 1 ≤ n ≤ N; S3, construct a hybrid model, including a simplified 3DGS model, a simplified NeRF model, and a hybrid unit; The simplified 3DGS model is used to initialize each point of the sparse point cloud as a three-dimensional Gaussian distribution, and generate a first rendering image and a depth image of each image. The color of the three-dimensional Gaussian distribution is represented by 0th order SH coefficients, and the first rendering image corresponding to image P n is , and the depth image is D n , and the color of the pixel points in the first rendering image is an independent color; The simplified NeRF model dynamically samples to generate sampling points on the ray from the camera to the pixel point based on the depth value of the pixel point in the depth map, inputs the coordinates of the sampling points and the viewing direction from the camera to the sampling points, outputs the color of the sampling points, and then uses the alpha blending technique to render to obtain the second rendered image, image P n The corresponding second rendered image is , and the color of the pixel point in the second rendered image is the viewing color; The mixing unit is used to and perform weighted summation of the independent color and the viewing color of the pixel points in to generate a mixed rendered image ; S4, construct the total loss Loss of the hybrid model; , In the formula, is the L1 loss, is to calculate the structural similarity, is the L2 loss, and λ1, λ2, and λ3 are the weights of the L1 loss, the SSIM loss, and the L2 loss respectively, and the sum of the three is 1; S5, use the image set to train the hybrid model to convergence by minimizing the total loss to obtain a lightweight 3D reconstruction model; S6, use the lightweight 3D reconstruction model to perform new view image reconstruction of the scene.

[0012] Preferably, S2 is specifically: use OpenCV to detect and remove blurred images, name the images in the acquisition order, and then use COLMAP to read the images and generate the sparse point cloud of the scene and the camera poses corresponding to each image.

[0013] Preferably, in S3, the construction method of the simplified 3DGS model is; Obtain a 3DGS model, where the attributes of one three-dimensional Gaussian distribution include position, opacity, diagonal scaling matrix, rotation matrix, and color; represent the color with 0th-order SH coefficients to obtain a simplified 3DGS model.

[0014] Preferably, the method for generating the first rendered image of the image is; Project each three-dimensional Gaussian distribution onto the two-dimensional imaging plane to obtain a two-dimensional Gaussian distribution; Construct a fast rasterizer to sort the two-dimensional Gaussian distributions; Use the alpha blending technique according to and the opacity σ i to generate the color of each pixel point in the imaging plane to obtain the first rendered image.

[0015] Preferably, in S3, the depth map D nThe calculation method of the depth value D(u) of a pixel u is as follows; According to the camera pose of P n a ray from the camera center to the pixel u is generated. This ray intersects Q three-dimensional Gaussian distributions in total, and D(u) is obtained according to the following formula; , , In the formula, d q (u) is the distance from the q-th three-dimensional Gaussian distribution on the ray to the camera center, σ q is the opacity of the q-th three-dimensional Gaussian distribution, T q is the cumulative occlusion value of the first q - 1 three-dimensional Gaussian distributions to the q-th three-dimensional Gaussian distribution, 1 ≤ q ≤ Q, σ j is the opacity of the j-th three-dimensional Gaussian distribution, 1 ≤ j ≤ q - 1, T Q+1 is the cumulative occlusion value of the first Q three-dimensional Gaussian distributions to the pixel u.

[0016] Preferably, the simplified NeRF model includes an input layer, three hidden layers, and an output layer; The input layer is used to input the position encoding of the sampling point, the camera position, and the viewing direction; The hidden layer is 128-dimensional; The output layer is 3-dimensional and is used to output the color of the sampling point.

[0017] Preferably, in S3, the method of dynamic sampling of the simplified NeRF model is as follows: If reconstructing P n , read the camera pose of P n generate a ray from the camera center to each pixel on the imaging plane. For a pixel u in the imaging plane, within the range (D(u) - k, D(u) + k) from the camera center on its ray, randomly sample N u sampling points at the sampling distance k u ; , , , In the formula, N set is the preset number of sampling points, k set is the preset sampling distance, D(u) is the depth value of the pixel u in the depth map D n , D n (u) is the normalized value corresponding to D(u), D min , D max are respectively the minimum and maximum values of the depth values in D n .

[0018] Preferably, when training the hybrid model, a preset number of iterations is set. The total loss is calculated once per iteration, and the parameters of the hybrid model are adjusted using the Adam optimizer.

[0019] The idea of the present invention is as follows: (1) Design two simplified models, namely the simplified 3DGS model and the simplified NeRF model. Among them, in the simplified 3DGS model, the color of the three-dimensional Gaussian distribution in the original 3DGS model is represented by 0th-order SH coefficients, and only 3 floating-point numbers are required to represent the color, while the traditional 3DGS model requires 48 spherical harmonic coefficients (SH coefficients), thus greatly reducing the number of three-dimensional Gaussian distributions and optimization parameters. At the same time, the simplified 3DGS model of the present invention provides a first rendered image and a depth map generated based on 0th-order SH coefficients. The simplified NeRF model reduces the expression of opacity in the existing NeRF model. The existing NeRF model needs to generate the color and opacity at the spatial position during the rendering process, and the present invention only needs it to generate the color at a certain position, which is used as a color component in the hybrid rendered image. Of course, the NeRF model in the prior art can also be used to provide colors in the present invention, but the simplified NeRF model is faster and has a lower computational cost.

[0020] (2) Use the depth map generated by the simplified 3DGS model to guide the dynamic sampling of the simplified NeRF model to generate sampling points. According to the depth value of a certain pixel point in the depth map, it is used to guide the generation of sampling points within a certain sampling range on the ray at a dynamic sampling distance and a dynamic number of sampling points, and generate the sampling point colors, which are used to generate the second rendered image.

[0021] (3) Then, use the color of the pixel point in the first rendered image as the independent color and the color in the second rendered image as the observed color, and perform weighted summation to generate the mixed color and obtain the hybrid rendered image.

[0022] Compared with the prior art, the advantages of the present invention are as follows: The present invention proposes a hybrid model that combines Neural Radiance Field (NeRF) and Gaussian Splatting (3D Gaussian Splatting) for high-quality rendering and efficient storage of three-dimensional models. Specifically: (1) Reduced storage occupancy: Through geometric-optical decoupled expression, the color is divided into independent color and observed color related to light and viewing angle such as reflection and shadow. Among them, the independent color is expressed by the simplified 3DGS model, which only requires 0th-order spherical harmonic coefficients (3 floating-point numbers), while the traditional method requires 48 spherical harmonic coefficients; the observed color is represented by the simplified NeRF model. The combined way of expressing color can greatly reduce the storage requirements.

[0023] (2) Dynamic Sampling Acceleration: The depth map rendered by the 3DGS model guides the sampling on the rays in NeRF, avoiding redundant calculations in open areas, reducing the number of parameters to be optimized, and thus improving the training and rendering efficiency.

[0024] (3) Rendering Quality Enhancement: Hybrid Rendering Pipeline: NeRF can simulate complex light reflection, transparency, and shadow effects, while 3DGS is responsible for efficient shape and basic color representation. The present invention uses a dual-path asynchronous rendering method to perform weighted superposition on the outputs of 3DGS and NeRF, and the resulting hybrid rendering map significantly improves the rendering quality.

[0025] In summary, the present invention proposes a hybrid model combining NeRF and 3DGS. Through the way of division of labor and cooperation, it not only retains the advantages of NeRF in efficient storage and complex light and reflection effects, but also utilizes the fast rendering ability of 3DGS, significantly reducing the storage occupancy and at the same time improving the rendering quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 is the schematic diagram of the present invention; Figure 2 is the simplified NeRF model structure diagram; Figure 3 is an image in a scene; Figure 4 is Figure 3 the depth map of. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0027] The present invention will be further described below in conjunction with the embodiments and the drawings.

[0028] Embodiment 1: Refer to Figures 1 to 2 , a lightweight three-dimensional reconstruction method for large scenes based on the hybrid representation of NeRF and 3DGS, including the following steps; S1, Collect multi-view images of the scene and the objects in the scene; S2, After preprocessing the images, obtain N images, generate the sparse point cloud of the scene and the camera poses of each image, and store the images in the image set in the acquisition order, where the nth image is P n , 1 ≤ n ≤ N; S3, Construct a hybrid model, including a simplified 3DGS model, a simplified NeRF model, and a hybrid unit; The simplified 3DGS model is used to initialize each point of the sparse point cloud as a three-dimensional Gaussian distribution, and generate the first rendering map and the depth map of each image. The color of the three-dimensional Gaussian distribution is represented by the 0th order SH coefficient. The first rendering map corresponding to the image P n is and the depth map is D n, the color of the pixel points in the first rendering is an independent color; The simplified NeRF model dynamically samples to generate sampling points on the ray from the camera to the pixel point based on the depth value of the pixel point in the depth map, inputs the coordinates of the sampling points and the viewing direction from the camera to the sampling points, outputs the color of the sampling points, and then uses the alpha blending technique to render to obtain the second rendering, image P n The corresponding second rendering is , the color of the pixel points in the second rendering is the viewing color; The mixing unit is used to and perform a weighted sum of the independent color and the viewing color of the pixel points in to generate a mixed rendering S4. Construct the total loss Loss of the mixed model; , In the formula, is the L1 loss, is to calculate the structural similarity, is the L2 loss, and λ1, λ2, and λ3 are the weights of the L1 loss, the SSIM loss, and the L2 loss respectively, and the sum of the three is 1; S5. Use the image set to train the mixed model to convergence with the minimum total loss to obtain a lightweight 3D reconstruction model; S6. Use the lightweight 3D reconstruction model to perform new view image reconstruction of the scene.

[0029] In this embodiment, S2 is specifically: using OpenCV to detect and remove blurred images, naming the images in the format according to the acquisition order, and then using COLMAP to read the images and generate the sparse point cloud of the scene and the camera poses corresponding to each image.

[0030] The construction method of the simplified 3DGS model is: obtain a 3DGS model, and the attributes of one three-dimensional Gaussian distribution include position, opacity, diagonal scaling matrix, rotation matrix, and color; represent the color with the 0th-order SH coefficient to obtain the simplified 3DGS model.

[0031] The method for generating the first rendering of the image is: project each three-dimensional Gaussian distribution onto the two-dimensional imaging plane to obtain a two-dimensional Gaussian distribution; construct a fast rasterizer and sort the two-dimensional Gaussian distribution; use the alpha blending technique according to and the opacity σ i to generate the color of each pixel point in the imaging plane to obtain the first rendering.

[0032] In S3, the calculation method of the depth value D(u) of a pixel point u in the depth map D n is; According to Pn Given the camera pose, a ray is generated from the camera center to the pixel point u. This ray intersects Q three-dimensional Gaussian distributions. Then, D(u) is obtained according to the following formula; , , In the formula, d q (u) is the distance from the q-th three-dimensional Gaussian distribution on the ray to the camera center, σ q is the opacity of the q-th three-dimensional Gaussian distribution, T q is the cumulative occlusion value of the first q - 1 three-dimensional Gaussian distributions on the q-th three-dimensional Gaussian distribution, 1 ≤ q ≤ Q, σ j is the opacity of the j-th three-dimensional Gaussian distribution, 1 ≤ j ≤ q - 1, T Q+1 is the cumulative occlusion value of the first Q three-dimensional Gaussian distributions on the pixel point u.

[0033] The simplified NeRF model includes an input layer, 3 hidden layers, and an output layer. Among them, the input layer is used to input the position encoding of the sampling point, the camera position, and the viewing direction; the hidden layer is 128-dimensional; the output layer is 3-dimensional and is used to output the color of the sampling point. The method of dynamic sampling of the simplified NeRF model is as follows: If reconstructing P n , read the camera pose of P n . Generate rays from the camera center to each pixel point on the imaging plane. For a pixel point u in the imaging plane, randomly sample N u sampling points within the range (D(u) - k, D(u) + k) from the camera center on its ray at the sampling distance k u ; , , , In the formula, N set is the preset number of sampling points, k set is the preset sampling distance, D(u) is the depth value of the pixel point u in the depth map D n , D n (u) is the normalized value corresponding to D(u), D min , D max are respectively the minimum and maximum values of the depth values in D n .

[0034] When training the hybrid model, a preset number of iterations is set. The total loss is calculated once for each iteration, and the parameters of the hybrid model are adjusted using the Adam optimizer.

[0035] Example 2: See Figures 1 to 4, on the basis of Embodiment 1, a more detailed implementation method is given. A lightweight three-dimensional reconstruction method for large scenes based on the hybrid representation of NeRF and 3DGS includes the following steps; S1. Collect multi-view images of the scene and objects in the scene, which can be achieved by video shooting with a camera; S2. Preprocess to generate N images and a sparse point cloud; For the video obtained in S1, use the open-source software ffmpeg to segment the captured video, and segment it at a rate of 10 frames per second to obtain the captured images; use the open-source software OpenCV to screen the obtained captured images to remove blurred images; rename the obtained images in the order of shooting time, starting from 0000, adding 1 to each image, and a total of N images are obtained and stored in the image set, which are sequentially marked as P1~P N ; Use the open-source software COLMAP based on SFM (motion pose estimation) to process the N preprocessed images, generate a sparse point cloud of the scene and the camera pose corresponding to each image, and the internal parameter matrix of the camera uses the default pinhole imaging model matrix.

[0036] S3. Construct a hybrid model, including a simplified 3DGS model, a simplified NeRF model, and a hybrid unit.

[0037] S4. Construct the total loss Loss of the hybrid model.

[0038] S5. Train the hybrid model to obtain a lightweight three-dimensional reconstruction model, specifically including S51~S56; S51. Read the sparse point cloud, camera pose, and image set generated in step S2, and initialize them as the training set; preset the number of iterations to 30000 times; S52. Initialize each point of the sparse point cloud as a three-dimensional Gaussian distribution using the simplified 3DGS model: First, define the attributes of the three-dimensional Gaussian distribution, including position, opacity, diagonal scaling matrix, rotation matrix, and color; Then read the sparse point cloud, and for a point i among them, obtain its RGB color and three-dimensional coordinates; Use the three-dimensional coordinates as the position μ of the three-dimensional Gaussian distribution corresponding to this point i ; Use the sigmod function to map the opacity parameter ‌self._opacity‌ to the interval [0,1] to obtain the opacity σ i ; Adopt the K-nearest neighbor algorithm to find the average distance d from the 5 points closest to point i to point i i , and use d i as the scaling parameters in the three-axis directions of the coordinate system to obtain the diagonal scaling matrix S; Define the quaternion (0, 0, 0, 1) and construct the rotation matrix R; Convert the RGB color of point i to the 0th-order SH representation as the color of the three-dimensional Gaussian distribution; Then generate the three-dimensional Gaussian distribution of point i according to the following formula; , , In the formula, p is the three-dimensional coordinate of a point in the scene, e is the natural constant, is the covariance matrix of the Gaussian distribution, and T is the transpose operation.

[0039] S53. Initialize the simplified NeRF model. Its input is the sampled point coordinates and the viewing direction from the camera to the sampled point. It is necessary to map the input to 33-dimensional data, which are the 27-dimensional sampled point position encoding, the 3-dimensional camera position, and the 3-dimensional viewing direction. Taking a sampled point on any ray as an example, if its three-dimensional coordinates are (x, y, z), the position encoding is mapped to 27-dimensional data such as (x, y, z, sin(2 i πx), sin(2 i πy), sin(2 i πz), cos(2 i πx), cos(2 i πy), cos(2 i πz)). In this formula, i = (1, 2, 3, 4), i represents the i-th power of 2. The hidden layer is a fully connected layer, and the sigmod function is used for all activation functions. After passing through the sigmod function in the output layer, the color value is multiplied by 255 and mapped to the range of 0 - 255 to meet the requirements of RGB images.

[0040] S54. Generate the image P n corresponding first rendering image, depth map, and second rendering image; Generate the first rendering image: Read the camera pose corresponding to P n , convert the world coordinates to camera coordinates, calculate the projection matrix, project each three-dimensional Gaussian distribution onto the two-dimensional imaging plane to obtain a two-dimensional Gaussian distribution; construct a fast rasterizer to sort the two-dimensional Gaussian distribution; use the α-blending technique to generate the color of each pixel point in the imaging plane according to and the opacity σ i to obtain the first rendering image; Generate the depth map: According to the steps described in S3, generate the ray from the camera center to pixel point u based on the camera pose of P n . This ray intersects with Q three-dimensional Gaussian distributions. Then, according to the formula of D(u) and T q , calculate the depth value D(u) of pixel point u, and finally generate the depth map; Generate the second rendering image: Refer to Figure 3 and Figure 4 , the scene and objects are divided into foreground and background, and there is a large amount of empty space between the foreground and the background. Taking Figure 3 and Figure 4 as an example, the objects closer to the camera have a brighter representation in the depth map, while the farther ones tend to be darker. Accordingly, the present invention designs a dynamic sampling method. For a pixel point u in the imaging plane, within the range (D(u) - k, D(u) + k) from the camera center on its ray, sample N u random sampling points at a sampling distance k u ; then use the alpha blending technique for the sampling points on the ray to obtain the color of the pixel point as the observed color described in the present invention, and the image composed of the observed colors is the second rendering image; S55, generate the mixed rendering image; To avoid NeRF learning too much color information and strengthen the expression ability of 3dGS, add the pixel colors in the first rendering image and the second rendering image with different weights respectively to obtain the mixed rendering image; specifically, set the weight of NeRF to 0 in the first 15000 iterations at the beginning of training, and gradually increase it from 0 to the set threshold in the 15000 - 25000 iterations.

[0041] S56, train the mixed model, calculate the total loss once for each training, backpropagate the gradient, and use the Adam optimizer to update all the parameters of the mixed model until the number of iterations is reached to obtain the lightweight three - dimensional reconstruction model; for the trained model, only retain the training weights of the simplified NeRF model and the three - dimensional model file of the simplified 3DGS model for subsequent image reconstruction from new viewpoints.

[0042] S6, perform new - viewpoint image reconstruction of the scene with the lightweight three - dimensional reconstruction model. This reconstruction is actually a process of image rendering: According to the new viewpoint to be reconstructed, generate the camera position and imaging plane at this viewpoint, project the three - dimensional Gaussian distribution onto the imaging plane using the simplified 3DGS model, and obtain the first rendering image through means such as sorting and alpha blending techniques, and generate the depth map based on the camera position, imaging plane, and three - dimensional Gaussian distribution; According to the new viewpoint to be reconstructed, the viewing direction from the camera center to each pixel point in the imaging plane can be generated. Use the simplified NeRF model to perform dynamic sampling according to the depth value of each pixel point in the depth map to generate sampling points for each ray. For each ray, according to the sampling points and the viewing direction, output the RGB value of the color of the sampling point in the viewing direction, and use the alpha blending technique to render to obtain the second rendering image. Weight - sum the independent color of the pixel point in the first rendering image and the observed color of the corresponding pixel point in the second rendering image to generate the mixed color of this pixel point, and finally obtain the mixed rendering image.

[0043] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included within the protection scope of the present invention.

Claims

1. A lightweight 3D reconstruction method for large scenes based on NeRF and 3DGS hybrid representation, characterized in that: The steps include: S1, collects multi-view images of the scene and objects in the scene; S2, after image preprocessing, obtains N images, generates a sparse point cloud of the scene and the camera pose of each image, and stores the images in the image set in the order of acquisition, where the nth image is P n , 1≤n≤N; S3, constructing a hybrid model including a simplified 3DGS model, a simplified NeRF model and a hybrid unit; The simplified 3DGS model is used to initialize each point of the sparse point cloud into a three-dimensional Gaussian distribution and generate a first rendering image and a depth map of each image. The color of the three-dimensional Gaussian distribution is represented by the 0th order SH coefficient. n The corresponding first rendering is , the depth map is D n , the color of the pixel in the first rendering is an independent color; The simplified NeRF model dynamically samples the sampling points on the ray from the camera to the pixel point based on the depth value of the pixel point in the depth map, inputs the sampling point coordinates and the observation direction from the camera to the sampling point, outputs the color of the sampling point, and then uses the alpha blending technology to render the second rendering image, image P n The corresponding second rendering is , the color of the pixel in the second rendering is the observed color; The mixing unit is used to and The independent color of the pixel and the observed color are weighted summed to generate a mixed rendering ; S4, the total loss Loss of constructing the hybrid model; ， In the formula, is the L1 loss, To calculate the structural similarity, is the L2 loss, λ1, λ2, and λ3 are the weights of L1 loss, SSIM loss, and L2 loss, respectively, and the sum of the three is 1; S5, using the image set to train the hybrid model until convergence by minimizing the total loss, and obtaining a lightweight 3D reconstruction model; S6, reconstructing new perspective images of scenes using lightweight 3D reconstruction models.

2. The large scene lightweight 3D reconstruction method based on NeRF and 3DGS hybrid representation according to claim 1 is characterized in that: S2 specifically includes: using OpenCV to detect and remove blurred images, naming the images in a format set according to the acquisition order, and then using COLMAP to read the images and generate a sparse point cloud of the scene and the camera pose corresponding to each image.

3. The large scene lightweight 3D reconstruction method based on NeRF and 3DGS hybrid representation according to claim 1 is characterized in that: In S3, the construction method of the simplified 3DGS model is: A 3DGS model is obtained, wherein the attributes of a three-dimensional Gaussian distribution include position, opacity, diagonal scaling matrix, rotation matrix and color; the color is represented by a 0-order SH coefficient to obtain a simplified 3DGS model.

4. The large scene lightweight 3D reconstruction method based on NeRF and 3DGS hybrid representation according to claim 1 is characterized in that: The method of generating a first rendering of an image is: Project each three-dimensional Gaussian distribution onto a two-dimensional imaging plane to obtain a two-dimensional Gaussian distribution; Build a fast rasterizer to sort 2D Gaussian distributions; Using alpha blending technology according to and opacity σ i The color of each pixel in the imaging plane is generated to obtain a first rendering image.

5. The method for lightweight 3D reconstruction of large scenes based on NeRF and 3DGS hybrid representation according to claim 1, characterized in that: In S3, the depth map D n The depth value D(u) of a pixel u is calculated as follows: According to P n The camera pose generates a ray from the camera center to the pixel u. The ray intersects with Q three-dimensional Gaussian distributions. Then D(u) is obtained according to the following formula: ， ， Where, d q (u) is the distance from the qth three-dimensional Gaussian distribution on the ray to the camera center, σ q is the opacity of the qth three-dimensional Gaussian distribution, T q is the cumulative occlusion value of the first q-1 three-dimensional Gaussian distributions on the qth three-dimensional Gaussian distribution, 1≤q≤Q, σ j is the opacity of the j-th three-dimensional Gaussian distribution, 1≤j≤q-1, T Q+1 is the cumulative occlusion value of the first Q three-dimensional Gaussian distributions for pixel u.

6. The large scene lightweight 3D reconstruction method based on NeRF and 3DGS hybrid representation according to claim 1 is characterized in that: The simplified NeRF model includes an input layer, three hidden layers and an output layer; The input layer is used to input sampling point position coding, camera position and observation direction; The hidden layer is 128-dimensional; The output layer is 3-dimensional and is used to output the color of the sampling point.

7. The method for lightweight 3D reconstruction of large scenes based on NeRF and 3DGS hybrid representation according to claim 1, characterized in that: In S3, the method to simplify the dynamic sampling of the NeRF model is: If we rebuild P n , read P n The camera pose is generated from the camera center to each pixel point on the imaging plane. For a pixel point u in the imaging plane, the sampling distance k is calculated within the range of (D(u)-k, D(u)+k) from the camera center on its ray. u Random sampling N u sampling points; ， ， ， Where N set is the preset number of sampling points, k set is the preset sampling distance, D(u) is the depth map D n The depth value of pixel u in the n (u) is the normalized value corresponding to D(u), D min , D max D n The minimum and maximum depth values.

8. The large scene lightweight 3D reconstruction method based on NeRF and 3DGS hybrid representation according to claim 1 is characterized in that: When training the hybrid model, the number of iterations is preset, the total loss is calculated once per iteration, and the parameters of the hybrid model are adjusted using the Adam optimizer.

Images