Efficient three-dimensional reconstruction method based on data-driven compression and adaptive splitting

Abstract

The invention discloses an efficient three-dimensional reconstruction method based on data-driven compression and adaptive splitting. The method comprises the following steps: acquiring a multi-view image to generate a 3D Gaussian; constructing a mask generation network and a discrimination network, generating a mask by the mask generation network, and screening the eliminated 3D gauss; the attributes of 3D Gaussian are optimized through a microgradable grating renderer; gaussian selection and elimination are carried out by using the mask, and a rendering image after partial Gaussian elimination is generated; if the preset number of iterations is reached, the contribution to the scene is reserved or eliminated according to 3D Gaussian; otherwise, performing Gaussian splitting or copying operation. And through antagonistic learning and reconstruction loss optimization of the mask generation network and the discrimination network, high-precision three-dimensional scene reconstruction is realized while the Gaussian number is reduced. The method is suitable for various three-dimensional reconstruction scenes, can greatly improve the rendering speed, reduces the consumption of computing resources, maintains the visual quality of a reconstruction result, and is suitable for the fields of virtual reality, augmented reality, movie special effects and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to an efficient three-dimensional reconstruction method based on data-driven compression and adaptive splitting. Background Art

[0002] Voice 3D scene reconstruction has important application value in the fields of virtual reality, augmented reality, film and television special effects, etc. Traditional 3D scene reconstruction methods mainly rely on geometric representations such as voxel grids and triangular grids. Although these methods can achieve good results in specific scenarios, they face problems of storage and computational efficiency when dealing with complex scenes. With the advancement of computing technology, 3D Gaussian Splatting (3DGS) technology is regarded as a new, feasible and efficient 3D scene representation method. 3DGS constructs scenes using millions of elliptical 3D Gaussians, which can render images in a high-quality manner. Each 3D Gaussian is defined by parameters such as its position, transparency, covariance matrix, and spherical harmonic coefficients, thereby achieving high-fidelity image generation.

[0003] However, existing 3DGS methods may suffer from image blurring and detail loss in high-frequency areas. Typically, 3DGS requires the management and processing of a large number of 3D Gaussians, which results in huge memory consumption and computational resource requirements, especially in real-time applications, which poses a great challenge to the performance of the system. In addition, most existing methods rely on fixed rules to reduce the number of 3D Gaussians and lack the ability to dynamically adjust the Gaussian distribution according to scene details, which may result in insufficient preservation of details in key areas of the image.

[0004] Therefore, the present invention proposes an efficient 3D reconstruction method based on data-driven compression and adaptive splitting, aiming to make full use of the key information in the scene, reduce redundant 3D Gaussians, and maintain the high fidelity of the scene. First, the 3D Gaussians are adaptively split based on the image gradient information, and more Gaussians are retained in areas with rich image details to reduce image blur. Secondly, a pruning mechanism based on a generative adversarial network (GAN) is used to effectively remove redundant 3D Gaussians. Finally, the contribution of 3D Gaussians to rendering quality is dynamically evaluated through importance weight sampling technology, and the selection and retention of Gaussians are further optimized, thereby significantly improving the efficiency of scene reconstruction. Summary of the invention

[0005] The purpose of the present invention is to provide an efficient three-dimensional reconstruction method based on data-driven compression and adaptive splitting. The method achieves high-precision modeling of scenes using a small number of Gaussians by effectively reducing the number of redundant Gaussians and adaptively adjusting the Gaussian distribution, thereby reducing computing resource consumption while maintaining visual effects.

[0006] To achieve the above functions, the present invention designs an efficient 3D reconstruction method based on data-driven compression and adaptive splitting, and executes the following steps S1 to S7 to complete the 3D reconstruction of the target scene: Step S1: Acquire real images of the target scene from different perspectives, generate sparse point clouds through the SfM method, and initialize the sparse point clouds into multiple 3D Gaussians. The attributes of the 3D Gaussians include the center position, size, shape, color, transparency, and learnable features in the three-dimensional space. Step S2: construct a mask generation network and a discriminant network. The mask generation network selects the eliminated 3D Gaussian based on the learnable features of the 3D Gaussian, generates a rendered image based on data-driven, and outputs a mask; the discriminant network is used to judge the difference between the rendered image and the real image; Step S3: construct a differentiable raster renderer to optimize the properties of the 3D Gaussian; Step S4: randomly select a viewing angle, input the real image of the viewing angle into the differentiable grating renderer, render the 3D Gaussian at the viewing angle through the differentiable grating renderer, and generate a complete 3D Gaussian rendering image; Step S5: the mask generation network outputs the importance score of the 3D Gaussian according to the learnable features and perspective information of the 3D Gaussian, and generates a mask according to the threshold; uses the generated mask to eliminate the specific 3D Gaussian; obtains the image after the Gaussian elimination, renders the image after the Gaussian elimination, and generates a rendering image with part of the 3D Gaussian eliminated; Step S6: the discriminant network compares the complete 3D Gaussian rendering, the rendering with some 3D Gaussians eliminated, and the real image; constructs the loss function of the mask generation network, the discriminant network, and the scene reconstruction, and uses the Adam optimizer to iteratively optimize and update the mask generation network and the discriminant network parameters and the properties of the 3D Gaussian through back propagation; Step S7: Determine whether the preset number of iterations has been reached. If so, proceed to the Gaussian reduction stage. Based on the mask generated by the mask generation network and the importance weighted sampling method, calculate the total importance score of the 3D Gaussian, evaluate the contribution of the 3D Gaussian to the target scene, select the 3D Gaussian with a total importance score greater than the preset value for retention, eliminate other 3D Gaussians, and output the optimized Gaussian representation; If the preset number of iterations is not reached, the Gaussian growth stage is performed to screen out incompletely reconstructed 3D Gaussians, process them according to their scale, adaptively split the 3D Gaussians whose scales are greater than the preset value, and copy the other 3D Gaussians; then return to step S2 to continue iterating until the preset number of iterations is reached, and the three-dimensional reconstruction of the target scene is completed.

[0007] Beneficial effects: Compared with the prior art, the advantages of the present invention include: The present invention proposes an efficient 3D reconstruction method based on data-driven compression and adaptive splitting, which makes full use of the key details in the scene and reduces the number of redundant 3D Gaussians, thereby reducing storage requirements and computing costs. The method uses a data-driven masked generative adversarial network to dynamically reduce the number of Gaussians, ensuring that the high fidelity of the scene is maintained while reducing the number of Gaussians. The Gaussian distribution of the scene is effectively optimized by combining the masked generative network and the generative adversarial network. The method has significantly improved the speed and quality of 3D reconstruction and is suitable for virtual reality, augmented reality and other scenes that require efficient rendering. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 is a flow chart of an efficient 3D reconstruction method based on data-driven compression and adaptive splitting provided according to an embodiment of the present invention; Figure 2 is a schematic diagram of the structure of a mask generation network provided according to an embodiment of the present invention; Figure 3 is a schematic diagram of the structure of a discriminant network provided according to an embodiment of the present invention; Figure 4 is a flowchart of adversarial training of a mask generation network and a discriminant network provided according to an embodiment of the present invention; Figure 5 3D Gaussian adaptive density adjustment process according to an embodiment of the present invention is a schematic diagram. DETAILED DESCRIPTION

[0009] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0010] The efficient 3D reconstruction method based on data driven compression and adaptive splitting provided by the embodiment of the present invention is referred to Figure 1 , perform the following steps S1 to S7 to complete the three-dimensional reconstruction of the target scene: Step S1: Acquire real images of the target scene from different perspectives, generate sparse point clouds through the SfM method, and initialize the sparse point clouds into multiple 3D Gaussians. The attributes of the 3D Gaussians include the center position, size, shape, color, transparency, and learnable features in the three-dimensional space. The 3D Gaussian in step S1 is defined as follows: ; In the formula, G i ( x ) is a 3D Gaussian, defined as an ellipsoid with specific properties, x Represents each point in the area where the ellipsoid lies,μ i , Σ i are the properties of a 3D Gaussian, where μ i is the center position of the 3D Gaussian in three-dimensional space; Σ i Indicates size The covariance matrix defines the size of the 3D Gaussian and its shape in space; The properties of 3D Gaussian also include color, transparency, and learnable features, among which: Color: It is represented by the coefficients of the 4th-order spherical harmonic function, which describes the color changes of 3D Gaussian under different lighting conditions; Opacity: Use a scalar α i Indicates the visibility of 3D Gaussian in the final rendered image. This attribute is used to control the visual contribution of 3D Gaussian and affect the superposition and fusion effects between different 3D Gaussian. Learnable Feature: Using a nine-dimensional first-order tensor This feature is passed as input to the mask generation network to describe the importance of 3D Gaussians; it is optimized during the training process to help generate masks and decide which 3D Gaussians need to be retained or eliminated.

[0011] Step S2: constructing a mask generation network and a discriminant network. The mask generation network selects the 3D Gaussian to be eliminated based on the learnable features of the 3D Gaussian, generates a rendering image with some 3D Gaussian eliminated based on data-driven, and outputs a mask; the discriminant network is used to judge the difference between the rendering image with some 3D Gaussian eliminated and the real image; The structures and functions of the mask generation network and the discriminant network described in step S2 are as follows: The mask generation network is implemented using tiny-cuda-nn, a neural network library designed for efficient parallel computing. ReLU is used as the activation function to enhance nonlinear processing capabilities. The input of the mask generation network includes learnable features of 3D Gaussian. , perspective information, the core function of the mask generation network is to generate accurate masks based on these inputs, thereby optimizing the representation and rendering quality of the scene. The mask generation network determines whether each 3D Gaussian should be retained or eliminated based on the learnable features; the perspective information helps the mask generation network process the 3D Gaussian from the best angle and optimize the rendering; the mask generation network processes the 3D Gaussian based on the perspective information and generates a rendering image that eliminates some 3D Gaussians, and outputs a mask; The structure of the mask generation network is as follows Figure 2As shown in Figure 2. Inside the mask generation network, the input learnable features and view information pass through a series of hidden layers to finally generate a mask that is used to decide which 3D Gaussians should be retained or eliminated. This mechanism can improve rendering efficiency and quality by optimizing Gaussian distribution.

[0012] The discriminant network adopts the convolutional neural network (CNN) structure. The structure of the discriminant network is as follows: Figure 3 As shown in the figure, the generated rendering image with some 3D Gaussians removed is subjected to feature extraction and authenticity evaluation through multiple convolutional layers and pooling layers. The generated rendering image with some 3D Gaussians removed is compared with the real image, and the discrimination result is output. The mask generation network gradually optimizes the generated mask by minimizing the feedback of the discriminant network. This adversarial learning ensures that the mask generation network can maintain the high accuracy and visual effect of the scene while reducing the number of 3D Gaussians. This adversarial learning mechanism can be found in Figure 4 Among them, the interactive process between the mask generation network and the discriminant network ensures efficient mask generation and image quality improvement.

[0013] Step S3: construct a differentiable raster renderer to optimize the properties of the 3D Gaussian; Step S4: randomly select a viewing angle, input the real image of the viewing angle into the differentiable grating renderer, render the 3D Gaussian at the viewing angle through the differentiable grating renderer, and generate a complete 3D Gaussian rendering image; Step S5: the mask generation network outputs the importance score of the 3D Gaussian according to the learnable features and perspective information of the 3D Gaussian, and generates a mask according to the threshold; uses the generated mask to eliminate the specific 3D Gaussian; obtains the image after the Gaussian elimination, renders the image after the Gaussian elimination, and generates a rendering image with part of the 3D Gaussian eliminated; The specific process of step S5 is as follows: Step S5.1: The mask generation network uses a two-layer multi-layer perceptron (MLP), denoted as , the mask generates the network input learnable features and viewing angle information , generating the corresponding 3D Gaussian importance score , the importance score generation process is as follows: ; in, Indicates that the features can be learned F i and viewing angle information θ i Concatenate as input to the mask generation network, I i Represents the importance score of 3D Gaussian. The importance score reflects the importance of each 3D Gaussian to the scene rendering at the current perspective. Step S5.2: The mask generation process not only relies on the application of thresholds, but also combines more complex strategies; the importance score of each 3D Gaussian I i Used to generate the mask M i , the generation formula is as follows: ; in, is the indicator function, It stops the gradient operation, and threshold represents the threshold value; The threshold is calculated by the following formula: ; in, , , N is the current number of 3D Gaussians, is the proportionality coefficient, is the maximum number of 3D Gaussians; The above formula is calculated by the current number of 3D Gaussians N Logarithmic transformation is performed so that the threshold can be smoothly adjusted according to the change in the number of 3D Gaussians, thereby effectively screening out 3D Gaussians with smaller contributions.

[0014] Step S5.3: Generated mask M i It is used to select and process the corresponding 3D Gaussians, and use the threshold threshold to filter the 3D Gaussians. The filtered 3D Gaussians will continue to be rendered through the 3D Gaussian point algorithm to generate a rendering image with some 3D Gaussians eliminated.

[0015] Step S6: the discriminant network compares the complete 3D Gaussian rendering, the rendering with some 3D Gaussians eliminated, and the real image; constructs the loss function of the mask generation network, the discriminant network, and the scene reconstruction, and uses the Adam optimizer to iteratively optimize and update the mask generation network and the discriminant network parameters and the properties of the 3D Gaussian through back propagation; The loss function of the mask generation network in step S6 Loss G As follows: ; in, represents the L2 norm of the learnable features, σ Represents the Sigmod operation; by minimizing the loss function Loss G, the mask generation network can dynamically optimize the generated mask to ensure that the importance of 3D Gaussians can be effectively evaluated. As many 3D Gaussians as possible are selected to retain more scene information, but at the same time, the number of these 3D Gaussians does not negatively affect the final reconstruction quality. Loss function Loss G By controlling the selection of 3D Gaussians, the number of unnecessary 3D Gaussians can be effectively reduced, thus improving computational efficiency.

[0016] Loss function of the discriminative network Loss D As follows: ; in, Loss R represents the loss of the real image, Loss F Indicates the loss of rendering by eliminating some 3D Gaussians; The discriminative network receives two types of images as input: real images Img R And the rendering with some 3D Gaussians removed Img F ; Loss of real images Loss R And eliminate some of the loss of 3D Gaussian renderings Loss F The calculation formula is as follows: ; ; Among them, D( Img R ) and D( Img F ) represent the output of the discriminant network for the real image and the rendering image with some 3D Gaussians eliminated, respectively, BCE represents the binary cross entropy loss, and MSE represents the mean square error loss; by minimizing the loss of the discriminant network, the discriminant network can improve the ability to distinguish between the real image and the rendering image with some 3D Gaussians eliminated, thereby providing more powerful feedback to the mask generation network, prompting it to generate higher quality masks.

[0017] Loss function for scene reconstruction Loss Restruction As follows: ; in, Img Represents a complete 3D Gaussian rendering. Img R is a real image, λ is a hyperparameter; L1 represents L1 loss, SSIM represents structural similarity loss; loss functionLoss Restruction The L1 loss and structural similarity loss are combined to ensure that the difference between the complete Gaussian rendered image and the real image is minimized in the case of 3D Gaussian elimination. The L1 loss measures the pixel differences between images, while SSIM ensures that the images are structurally consistent.

[0018] Total loss function Loss total As follows: ; in, λ G represents the hyperparameters of the mask generation network, λ D represents the hyperparameters of the discriminative network. The total loss function combines the loss function of scene reconstruction Loss Restruction , the loss function of the mask generation network Loss G And the loss function of the discriminant network Loss D . Loss Restruction Ensure that the difference between the generated rendering with some 3D Gaussian removed and the real image is minimal, Loss G Optimize the selection and elimination of 3D Gaussians, while Loss D The feedback from the discriminative network helps the mask generation network generate more realistic images. λ G and λ D Control Loss G and Loss D exist Loss total The weights in , thus ensuring that the mask generation network can maintain image quality while reducing the number of 3D Gaussians.

[0019] Step S7: Determine whether the preset number of iterations has been reached. If so, proceed to the Gaussian reduction stage. Based on the mask generated by the mask generation network and the importance weighted sampling method, calculate the total importance score of the 3D Gaussian, evaluate the contribution of the 3D Gaussian to the target scene, select the 3D Gaussian with a total importance score greater than the preset value for retention, eliminate other 3D Gaussians, and output the optimized Gaussian representation; If the preset number of iterations is not reached, the Gaussian growth stage is performed to screen out incompletely reconstructed 3D Gaussians, process them according to their scale, adaptively split the 3D Gaussians whose scales are greater than the preset value, and copy the other 3D Gaussians; then return to step S2 to continue iterating until the preset number of iterations is reached, and the three-dimensional reconstruction of the target scene is completed.

[0020] In one embodiment, the preset number of iterations is half of the total number of iterations; that is, when the number of iterations is greater than or equal to half of the total number of iterations, it is regarded as the Gaussian reduction stage, and when the number of iterations is less than half of the total number of iterations, it is regarded as the Gaussian growth stage.

[0021] The Gaussian growth phase in step S7 specifically includes the following processes: Step S7.1.1: Filter out 3D Gaussians whose scale is larger than a preset value, define them as large Gaussians, and use the Scharr edge detection operator to calculate the complete 3D Gaussian rendering Img and real images Img R The gradient of is as follows: ; ; In the formula, G img Represents a complete 3D Gaussian rendering Img The gradient of G ImgR Represents a real image Img R The gradient of Compute gradient differences G d : ; Step S7.1.2: Project the center position of each large Gaussian from the three-dimensional space to the image plane, and calculate the image gradient at the projection point; if the gradient difference value at the projection point of the 3D Gaussian is less than 25% of the maximum gradient difference, the 3D Gaussian is marked as incompletely reconstructed, and the formula is as follows: ; Where, Threshold represents the threshold; Step S7.1.3: Use a voxel grid to discretize the target scene into small spatial regions (voxels). Each voxel represents a specific spatial region. The density of the space is obtained by calculating the number of 3D Gaussians within each voxel. The formula is as follows: ; in, d v It is a voxel v The density of It is a voxel v The number of Gaussians in vol ( v ) is a voxel v In this way, a 3D Gaussian distribution density map of the entire scene can be obtained, which provides a basis for subsequent splitting operations. Step S7.1.4: For the large Gaussian that is marked as incompletely reconstructed, the number of Gaussian splits is dynamically adjusted based on the density of the space in which it is located. The formula for the number of Gaussian splits is as follows: ; in, N i Indicates i The number of splits of a large Gaussian, d i It is i The density of the space where the large Gaussian is located, min(D) and max(D) are the minimum and maximum spatial densities respectively; Step S7.1.5: Filter out 3D Gaussians whose scale is less than or equal to a preset value, define them as small Gaussians, calculate the gradients of the center positions of the small Gaussians through back propagation, and judge whether these gradients are less than a preset threshold, so as to determine whether these small Gaussians fail to fully reconstruct the details; for small Gaussians whose gradient values ​​are lower than the preset threshold, perform a copy operation to enhance their expressiveness in reconstruction, thereby improving detail recovery.

[0022] 3D Gaussian adaptive density adjustment process diagram reference Figure 5 , Figure 5 Two processing situations of 3D Gaussian splitting and replication are shown.

[0023] The Gaussian reduction stage in step S7 specifically includes the following processes: Step S7.2.1: At the beginning of the Gaussian reduction phase, every 25 iterations, the mask generation network generates a 3D Gaussian G i Generate its importance score I i ; Step S7.2.2: Importance score for mask generation network output I i , perform binarization and generate a mask M i , the mask is used to identify which 3D Gaussians should be retained and which 3D Gaussians should be eliminated; the mask is generated every 25 iterations; Step S7.2.3: In order to maintain image quality while reducing the number of 3D Gaussians, importance weighted sampling is performed every 1000 iterations, and 95% of the 3D Gaussians are randomly selected and retained each time; the weight of each 3D Gaussian at the viewing angle is calculated.j The blending weights and hit counts on each pixel in; 3D Gaussian G i The mixed weight ω i The calculation of is as follows: ; in, Is the indicator function, used to judge the 3D Gaussian G i Is it related to the ray that passes through the pixel (u,v) from the center of the camera? r u,v intersect; σ i For 3D Gaussian G i The opacity, H and W are the height and width of the image respectively; 3D Gaussian G i Hit count c i Represents the number of times a 3D Gaussian covers a pixel, calculated as follows: ; Calculate 3D Gaussian G i Total importance score S i As follows: ; In the formula, N v Indicates the total number of viewing angles; The total importance score of each 3D Gaussian is normalized as follows: ; In the formula, is the normalized 3D Gaussian G i The total importance score of . The 3D Gaussian normalized G i The total importance score of , each time weighted sampling is performed, the probability of 3D Gaussian being selected is proportional to its importance, thereby ensuring that 3D Gaussian with high importance has a higher chance of being retained and retaining the key details of the image.

[0024] In order to verify the effectiveness of the method of the present invention, comparative experiments and ablation experiments are conducted. First, the dataset and training details used are introduced, and then the comparative experimental results of different algorithms on the dataset are presented. A series of ablation experiments are performed to evaluate the effectiveness of the adversarial network composed of the mask generation network and the discriminant network, the importance weighted sampling, and the adaptive splitting module.

[0025] The initial learning rate of the model is set to 0.0001, which is gradually adjusted to 0.00005 during the training process. The batch size used is 1. In order to optimize the reduction of the number of Gaussians, no Gaussian reduction operation is performed in the first half of the training, and the focus is entirely on the training of the network. After reaching half of the total number of training iterations, Gaussian reduction begins. During the reduction phase, only the reconstruction loss is retained, and the adversarial network is no longer trained. During the reduction process, the Gaussian reduction frequencies of the adversarial network and importance sampling are different: the adversarial network performs Gaussian reduction once every 25 iterations, while importance sampling performs reduction once every 1000 iterations. This different reduction frequency ensures that in the later stages of model training, unimportant Gaussians can be reduced more accurately, thereby improving the rendering efficiency of the model while maintaining a high reconstruction accuracy.

[0026] The method proposed in this paper is compared with several current mainstream 3D reconstruction technologies. The experimental data sets include MipNeRF360 and Deep Blending. In the comparative experiment, the better results in the experimental data and pre-trained models are selected, covering models such as Compact3DGS, LightGaussian, and Mini-Splatting. The experimental results are shown in Table 1:

[0027] In order to further verify the effectiveness of each module in the model, an ablation experiment was conducted. In the experiment, the adversarial network, importance weighted sampling, and adaptive split modules were removed respectively to evaluate the impact of these modules on the overall effect, and the model performance after different modules were removed was compared with the complete model. The ablation experiment results are shown in Table 2. The check marks in Table 2 indicate that the corresponding modules are retained:

[0028] Among the evaluation indicators, PSNR and SSIM represent peak signal-to-noise ratio and structural similarity respectively, FPS represents frames per second, and LPIPS measures the perceived similarity. PSNR mainly reflects the clarity of image reconstruction. The higher the value, the closer the reconstructed image is to the original image, while SSIM takes into account the similarity of the image in brightness, contrast, and structure. LPIPS is used to measure the difference in image perception. The lower the value, the closer the reconstructed image is to the real image in perception.

[0029] As can be seen from Table 1, in the MipNeRF360 dataset, the method proposed in the present invention performs 28.20, 0.845 and 0.226 in PSNR, SSIM and LPIPS respectively, and is particularly outstanding in optimizing the number of Gaussians, requiring only 0.49M Gaussians. These results have obvious advantages over other comparative methods, especially in significantly reducing the number of Gaussians while maintaining high-quality reconstruction. In the Deep Blending dataset, the PSNR, SSIM and LPIPS of the present invention are 35.56, 0.939 and 0.203 respectively, and also maintains excellent reconstruction quality when the number of Gaussians is reduced to 0.51M.

[0030] As can be seen from Table 2, after adding the adversarial network, the method proposed in the present invention improves PSNR, SSIM and LPIPS by 0.74% (from 28.10 to 28.20), 0.25% (from 0.845 to 0.847) and 0.52% (from 0.226 to 0.225) respectively. It can be seen that the adversarial network has significantly improved the reconstruction quality of the model, especially in detail preservation. After introducing the adaptive split module, PSNR and SSIM increased by 1.12% (from 28.20 to 28.52) and 0.52% (from 0.847 to 0.851), respectively, indicating that the module is more effective in Gaussian distribution optimization and visual detail enhancement. Importance weighted sampling further improves the performance of the model, so that PSNR and SSIM maintain high accuracy while reducing the number of Gaussians. Based on the results in Table 2, each module in the method proposed in the present invention has brought stable improvement to the model in the ablation experiment, which verifies the effectiveness of each module in improving the reconstruction quality and reducing the number of Gaussians.

[0031] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An efficient 3D reconstruction method based on data-driven compression and adaptive splitting, characterized in that: Execute the following steps S1 to S7 to complete the three-dimensional reconstruction of the target scene: Step S1: Acquire real images of the target scene from different perspectives, generate sparse point clouds through the SfM method, and initialize the sparse point clouds into multiple 3D Gaussians. The attributes of the 3D Gaussians include the center position, size, shape, color, transparency, and learnable features in the three-dimensional space. Step S2: construct a mask generation network and a discriminant network. The mask generation network selects the 3D Gaussian to be eliminated based on the learnable features of the 3D Gaussian, generates a rendering image with some 3D Gaussian eliminated based on data-driven, and outputs a mask; the discriminant network is used to judge the difference between the rendering image with some 3D Gaussian eliminated and the real image; Step S3: construct a differentiable raster renderer to optimize the properties of the 3D Gaussian; Step S4: randomly select a viewing angle, input the real image of the viewing angle into the differentiable grating renderer, render the 3D Gaussian at the viewing angle through the differentiable grating renderer, and generate a complete 3D Gaussian rendering image; Step S5: the mask generation network outputs the importance score of the 3D Gaussian according to the learnable features and perspective information of the 3D Gaussian, and generates a mask according to the threshold; uses the generated mask to eliminate the specific 3D Gaussian; obtains the image after the Gaussian elimination, renders the image after the Gaussian elimination, and generates a rendering image with part of the 3D Gaussian eliminated; Step S6: the discriminant network compares the complete 3D Gaussian rendering, the rendering with some 3D Gaussians eliminated, and the real image; Construct the loss function of the mask generation network, the discriminant network and the scene reconstruction, and use the Adam optimizer to iteratively optimize and update the mask generation network and the discriminant network parameters and the properties of the 3D Gaussian through back propagation; Step S7: Determine whether the preset number of iterations has been reached. If so, proceed to the Gaussian reduction stage. Based on the mask generated by the mask generation network and the importance weighted sampling method, calculate the total importance score of the 3D Gaussian, evaluate the contribution of the 3D Gaussian to the target scene, select the 3D Gaussian with a total importance score greater than the preset value for retention, eliminate other 3D Gaussians, and output the optimized Gaussian representation; If the preset number of iterations is not reached, the Gaussian growth stage is performed to screen out incompletely reconstructed 3D Gaussians, process them according to their scale, adaptively split the 3D Gaussians whose scales are greater than the preset value, and copy the other 3D Gaussians; then return to step S2 to continue iterating until the preset number of iterations is reached, and the three-dimensional reconstruction of the target scene is completed.

2. The efficient 3D reconstruction method based on data-driven compression and adaptive splitting according to claim 1, characterized in that: The 3D Gaussian in step S1 is defined as follows: ； In the formula, G i ( x ) is a 3D Gaussian, defined as an ellipsoid with specific properties, x Represents each point in the area where the ellipsoid lies, μ i , Σ i are the properties of a 3D Gaussian, where μ i is the center position of the 3D Gaussian in three-dimensional space; Σ i Indicates size The covariance matrix defines the size of the 3D Gaussian and its shape in space; The attributes of 3D Gaussian also include color, transparency, and learnable features, among which the learnable feature F i Using a nine-dimensional first-order tensor express.

3. The efficient 3D reconstruction method based on data-driven compression and adaptive splitting according to claim 1, characterized in that: The structures and functions of the mask generation network and the discriminant network described in step S2 are as follows: The mask generation network is implemented using tiny-cuda-nn, using ReLU as the activation function. The input of the mask generation network includes the learnable features of 3D Gaussian. , viewing angle information, the mask generation network determines whether each 3D Gaussian should be retained or eliminated according to the learnable features; the mask generation network processes the 3D Gaussian according to the viewing angle information and generates a rendering image that eliminates some 3D Gaussians, and outputs a mask; The discriminant network adopts a convolutional neural network structure, compares the generated rendering image with part of the 3D Gaussian eliminated with the real image, and outputs the discrimination result. The mask generation network gradually optimizes the generated mask by minimizing the feedback of the discriminant network.

4. The efficient 3D reconstruction method based on data-driven compression and adaptive splitting according to claim 1, characterized in that: The specific process of step S5 is as follows: Step S5.1: The mask generation network uses a two-layer multilayer perceptron, denoted as , the mask generates the network input learnable features and viewing angle information , generating the corresponding 3D Gaussian importance score , the importance score generation process is as follows: ； in, Indicates that the features can be learned F i and viewing angle information θ i Concatenate as input to the mask generation network, I i represents the importance score of 3D Gaussian; Step S5.2: Importance score for each 3D Gaussian I i Used to generate the mask M i , the generation formula is as follows: ； in, is the indicator function, It stops the gradient operation, and threshold represents the threshold value; The threshold is calculated by the following formula: ； in, , , N is the current number of 3D Gaussians, is the proportionality coefficient, is the maximum number of 3D Gaussians; Step S5.3: Generated mask M i It is used to select and process the corresponding 3D Gaussians, and use the threshold threshold to filter the 3D Gaussians. The filtered 3D Gaussians will continue to be rendered through the 3D Gaussian point algorithm to generate a rendering image with some 3D Gaussians eliminated.

5. The efficient 3D reconstruction method based on data-driven compression and adaptive splitting according to claim 1, characterized in that: The loss function of the mask generation network in step S6 Loss G As follows: ； in, represents the L2 norm of the learnable feature, and σ represents the Sigmod operation; Loss function of the discriminative network Loss D As follows: ； in, Loss R represents the loss of the real image, Loss F Indicates the loss of rendering by eliminating some 3D Gaussians; The discriminative network receives two types of images as input: real images Img R And the rendering with some 3D Gaussians removed Img F ; Loss of real image Loss R And eliminate some of the loss of 3D Gaussian renderings Loss F The calculation formula is as follows: ； ； in, D ( Img R )and D ( Img F ) represent the output of the discriminant network for the real image and the rendering image with some 3D Gaussians eliminated, BCE represents the binary cross entropy loss, and MSE represents the mean square error loss; Loss function for scene reconstruction Loss Restruction As follows: ； in, Img Represents a complete 3D Gaussian rendering. Img R For real images, λ is a hyperparameter; L1 represents L1 loss, SSIM represents structural similarity loss; Total loss function Loss total As follows: ； in, λ G represents the hyperparameters of the mask generation network, λ D Represents the hyperparameters of the discriminative network.

6. The efficient 3D reconstruction method based on data-driven compression and adaptive splitting according to claim 1, characterized in that: The Gaussian growth phase in step S7 specifically includes the following processes: Step S7.1.1: Filter out 3D Gaussians whose scale is larger than a preset value, define them as large Gaussians, and use the Scharr edge detection operator to calculate the complete 3D Gaussian rendering Img and real images Img R The gradient of is as follows: ； ； In the formula, G img Represents a complete 3D Gaussian rendering Img The gradient of G ImgR Represents a real image Img R The gradient of Compute gradient differences G d : ； Step S7.1.2: Project the center position of each large Gaussian from the three-dimensional space to the image plane, and calculate the image gradient at the projection point; if the gradient difference value at the projection point of the 3D Gaussian is less than 25% of the maximum gradient difference, the 3D Gaussian is marked as incompletely reconstructed, and the formula is as follows: ； Where, Threshold represents the threshold; Step S7.1.3: Discretize the target scene into voxels using a voxel grid, and calculate the density of the space by calculating the number of 3D Gaussians within each voxel, as follows: ； in, d v It is a voxel v The density of It is a voxel v The number of Gaussians in vol ( v ) is a voxel v Volume; Step S7.1.4: For the large Gaussian that is marked as incompletely reconstructed, the number of Gaussian splits is dynamically adjusted based on the density of the space where it is located. The formula for the number of Gaussian splits is as follows: ； in, N i Indicates i The number of splits of a large Gaussian, d i It is i The density of the space where the large Gaussian is located, min(D) and max(D) are the minimum and maximum spatial densities respectively; Step S7.1.5: Filter out 3D Gaussians whose scale is less than or equal to the preset value, define them as small Gaussians, calculate the gradient of the center position of the small Gaussians through back propagation, determine whether these gradients are less than the preset threshold, and perform a copy operation on the small Gaussians whose gradient values ​​are lower than the preset threshold.

7. The efficient 3D reconstruction method based on data-driven compression and adaptive splitting according to claim 1, characterized in that: The Gaussian reduction stage in step S7 specifically includes the following processes: Step S7.2.1: At the beginning of the Gaussian reduction phase, every 25 iterations, the mask generation network generates a 3D Gaussian G i Generate its importance score I i ; Step S7.2.2: Importance score for mask generation network output I i , perform binarization and generate a mask M i , the mask is generated every 25 iterations; Step S7.2.3: Perform importance weighted sampling every 1000 iterations, randomly select and retain 95% of the 3D Gaussians each time; calculate the value of each 3D Gaussian at the viewing angle j The blending weights and hit counts on each pixel in; 3D Gaussian G i The mixed weight ω i The calculation of is as follows: ； in, Is the indicator function, used to judge the 3D Gaussian G i Is it consistent with the ray r that passes through the pixel (u,v) from the center of the camera? u,v intersect; σ i For 3D Gaussian G i The opacity, H and W are the height and width of the image respectively; 3D Gaussian G i Hit count c i The calculation of is as follows: ； Calculate 3D Gaussian G i Total importance score S i As follows: ； In the formula, N v Indicates the total number of viewing angles; The total importance score of each 3D Gaussian is normalized as follows: ； In the formula, is the normalized 3D Gaussian G i The total importance score of .

8. The efficient 3D reconstruction method based on data-driven compression and adaptive splitting according to claim 1, characterized in that: The preset number of iterations in step S7 is half of the total number of iterations.

Images