Efficient 3D Gaussian scene reconstruction method based on multimodal depth distribution supervision

Abstract

The present invention discloses an efficient 3D Gaussian scene reconstruction method based on multimodal depth distribution supervision. Compared with the prior art, the pixel-level deep dense point cloud is directly extracted in the Gaussian initialization stage to generate a multimodal depth distribution map, which effectively avoids the random initialization of dense Gaussian positions according to the COLMAP sparse point cloud, solves the problem that the point cloud obtained by COLMAP is too sparse and provides insufficient geometric constraints in the case of few views, and the randomly initialized Gaussian is hindered from moving to the correct position due to the local minimum. The present invention applies an image segmentation algorithm based on edge detection to the image frame, integrates a large number of Gaussian primitives that are far away from the edge points and have similar depths, effectively prunes the number of 3D Gaussians, and saves GPU memory consumption. The present invention introduces an additional isotropic regularization loss to the global Gaussian to avoid the problem of highly elongated Gaussian distribution that produces artifacts along the line of sight.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional reconstruction and new perspective synthesis, and specifically relates to an efficient 3D Gaussian scene reconstruction method based on multi-modal depth distribution supervision. Background Art

[0002] The development of neural radiance fields has revolutionized the field of 3D reconstruction and new view synthesis by generating high-quality, realistic 3D models that can render realistic surface and texture details of objects at any angle and distance. However, the volume rendering technology used in it usually requires dozens of queries on the neural radiance field parameters for each ray. This high computational demand has become a key obstacle to achieving real-time rendering. The recently proposed 3D reconstruction technology based on Gaussian splashing uses Gaussian primitives to explicitly represent the scene and render it through rasterization, which effectively solves this problem. It achieves photo-realistic new view synthesis and fast reconstruction speed, becoming another remarkable progress in the study of reconstructing 3D scenes from 2D images. However, the key challenge of fitting the input view based on Gaussian primitives is the sensitivity to local minima, which is a well-known non-convex problem. The 3D Gaussian initialized at a random position has a local minimum due to the existence of local support, which hinders it from moving to the correct position in space. It is easy to fall into overfitting when only based on multi-view color supervision without global structure, resulting in incorrect geometric reconstruction. This problem is more obvious when there are fewer images used to optimize the 3D scene. Supplementing with additional geometric clues, depth, helps solve this problem. Depth provides intuitive geometric information and has been proven to be of great value for 3D scene reconstruction. Currently, the application of depth in 3D Gaussian-based work can be divided into two categories: First, using depth point clouds to initialize Gaussians, intuitively completing a simple depth-aware Gaussian, but not using depth for geometric supervision, and the geometric structure of the scene still relies only on RGB views; second, applying a deep rendering loss similar to NeRF to RGBD input or using a few-view input scene with monocular depth estimation, but rendering depth not only leads to blurred geometric shapes, but also uses the Gaussian mean position as the entire Gaussian depth and ignores the Gaussian shape, resulting in rough surface reconstruction and free space artifacts. In addition, another important input camera pose in current 3D Gaussian-based work mainly relies on the acquisition of COLMAP, but COLMAP has difficulty in extracting features for scenes with textureless or repetitive texture surfaces, which may ultimately lead to the failure of scene reconstruction. Summary of the invention

[0003] In view of the above-mentioned deficiencies in the prior art, the present invention provides an efficient 3D Gaussian scene reconstruction method based on multimodal depth distribution supervision, which solves the problems in the prior art that only color supervision is prone to cause local overfitting and COLMAP is prone to feature extraction failure leading to reconstruction failure.

[0004] In order to achieve the above-mentioned invention object, the technical solution adopted by the present invention is: an efficient 3D Gaussian scene reconstruction method based on multimodal depth distribution supervision, comprising the following steps:

[0005] S1, inputting a video sequential image frame and using it as the current image frame, and obtaining a multimodal depth distribution map of the current image frame through a fuzzy-aware depth estimation module;

[0006] S2, extract pixel-level multimodal point cloud according to the multimodal depth map, initialize 3D Gaussian position distribution, and learn 3D Gaussian set with all attributes of mean, scaling matrix, rotation matrix, opacity and spherical harmonic coefficients;

[0007] S3, based on the learned 3D Gaussian set, minimize the photometric loss of the rendered image and the current image frame to obtain the trained 3D Gaussian set;

[0008] S4, initialize the learnable SE-3 affine transformation, fix the 3D Gaussian set attributes, use the trained 3D Gaussian set to render the next image frame, minimize the photometric loss between the rendered image and the next image frame to optimize the SE-3 affine transformation, and obtain the estimated camera pose of the next image frame;

[0009] S5, adding the trained 3D Gaussian set to the global 3D Gaussian set, and iteratively optimizing the 3D Gaussian properties and the pose of the next image frame through the photometric loss combined with the multimodal depth distribution consistency strategy according to the current image frame and the estimated camera pose of the current image frame and the next image frame;

[0010] S6, applying the traditional image segmentation algorithm based on edge detection to the current image frame, retaining the Gaussian primitives close to the edge points, fusing the Gaussian primitives far from the edge points and with similar depths, passing the camera pose of the next image frame to the optimized camera pose set, and judging whether the input of the video sequence image frame is finished, if so, entering S7, if not, obtaining the next video sequence image frame, and returning to S1;

[0011] S7. Based on the final global 3D Gaussian set, according to the photometric loss and the isotropic regularization loss, the final global 3D Gaussian set and the camera pose of the camera pose set are iteratively optimized to complete the 3D Gaussian scene reconstruction.

[0012] Further: In S1, the method of obtaining the multimodal depth distribution map of the current image frame by the fuzzy perception depth estimation module is specifically as follows:

[0013] The multimodal distribution of depth estimation is learned using conditional implicit maximum likelihood estimation. The image and several sets of latent codes are input into the fuzzy-aware depth estimation module, and a multimodal depth distribution map is output.

[0014] Further: In S2, the initialized 3D Gaussian distribution includes the following properties:

[0015] Center position, scaling factor, rotation factor, opacity and spherical harmonic coefficients, where the scaling factor and rotation factor constitute the covariance matrix describing the 3D Gaussian ellipsoid , its specific expression is:

[0016]

[0017] in, S is the scaling matrix, , For Gauss x The scaling factor of the axis direction, s y For Gauss y The scaling factor of the axis direction, For Gauss z The scaling factor of the axis direction, R is the rotation matrix, , is the distance from the Gaussian center to the origin in the world coordinate system, is the Gaussian center in the world coordinate system x The projection coordinates on the axis, r y is the Gaussian center in the world coordinate system y The projection coordinates on the axis, is the Gaussian center in the world coordinate system z The projection coordinates on the axis, is the formula for constructing a rotation matrix from quaternions, T is the transpose symbol.

[0018] Further: In S2, the method for extracting pixel-level multimodal point cloud according to the multimodal depth map is specifically:

[0019] According to the camera intrinsic parameter matrix Calculate the multimodal depth map for each pixel Three-dimensional coordinates in the camera coordinate system ;

[0020]

[0021] In the formula, is the camera coordinate system x Axis coordinates, is the camera coordinate system y Axis coordinates, is the camera coordinate system z Axis coordinates, u is the coordinate of the pixel on the horizontal axis in the image coordinate system,v is the coordinate of the pixel on the vertical axis in the image coordinate system, the camera intrinsic parameter matrix The expression is:

[0022]

[0023] In the formula, For the camera x The focal length of the axis, f y For the camera y The focal length of the axis, For The offset of the image center in the axis direction in the camera coordinate system, c y For The offset of the image center in the axial direction in the camera coordinate system;

[0024] The three-dimensional coordinates in the camera coordinate system are transformed into the point cloud position in the world coordinate system through the camera pose matrix ;

[0025]

[0026] In the formula, In the world coordinate system x Axis coordinates, In the world coordinate system y Axis coordinates, In the world coordinate system z Axis coordinates, T is the camera pose matrix.

[0027] Further: In S3, the color of the minimized rendered image is represented as an alpha blend of all ordered points covering the pixel, and the color of the pixel finally rendered is The specific expression is:

[0028]

[0029] In the formula, The pixel's i The color of the ordered points, is the pixel i The opacity of the ordered points, is the pixel j The opacity of the ordered points, N is the number of ordered points covering the pixel, given by the covariance matrix The weighted learnable per-point opacity and SH color coefficients are calculated, N is the number of ordered points covering the pixel, The final rendered color of the pixel.

[0030] Further: In S3, the luminosity loss The specific expression is:

[0031]

[0032] In the formula, For color loss, is the D-SSIM loss, is the first adjustment parameter.

[0033] Further: In S5, the method of iteratively optimizing the 3D Gaussian properties and the camera pose of the next image frame by combining the photometric loss with the multimodal depth distribution consistency strategy comprises the following sub-steps:

[0034] S51, setting the opacity according to the multimodal depth distribution consistency strategy, assigning the gradient of the photometric loss relative to the opacity to the gradient of the depth probability, increasing the opacity of the Gaussian at the correct spatial position by gradient descent, degenerating the opacity of the Gaussian at the incorrect position, iteratively optimizing the opacity, covariance matrix and spherical harmonic coefficients of the 3D Gaussian attribute according to a preset first iteration threshold, thereby determining a Gaussian mean that is consistent in multiple views, where the multiple views include the current image frame and the next image frame;

[0035] S52, fixing the current Gaussian property, rendering the next frame of image, iteratively minimizing the photometric loss of the rendered image and the next frame of true value image according to a preset second iteration threshold, and optimizing the camera pose of the next image frame.

[0036] Further: In S7, the isotropic regularization loss The specific expression is:

[0037]

[0038] In the formula, For the i Gauss The scaling matrix, is the mean of the values ​​in the three directions of the scaling matrix, For isomorphic matrix, G is the global 3D Gaussian set.

[0039] The beneficial effects of the present invention are as follows: the present invention provides an efficient 3D Gaussian scene reconstruction method based on multimodal depth distribution supervision, which has the following advantages over the prior art:

[0040] (1) In the Gaussian initialization stage, a pixel-level deep dense point cloud is directly extracted to generate a multimodal depth distribution map, which effectively avoids the random initialization of dense Gaussian positions based on the COLMAP sparse point cloud. This solves the problem that the point cloud obtained by COLMAP is too sparse in the few-view case and the geometric constraints are insufficient. At the same time, the randomly initialized Gaussian is hindered from moving to the correct position due to the local minimum.

[0041] (2) The present invention retains the inherent ambiguity of monocular depth estimation through a fuzzy-aware depth estimation module. Only when information from multiple views is available can the depth estimation algorithm be selected from all views. Figure 1 The present invention supervises the mode of depth distribution rather than the moment of depth distribution. The moment of depth distribution, i.e., rendering depth, only constrains the integral value gathered along the ray, and cannot remove translucent artifacts before the actual depth, resulting in blurred geometric shapes. The mode of depth distribution provides sufficient geometric constraints for the Gaussian primitives used to fit few views, making the reconstruction result more accurate.

[0042] (3) The present invention performs photometric loss on the next frame image by Gaussian rendering of the current frame and the true RGB image, and learns the relative affine transformation of the image frame. This makes the method of the present invention no longer limited to the performance limitation of COLMAP, which has difficulty in processing scenes with smooth or repetitive textures and falls into collapse, and shows a more robust reconstruction result.

[0043] (4) The present invention applies an edge detection-based image segmentation algorithm to the image frame, fuses a large number of Gaussian primitives that are far away from the edge points and have similar depths, effectively prunes the number of 3D Gaussians, and saves GPU memory consumption.

[0044] (5) The present invention introduces an additional isotropic regularization loss to the global Gaussian, avoiding the problem of highly elongated Gaussian distribution that produces artifacts along the line of sight.

[0045] In summary, the method provided by the present invention achieves higher quality and more robust new view synthesis and 3D scene reconstruction results. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 Flowchart of the efficient 3D Gaussian scene reconstruction method based on multimodal depth distribution supervision. DETAILED DESCRIPTION

[0047] The specific implementation modes of the present invention are described below so that those skilled in the art can understand the present invention. However, it should be clear that the present invention is not limited to the scope of the specific implementation modes. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations utilizing the concept of the present invention are protected.

[0048] like Figure 1 As shown, in one embodiment of the present invention, an efficient 3D Gaussian scene reconstruction method based on multimodal depth distribution supervision includes the following steps:

[0049] S1, inputting a video sequential image frame and using it as the current image frame, and obtaining a multimodal depth distribution map of the current image frame through a fuzzy-aware depth estimation module;

[0050] S2, extract pixel-level multimodal point cloud according to the multimodal depth map, initialize 3D Gaussian position distribution, and learn 3D Gaussian set with all attributes of mean, scaling matrix, rotation matrix, opacity and spherical harmonic coefficients;

[0051] S3, based on the learned 3D Gaussian set, minimize the photometric loss of the rendered image and the current image frame to obtain the trained 3D Gaussian set;

[0052] S4, initialize the learnable SE-3 affine transformation, fix the 3D Gaussian set attributes, use the trained 3D Gaussian set to render the next image frame, minimize the photometric loss between the rendered image and the next image frame to optimize the SE-3 affine transformation, and obtain the estimated camera pose of the next image frame;

[0053] S5, adding the trained 3D Gaussian set to the global 3D Gaussian set, and iteratively optimizing the 3D Gaussian properties and the pose of the next image frame through the photometric loss combined with the multimodal depth distribution consistency strategy according to the current image frame and the estimated camera pose of the current image frame and the next image frame;

[0054] S6, applying the traditional image segmentation algorithm based on edge detection to the current image frame, retaining the Gaussian primitives close to the edge points, fusing the Gaussian primitives far from the edge points and with similar depths, passing the camera pose of the next image frame to the optimized camera pose set, and judging whether the input of the video sequence image frame is finished, if so, entering S7, if not, obtaining the next video sequence image frame, and returning to S1;

[0055] S7. Based on the final global 3D Gaussian set, according to the photometric loss and the isotropic regularization loss, the final global 3D Gaussian set and the camera pose of the camera pose set are iteratively optimized to complete the 3D Gaussian scene reconstruction.

[0056] In S1, the method for obtaining the multimodal depth distribution map of the current image frame through the blur-aware depth estimation module is specifically as follows:

[0057] The multimodal distribution of depth estimation is learned using conditional implicit maximum likelihood estimation. The image and several sets of latent codes are input into the fuzzy-aware depth estimation module, and a multimodal depth distribution map is output.

[0058] In this embodiment, an image and M groups of latent codes are input into a blur-aware depth estimation module, which outputs a multimodal depth distribution map, with M depth maps for each image.

[0059] The multimodal depth distribution map of each image is obtained through the fuzzy-aware depth estimation module, and a dense point cloud based on the multimodal depth distribution is directly extracted for the scene to generate a multimodal depth distribution map. This effectively solves the problem that the point cloud obtained by COLMAP in the few-view case is too sparse and provides insufficient geometric constraints, and the randomly initialized Gaussian is hindered from moving to the correct position due to the local minimum.

[0060] However, there is an inherent scale ambiguity in monocular depth estimation, which causes key geometric conflict problems in multi-views. The present invention introduces conditional implicit maximum likelihood estimation (cIMAX) to retain the inherent ambiguity of monocular depth. When information from multiple views is available, the present invention defines a principled loss based on the probability distribution of depth estimates from different views to solve the scale ambiguity of monocular depth. The loss selects a subset of patterns with consistent probability in all views, providing effective geometric supervision based on multi-modal depth distribution for Gaussian optimization.

[0061] In S2, the initialized 3D Gaussian distribution has the following properties:

[0062] Center position, scaling factor, rotation factor, opacity and spherical harmonic coefficients, where the scaling factor and rotation factor constitute the covariance matrix describing the 3D Gaussian ellipsoid , its specific expression is:

[0063]

[0064] in, S is the scaling matrix, , For Gauss x The scaling factor of the axis direction, s y For Gauss y The scaling factor of the axis direction, For Gauss z The scaling factor of the axis direction, R is the rotation matrix, , is the distance from the Gaussian center to the origin in the world coordinate system, is the Gaussian center in the world coordinate system x The projection coordinates on the axis, r y is the Gaussian center in the world coordinate system y The projection coordinates on the axis, is the Gaussian center in the world coordinate system zThe projection coordinates on the axis, is the formula for constructing a rotation matrix from quaternions, T is the transpose symbol.

[0065] In S2, the method for extracting pixel-level multimodal point cloud according to the multimodal depth map is specifically as follows:

[0066] According to the camera intrinsic parameter matrix Calculate the multimodal depth map for each pixel Three-dimensional coordinates in the camera coordinate system ;

[0067]

[0068] In the formula, is the camera coordinate system x Axis coordinates, is the camera coordinate system y Axis coordinates, is the camera coordinate system z Axis coordinates, u is the coordinate of the pixel on the horizontal axis in the image coordinate system, v is the coordinate of the pixel on the vertical axis in the image coordinate system, the camera intrinsic parameter matrix The expression is:

[0069]

[0070] In the formula, For the camera x The focal length of the axis, f y For the camera y The focal length of the axis, For The offset of the image center in the axis direction in the camera coordinate system, c y For The offset of the image center in the axial direction in the camera coordinate system;

[0071] The three-dimensional coordinates in the camera coordinate system are transformed into the point cloud position in the world coordinate system through the camera pose matrix ;

[0072]

[0073] In the formula, In the world coordinate system x Axis coordinates, In the world coordinate system y Axis coordinates, In the world coordinate system z Axis coordinates,T is the camera pose matrix.

[0074] In S3, the method of minimizing the rendered image is specifically: the process of rendering the Gaussian primitives from a given camera view onto the image plane is achieved by approximating the projection of the 3D Gaussian along the depth dimension as pixel coordinates. Given a perspective transformation , the covariance matrix in camera coordinates It can be expressed as:

[0075]

[0076] In the formula, is the Jacobian matrix of the affine approximation of the projective transformation, W is the view transformation matrix.

[0077] For each pixel, calculate the color and opacity of all Gaussian distributions using the following formula;

[0078]

[0079] In the formula, is a three-dimensional point, is the Gaussian mean, is the probability density function of the three-dimensional Gaussian distribution, that is, the probability density corresponding to the three-dimensional point.

[0080] Minimally the color of the rendered image is represented as an alpha blend of all the ordered points covering the pixel.

[0081]

[0082] In the formula, The pixel's i The color of the ordered points, is the pixel i The opacity of the ordered points, is the pixel j The opacity of the ordered points, N is the number of ordered points covering the pixel, given by the covariance matrix The weighted learnable per-point opacity and SH color coefficients are calculated, N is the number of ordered points covering the pixel, The final rendered color of the pixel.

[0083] In S3, the photometric loss The specific expression is:

[0084]

[0085] In the formula, For color loss, is the D-SSIM loss, is the first adjustment parameter.

[0086] In this embodiment, the luminosity loss is a combination of color loss and D-SSIM loss, and the first adjustment parameter is set to 0.2, which can make the running process faster.

[0087] In particular, for the affine transformation in S4 , the present invention observes that for a given set of The three-dimensional Gaussian point at the center is transformed by the perspective matrix Projecting them gives: , Express The projection transformation, express The distance to the plane along the projection direction. 2D projection From a set of rigidly transformed points, orthogonal directions can be Get, that is ,get Then the estimated viewing angle transformation matrix is , which is equivalent to estimating the transformation of a set of three-dimensional Gaussian points. The perspective transformation matrix , also called the camera extrinsic matrix, is similar to the camera pose matrix are inverse matrices of each other.

[0088] In this embodiment, for another important input required in the existing algorithm - camera pose, the present invention adopts a local 3DGS camera pose estimation method, learns an affine transformation for each sequential frame of the input video, and fixes the 3D Gaussian set in the affine transformation optimization process. The image frame to be estimated is rendered with Gaussian primitives trained with images of known poses, and the affine transformation is optimized according to the photometric loss to obtain the camera pose of the current image frame and the next image frame. At the same time, the Gaussian set is synchronously grown to achieve scene expansion.

[0089] In S5, the method for iteratively optimizing 3D Gaussian properties and the camera pose of the next image frame by combining photometric loss with a multimodal depth distribution consistency strategy includes the following sub-steps:

[0090] S51. Set the opacity according to the multimodal depth distribution consistency strategy, assign the gradient of the photometric loss relative to the opacity to the gradient of the depth probability, increase the opacity of the Gaussian at the correct spatial position by gradient descent, degrade the opacity of the Gaussian at the incorrect position, and iteratively optimize the opacity of the 3D Gaussian attribute according to the preset first iteration threshold. , covariance matrix Spherical harmonic coefficients SH , thereby determining a consistent Gaussian mean across multiple views , the multi-view includes the current image frame and the next image frame;

[0091] S52, fixing the current Gaussian property, rendering the next frame of image, iteratively minimizing the photometric loss of the rendered image and the next frame of true value image according to a preset second iteration threshold, and optimizing the camera pose of the next image frame.

[0092] In this embodiment, the opacity , The probability of the depth sampled from the blur-aware depth estimation module is estimated by conditional implicit maximum likelihood for the latent code and renormalized to sum to 1. Relative to Opacity The gradient of is assigned to the gradient of the depth probability, that is , so that the gradient descent increases the opacity of the correct Gaussian and degenerates the opacity of the incorrect Gaussian, thereby determining a consistent Gaussian mean in multiple views Location.

[0093] The present invention introduces a multimodal depth distribution consistency strategy to obtain a consistent Gaussian mean position in multiple views, and applies an isotropic regularization loss to the Gaussian primitives to avoid artifacts caused by excessive elongation of the Gaussian primitive shape.

[0094] Preferably, the traditional image segmentation algorithm based on edge detection used in this embodiment is easy to understand that the grayscale values ​​of pixels on the boundaries of different regions change dramatically. Then the image is converted to the frequency domain through Fourier transform, and the regions with relatively high frequencies correspond to the boundaries of different regions of the image. Although it is difficult for this algorithm to ensure the continuity and closure of the edges, and there are a large number of broken edges in high-detail regions, making it difficult to form large regions, it achieves fast and accurate edge positioning. The present invention retains the existence of such a large number of broken edges and small Gaussians close to the edge lines. For Gaussians far away from the edge points and with similar depths, the present invention fuses them to represent large and flat regions with relatively few details, thereby achieving effective pruning of Gaussian primitives.

[0095] The present invention applies a fusion strategy to Gaussian primitives in the adaptive density control stage. This embodiment applies a traditional image segmentation algorithm based on edge detection to the image frame. This solution can effectively reduce the number of Gaussians and the memory consumption of the GPU, and achieve efficient rendering and 3D scene reconstruction.

[0096] In S7, the homogeneous regularization loss The specific expression is:

[0097]

[0098] In the formula, For the iGauss The scaling matrix, is the mean of the values ​​in the three directions of the scaling matrix, For isomorphic matrix, G is the global 3D Gaussian set.

[0099] The final global loss for:

[0100]

[0101] In the formula, is the second adjustment parameter.

[0102] In the description of the present invention, it is necessary to understand that the orientation or positional relationship indicated by the terms "center", "thickness", "upper", "lower", "horizontal", "top", "bottom", "inner", "outer", "radial", etc. is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", and "third" are used only for descriptive purposes, and cannot be understood as indicating or implying the relative importance or the number of implicitly specified technical features. Therefore, the features defined by "first", "second", and "third" may explicitly or implicitly include one or more of the features.

Claims

1. An efficient 3D Gaussian scene reconstruction method based on multimodal depth distribution supervision, characterized in that: The following steps are involved: S1, inputting a video sequential image frame and using it as the current image frame, and obtaining a multimodal depth distribution map of the current image frame through a fuzzy-aware depth estimation module; S2, extract pixel-level multimodal point cloud according to the multimodal depth map, initialize 3D Gaussian position distribution, and learn 3D Gaussian set with all attributes of mean, scaling matrix, rotation matrix, opacity and spherical harmonic coefficients; S3, based on the learned 3D Gaussian set, minimize the photometric loss of the rendered image and the current image frame to obtain the trained 3D Gaussian set; S4, initialize the learnable SE-3 affine transformation, fix the 3D Gaussian set attributes, use the trained 3D Gaussian set to render the next image frame, minimize the photometric loss between the rendered image and the next image frame to optimize the SE-3 affine transformation, and obtain the estimated camera pose of the next image frame; S5, adding the trained 3D Gaussian set to the global 3D Gaussian set, and iteratively optimizing the 3D Gaussian properties and the pose of the next image frame through the photometric loss combined with the multimodal depth distribution consistency strategy according to the current image frame and the estimated camera pose of the current image frame and the next image frame; S6, applying the traditional image segmentation algorithm based on edge detection to the current image frame, retaining the Gaussian primitives close to the edge points, fusing the Gaussian primitives far from the edge points and with similar depths, passing the camera pose of the next image frame to the optimized camera pose set, and judging whether the input of the video sequence image frame is finished, if so, entering S7, if not, obtaining the next video sequence image frame, and returning to S1; S7, based on the final global 3D Gaussian set, according to the photometric loss and the isotropic regularization loss, iteratively optimize the final global 3D Gaussian set and the camera pose of the camera pose set to complete the 3D Gaussian scene reconstruction; In S1, the method for obtaining the multimodal depth distribution map of the current image frame through the blur-aware depth estimation module is specifically as follows: The multimodal distribution of depth estimation is learned using conditional implicit maximum likelihood estimation. The image and several sets of latent codes are input into the fuzzy-aware depth estimation module, and a multimodal depth distribution map is output.

2. The efficient 3D Gaussian scene reconstruction method based on multimodal depth distribution supervision according to claim 1, characterized in that: In S2, the initialized 3D Gaussian distribution has the following properties: Center position, scaling factor, rotation factor, opacity and spherical harmonic coefficients, where the scaling factor and rotation factor constitute the covariance matrix describing the 3D Gaussian ellipsoid , its specific expression is: in, S is the scaling matrix, , For Gauss x The scaling factor of the axis direction, s y For Gauss y The scaling factor of the axis direction, For Gauss z The scaling factor of the axis direction, R is the rotation matrix, , is the distance from the Gaussian center to the origin in the world coordinate system, is the Gaussian center in the world coordinate system x The projection coordinates on the axis, r y is the Gaussian center in the world coordinate system y The projection coordinates on the axis, is the Gaussian center in the world coordinate system z The projection coordinates on the axis, is the formula for constructing a rotation matrix from quaternions, T is the transpose symbol.

3. The efficient 3D Gaussian scene reconstruction method based on multimodal depth distribution supervision according to claim 1 is characterized in that: In S2, the method for extracting pixel-level multimodal point cloud according to the multimodal depth map is specifically as follows: According to the camera intrinsic parameter matrix Calculate the multimodal depth map for each pixel Three-dimensional coordinates in the camera coordinate system ; In the formula, is the camera coordinate system x Axis coordinates, is the camera coordinate system y Axis coordinates, is the camera coordinate system z Axis coordinates, u is the coordinate of the pixel on the horizontal axis in the image coordinate system, v is the coordinate of the pixel on the vertical axis in the image coordinate system, the camera intrinsic parameter matrix The expression is: In the formula, For the camera x The focal length of the axis, f y For the camera y The focal length of the axis, For The offset of the image center in the axis direction in the camera coordinate system, c y For The offset of the image center in the axial direction in the camera coordinate system; The three-dimensional coordinates in the camera coordinate system are transformed into the point cloud position in the world coordinate system through the camera pose matrix ; In the formula, In the world coordinate system x Axis coordinates, In the world coordinate system y Axis coordinates, In the world coordinate system z Axis coordinates, T is the camera pose matrix.

4. The efficient 3D Gaussian scene reconstruction method based on multimodal depth distribution supervision according to claim 1, characterized in that: In S3, the color of the minimized rendered image is represented as an alpha blend of all ordered points covering the pixel, and the color of the pixel finally rendered The specific expression is: In the formula, The pixel's i The color of the ordered points, is the pixel i The opacity of the ordered points, is the pixel j The opacity of the ordered points, N is the number of ordered points covering the pixel, given by the covariance matrix The weighted learnable per-point opacity and SH color coefficients are calculated, N is the number of ordered points covering the pixel, The final rendered color of the pixel.

5. The efficient 3D Gaussian scene reconstruction method based on multimodal depth distribution supervision according to claim 1, characterized in that: In S3, the photometric loss The specific expression is: In the formula, For color loss, is the D-SSIM loss, is the first adjustment parameter.

6. The efficient 3D Gaussian scene reconstruction method based on multimodal depth distribution supervision according to claim 1, characterized in that: In S5, the method for iteratively optimizing 3D Gaussian properties and the camera pose of the next image frame by combining photometric loss with a multimodal depth distribution consistency strategy includes the following sub-steps: S51, setting the opacity according to the multimodal depth distribution consistency strategy, assigning the gradient of the photometric loss relative to the opacity to the gradient of the depth probability, increasing the opacity of the Gaussian at the correct spatial position by gradient descent, degenerating the opacity of the Gaussian at the incorrect position, iteratively optimizing the opacity, covariance matrix and spherical harmonic coefficients of the 3D Gaussian attribute according to a preset first iteration threshold, thereby determining a Gaussian mean that is consistent in multiple views, where the multiple views include the current image frame and the next image frame; S52, fixing the current Gaussian property, rendering the next frame of image, iteratively minimizing the photometric loss of the rendered image and the next frame of true value image according to a preset second iteration threshold, and optimizing the camera pose of the next image frame.

7. The efficient 3D Gaussian scene reconstruction method based on multimodal depth distribution supervision according to claim 1, characterized in that: In S7, the homogeneous regularization loss The specific expression is: In the formula, For the i Gauss The scaling matrix, is the mean of the values ​​in the three directions of the scaling matrix, For isomorphic matrix, G is the global 3D Gaussian set.

Images