A perspective synthesis method based on 3D Gaussian splashing technology based on spatial coupling

Abstract

The present invention discloses a perspective synthesis method based on 3D Gaussian splashing technology of spatial coupling. In the method, a multi-scale gridding module structure is used to regularize point clouds at different scales, and the center point of the grid is used as an initialization control point to guide the generation of local 3D Gaussian; the control point itself has its own learnable feature vector as a local context feature, and the local position perception module fuses it with the 3D Gaussian position feature of the current local space; the 3D Gaussian attribute decoding module uses the fusion feature reasoning to estimate the 3D Gaussian attribute generated by the control point; the redundant point clipping module clips the control points that do not meet the rules according to the rule settings in the optimization process; the differentiable rasterization module projects the 3D Gaussian into the 2D image space for rendering according to the specified camera internal and external parameters, and obtains the new perspective synthesis result under the perspective. The present invention is used to solve the problem of slow optimization convergence and easy generation of more redundant Gaussians under 3D Gaussian representation.

Description

Technical Field

[0001] The present invention relates to the fields of computer vision and graphics, and in particular to a perspective synthesis method based on a 3D Gaussian splashing technology with spatial coupling. Background Art

[0002] 3D Gaussian Splatting (3D GS) is a revolutionary technology that has emerged in the field of explicit radiation fields and computer graphics in recent years. This innovative method is characterized by the use of millions of 3D Gaussians, which is very different from the Neural Radiance Field (NeRF) method, which mainly uses implicit, coordinate-based models to map spatial coordinates to pixel values. 3D GS has an explicit scene representation and a differentiable rasterization algorithm. It supports fast rendering while having powerful scene expression capabilities. The following parameters are stored in the 3D Gaussian: position coordinates, covariance matrix (determines the shape of the Gaussian), opacity, spherical harmonics (fits the view-dependent appearance), and the parameters are updated during the iterative optimization process.

[0003] However, many new perspective synthesis methods based on 3D GS are still relatively low in memory efficiency, that is, there are many redundant 3D Gaussians after optimization, which will have an adverse effect on real-time rendering; and there is still room for further improvement in convergence speed and image quality. Summary of the invention

[0004] In order to overcome the shortcomings of the prior art, the present invention provides a perspective synthesis method based on spatially coupled 3D Gaussian splattering technology, which organizes the adaptive adjustment of 3D Gaussian in a structured manner, and fully considers the 3D Gaussian distribution information of the local space during the optimization process, thereby significantly accelerating the convergence process and having high memory efficiency while ensuring a certain rendering quality.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention provides a method for 3D Gaussian splattering technology perspective synthesis based on spatial coupling, which specifically includes the following steps:

[0007] Step 1: First, use the Structure from Motion (SfM) method to estimate the point cloud from a set of images, and send the point cloud to the multi-scale gridding module structure. The point cloud is voxelized using grids of different resolutions. The center point of the grid is called the control point V. For a specific control point v, its initialization coordinates are the coordinates of the grid center point. The control point also initializes a learnable feature vector As a local context feature, one control point corresponds to the generation of k 3D Gaussians, and each control point v is also accompanied by a learnable offset vector ∈R k×3 , and a learnable scale factor ∈R 3 ;

[0008] Step 2: Set the offset vector relative to the control point In order to make the k offset coordinates of the input from the local position perception module have permutation invariance, the module is designed as a symmetric function to aggregate the position information of each point. The module output represents the position distribution characteristics of k 3D Gaussians in the current local space. ;

[0009] Step 3: Local position features The characteristics of the control point itself Splice to get fusion features ;

[0010] Step 4: The viewing direction of a specific control point v relative to the camera is concatenated with the previous fusion features and sent to the 3D Gaussian attribute decoding module to decode the opacity, covariance matrix, and color of k 3D Gaussians respectively;

[0011] Step 5: According to the coordinates of the control points , scale control factor , and the offset vector of the 3D Gaussian, get the final 3D Gaussian position, and set a threshold according to the opacity and remove the 3D Gaussian below the threshold;

[0012] Step 6: According to the specified camera internal and external parameters, the 3D Gaussian at this perspective is sent to the differentiable rasterization module, including the center position, opacity, 3D covariance matrix and color of the 3D Gaussian body, which are projected into the 2D image space for rendering to obtain the new perspective synthesis result at this perspective, and the loss function is calculated with the true value of the image at this perspective, including loss and structural similarity loss ; In order to constrain the scale of the 3D Gaussian not to be too large and produce too much overlap, a regularization term is introduced into the loss function ; The final loss function is in and are two hyperparameters used to control the respective weights of the two losses. Finally, the model parameters are optimized using the back propagation algorithm until convergence.

[0013] The beneficial effects of the present invention compared with the prior art are:

[0014] The multi-scale gridding module structure of the present invention structures disordered point clouds into differentiable grids, which can control the Gaussian balls to be distributed as evenly as possible in space, while avoiding excessive redundant generation of 3D Gaussians during the optimization process; the local position perception module of the present invention can utilize the 3D Gaussian spatial distribution characteristics in the current state to assist in the prediction of 3D Gaussian attributes, thereby accelerating convergence; the present invention directly uses a very shallow multi-layer perceptron (MLP) to regress the colors of multiple Gaussians at one time, and anisotropy is also introduced in this process, which also improves the reconstruction of highlight scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a principle diagram of a perspective synthesis method based on a spatially coupled 3D Gaussian splashing technology of the present invention;

[0016] Figure 2 is a schematic diagram of a local position sensing module of the present invention;

[0017] Figure 3 is a schematic diagram of a 3D Gaussian attribute decoding module of the present invention;

[0018] Figure 4 This is the effect diagram synthesized from the new perspective of the present invention. DETAILED DESCRIPTION

[0019] In order to more clearly illustrate the purpose, technical solutions and advantages of the present invention, the specific implementation methods of the present invention are further described below in conjunction with the accompanying drawings.

[0020] The present invention provides a perspective synthesis method based on a spatially coupled 3D Gaussian splashing technology, which is implemented by a multi-scale gridding module, a local position sensing module, a 3D Gaussian attribute decoding module, a redundant control point clipping module, and a differentiable rasterization module. The 3D Gaussian attributes include opacity, covariance matrix, and color.

[0021] like Figure 1 As shown, a perspective synthesis method based on a spatially coupled 3D Gaussian splattering technology of the present invention specifically comprises the following steps:

[0022] First, a point cloud is estimated from a set of images using the Structure from Motion (SfM) method. ∈R M×3 , M is the number of initial point clouds, the point cloud is sent to the multi-scale gridding module, and the point cloud is voxelized using grids of different resolutions:

[0023] ;

[0024] The center point of the grid is called the control point V∈R N×3 , N is the number of control points, is the grid size, and the multi-scale gridding module in the present invention is Three values ​​are set. For a specific control point v∈V, its initialization coordinates are the coordinates of the grid center point; in addition, the control point v will also initialize a learnable feature vector As a local context feature; one control point v corresponds to the generation of k 3D Gaussians, and k is 10 in the specific implementation; each control point v is also accompanied by a learnable offset vector ∈R k×3 and a learnable scale factor ∈R 3 .

[0025] Then, for a specific camera, within its view frustum, k 3D Gaussians need to be generated from each control point v. The parameterized expression of the 3D Gaussian is It is defined by the 3D covariance matrix Σ in world coordinates and the center point μ:

[0026] ;

[0027] Where x is an arbitrary position in the 3D scene, ∑ represents the covariance matrix of the 3D Gaussian, and the superscript T represents the transpose of the matrix.

[0028] In addition, each Gaussian has its own opacity and color attributes. When generating 3D Gaussians from control points, ablation experiments show that combining position information (offset vector ) perception, which helps the most with the results.

[0029] Specifically, the process of fusing position information includes: first, the offset vector relative to the control point v is In order to make the k input offset coordinates from the output features of the local position perception module With permutation invariance, the module is designed as a symmetric function to aggregate the position information of each point, such as Figure 2 As shown in the figure, drawing on the design concept of pointnet, the present invention also uses the maximization operation as a symmetric operator; at the same time, in order to reduce the loss of geometric information, the point cloud is mapped to a higher dimension using MLP. Mapping to a higher dimension will produce redundant information, and the symmetric operation Maxpooling (maximum pooling layer) is taken in a higher dimension to obtain redundant information, thereby achieving the purpose of reducing the loss of geometric information. Then, the output features of the local position perception module are The characteristics of the control points themselves Splice to get fusion features In the present invention and The dimension is 16.

[0030] Next, get the viewing direction of a specific control point v relative to the camera :

[0031] ;

[0032] in and They are the position coordinates of the camera and control point in the world coordinate system respectively.

[0033] Will And the previous fusion features After splicing, they are sent to the 3D Gaussian attribute decoding module, i.e. the opacity decoding module ,Covariance decoding module , color decoding module In the example, they are all small MLPs consisting of two fully connected layers. The specific structure is as follows Figure 3 Decoding module with opacity Generate k 3D Gaussian opacity values ​​for each control point , as follows:

[0034] ;

[0035] For opacity, the output is activated by Tanh, where the value 0 is used as the threshold for selecting valid samples, and the final valid value can cover the entire range of [0,1]. For color, the final output is activated using the sigmoid function. Since it is difficult to guarantee its semi-positive finalization by directly optimizing the three-dimensional covariance matrix, the 3D covariance matrix Σ is represented as a combination of scaling transformation S and rotation transformation R:

[0036] ;

[0037] Specifically, the scaling S is expressed by a three-dimensional vector, and the rotation R is expressed by a quaternion q, so the covariance matrix of a 3D Gaussian corresponds to an output dimension of 7. In specific implementation, The first three dimensions F cov [:3] The output is used to represent the scaling, and then the scale control factor of the control point is used First three dimensions l v [:3] Make adjustments; the last four digits F cov [3:7] After normalization, it is used as a quaternion to represent the rotation. The scale of the final k 3D Gaussians is , as shown below:

[0038] s 0 ,…, s k -1 =Sigmoid F cov [:3] ⋅ l v [:3] ;

[0039] in, [:3] Represents the elements before the 3rd column.

[0040] In specific implementation, the scale control factor of the control point is used Post-3D l v [3:6] Make k Gaussian offset vectors Adjustment, for a position coordinate of The final position of the k Gaussians controlled by the control point v is:

[0041] μ 0 ,…, μ k -1 = x v + O 0 ,…, O k -1 ⋅ l v [3 :6 ] ;

[0042] The offset vector ∈R k×3 Next, in order to reduce redundancy and improve efficiency, a threshold is set according to the opacity and 3D Gaussians below the threshold are removed. Then, according to the specified camera internal and external parameters, only the 3D Gaussians under this perspective are screened out and sent to the differentiable rasterization module, including the center position, opacity, three-dimensional covariance matrix and color of the three-dimensional Gaussian body, and they are projected into the 2D image space for rendering. In this step, a perspective transformation W is first performed to convert the reference coordinate system from the world coordinate system to the camera coordinate system, mainly rotation and translation, both of which are affine; then a projection transformation is needed to align the perspective space with the pixels before rasterization can be performed, but because the perspective transformation is not affine, the 3D Gaussian after the perspective transformation no longer satisfies the Gaussian distribution. In order to make it still a 3D Gaussian after the transformation, the Jacobian matrix J is used instead of the perspective transformation, that is, a point is taken to perform a second-order Taylor expansion on the projection transformation matrix to obtain the approximate result of the perspective transformation, the 3D covariance matrix Σ in the world coordinate system, the transformation result as follows:

[0043] ;

[0044] It is also obtained from the original world coordinate system after the above transformation. The transformed distribution is aligned with the canvas pixels, and the integration along the third dimension can get the coloring of the ellipsoid on a certain pixel. According to the characteristics of 3D Gaussian, the result of integration along a certain axis is a 2D Gaussian, so here we can directly replace the integration process with 2D Gaussian, and the three-dimensional Gaussian G(x) is transformed into a two-dimensional Gaussian G'(x) on the image plane.

[0045] ;

[0046] The opacity of the projected 2D Gaussian needs to be adjusted based on the opacity and covariance matrix of the 3D Gaussian. Specifically, the opacity of the projected 2D Gaussian It can be calculated using the following formula

[0047] ;

[0048] A tile-based rasterizer is then used to sort the 2D Gaussian distribution by depth and alpha-blending is used for final shading. For a given pixel, the final color is for:

[0049] ;

[0050] in, denote the number, color, and opacity of the sorted 2D Gaussians associated with the query pixel, respectively.

[0051] After obtaining the new perspective synthesis result under this perspective, the loss function is calculated with the true value of the image under this perspective, which includes loss and structural similarity loss ; In order to constrain the scale of the 3D Gaussian not to be too large and produce too much overlap, a regularization term is introduced into the loss function ;

[0052] ;

[0053] where Prod(·) is the product of vector values, is the number of all 3D Gaussians.

[0054] The final loss function is , and are two hyperparameters used to control the weight of the two losses. Finally, the model parameters are optimized by back propagation algorithm until convergence. Figure 4 As shown, the initial point cloud and multiple supervised images are input to obtain the effect image of the new perspective synthesis output of the present invention.

[0055] In the optimization process, it is necessary to adaptively control the 3D Gaussian to fill the blank area; in the present invention, a new 3D Gaussian is generated for a new control point, and for the generation of the control point, the present invention calculates the average gradient of the Gaussian with a large position gradient in the view space in N training iterations (for example, greater than 0.001), which is recorded as Then, The 3D Gaussian is considered as a significant Gaussian, where is a predefined threshold; the significant Gaussian is sent to the multi-scale gridding module structure. If there is no control point established at the center of a grid before, a new control point is deployed at the center of the grid.

[0056] In order to prevent floating objects from being learned close to the camera, which would cause errors in the adaptive optimization process of the 3D Gaussian, it is necessary to periodically prune redundant control points. The present invention accumulates the opacity values ​​of their corresponding 3D Gaussians (for example, greater than 0.005) in N training iterations; if a control point cannot generate a group of 3D Gaussians with a higher opacity level, it will be deleted from the scene.

[0057] In addition, in order to further reduce the redundant 3D Gaussians in the optimization process, pruning is performed globally, and a significance score is assigned to each 3D Gaussian according to its contribution in all training views. Specifically, for a certain 3D Gaussian , traverse all training pixels to calculate the hit count of each Gaussian, and then combine the volume γ(Σ) and opacity σ of the 3D Gaussian to obtain the global saliency score corresponding to the 3D Gaussian :

[0058] ;

[0059] Where M, H, and W represent the number of training views, image height, and image width, respectively. It is a ray connecting the camera's optical center to a certain pixel. is the indicator function used to determine whether the Gaussian is consistent with a given However, using the Gaussian volume directly will exaggerate the importance of the background Gaussian, resulting in over-clipping of the Gaussian for modeling complex geometry. Therefore, a more adaptive method is used to weigh the impact of its volume, that is, using γ( , the specific formula is as follows:

[0060] ;

[0061] ;

[0062] First, the volume of all 3D Gaussians Sort from small to large and use the Gaussian volume at the 90% position All Gaussian volumes are normalized and then clipped to the range between 0 and 1 to obtain , and introduces a hyperparameter β to provide additional flexibility.

[0063] Since global pruning requires statistical counting of each 3D Gaussian, which is a time-consuming process, periodic pruning is not used. Instead, it can be performed once or twice at an appropriate time during the entire optimization process.

[0064] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A perspective synthesis method based on 3D Gaussian splashing technology with spatial coupling, characterized in that: The steps include: Step 1: First, use the motion structure recovery method to estimate the point cloud from a set of images, send the point cloud to the multi-scale gridding module structure, and voxelize the point cloud using grids of different resolutions. The center point of the grid is called the control point v. For a specific control point v, its initialization coordinates are the coordinates of the grid center point. The control point also initializes a learnable feature vector As a local context feature, one control point corresponds to the generation of k 3D Gaussians, and each control point v is accompanied by a learnable offset vector ∈R k×3 and a learnable scale factor ∈R 3 ; Step 2: Set the offset vector relative to the control point The local position perception module uses a symmetric function to aggregate the position information of each point to make the k offset coordinates of the input have permutation invariance from the output features of the local position perception module. The module outputs the position distribution features of k 3D Gaussians in the current local space. ; Step 3: Local position features The characteristics of the control point itself Splice to get fusion features ; Step 4: Relate a specific control point v to the camera’s viewing direction and fusion features After splicing, the data are sent to the 3D Gaussian attribute decoding module to decode the opacity, covariance matrix, and color of k 3D Gaussians respectively; The 3D Gaussian attribute decoding modules are all composed of shallow MLPs, including opacity MLP, covariance matrix MLP, and color MLP, which obtain their own attributes respectively; for a control point, k 3D Gaussian attributes are directly decoded at one time; the 3D covariance matrix is ​​represented as a combination of scaling transformation S and rotation transformation R; the scaling transformation S is expressed by a three-dimensional vector, and the rotation transformation R is expressed by a quaternion; Step 5: According to the coordinates of the control points , scale control factor and the offset vector of the 3D Gaussian to obtain the final 3D Gaussian position, and according to the opacity, set a threshold and remove the 3D Gaussian below the threshold; For a position coordinate The control point of k Gaussians is: ； in, ∈R k×3 is a learnable offset vector, is the scale control factor associated with the control point, is the position of the k Gaussians corresponding to the control point; Step 6: According to the specified camera internal and external parameters, the 3D Gaussian under the specified camera perspective is sent to the differentiable rasterization module. The parameters of each 3D Gaussian include center position, opacity, three-dimensional covariance matrix and color. The 3D Gaussian is projected into the 2D image space for rendering to obtain the new perspective synthesis result under this perspective, and the loss function is calculated with the true value of the image under this perspective.

2. The perspective synthesis method based on the spatially coupled 3D Gaussian splattering technology according to claim 1, characterized in that: In step 2, in order to avoid different results due to different input sorting, the same data input is designed, that is, the k offset vectors in the local coordinate system are processed separately using MLP, and then sent to the symmetric function maximum pooling layer for processing to achieve permutation invariance; MLP represents a multi-layer perceptron.

3. The perspective synthesis method based on spatially coupled 3D Gaussian splattering technology according to claim 1, characterized in that: For the generation of control points, the average gradient of the Gaussian greater than 0.001 in the view space during N training iterations is calculated and recorded as ; Then, The 3D Gaussian is considered as a significant Gaussian, where is a predefined threshold; the significant Gaussian is sent to the multi-scale gridding module structure. If there is no control point established at the center of a grid before, a new control point is deployed at the center of the grid.

4. The perspective synthesis method based on spatially coupled 3D Gaussian splattering technology according to claim 1, characterized in that: The opacity values ​​of the 3D Gaussians associated with redundant control points are accumulated over N training iterations; if a control point cannot produce a 3D Gaussian with an opacity level higher than 0.005, it is removed from the scene.

Images