Asymmetric hybrid quantization three-dimensional Gaussian scene expression method
By performing asymmetric mixed quantization processing and adjustment of the three-dimensional Gaussian distribution parameters of sparse point clouds, the effect of reducing GPU occupation during the three-dimensional Gaussian sputtering training and ensuring image quality is achieved, and GPU resources in the existing technology are solved High occupation problem.
Patent Information
- Application Number
- CN202411503797.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-10-25
- Publication Date
- 2025-02-11
AI Technical Summary
The existing three-dimensional Gaussian sputtering technology occupies high GPU resources, resulting in inefficient computing.
A three-dimensional Gaussian scene expression method with asymmetric mixed quantization is used to perform asymmetric mixed quantization of the three-dimensional Gaussian distribution parameters of sparse point clouds to obtain the three-dimensional quantized Gaussian distribution, and the three-dimensional Gaussian distribution of sparse point clouds is adjusted based on the loss function value.
The GPU occupation during the three-dimensional Gaussian sputtering training process is reduced, while ensuring high image quality, solving the problem of high GPU resource occupation in the prior art.
Smart Images
Figure CN119417965A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a method for expressing a three-dimensional Gaussian scene using asymmetric hybrid quantization. Background Art
[0002] 3D Gaussian sputtering is a technology that uses 3D Gaussian distribution to describe and render 3D scenes. Compared with traditional rendering technology, 3D Gaussian sputtering provides higher computational efficiency and comprehensive performance, and has good flexibility and can adapt to various complex 3D scenes.
[0003] However, 3D Gaussian sputtering requires millions of 3D Gaussian spheres to achieve high-quality rendering images, which requires a high level of GPU resources. Summary of the invention
[0004] In view of this, an object of an embodiment of the present application is to provide an asymmetric hybrid quantized three-dimensional Gaussian scene expression method to solve the technical problem that the existing three-dimensional Gaussian sputtering occupies a high GPU resource.
[0005] In a first aspect, an embodiment of the present application provides a method for expressing a three-dimensional Gaussian scene using asymmetric hybrid quantization, the method comprising:
[0006] Acquire initial point cloud data of the target area; wherein the initial point cloud data includes a plurality of sparse point clouds;
[0007] Asymmetric hybrid quantization processing is performed on Gaussian distribution parameters corresponding to the three-dimensional Gaussian distribution of each of the sparse point clouds to obtain a three-dimensional quantized Gaussian distribution; wherein the process of the asymmetric hybrid quantization processing includes: asymmetric quantization processing is performed on low-sensitivity parameters in the Gaussian distribution parameters; the low-sensitivity parameters include at least one of geometric shape parameters and color parameters; during the asymmetric quantization processing, the zero point of the data before quantization of the low-sensitivity parameters is inconsistent with the zero point of the data after quantization;
[0008] Calculating a loss function value corresponding to the three-dimensional quantized Gaussian distribution;
[0009] The three-dimensional Gaussian distribution of the sparse point cloud is adjusted based on the loss function value to obtain a three-dimensional Gaussian scene expression result of the target area.
[0010] In the above implementation process, the asymmetric hybrid quantization three-dimensional Gaussian scene expression method obtains a three-dimensional quantized Gaussian distribution by performing asymmetric hybrid quantization processing on the Gaussian distribution parameters corresponding to the three-dimensional Gaussian distribution of each sparse point cloud; calculates the loss function value corresponding to the three-dimensional quantized Gaussian distribution; and adjusts the three-dimensional Gaussian distribution of the sparse point cloud based on the calculated loss function value to obtain the three-dimensional Gaussian scene expression result of the target area. The process of asymmetric hybrid quantization processing includes: asymmetric quantization processing of low-sensitivity parameters such as geometric shape parameters and color parameters in the Gaussian distribution parameters; and the zero point of the data before quantization of the low-sensitivity parameters is inconsistent with the zero point of the data after quantization. The method calculates the loss function value through the three-dimensional quantized Gaussian distribution after asymmetric hybrid quantization processing, and adjusts the three-dimensional Gaussian distribution of the sparse point cloud based on the calculated loss function value, thereby reducing the GPU occupancy during the three-dimensional Gaussian sputtering training process. In addition, due to the asymmetric hybrid quantization 3D Gaussian scene expression method provided by the present application, only low-sensitivity parameters such as geometric shape parameters and / or color parameters are asymmetric quantized, which reduces the GPU occupancy during the 3D Gaussian sputtering training process while ensuring high image quality as much as possible. This solves the technical problem of the existing 3D Gaussian sputtering occupying high GPU resources.
[0011] Optionally, in an embodiment of the present application, the three-dimensional quantized Gaussian distribution includes multiple three-dimensional quantized Gaussian parameters; the three-dimensional Gaussian distribution of the sparse point cloud is adjusted based on the loss function value to obtain the three-dimensional Gaussian scene expression result of the target area, including: obtaining the parameter gradient of each of the three-dimensional quantized Gaussian parameters relative to the loss function value; adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the parameter gradient and the loss function value until the loss function value corresponding to the adjusted three-dimensional Gaussian distribution meets a preset loss condition; based on the adjusted three-dimensional Gaussian distribution, the three-dimensional Gaussian scene expression result of the target area is obtained.
[0012] In the above implementation process, by obtaining the parameter gradient of each three-dimensional quantized Gaussian parameter relative to the loss function value, the three-dimensional Gaussian distribution of the sparse point cloud can be adjusted more accurately according to the parameter gradient and the loss function value, thereby improving the training speed of three-dimensional Gaussian sputtering.
[0013] Optionally, in an embodiment of the present application, the three-dimensional quantized Gaussian parameters specifically include: unquantized Gaussian distribution parameters and quantized low-sensitivity parameters; the parameter gradient includes: an unquantized gradient corresponding to the unquantized Gaussian distribution parameters, and a quantized gradient corresponding to the quantized low-sensitivity parameter; adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the parameter gradient and the loss function value includes: dequantizing the quantized gradient to obtain a dequantized gradient; adjusting the three-dimensional Gaussian distribution of the sparse point cloud based on the unquantized gradient and the dequantized gradient.
[0014] In the above implementation process, by dequantizing the quantized gradients corresponding to the quantized low-sensitivity parameters and adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the dequantized gradients and the unquantized gradients, the adjustment accuracy of the three-dimensional Gaussian distribution can be improved; thereby, while reducing the GPU occupancy during the three-dimensional Gaussian sputtering training process, the image quality is guaranteed as high as possible.
[0015] Optionally, in an embodiment of the present application, the precision format of the unquantized Gaussian distribution parameter includes a half-precision floating point format; the precision format of the quantized low-sensitivity parameter includes a half-precision floating point format or an integer format.
[0016] In the above implementation process, the low-sensitivity parameter can select the appropriate quantization accuracy according to actual needs to reduce the accuracy loss caused by the quantization process; thereby reducing the GPU occupancy during the three-dimensional Gaussian sputtering training process while ensuring a high image quality as much as possible.
[0017] Optionally, in an embodiment of the present application, adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the parameter gradient and the loss function value specifically includes: performing gradient clipping on the parameter gradient to obtain a clipped gradient; wherein the clipped gradient includes an unquantized clipped gradient and a quantized clipped gradient; dequantizing the quantized clipped gradient to obtain a dequantized clipped gradient; and adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the unquantized clipped gradient and the dequantized clipped gradient.
[0018] In the above implementation process, by performing gradient clipping on the parameter gradient before adjusting the three-dimensional Gaussian distribution of the sparse point cloud, it is helpful to control the gradient size and prevent problems such as gradient disappearance or explosion during training.
[0019] Optionally, in an embodiment of the present application, adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the parameter gradient and the loss function value includes: adjusting the Gaussian distribution parameters of the sparse point cloud according to the parameter gradient and the loss function value; and / or adjusting the point cloud distribution state of the sparse point cloud according to the parameter gradient and the loss function value.
[0020] In the above implementation process, the three-dimensional Gaussian distribution of the sparse point cloud can be adjusted by adjusting the Gaussian distribution parameters and / or the point cloud distribution state of the sparse point cloud. It is also possible to select a suitable three-dimensional Gaussian distribution adjustment method according to actual needs to obtain a better adjustment effect and improve the training speed of the three-dimensional Gaussian sputtering.
[0021] Optionally, in an embodiment of the present application, the calculation of the loss function value corresponding to the three-dimensional quantized Gaussian distribution includes: rendering the three-dimensional quantized Gaussian distribution from at least one angle to obtain at least one intermediate processed image of the initial point cloud data; based on the intermediate processed image and the actual captured image of the target area, calculating the loss function value corresponding to the three-dimensional quantized Gaussian distribution.
[0022] In the above implementation process, by rendering the three-dimensional quantized Gaussian distribution from at least one angle, the expression image of the corresponding angle in the current three-dimensional Gaussian scene expression result can be obtained; according to the expression image of the corresponding angle and the actual captured image of the angle, the loss function component of the corresponding angle can be calculated; and then the loss function value corresponding to the three-dimensional quantized Gaussian distribution is determined according to the loss function components of multiple angles. The three-dimensional quantized Gaussian distribution is obtained by asymmetric quantization of the low-sensitivity parameters in the three-dimensional Gaussian distribution; the GPU occupancy in the above rendering process can be reduced, the GPU occupancy in the loss function value calculation process can be reduced, and then the GPU occupancy in the three-dimensional Gaussian sputtering training process can be reduced.
[0023] In a second aspect, an embodiment of the present application provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the asymmetric hybrid quantized three-dimensional Gaussian scene expression method as described in any one of the first aspects.
[0024] In a third aspect, an embodiment of the present application further provides an electronic device; the electronic device includes:
[0025] Memory;
[0026] processor;
[0027] The memory stores a computer program executable by the processor. When the computer program is executed by the processor, the asymmetric hybrid quantized three-dimensional Gaussian scene expression method described in any one of the first aspects is executed.
[0028] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which computer program instructions are stored. When the computer program instructions are executed by a processor, the asymmetric hybrid quantized three-dimensional Gaussian scene expression method as described in any one of the first aspects is executed.
[0029] In a fifth aspect, an embodiment of the present application provides an asymmetric hybrid quantized three-dimensional Gaussian scene expression device, the device comprising:
[0030] A point cloud data acquisition module, used to acquire initial point cloud data of a target area; wherein the initial point cloud data includes a plurality of sparse point clouds;
[0031] A quantization processing module, used for performing asymmetric quantization processing on the low-sensitivity parameters in the three-dimensional Gaussian distribution of each of the sparse point clouds to obtain a three-dimensional quantized Gaussian distribution; wherein the low-sensitivity parameters include at least one of a geometric shape parameter and a color parameter; during the asymmetric quantization processing, the zero point of the data before quantization of the low-sensitivity parameters is inconsistent with the zero point of the data after quantization;
[0032] A loss function calculation module, used to calculate the loss function value corresponding to the three-dimensional quantized Gaussian distribution;
[0033] A Gaussian distribution adjustment module is used to adjust the three-dimensional Gaussian distribution of the sparse point cloud based on the loss function value to obtain a three-dimensional Gaussian scene expression result of the target area.
[0034] The beneficial effects of the present application include at least: obtaining a three-dimensional quantized Gaussian distribution by performing asymmetric hybrid quantization processing on the Gaussian distribution parameters corresponding to the three-dimensional Gaussian distribution of each sparse point cloud; calculating the loss function value corresponding to the three-dimensional quantized Gaussian distribution; and adjusting the three-dimensional Gaussian distribution of the sparse point cloud based on the calculated loss function value to obtain the three-dimensional Gaussian scene expression result of the target area. The process of asymmetric hybrid quantization processing includes: performing asymmetric quantization processing on low-sensitivity parameters such as geometric shape parameters and color parameters in the Gaussian distribution parameters; and the zero point of the data before quantization of the low-sensitivity parameters is inconsistent with the zero point of the data after quantization. The method calculates the loss function value through the three-dimensional quantized Gaussian distribution after asymmetric hybrid quantization processing, and adjusts the three-dimensional Gaussian distribution of the sparse point cloud based on the calculated loss function value, thereby reducing the GPU occupancy during the three-dimensional Gaussian sputtering training process. And because the asymmetric hybrid quantized three-dimensional Gaussian scene expression method provided by this application only performs asymmetric quantization processing on low-sensitivity parameters such as geometric shape parameters and / or color parameters, while reducing the GPU occupancy and model storage space during the three-dimensional Gaussian sputtering training process, it also ensures as high an image quality (equivalent to the original three-dimensional Gaussian sputtering method) as possible. The technical problem that the existing three-dimensional Gaussian sputtering occupies a high GPU resource is solved. This is of great significance for application scenarios that require high-quality rendering on resource-constrained devices. BRIEF DESCRIPTION OF THE DRAWINGS
[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments of the present application will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0036] Figure 1A schematic diagram of a flow chart of an asymmetric hybrid quantized three-dimensional Gaussian scene expression method provided in an embodiment of the present application;
[0037] Figure 2 A schematic diagram of a specific implementation process of a quantization operation provided in an embodiment of the present application;
[0038] Figure 3 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0039] The following embodiments of the technical solution of the present application are described in detail in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present application, and are therefore only used as examples, and cannot be used to limit the scope of protection of the present application.
[0040] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by technicians in the technical field to which this application belongs; the terms used herein are only for the purpose of describing specific embodiments and are not intended to limit this application.
[0041] In the description of the embodiments of the present application, the technical terms "first", "second", etc. are only used to distinguish different objects, and cannot be understood as indicating or implying relative importance or implicitly indicating the number, specific order or primary and secondary relationship of the indicated technical features. In the description of the embodiments of the present application, the meaning of "multiple" is two or more, unless otherwise clearly and specifically defined.
[0042] See also Figure 1 The flowchart of a method for expressing a three-dimensional Gaussian scene with asymmetric hybrid quantization provided by an embodiment of the present application is shown. The method for expressing a three-dimensional Gaussian scene with asymmetric hybrid quantization may include the following steps:
[0043] S101, obtaining initial point cloud data of a target area; wherein the initial point cloud data includes a plurality of sparse point clouds.
[0044] Wherein, in step 101, the target area may include one or more target objects, and the target objects may be trees, vehicles or buildings, etc. The actual captured image data of the target area at multiple angles may be obtained by taking photos or recording videos of the target area; based on the three-dimensional reconstruction algorithm and the actual captured image data (exemplarily, the actual captured image data may be processed based on colmap or other software that can realize three-dimensional reconstruction), the initial point cloud data of the target area may be obtained. It is also possible to directly emit laser pulses to the target area based on a laser radar, measure the distance between the target object and the laser radar, and generate the initial point cloud data of the target area based on the measured distance data. It is also possible to project an optical fiber with a known pattern to the target area based on a 3D scanner or a structured light scanner, capture the reflected light pattern changes through a camera, and calculate the initial point cloud data of the target area based on the light pattern changes. The area size of the target area and the number of sparse point clouds and other data may be adjusted according to actual application requirements, and this application does not specifically limit this.
[0045] S102. Perform asymmetric hybrid quantization processing on Gaussian distribution parameters corresponding to the three-dimensional Gaussian distribution of each of the sparse point clouds to obtain a three-dimensional quantized Gaussian distribution; wherein the asymmetric hybrid quantization processing process includes: performing asymmetric quantization processing on low-sensitivity parameters in the Gaussian distribution parameters; the low-sensitivity parameters include at least one of geometric shape parameters and color parameters; during the asymmetric quantization processing process, the zero point of the low-sensitivity parameter before quantization is inconsistent with the zero point of the data after quantization.
[0046] In step 102, the three-dimensional Gaussian distribution G of the sparse point cloud s (x) can be used Indicates; where det(Σ) represents the determinant of Σ; e is the natural logarithm; (x-μ) T represents the transpose of (x-μ); x represents the three-dimensional coordinate vector (position parameter); μ represents the center of the ellipsoid (position parameter); Σ represents the covariance matrix (which may include the scaling matrix S and the rotation matrix R; specifically, Σ=R*S*S T *R T, S corresponds to size parameters, and R corresponds to rotation parameters). Low-sensitivity parameters refer to parameters that have little impact on the three-dimensional Gaussian model, excluding opacity parameters and position parameters. Low-sensitivity parameters may include geometric shape parameters (e.g., scale parameters, rotation parameters) or color parameters (e.g., DC components of colors, spherical harmonic functions), or may include both geometric shape parameters and color parameters. According to actual needs, you can choose to perform asymmetric quantization processing on only geometric shape parameters (which may be scale parameters and / or rotation parameters, etc.) or color parameters (which may be DC components of colors and / or spherical harmonic functions, etc.), or you can perform asymmetric quantization processing on both geometric shape parameters and color parameters at the same time.
[0047] Among them, the quantization process can convert the parameters with higher precision into the parameters with lower precision. Usually, the data distribution of the low-sensitivity parameters before quantization is uneven. By performing asymmetric quantization on the low-sensitivity parameters, the problem of poor quantization effect caused by uneven data distribution can be effectively solved. The zero point of the low-sensitivity parameter before quantization is inconsistent with the zero point of the data after quantization, that is, the zero point of the data after quantization is not zero. The single-precision floating point format (FP32) before quantization and the value range is [a f , b f ] low-sensitivity parameter x float For example, based on Implement x float quantization processing; where round() means rounding the parameter in the brackets to a specified number of digits (the specified number of digits can be determined according to the preset parameter accuracy after quantization processing), x quant ∈[a q , b q ], As long as z≠0, we can Realize low sensitivity parameter x float Asymmetric quantization processing. According to the low sensitivity parameter x float The minimum floating value of a f The value of the low-sensitivity parameter x float The maximum floating value of b f The specific quantization accuracy of the low-sensitivity parameter can be adjusted according to actual needs, and the quantization accuracy of multiple low-sensitivity parameters can be the same or different. Taking the accuracy of the unquantized Gaussian distribution parameters as single-precision floating point format (FP32) as an example, the quantization accuracy of the scale parameter or rotation parameter can be half-precision floating point format (FP16), while the quantization accuracy of the DC component of the color or the spherical harmonic function can be eight-bit integer format (INT8). The three-dimensional quantized Gaussian distribution may include: a three-dimensional Gaussian distribution determined based on the unquantized Gaussian distribution parameters and the quantized low-sensitivity parameters.
[0048] S103: Calculate a loss function value corresponding to the three-dimensional quantized Gaussian distribution.
[0049] In step 103, a model quantized image of a three-dimensional quantized Gaussian model corresponding to the three-dimensional quantized Gaussian distribution can be obtained by image rendering, and the loss function value can be determined based on the similarity between the model quantized image and the actual captured image. The loss function value can also be determined based on the optimal transmission theory, the model quantized image, and the actual captured image.
[0050] S104. Adjust the three-dimensional Gaussian distribution of the sparse point cloud based on the loss function value to obtain a three-dimensional Gaussian scene expression result of the target area.
[0051] Among them, in step 104, the three-dimensional Gaussian distribution of the sparse point cloud can be adjusted according to the loss function value by adjusting the Gaussian distribution parameters of the sparse point cloud or the point cloud distribution state; and the adjusted three-dimensional Gaussian distribution of the sparse point cloud is determined as the three-dimensional Gaussian scene expression result of the target area.
[0052] It can be seen that the asymmetric hybrid quantization three-dimensional Gaussian scene expression method provided in the embodiment of the present application obtains a three-dimensional quantized Gaussian distribution by performing asymmetric hybrid quantization processing on the Gaussian distribution parameters corresponding to the three-dimensional Gaussian distribution of each sparse point cloud; calculates the loss function value corresponding to the three-dimensional quantized Gaussian distribution; and adjusts the three-dimensional Gaussian distribution of the sparse point cloud based on the calculated loss function value to obtain the three-dimensional Gaussian scene expression result of the target area. The process of asymmetric hybrid quantization processing includes: asymmetric quantization processing of low-sensitivity parameters such as geometric shape parameters and color parameters in the Gaussian distribution parameters; and the zero point of the data before quantization of the low-sensitivity parameters is inconsistent with the zero point of the data after quantization. The method calculates the loss function value through the three-dimensional quantized Gaussian distribution after asymmetric hybrid quantization processing, and adjusts the three-dimensional Gaussian distribution of the sparse point cloud based on the calculated loss function value, thereby reducing the GPU occupancy during the three-dimensional Gaussian sputtering training process. And because the asymmetric hybrid quantized three-dimensional Gaussian scene expression method provided by this application only performs asymmetric quantization processing on low-sensitivity parameters such as geometric shape parameters and / or color parameters, while reducing the GPU occupancy and model storage space during the three-dimensional Gaussian sputtering training process, it also ensures as high an image quality (equivalent to the original three-dimensional Gaussian sputtering method) as possible. The technical problem that the existing three-dimensional Gaussian sputtering occupies a high GPU resource is solved. This is of great significance for application scenarios that require high-quality rendering on resource-constrained devices.
[0053] In some optional embodiments, the three-dimensional quantized Gaussian distribution includes multiple three-dimensional quantized Gaussian parameters; S104, adjusting the three-dimensional Gaussian distribution of the sparse point cloud based on the loss function value to obtain the three-dimensional Gaussian scene expression result of the target area, including: obtaining the parameter gradient of each of the three-dimensional quantized Gaussian parameters relative to the loss function value; adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the parameter gradient and the loss function value until the loss function value corresponding to the adjusted three-dimensional Gaussian distribution meets the preset loss condition; based on the adjusted three-dimensional Gaussian distribution, obtaining the three-dimensional Gaussian scene expression result of the target area.
[0054] Among them, the three-dimensional quantized Gaussian parameters may specifically include unquantized Gaussian distribution parameters (for example, opacity parameters, position parameters) and quantized low-sensitivity parameters (quantized scale parameters, rotation parameters, DC components of colors or spherical harmonic functions). The unquantized parameter gradient of each unquantized Gaussian distribution parameter relative to the loss function value and the quantized parameter gradient of each quantized low-sensitivity parameter relative to the loss function value may be calculated respectively; the unquantized parameters in the Gaussian distribution parameters are adjusted according to the unquantized parameter gradient, and the quantized parameters in the Gaussian distribution parameters are adjusted according to the quantized parameter gradient to obtain the adjusted three-dimensional Gaussian distribution of the sparse point cloud. The preset loss condition may include a loss function threshold. When the loss function value corresponding to the adjusted three-dimensional Gaussian distribution is less than or equal to the loss function threshold, it is considered that the loss function value corresponding to the adjusted three-dimensional Gaussian distribution meets the preset loss condition. By obtaining the parameter gradient of each three-dimensional quantized Gaussian parameter relative to the loss function value, each Gaussian distribution parameter can be accurately adjusted according to the parameter gradient and the loss function value, thereby improving the adjustment accuracy of the three-dimensional Gaussian distribution of the sparse point cloud and the three-dimensional Gaussian sputtering training speed.
[0055] In some optional embodiments, the three-dimensional quantized Gaussian parameters specifically include: unquantized Gaussian distribution parameters and quantized low-sensitivity parameters; the parameter gradient includes: an unquantized gradient corresponding to the unquantized Gaussian distribution parameters, and a quantized gradient corresponding to the quantized low-sensitivity parameters; adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the parameter gradient and the loss function value includes: dequantizing the quantized gradient to obtain a dequantized gradient; adjusting the three-dimensional Gaussian distribution of the sparse point cloud based on the unquantized gradient and the dequantized gradient.
[0056] Among them, by dequantizing the quantized gradient and adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the unquantized gradient and the dequantized gradient, the adjustment accuracy of the three-dimensional Gaussian distribution can be improved; thereby, while reducing the GPU occupancy during the three-dimensional Gaussian sputtering training process, a higher image quality is guaranteed as much as possible.
[0057] In some optional embodiments, adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the parameter gradient and the loss function value specifically includes: performing gradient clipping on the parameter gradient to obtain a clipped gradient; wherein the clipped gradient includes an unquantized clipped gradient and a quantized clipped gradient; dequantizing the quantized clipped gradient to obtain a dequantized clipped gradient; and adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the unquantized clipped gradient and the dequantized clipped gradient.
[0058] Among them, the gradient clipping operation of the parameter gradient can be implemented by the straight-through estimator (STE). By performing gradient clipping on the parameter gradient before adjusting the three-dimensional Gaussian distribution of the sparse point cloud, it is helpful to control the gradient size and prevent problems such as gradient disappearance or explosion during training. For example, STE can be applied using the clip function to reduce the sensitivity of the low-sensitivity parameter x float For example, based on Realize low sensitivity parameter x float Quantitative processing of Indicates that That is x quant , limited to [a q , b q ]. Accordingly, we can use Represents the gradient clipping process of STE; where, Indicates that the gradient Input STE and you will get the output gradient Represents the quantitative low-sensitivity parameter x quant The parameter gradient relative to the loss function value Loss, Represents the unquantized low-sensitivity parameter x float The parameter gradient with respect to the loss function value Loss.
[0059] Among them, based on the above Realize low sensitivity parameter x float In the case of quantization processing, the quantization process can be expressed as Accordingly, the dequantization process can be expressed as f dequant (x quant ,k,z)=k*(x quant -z); where f quant represents the quantization function, f dequant Represents the inverse quantization function. By inverse quantizing the quantized clipped gradient and adjusting the 3D Gaussian distribution of the sparse point cloud according to the unquantized clipped gradient and the inverse quantized clipped gradient, the adjustment accuracy of the 3D Gaussian distribution can be improved; thereby, while reducing the GPU usage during the 3D Gaussian sputtering training process, the image quality is guaranteed as high as possible.
[0060] Among them, the basic quantization calculation can include quantization addition calculation, quantization multiplication calculation and quantization matrix multiplication calculation. Other quantization algorithms can be simplified into these three basic quantization calculations. Here, only the implementation methods of the above three basic quantization calculations are explained.
[0061] For quantized addition calculations, z float =x float +y float For example, based on f dequant (x quant ,k,z)=k*(x quant -z), we can get k z *(z quant -z z ) = k x *(x quant -z x )+k y *(y quant -z y ), where x quant ,y quant and z quant Respectively represent x float ,y float and z float The quantized value, z x 、z y and z z Represents the variable x quant ,y quant and z quant The zero point, k x , k y and k z Respectively represent the variable x quant ,y quant and z quant The quantization scale factor of Mul(t xy )=2*max(k x , k y ), max(k x , k y ) represents k x and k y The maximum value in can be obtained: make You can get: k z *(z quant -z z )=Mul(t xy )*(Mul(t x )*(x quant -z x))+Mul(t y )*(y quant -z y ), accordingly, z quant =Mul(t xyz )*(Mul(t x )*(x quant -z x ))+Mul(t y )*(y quant -z y )+k z ; z quant It means x quant and quant sum.
[0062] For quantized multiplication calculations, z float =x float *y float For example, based on f dequant (x quant ,k,z)=k*(x quant -z), we can get: k z *(z quant -z z ) = k x *(x quant -z x )*k y *(y quant -z y ), where x quant ,y quant and z quant Respectively represent x float ,y float and z float The quantized value, z x 、z y and z z Represents the variable x quant ,y quant and z quant The zero point, k x , k y and k z Represents the variable x quant ,y quant and z quant Quantization scale factor. make We can get: quant =Mul ′ (t xyz )*(x quant -z x )*(y quant -z y)+z z .
[0063] For quantization matrix multiplication calculation, M = X*Y+c, X∈R m*p , Y∈R p*n , c∈R n , M∈R m*n For example, Based on f dequant (x quant ,k,z)=k*(x quant -z), we can get: therefore, Due to M i,j =k M *(M q,i,j -z M ), we can get: Among them, X q , Y q 、c q and M q Respectively represent the quantized values of X, Y, c, and M, z x 、z Y 、z c and z M Respectively represent the zero points of variables X, Y, c and M, k X , k Y , k c and k M Represent the quantization scale factors of variables X, Y, c, and M respectively. It should be noted that as well as It is a constant during the training process and can be calculated in advance; what needs to be calculated is
[0064] In some optional embodiments, adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the parameter gradient and the loss function value includes: adjusting the Gaussian distribution parameters of the sparse point cloud according to the parameter gradient and the loss function value; and / or adjusting the point cloud distribution state of the sparse point cloud according to the parameter gradient and the loss function value.
[0065] Among them, adjusting the point cloud distribution state of the sparse point cloud may include: point cloud segmentation and point cloud merging. Specifically, when the three-dimensional Gaussian distribution of a sparse point cloud is too large to fully fit the details at that location, the sparse point cloud may be segmented, and two or more point clouds may be used to express the details at that location; or when multiple sparse point clouds are too dense, two or more of the point clouds may be merged and expressed with one merged point cloud. The three-dimensional Gaussian distribution of the sparse point cloud may be adjusted by adjusting the Gaussian distribution parameters and / or the point cloud distribution state of the sparse point cloud. It is also possible to select a suitable three-dimensional Gaussian distribution adjustment method according to actual needs to obtain a better adjustment effect and improve the training speed of three-dimensional Gaussian sputtering.
[0066] In some optional embodiments, S103, calculating the loss function value corresponding to the three-dimensional quantized Gaussian distribution, includes: rendering the three-dimensional quantized Gaussian distribution from at least one angle to obtain at least one intermediate processed image of the initial point cloud data; based on the intermediate processed image and the actual captured image of the target area, calculating the loss function value corresponding to the three-dimensional quantized Gaussian distribution.
[0067] Among them, projection can be performed based on the camera extrinsics used to collect the actual captured image at the corresponding angle, and the intermediate processed image at the corresponding projection angle can be obtained by using differentiable rasterization rendering. Then, the above-mentioned loss function value is determined based on the similarity between the intermediate processed image and the actual captured image at the corresponding angle. By rendering the three-dimensional quantized Gaussian distribution from at least one angle, the intermediate processed image of the corresponding angle in the three-dimensional Gaussian scene expression result can be obtained; according to the intermediate processed image of the corresponding angle and the actual captured image of the angle, the loss function component of the corresponding angle can be calculated; and then the loss function value corresponding to the three-dimensional quantized Gaussian distribution is determined according to the loss function components of multiple angles. By performing asymmetric quantization processing on the low-sensitivity parameters in the three-dimensional Gaussian distribution, a three-dimensional quantized Gaussian distribution is obtained; the GPU occupancy in the above-mentioned rendering process can be reduced, and the GPU occupancy in the loss function value calculation process can be reduced, thereby reducing the GPU occupancy in the three-dimensional Gaussian sputtering training process.
[0068] Please refer to Figure 2 , Figure 2 A schematic diagram of a specific implementation process of a quantization operation provided in an embodiment of the present application. Figure 2As shown, the low-sensitivity parameters in the Gaussian distribution parameters of the initial sparse point cloud can be asymmetrically quantized; specifically, for the unquantized low-sensitivity parameters in the single-precision floating point format (FP32), the quantized low-sensitivity parameters in the half-precision floating point format (FP16) or the eight-bit integer format (INT8) can be obtained through asymmetrical quantization. Then, based on the unquantized Gaussian distribution parameters (FP32) and the quantized low-sensitivity parameters (FP16 or INT8), the three-dimensional quantized Gaussian distribution of the sparse point cloud is rendered to calculate the loss function value and parameter gradient corresponding to the three-dimensional quantized Gaussian distribution. The parameter gradient is gradient clipped to obtain the unquantized clipped gradient and the quantized clipped gradient; and the quantized clipped gradient corresponding to the quantized low-sensitivity parameter is dequantized to obtain the dequantized dequantized clipped gradient; based on the dequantized gradient (including the unquantized clipped gradient and the dequantized clipped gradient), the three-dimensional Gaussian distribution of the sparse point cloud is adjusted (the point cloud distribution of the sparse point cloud is adjusted and / or the Gaussian distribution parameters of the sparse point cloud are adjusted). When the adjusted three-dimensional Gaussian distribution of the sparse point cloud does not meet the preset loss condition, the low-sensitivity parameters in the adjusted three-dimensional Gaussian distribution of the sparse point cloud are asymmetrically quantized again. Until the adjusted three-dimensional Gaussian distribution of the sparse point cloud meets the preset loss condition, the adjusted three-dimensional Gaussian distribution of the sparse point cloud that meets the preset loss condition is determined as the three-dimensional Gaussian scene expression result of the target area. It should be noted that Figure 2 It only shows the situation where "the unquantized low-sensitivity parameter / unquantized Gaussian distribution parameter is data in single-precision floating point format (FP32), and the quantized low-sensitivity parameter is data in half-precision floating point format (FP16) or eight-bit integer format (INT8)" by way of example.
[0069] In some optional embodiments, the three-dimensional quantized Gaussian parameters may include: unquantized Gaussian distribution parameters in double-precision floating point format, and quantized low-sensitivity parameters in single-precision floating point format, half-precision floating point format, or integer format.
[0070] An embodiment of the present application also provides an asymmetric hybrid quantized three-dimensional Gaussian scene expression device, which includes: a point cloud data acquisition module, used to acquire initial point cloud data of a target area; wherein the initial point cloud data includes multiple sparse point clouds; a quantization processing module, used to perform asymmetric hybrid quantization processing on Gaussian distribution parameters corresponding to the three-dimensional Gaussian distribution of each of the sparse point clouds to obtain a three-dimensional quantized Gaussian distribution; wherein the asymmetric hybrid quantization processing process includes: performing asymmetric quantization processing on low-sensitivity parameters in the Gaussian distribution parameters; the low-sensitivity parameters include at least one of geometric shape parameters and color parameters; during the asymmetric quantization processing process, the zero point of the data before quantization of the low-sensitivity parameters is inconsistent with the zero point of the data after quantization; a loss function calculation module, used to calculate the loss function value corresponding to the three-dimensional quantized Gaussian distribution; a Gaussian distribution adjustment module, used to adjust the three-dimensional Gaussian distribution of the sparse point cloud based on the loss function value to obtain a three-dimensional Gaussian scene expression result of the target area.
[0071] It should be understood that the asymmetric hybrid quantized three-dimensional Gaussian scene expression device corresponds to the above-mentioned asymmetric hybrid quantized three-dimensional Gaussian scene expression method embodiment, and can execute the various steps involved in the above-mentioned method embodiment. The specific functions of the asymmetric hybrid quantized three-dimensional Gaussian scene expression device can be found in the description above. To avoid repetition, the detailed description is appropriately omitted here. The asymmetric hybrid quantized three-dimensional Gaussian scene expression device includes at least one software function module that can be stored in a memory in the form of software or firmware or solidified in the operating system (OS) of the device.
[0072] An embodiment of the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the asymmetric hybrid quantized three-dimensional Gaussian scene expression method as described in any one of the first aspects.
[0073] Please refer to Figure 3 , Figure 3 A schematic diagram of the structure of an electronic device 200 provided in an embodiment of the present application. The electronic device 200 includes: a memory 202 and a processor 201; the memory 202 stores a computer program executable by the processor 201, and when the computer program is executed by the processor 201, the asymmetric hybrid quantized three-dimensional Gaussian scene expression method described in any one of the first aspects is executed.
[0074] The memory 202 and the processor 201 may be interconnected and communicate with each other via a communication bus 203 and / or other forms of connection mechanisms (not shown). The memory 202 stores a computer program executable by the processor 201. When the computer program is executed by the processor 201, the asymmetric hybrid quantized three-dimensional Gaussian scene expression method described in the first aspect above is executed.
[0075] The embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by the processor 201, the asymmetric hybrid quantized three-dimensional Gaussian scene expression method described in the first aspect above is executed.
[0076] Among them, the storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable red-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0077] In several embodiments provided by the embodiments of the present application, it should be understood that the disclosed device / system and method can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and a part of the module, program segment or code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.
[0078] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.
[0079] The above description is only an optional implementation manner of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or replacements within the technical scope disclosed in the embodiments of the present application, which should be covered within the protection scope of the embodiments of the present application.
Claims
1. A three-dimensional Gaussian scene expression method based on asymmetric hybrid quantization, characterized in that: The method comprises: Acquire initial point cloud data of the target area; wherein the initial point cloud data includes a plurality of sparse point clouds; Asymmetric hybrid quantization processing is performed on Gaussian distribution parameters corresponding to the three-dimensional Gaussian distribution of each of the sparse point clouds to obtain a three-dimensional quantized Gaussian distribution; wherein the process of the asymmetric hybrid quantization processing includes: asymmetric quantization processing is performed on low-sensitivity parameters in the Gaussian distribution parameters; the low-sensitivity parameters include at least one of geometric shape parameters and color parameters; during the asymmetric quantization processing, the zero point of the data before quantization of the low-sensitivity parameters is inconsistent with the zero point of the data after quantization; Calculating a loss function value corresponding to the three-dimensional quantized Gaussian distribution; The three-dimensional Gaussian distribution of the sparse point cloud is adjusted based on the loss function value to obtain a three-dimensional Gaussian scene expression result of the target area.
2. The method according to claim 1, characterized in that in, The three-dimensional quantized Gaussian distribution includes a plurality of three-dimensional quantized Gaussian parameters; The step of adjusting the three-dimensional Gaussian distribution of the sparse point cloud based on the loss function value to obtain a three-dimensional Gaussian scene expression result of the target area includes: Obtaining a parameter gradient of each of the three-dimensional quantized Gaussian parameters relative to the loss function value; According to the parameter gradient and the loss function value, adjusting the three-dimensional Gaussian distribution of the sparse point cloud until the loss function value corresponding to the adjusted three-dimensional Gaussian distribution meets a preset loss condition; Based on the adjusted three-dimensional Gaussian distribution, a three-dimensional Gaussian scene expression result of the target area is obtained.
3. The method according to claim 2, characterized in that in, The three-dimensional quantized Gaussian parameters specifically include: unquantized Gaussian distribution parameters and quantized low-sensitivity parameters; the parameter gradient includes: an unquantized gradient corresponding to the unquantized Gaussian distribution parameters, and a quantized gradient corresponding to the quantized low-sensitivity parameters; The adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the parameter gradient and the loss function value includes: Dequantizing the quantized gradient to obtain a dequantized gradient; Based on the unquantized gradient and the dequantized gradient, a three-dimensional Gaussian distribution of the sparse point cloud is adjusted.
4. The method according to claim 3, characterized in that in, The precision format of the unquantized Gaussian distribution parameter includes a half-precision floating point format; the precision format of the quantized low-sensitivity parameter includes a half-precision floating point format or an integer format.
5. The method according to claim 3, characterized in that: The adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the parameter gradient and the loss function value specifically includes: Performing gradient clipping on the parameter gradient to obtain a clipped gradient; wherein the clipped gradient includes an unquantized clipped gradient and a quantized clipped gradient; Dequantizing the quantized clipping gradient to obtain a dequantized clipping gradient; The three-dimensional Gaussian distribution of the sparse point cloud is adjusted according to the unquantized clipping gradient and the inverse quantized clipping gradient.
6. The method according to claim 2, characterized in that The adjusting the three-dimensional Gaussian distribution of the sparse point cloud according to the parameter gradient and the loss function value includes: According to the parameter gradient and the loss function value, adjusting the Gaussian distribution parameters of the sparse point cloud; and / or, According to the parameter gradient and the loss function value, the point cloud distribution state of the sparse point cloud is adjusted.
7. The method according to any one of claims 1 to 6, characterized in that: in, The calculating a loss function value corresponding to the three-dimensional quantized Gaussian distribution includes: Rendering the three-dimensional quantized Gaussian distribution from at least one angle to obtain at least one intermediate processed image of the initial point cloud data; Based on the intermediate processed image and the actual captured image of the target area, a loss function value corresponding to the three-dimensional quantized Gaussian distribution is calculated.
8. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.
9. An electronic device, characterized in that: The electronic device comprises: Memory; processor; The memory stores a computer program executable by the processor, and when the computer program is executed by the processor, the method according to any one of claims 1 to 7 is performed.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the method according to any one of claims 1 to 7 is executed.