Three-dimensional modeling method, electronic equipment, storage medium and computer program product

Abstract

The invention discloses a three-dimensional modeling method, electronic equipment, a storage medium and a computer program product, and relates to the technical field of data processing, and the method comprises the steps: retrieving a four-dimensional bilateral affine grid based on a spatial dimension index and a color dimension index of a three-dimensional Gaussian point of a training view to obtain an affine transformation matrix, adjusting the color dimension index through the affine transformation matrix to obtain an adjusted color dimension index; rendering the training view based on the adjusted color dimension index through a three-dimensional Gaussian splash model to obtain a rendered image; and based on the luminosity loss between the rendering image and the training view, performing parameter optimization on the four-dimensional bilateral affine grid and the three-dimensional Gaussian splash model to obtain an optimized three-dimensional Gaussian splash model, and performing three-dimensional modeling on the multi-view image based on the optimized three-dimensional Gaussian splash model. According to the method, the floating objects in the scene are reduced, and the geometric modeling accuracy of the scene is improved, so that the rendering quality and the reconstruction accuracy in the three-dimensional modeling process are improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a three-dimensional modeling method, an electronic device, a storage medium, and a computer program product. Background Art

[0002] The main goal of 3D reconstruction and rendering is to achieve multi-view from a set of images with known camera poses. Figure 1 Neural Radiance Fields (NeRF) has made significant progress in this area, aiming to obtain a consistent 3D representation that enables realistic rendering from new perspectives. Currently, 3D Gaussian Splatting has attracted widespread attention due to its superior performance in terms of visual quality and rendering speed. However, existing 3D Gaussian Splatting ignores the multi-view inconsistencies brought by camera capture, and simply treating all views equally may cause these inconsistencies to be overfitted into the radiance field, resulting in erroneous geometry and floating objects.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of the present application is to provide a three-dimensional modeling method, electronic device, storage medium and computer program product, aiming to solve the technical problem of low accuracy of geometric modeling results of three-dimensional Gaussian splash reconstruction caused by inconsistent multi-view image enhancement.

[0005] To achieve the above objectives, the present application proposes a three-dimensional modeling method, which includes:

[0006] Based on the spatial dimension index and color dimension index of the three-dimensional Gaussian point of the training view, retrieve the four-dimensional bilateral affine grid to obtain an affine transformation matrix, and adjust the color dimension index by the affine transformation matrix to obtain an adjusted color dimension index;

[0007] Rendering the training view based on the adjusted color dimension index using a three-dimensional Gaussian splash model to obtain a rendering image;

[0008] Based on the photometric loss between the rendering and the training view, the four-dimensional bilateral affine grid and the three-dimensional Gaussian splash model are parameter optimized to obtain an optimized three-dimensional Gaussian splash model, wherein three-dimensional modeling of multi-view images is performed based on the optimized three-dimensional Gaussian splash model.

[0009] In one embodiment, the step of retrieving an affine transformation matrix from a four-dimensional bilateral affine grid based on the spatial dimension index and the color dimension index of the three-dimensional Gaussian points of the training view includes:

[0010] Normalize the point coordinates of the three-dimensional Gaussian points of the training view to obtain a spatial dimension index, and normalize the point colors of the three-dimensional Gaussian points and map them to a grayscale space to obtain a color dimension index;

[0011] Retrieve an affine transformation matrix from the four-dimensional bilateral affine grid based on the spatial dimension index and the color dimension index.

[0012] In one embodiment, the dimensions of the four-dimensional bilateral affine grid include a spatial dimension and a color dimension;

[0013] The step of retrieving an affine transformation matrix from the four-dimensional bilateral affine grid based on the spatial dimension index and the color dimension index includes:

[0014] Map the spatial dimension index to the spatial dimension of the four-dimensional bilateral affine grid to obtain a spatial dimension index, and map the color dimension index to the color dimension of the four-dimensional bilateral affine grid to obtain a color dimension index;

[0015] Determine the weight function of the three-dimensional Gaussian point at the grid point, where the grid point is the point determined based on the spatial dimension index and the color dimension index;

[0016] Retrieve the transformation matrix of the nearby points from the four-dimensional bilateral affine grid, and perform weighted processing on the transformation matrices of the respective nearby points through the weight function to obtain an affine transformation matrix, where the nearby points are the grid points whose distance from the three-dimensional Gaussian point is less than a preset distance.

[0017] In one embodiment, the step of adjusting the color dimension index through the affine transformation matrix to obtain an adjusted color dimension index includes:

[0018] Expand the color dimension index into a four-dimensional vector to obtain a color vector, and transform the color vector through the affine transformation matrix to obtain a transformed color vector;

[0019] Perform a clamp operation on the transformed color vector based on a preset value range to obtain an adjusted color dimension index.

[0020] In one embodiment, before the step of retrieving an affine transformation matrix from the four-dimensional bilateral affine grid based on the spatial dimension index and the color dimension index of the three-dimensional Gaussian points of the training view, it further includes:

[0021] Decompose a preset four-dimensional grid into the sum of the outer products of multiple vectors through tensor decomposition to obtain a decomposed grid, and determine the decomposed grid as a four-dimensional bilateral affine grid.

[0022] In one embodiment, the step of rendering the training view based on the adjusted color dimension index through a three-dimensional Gaussian splash model to obtain a rendered image includes:

[0023] Project each three-dimensional Gaussian point in the training view into a two-dimensional space to obtain the two-dimensional Gaussian distribution of each three-dimensional Gaussian point;

[0024] Traverse each of the two-dimensional Gaussian distributions, and perform rasterization processing based on the coordinates of the two-dimensional Gaussian distribution, the adjusted color dimension index, and the rendering attribute data of the training view to obtain the color contribution value of each two-dimensional Gaussian distribution to each pixel;

[0025] Traverse each of the pixels, and superimpose the color contribution values of each two-dimensional Gaussian distribution to the pixel to obtain the pixel color;

[0026] Based on the pixel colors of each pixel, combine each pixel to obtain a rendered image.

[0027] In one embodiment, the step of optimizing the parameters of the four-dimensional bilateral affine grid and the three-dimensional Gaussian splash model based on the photometric loss between the rendered image and the training view to obtain an optimized three-dimensional Gaussian splash model includes:

[0028] Traverse each pixel of the training view, input the pixel value of the pixel in the rendered image and the pixel value of the pixel in the training view into a preset loss function to obtain the photometric loss between the rendered image and the training view;

[0029] Determine the first gradient of the photometric loss with respect to the model parameters in the three-dimensional Gaussian splash model, and determine the second gradient of the photometric loss with respect to the model parameters in the four-dimensional bilateral affine grid;

[0030] Update the model parameters in the three-dimensional Gaussian splash model based on the first gradient and a preset optimization algorithm to obtain an updated three-dimensional Gaussian splash model, and update the model parameters in the four-dimensional bilateral affine grid based on the second gradient and the preset optimization algorithm to obtain an updated four-dimensional bilateral affine grid. Until a preset optimization condition is reached, determine the updated three-dimensional Gaussian splash model as the optimized three-dimensional Gaussian splash model.

[0031] In addition, to achieve the above object, the present application further provides an electronic device, which includes: a memory, a processor, and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the three-dimensional modeling method as described above.

[0032] In addition, to achieve the above object, the present application further provides a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium, and when the computer program is executed by a processor, the steps of the three-dimensional modeling method as described above are implemented.

[0033] In addition, to achieve the above object, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the three-dimensional modeling method as described above are implemented.

[0034] In the present application, based on the spatial dimension index and color dimension index of the three-dimensional Gaussian points of the training view, a four-dimensional bilateral affine grid is retrieved to obtain an affine transformation matrix, and the color dimension index is adjusted by the affine transformation matrix to obtain an adjusted color dimension index; the joint representation of spatial and color information is utilized to dynamically adjust the color of the Gaussian points to keep them consistent under different perspectives, and the retrieved affine transformation matrix is used to adjust the color of the Gaussian points to reduce color inconsistencies in multi-view images.

[0035] The training view is rendered based on the adjusted color dimension index through a three-dimensional Gaussian splash model to obtain a rendered image; based on the photometric loss between the rendered image and the training view, the parameters of the four-dimensional bilateral affine grid and the three-dimensional Gaussian splash model are optimized to obtain an optimized three-dimensional Gaussian splash model, wherein three-dimensional modeling of multi-view images is performed based on the optimized three-dimensional Gaussian splash model. By comparing the photometric loss between the rendered image and the training view, the difference between the reconstruction result and the real scene is quantified, and based on the photometric loss, the parameters of the four-dimensional bilateral affine grid and the three-dimensional Gaussian splash model are optimized to minimize the photometric loss. Through the optimization process, the three-dimensional Gaussian splash model can more accurately reflect the geometric structure and appearance of the scene, reduce geometric modeling errors and floating object problems, thereby improving the accuracy of three-dimensional modeling.

[0036] In the present application, a bilateral affine grid is used as a post-processing module to decouple the enhancement inconsistencies in multi-view images, jointly guide the three-dimensional Gaussian splash for consistent reconstruction, reduce floating objects in the scene, and improve the geometric modeling accuracy of the scene, thereby improving the rendering quality and reconstruction accuracy in the three-dimensional modeling process. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings here are incorporated into the description and form a part of this description, showing embodiments consistent with this application, and are used together with the description to explain the principles of this application.

[0038] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following will briefly introduce the accompanying drawings required for use in the description of the embodiments or the prior art. Obviously, for those of ordinary skill in the art, without creative efforts, other drawings can also be obtained based on these drawings.

[0039] Figure 1 It is a schematic flowchart provided for the first embodiment of the three-dimensional modeling method of this application;

[0040] Figure 2 It is a schematic flowchart provided for the second embodiment of the three-dimensional modeling method of this application;

[0041] Figure 3 It is a schematic framework diagram of the three-dimensional modeling method provided for an embodiment of this application;

[0042] Figure 4 It is a schematic module structure diagram of the three-dimensional modeling device for the embodiment of this application;

[0043] Figure 5 It is a schematic device structure diagram of the hardware operating environment involved in the three-dimensional modeling method for the embodiment of this application.

[0044] The realization of the purpose, functional features and advantages of this application will be further described with reference to the embodiments and the accompanying drawings. Specific Embodiments

[0045] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of this application and are not used to limit this application.

[0046] To better understand the technical solutions of this application, the following will be described in detail in combination with the description drawings of the specification and specific embodiments.

[0047] The main solution of the embodiment of this application is: based on the spatial dimension index and color dimension index of three-dimensional Gaussian points in the training view, retrieve the four-dimensional bilateral affine grid to obtain the affine transformation matrix, and adjust the color dimension index through the affine transformation matrix to obtain the adjusted color dimension index; render the training view based on the adjusted color dimension index through the three-dimensional Gaussian splash model to obtain a rendered image; optimize the parameters of the four-dimensional bilateral affine grid and the three-dimensional Gaussian splash model based on the photometric loss between the rendered image and the training view to obtain an optimized three-dimensional Gaussian splash model, wherein three-dimensional modeling of multi-view images is performed based on the optimized three-dimensional Gaussian splash model.

[0048] In this embodiment, for the convenience of description, the following will be described with an electronic device as the execution subject.

[0049] Currently, the direct method to solve multi-view inconsistency is to use the original sensor data to recover the radiation field. However, since many cameras do not provide the original sensor data due to hardware or software limitations, and the original sensor data requires a large amount of storage space and is usually not available, the NeRF-based method proposes Generative Latent Optimization (GLO) with appearance embedding. However, it is not applicable to 3D Gaussian splatting because the rendering of 3D Gaussian splatting is performed through per-frame rasterization rather than MLPs (Multilayer Perceptrons).

[0050] This application provides a solution. By using a bilateral affine grid as a post-processing module, it decouples the enhanced inconsistency in multi-view images, jointly guides the 3D Gaussian splatting for consistent reconstruction, reduces the floating objects in the scene, and improves the geometric modeling accuracy of the scene, thereby improving the rendering quality and reconstruction accuracy in the 3D modeling process.

[0051] In addition, compared with the NeRF-based method, this application does not need to rely on the original sensor data, avoids the problem of unavailability caused by hardware or software limitations, is a general solution applicable to various camera devices, and enhances the applicability and flexibility of the technology.

[0052] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication, and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device, an electronic device, etc. that can implement the above functions. The following will take an electronic device as an example to illustrate this embodiment and the following embodiments.

[0053] Based on this, the embodiment of this application provides a 3D modeling method. Refer to Figure 1 , Figure 1 which is a schematic flowchart of the first embodiment of the 3D modeling method of this application.

[0054] In this embodiment, the 3D modeling method includes steps S10 to S40:

[0055] Step S10: Based on the spatial dimension index and color dimension index of the 3D Gaussian points of the training view, retrieve the four-dimensional bilateral affine grid to obtain an affine transformation matrix, and adjust the color dimension index through the affine transformation matrix to obtain an adjusted color dimension index;

[0056] Preprocess the Gaussian points in the training view, extract their coordinate and color information, and initialize a four-dimensional bilateral affine grid to ensure that it can cover all Gaussian points in the training view. According to the coordinate and color information of the three-dimensional Gaussian points, retrieve the relevant affine transformation matrix in the four-dimensional bilateral affine grid, and adjust the color dimension index of the Gaussian points through the retrieved affine transformation matrix. The adjusted color dimension index is used for the subsequent rendering process. It should be noted that a clamp operation can be performed on the adjusted color dimension index to ensure that it is within a reasonable range, and this range can be set according to actual needs and is not limited here.

[0057] It can be understood that by adjusting the color dimension index through the affine transformation matrix, the color deviation can be effectively corrected, the color accuracy of the rendered image can be improved, the adjusted color dimension index can reflect the true color of the training view, and the visual effect of the rendered image can be enhanced.

[0058] Step S20, render the training view based on the adjusted color dimension index through a three-dimensional Gaussian splash model to obtain a rendered image;

[0059] Project the three-dimensional Gaussian points into the two-dimensional space to obtain the two-dimensional distribution of the Gaussian points. According to the adjusted color dimension index and the two-dimensional distribution characteristics of the Gaussian points, render through the three-dimensional Gaussian splash model to generate a rendered image corresponding to the training view for subsequent model optimization.

[0060] Step S30, optimize the parameters of the four-dimensional bilateral affine grid and the three-dimensional Gaussian splash model based on the photometric loss between the rendered image and the training view to obtain an optimized three-dimensional Gaussian splash model, where three-dimensional modeling of multi-view images is performed based on the optimized three-dimensional Gaussian splash model.

[0061] Calculate the photometric loss by comparing the photometric differences between the rendered image and the training view. According to the photometric loss, optimize the parameters of the four-dimensional bilateral affine grid and the parameters of the three-dimensional Gaussian splash model. After optimization, discard the four-dimensional bilateral affine grid and retain the optimized three-dimensional Gaussian splash model for subsequent three-dimensional modeling of multi-view images.

[0062] In a feasible implementation manner, the step S30, based on the photometric loss between the rendered image and the training view, optimizes the parameters of the four-dimensional bilateral affine grid and the three-dimensional Gaussian splash model to obtain an optimized three-dimensional Gaussian splash model, includes:

[0063] Step S301, traverse each pixel of the training view, input the pixel value of the pixel in the rendered image and the pixel value of the pixel in the training view into a preset loss function to obtain the photometric loss between the rendered image and the training view;

[0064] Traverse the training view pixel by pixel, extract the pixel values of the corresponding pixels in the rendered image and the training view, input the pixel values of the corresponding pixels in the rendered image and the training view into a preset loss function, and calculate the photometric loss for subsequent gradient calculation. It should be noted that before determining the photometric loss, the training view and the rendered image can be aligned to ensure that the resolutions and coordinate systems of the two images are consistent.

[0065] It should be noted that the structure of the preset loss function is not limited here and can be set according to actual needs. Exemplarily, in a feasible implementation manner, the mean square error can be used to calculate the photometric loss, and the specific formula is Loss = ∑ u，v (Irendered(u,v)-Itrue(u,v)) 2 , where (u, v) represents a pixel, Irendered(u,v) is the pixel value of the rendered image pixel, and Itrue(u,v) is the pixel value of the training view pixel.

[0066] Step S302, determine the first gradient of the photometric loss with respect to the model parameters in the three-dimensional Gaussian splash model, and determine the second gradient of the photometric loss with respect to the model parameters in the four-dimensional bilateral affine grid;

[0067] Determine the gradient of the photometric loss with respect to the model parameters in the three-dimensional Gaussian splash model (i.e., the first gradient) through backpropagation, and determine the gradient of the photometric loss with respect to the model parameters in the four-dimensional bilateral affine grid (i.e., the second gradient) through backpropagation.

[0068] Step S303, update the model parameters in the three-dimensional Gaussian splash model based on the first gradient and a preset optimization algorithm to obtain an updated three-dimensional Gaussian splash model, and update the model parameters in the four-dimensional bilateral affine grid based on the second gradient and the preset optimization algorithm to obtain an updated four-dimensional bilateral affine grid, until a preset optimization condition is reached, and determine the updated three-dimensional Gaussian splash model as the optimized three-dimensional Gaussian splash model.

[0069] Update the parameters in the three-dimensional Gaussian splash model based on the first gradient and a preset optimization algorithm (such as the gradient descent algorithm), update the parameters in the four-dimensional bilateral affine grid based on the second gradient and the preset optimization algorithm, repeat the above update process until a preset optimization condition (such as the loss value converges or the maximum number of iterations is reached), and determine the updated three-dimensional Gaussian splash model as the optimized three-dimensional Gaussian splash model.

[0070] In this embodiment, a bilateral affine grid is used as a post-processing module to decouple the enhancement inconsistencies in multi-view images, jointly guide the three-dimensional Gaussian splash for consistent reconstruction, reduce the floating objects in the scene, and improve the geometric modeling accuracy of the scene, thereby improving the rendering quality and reconstruction accuracy in the three-dimensional modeling process.

[0071] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar content as that in the above-mentioned first embodiment can be referred to the above introduction and will not be repeated hereinafter. On this basis, please refer to Figure 2 , step S10, the step of retrieving an affine transformation matrix by retrieving a four-dimensional bilateral affine grid based on the spatial dimension index and the color dimension index of the three-dimensional Gaussian points of the training view, includes:

[0072] Step S101, normalizing the point coordinates of the three-dimensional Gaussian points of the training view to obtain a spatial dimension index, and normalizing the point colors of the three-dimensional Gaussian points and mapping them to a grayscale space to obtain a color dimension index;

[0073] Normalize the spatial coordinates (x, y, z) of the three-dimensional Gaussian points to a predefined range (such as [0, 1]) to adapt to the index range of the four-dimensional bilateral affine grid. Normalize the color dimension index c of the Gaussian points to a predefined range. The normalized spatial coordinates are used as the spatial dimension index to locate the information related to the spatial position in the four-dimensional bilateral affine grid. The normalization formula can be: (x′, y′, z′) = ((x - xmin) / (xmax - xmin), (y - ymin) / (ymax - ymin), (z - zmin) / (zmax - zmin)), where xmin, ymin, zmin and xmax, ymax, zmax are the minimum and maximum values of the preset coordinate range respectively.

[0074] The point colors of the three-dimensional Gaussian points are usually represented by multiple color channels (such as RGB), and their value ranges may also be different. First, normalize the color values, map them to a standard interval, and then map the multi-channel color information to the grayscale space, convert it to a single-channel grayscale value, and use the obtained grayscale value as the color dimension index to locate the information related to the color in the four-dimensional bilateral affine grid.

[0075] Step S102, retrieving an affine transformation matrix by retrieving the four-dimensional bilateral affine grid based on the spatial dimension index and the color dimension index.

[0076] According to the normalized spatial dimension index and color dimension index, retrieve the corresponding affine transformation matrix in the four-dimensional bilateral affine grid to obtain the affine transformation matrix related to the three-dimensional Gaussian points for subsequent color adjustment.

[0077] In a feasible implementation manner, the dimensions of the four-dimensional bilateral affine grid include a spatial dimension and a color dimension; in step S102, based on the spatial dimension index and the color dimension index, retrieving the four-dimensional bilateral affine grid to obtain an affine transformation matrix includes:

[0078] Step S1021, mapping the spatial dimension index to the spatial dimension of the four-dimensional bilateral affine grid to obtain a spatial dimension index, and mapping the color dimension index to the color dimension of the four-dimensional bilateral affine grid to obtain a color dimension index;

[0079] Mapping the normalized spatial dimension index (x′, y′, z′) to the spatial dimension of the four-dimensional bilateral affine grid to obtain a spatial dimension index (h, i, j), and the mapping formula can be: where H, I, and J are the resolutions of the grid in the spatial dimension.

[0080] Mapping the normalized color dimension index c′ to the color dimension of the four-dimensional bilateral affine grid to obtain a color dimension index k. The mapping formula can be: where K is the resolution of the grid in the color dimension, and fe(c′) is a color mapping function.

[0081] Step S1022, determining a weight function of the three-dimensional Gaussian point at the grid point, where the grid point is a point determined based on the spatial dimension index and the color dimension index;

[0082] According to the spatial dimension index and the color dimension index, determining the grid point closest to the Gaussian point, and assigning a weight function to each grid point. The weight function can be determined based on the distance (in the spatial and color dimensions) between the Gaussian point and the grid point, for example, it can be a Gaussian weight or a bilinear weight.

[0083] Step S1023, retrieving the transformation matrix of the nearby points from the four-dimensional bilateral affine grid, and performing weighted processing on the transformation matrices of each of the nearby points through the weight function to obtain an affine transformation matrix, where the nearby points are grid points whose distance from the three-dimensional Gaussian point is less than a preset distance.

[0084] Retrieving grid points (i.e., nearby points) from the four-dimensional bilateral affine grid whose distance from the Gaussian point is less than a preset threshold, extracting the affine transformation matrix corresponding to each nearby point from the grid, and using the weight function to perform weighted processing on the transformation matrix of each nearby point to obtain the final affine transformation matrix.

[0085] In a feasible implementation, step S20 of adjusting the color dimension index through the affine transformation matrix to obtain an adjusted color dimension index includes:

[0086] Step S201: Expand the color dimension index into a four-dimensional vector to obtain a color vector, and transform the color vector through the affine transformation matrix to obtain a transformed color vector;

[0087] Expand the color dimension index of the three-dimensional Gaussian point into a four-dimensional vector. For example, expand the RGB color dimension index (r, g, b) into (r, g, b, 1), and transform the expanded four-dimensional color vector through the affine transformation matrix to obtain a transformed color vector. It should be noted that the 3x4 transformation matrix multiplies with a 4x1 four-dimensional color vector, and the obtained transformed color vector is a 3x1 vector, that is, a three-dimensional color vector.

[0088] Step S202: Perform a clamp operation on the transformed color vector based on a preset value range to obtain an adjusted color dimension index.

[0089] In this implementation, the effective value range of the color dimension index is preset in advance, that is, the preset value range. The preset value range can be set according to actual requirements and actual application scenarios. For example, for 8-bit image data, the preset value range can be (0, 255). Perform a clamp operation on the transformed three-dimensional color vector based on the preset value range. The clamp operation is to limit the element values in the transformed color vector to prevent them from exceeding the set preset value range to obtain an adjusted color dimension index for subsequent rendering. The specific process of the clamp operation is not elaborated here.

[0090] In a feasible implementation, before step S10 of retrieving an affine transformation matrix by using the spatial dimension index and color dimension index of the three-dimensional Gaussian points in the training view to retrieve a four-dimensional bilateral affine grid, it further includes:

[0091] Step S01: Decompose a preset four-dimensional grid into a sum of outer products of multiple vectors through tensor decomposition to obtain a decomposed grid, and determine the decomposed grid as a four-dimensional bilateral affine grid.

[0092] Since there are a large number of blank areas in the scene modeled by three-dimensional Gaussian splashing, only a part of the high-dimensional bilateral affine grid can be optimized, and the rest will not be accessed. To solve this problem, use CP decomposition to decompose the 4D bilateral affine grid into a sum of outer products of vectors to construct a low-rank approximation tensor:

[0093]

[0094] where, vC,r ∈R D , v Z,r ∈R N , v Y,r ∈R N , v X,r ∈R N , b r ∈R 12 , where R is the number of decompositions and also the maximum upper bound of the tensor rank after CP decomposition. For an approximate low-rank tensor, to simplify the formula, stack the R b r vectors to obtain B ∈ R 12×R . At the same time, stack the linear interpolations of the remaining four dimensions under each component and perform a dot product with B. The retrieval process can be rewritten as:

[0095]

[0096] In a feasible implementation manner, in step S20, the step of rendering the training view based on the adjusted color dimension index through a three-dimensional Gaussian splash model to obtain a rendered image includes:

[0097] Step S201, project each three-dimensional Gaussian point in the training view into a two-dimensional space to obtain the two-dimensional Gaussian distribution of each three-dimensional Gaussian point;

[0098] Map the Gaussian points in three-dimensional space to a two-dimensional plane through a projection matrix (such as perspective projection or orthographic projection) to obtain the two-dimensional positions and distributions of each Gaussian point. According to the attributes of the three-dimensional Gaussian points (such as the mean and covariance matrix), calculate the Gaussian distribution parameters (mean and covariance) in the two-dimensional space to obtain the Gaussian distribution of each Gaussian point in the two-dimensional space, including its position, width, and direction.

[0099] Step S202, traverse each two-dimensional Gaussian distribution, and perform rasterization processing based on the coordinates of the two-dimensional Gaussian distribution, the adjusted color dimension index, and the rendering attribute data of the training view to obtain the color contribution value of each two-dimensional Gaussian distribution to each pixel;

[0100] Traverse the two-dimensional Gaussian distribution, and calculate the color contribution value of the Gaussian distribution to each pixel according to the coordinates of the two-dimensional Gaussian distribution, the adjusted color dimension index, and the rendering attributes of the training view (such as lighting, material, etc.) to obtain the color contribution value of each two-dimensional Gaussian distribution to the pixel.

[0101] Step S203, traverse each pixel, and superimpose the color contribution values of each two-dimensional Gaussian distribution to the pixel to obtain the pixel color;

[0102] Traverse and render the view pixel by pixel, superimpose the color contribution values of all two-dimensional Gaussian distributions to this pixel, obtain the final pixel color, and thus obtain the final color dimension index of each pixel. By superimposing the color contributions of multiple Gaussian distributions, smooth color transitions and blending effects can be achieved, preserving the continuity of the Gaussian distribution and avoiding the jaggedness and discontinuities that may occur in traditional rasterization.

[0103] Step S204, based on the respective pixel colors of each pixel, combine each pixel to obtain a rendered image.

[0104] Combine the color dimension indices of all pixels into a complete rendered image. It can be understood that post-processing can be performed on the rendered image, such as anti-aliasing, color correction, etc., to improve the overall quality of the rendered image.

[0105] In this embodiment, through a bilateral affine grid as a post-processing module, the enhancement inconsistencies in multi-view images are decoupled, jointly guiding the three-dimensional Gaussian splash for consistent reconstruction, reducing floating objects in the scene and improving the geometric modeling accuracy of the scene, thereby improving the rendering quality and reconstruction accuracy in the three-dimensional modeling process.

[0106] Exemplarily, to help understand the implementation process of the three-dimensional modeling method obtained based on the above embodiments, please refer to Figure 3 , Figure 3 A framework schematic diagram of a three-dimensional modeling method is provided. Specifically:

[0107] 4D (Four-Dimensional) bilateral affine grid Where N is the spatial resolution, D is the color resolution, and 12 corresponds to the length of a flattened 3×4 affine transformation. For a training view, use the color and spatial position data of the Gaussian basis elements to retrieve the corresponding 4D bilateral affine grid to achieve color transformation of the Gaussian points. Based on Figure 3 The framework shown, the specific process is as follows:

[0108] 1) Normalize the point coordinates of the 3D (Three-Dimensional) Gaussian basis elements in space to obtain {x, y, z} ∈ [0, 1] 3 , normalize the color to obtain c = {c R , c G , c B} ∈ [0, 1] 3 ;

[0109] 2) Use the normalized values to retrieve the 4D bilateral affine grid to obtain the corresponding affine transformation:

[0110]

[0111] w h,i,j,k (x, y, z, c g ) = g(N·x - h)g(N·y - i)g(N·z - j)g(D·c g - k).

[0112] Where g(t) = max(0, 1 - |t|) is a linear interpolation kernel function, and f θ (·) is a two - layer MLPs network that maps the RGB color c to a scalar c g ∈ [0, 1], and T h,i,j,k is an affine tensor indexed by h, i, j, k in the affine grid.

[0113] 3) Reshape into an affine transformation matrix Perform an affine transformation on the color c to obtain the transformed color c' = A[c|1] T , coordinates.

[0114] 4) Use the color attributes and other attributes (mean, i.e., center position, scale, rotation, opacity, etc.) after the 3D Gaussian basis element transformation to rasterize and obtain a rendered image. Calculate the photometric loss using the rendered image and the ground - truth image, and perform joint optimization.

[0115] 5) After the optimization is completed, perform 3D modeling on the multi - view images based on the optimized 3D Gaussian splash model.

[0116] It should be noted that the above examples are only for understanding this application and do not limit the 3D modeling method of this application. Based on this technical concept, more simple transformations in various forms are within the protection scope of this application.

[0117] This application also provides a 3D modeling device. Please refer to Figure 4 , and the 3D modeling device includes:

[0118] A four - dimensional affine transformation module 10, which is used to retrieve a four - dimensional bilateral affine grid based on the spatial - dimension index and color - dimension index of the 3D Gaussian points of the training views to obtain an affine transformation matrix, and adjust the color - dimension index through the affine transformation matrix to obtain an adjusted color - dimension index;

[0119] A rendering module 20, which is used to render the training views based on the adjusted color - dimension index through a 3D Gaussian splash model to obtain a rendered image;

[0120] The model optimization module 30 is configured to optimize the parameters of the four-dimensional bilateral affine grid and the three-dimensional Gaussian splash model based on the photometric loss between the rendered image and the training view, so as to obtain an optimized three-dimensional Gaussian splash model, wherein three-dimensional modeling is performed on the multi-view image based on the optimized three-dimensional Gaussian splash model.

[0121] Optionally, the four-dimensional affine transformation module 10 is configured to:

[0122] Normalize the point coordinates of the three-dimensional Gaussian points of the training view to obtain a spatial dimension index, and normalize the point colors of the three-dimensional Gaussian points and map them to the gray space to obtain a color dimension index;

[0123] Retrieve a four-dimensional bilateral affine grid based on the spatial dimension index and the color dimension index to obtain an affine transformation matrix.

[0124] Optionally, the dimensions of the four-dimensional bilateral affine grid include a spatial dimension and a color dimension;

[0125] The four-dimensional affine transformation module 10 is configured to:

[0126] Map the spatial dimension index to the spatial dimension of the four-dimensional bilateral affine grid to obtain a spatial dimension index, and map the color dimension index to the color dimension of the four-dimensional bilateral affine grid to obtain a color dimension index;

[0127] Determine a weight function of the three-dimensional Gaussian points at the grid points, where the grid points are points determined based on the spatial dimension index and the color dimension index;

[0128] Retrieve the transformation matrices of the nearby points from the four-dimensional bilateral affine grid, and weight the transformation matrices of the respective nearby points through the weight function to obtain an affine transformation matrix, where the nearby points are grid points whose distance from the three-dimensional Gaussian points is less than a preset distance.

[0129] Optionally, the four-dimensional affine transformation module 10 is configured to:

[0130] Expand the color dimension index into a four-dimensional vector to obtain a color vector, and transform the color vector through the affine transformation matrix to obtain a transformed color vector;

[0131] Perform a clamp operation on the transformed color vector based on a preset value range to obtain an adjusted color dimension index.

[0132] Optionally, the apparatus further includes a decomposition module, configured to:

[0133] Decompose a preset four-dimensional grid into the sum of outer products of multiple vectors through tensor decomposition to obtain a decomposed grid, and determine the decomposed grid as a four-dimensional bilateral affine grid.

[0134] Optionally, the rendering module 20 is configured to:

[0135] Project each three-dimensional Gaussian point in the training view into a two-dimensional space to obtain the two-dimensional Gaussian distribution of each three-dimensional Gaussian point;

[0136] Traverse each of the two-dimensional Gaussian distributions, and perform rasterization processing based on the coordinates of the two-dimensional Gaussian distribution, the adjusted color dimension index, and the rendering attribute data of the training view to obtain the color contribution value of each two-dimensional Gaussian distribution to each pixel;

[0137] Traverse each of the pixels, and superimpose the color contribution values of each two-dimensional Gaussian distribution to the pixel to obtain the pixel color;

[0138] Based on the pixel colors of each pixel, combine each pixel to obtain a rendered image.

[0139] Optionally, the model optimization module 30 is configured to:

[0140] Traverse each pixel of the training view, input the pixel value of the pixel in the rendered image and the pixel value of the pixel in the training view into a preset loss function to obtain the photometric loss between the rendered image and the training view;

[0141] Determine the first gradient of the photometric loss with respect to the model parameters in the three-dimensional Gaussian splash model, and determine the second gradient of the photometric loss with respect to the model parameters in the four-dimensional bilateral affine grid;

[0142] Update the model parameters in the three-dimensional Gaussian splash model based on the first gradient and a preset optimization algorithm to obtain an updated three-dimensional Gaussian splash model, and update the model parameters in the four-dimensional bilateral affine grid based on the second gradient and the preset optimization algorithm to obtain an updated four-dimensional bilateral affine grid, until a preset optimization condition is reached, and determine the updated three-dimensional Gaussian splash model as an optimized three-dimensional Gaussian splash model.

[0143] The three-dimensional modeling device provided by this application adopts the three-dimensional modeling method in the above-mentioned embodiment, and can solve the technical problem of low accuracy of the geometric modeling result of three-dimensional Gaussian splash reconstruction caused by inconsistent multi-view image enhancement. Compared with the prior art, the beneficial effects of the three-dimensional modeling device provided by this application are the same as those of the three-dimensional modeling method provided by the above-mentioned embodiment, and other technical features in the three-dimensional modeling device are the same as the features disclosed in the method of the above-mentioned embodiment, and will not be elaborated here.

[0144] The present application provides an electronic device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein, the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the 3D modeling method in the above embodiments.

[0145] Reference is made below Figure 5 , which shows a schematic structural diagram of an electronic device suitable for implementing the embodiments of the present application. The electronic device in the embodiments of the present application may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions: tablet computers), PMPs (Portable Media Players), in-vehicle terminals (such as in-vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is only an example and should not impose any limitations on the functions and usage scope of the embodiments of the present application.

[0146] As Figure 5 shown, the electronic device may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM: Random Access Memory) 1004. In the RAM 1004, various programs and data required for the operation of the electronic device are also stored. The processing device 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems may be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the electronic device to communicate with other devices wirelessly or wirelesly to exchange data. Although the figure shows an electronic device having various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or provided alternatively.

[0147] In particular, according to the embodiments disclosed in the present application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product that includes a computer program carried on a computer-readable medium, and the computer program contains program codes for executing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by a processing device 1001, the above-mentioned functions defined in the methods of the embodiments disclosed in the present application are executed.

[0148] The electronic device provided in the present application adopts the three-dimensional modeling method in the above-mentioned embodiment, and can solve the technical problem that the accuracy of the geometric modeling result of the three-dimensional Gaussian splash reconstruction is low due to inconsistent multi-view image enhancement. Compared with the prior art, the beneficial effects of the electronic device provided in the present application are the same as those of the three-dimensional modeling method provided in the above-mentioned embodiment, and other technical features in the electronic device are the same as those disclosed in the method of the previous embodiment, and will not be elaborated here.

[0149] It should be understood that the various parts disclosed in the present application can be implemented by hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in a suitable manner in any one or more embodiments or examples.

[0150] As described above, only the specific embodiments of the present application are provided, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed in the present application, and all of them should be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

[0151] The present application provides a computer-readable storage medium having computer-readable program instructions (i.e., computer programs) stored thereon, and the computer-readable program instructions are used to execute the three-dimensional modeling method in the above-mentioned embodiment.

[0152] The computer-readable storage medium provided by this application can be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM) or flash memory, optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above. In this embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system or device. The program code contained on the computer-readable storage medium can be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0153] The above computer-readable storage medium can be included in an electronic device; or can exist separately without being assembled into the electronic device.

[0154] The above computer-readable storage medium carries one or more programs. When the above one or more programs are executed by an electronic device, the electronic device is caused to: retrieve a four-dimensional bilateral affine grid based on the spatial dimension index and color dimension index of the three-dimensional Gaussian points of the training view to obtain an affine transformation matrix, and adjust the color dimension index through the affine transformation matrix to obtain an adjusted color dimension index; render the training view based on the adjusted color dimension index through a three-dimensional Gaussian splash model to obtain a rendered image; optimize the parameters of the four-dimensional bilateral affine grid and the three-dimensional Gaussian splash model based on the photometric loss between the rendered image and the training view to obtain an optimized three-dimensional Gaussian splash model, wherein three-dimensional modeling of a multi-view image is performed based on the optimized three-dimensional Gaussian splash model.

[0155] Computer program code for performing the operations of this application can be written in one or more programming languages or combinations thereof. The above-mentioned programming languages include object-oriented programming languages such as Java, Smalltalk, C++, and also include conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, executed as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN: Local Area Network) or a wide area network (WAN: Wide Area Network), or it can be connected to an external computer (for example, by connecting through the Internet using an Internet service provider).

[0156] The flowcharts and block diagrams in the accompanying drawings illustrate the possible architectures, functions, and operations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in the flowchart or block diagram can represent a module, a program segment, or a part of the code, and this module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks can occur in a different order than that marked in the accompanying drawings. For example, two consecutively represented blocks can actually be executed substantially in parallel, and they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and / or flowchart, as well as the combination of blocks in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system for performing the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0157] The modules described in the embodiments of this application can be implemented in software or in hardware. Among them, the name of the module does not constitute a limitation on the unit itself in some cases.

[0158] The readable storage medium provided by this application is a computer-readable storage medium. The computer-readable storage medium stores computer-readable program instructions (i.e., computer programs) for performing the above-mentioned three-dimensional modeling method, and can solve the technical problem of low accuracy of the geometric modeling results of three-dimensional Gaussian splash reconstruction caused by inconsistent multi-view image enhancement. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by this application are the same as those of the three-dimensional modeling method provided by the above embodiments, and will not be elaborated here.

[0159] The present application also provides a computer program product, including a computer program which, when executed by a processor, implements the steps of the three-dimensional modeling method as described above.

[0160] The computer program product provided by the present application can solve the technical problem that the accuracy of the geometric modeling result of three-dimensional Gaussian splash reconstruction is low due to inconsistent multi-view image enhancement. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as those of the three-dimensional modeling method provided by the above embodiments, and will not be elaborated here.

[0161] The above are only partial embodiments of the present application, and thus do not limit the patent scope of the present application. Any equivalent structural transformation made under the technical concept of the present application by using the content of the specification and drawings of the present application, or any direct / indirect application in other related technical fields, is included in the patent protection scope of the present application.

Claims

1. A three-dimensional modeling method, characterized in that: The three-dimensional modeling method comprises: Based on the spatial dimension index and color dimension index of the three-dimensional Gaussian point of the training view, retrieve the four-dimensional bilateral affine grid to obtain an affine transformation matrix, and adjust the color dimension index by the affine transformation matrix to obtain an adjusted color dimension index; Rendering the training view based on the adjusted color dimension index using a three-dimensional Gaussian splash model to obtain a rendering image; Based on the photometric loss between the rendering and the training view, the four-dimensional bilateral affine grid and the three-dimensional Gaussian splash model are parameter optimized to obtain an optimized three-dimensional Gaussian splash model, wherein three-dimensional modeling of multi-view images is performed based on the optimized three-dimensional Gaussian splash model.

2. The three-dimensional modeling method according to claim 1, characterized in that: The step of retrieving a four-dimensional bilateral affine grid to obtain an affine transformation matrix based on the spatial dimension index and the color dimension index of the three-dimensional Gaussian point of the training view comprises: Normalizing the point coordinates of the three-dimensional Gaussian points of the training view to obtain a spatial dimension index, and normalizing the point colors of the three-dimensional Gaussian points and mapping them to a grayscale space to obtain a color dimension index; Based on the spatial dimension index and the color dimension index, a four-dimensional bilateral affine grid is retrieved to obtain an affine transformation matrix.

3. The three-dimensional modeling method according to claim 2, characterized in that: The dimensions of the four-dimensional bilateral affine grid include a spatial dimension and a color dimension; The step of retrieving a four-dimensional bilateral affine grid to obtain an affine transformation matrix based on the spatial dimension index and the color dimension index comprises: Mapping the spatial dimension index to the spatial dimension of the four-dimensional bilateral affine grid to obtain a spatial dimension index, and mapping the color dimension index to the color dimension of the four-dimensional bilateral affine grid to obtain a color dimension index; Determine a weight function of the three-dimensional Gaussian point at a grid point, wherein the grid point is a point determined based on the spatial dimension index and the color dimension index; The transformation matrix of the close-distance point is retrieved from the four-dimensional bilateral affine grid, and the transformation matrix of each of the close-distance points is weighted by the weight function to obtain an affine transformation matrix, wherein the close-distance point is a grid point whose distance to the three-dimensional Gaussian point is less than a preset distance.

4. The three-dimensional modeling method according to claim 3, characterized in that: The step of adjusting the color dimension index by using the affine transformation matrix to obtain an adjusted color dimension index comprises: Expanding the color dimension index into a four-dimensional vector to obtain a color vector, and transforming the color vector using the affine transformation matrix to obtain a transformed color vector; A clamp operation is performed on the transformed color vector based on a preset value range to obtain an adjusted color dimension index.

5. The three-dimensional modeling method according to claim 3, characterized in that: Before the step of retrieving the four-dimensional bilateral affine grid to obtain the affine transformation matrix based on the spatial dimension index and the color dimension index of the three-dimensional Gaussian point of the training view, the method further includes: The preset four-dimensional grid is decomposed into the sum of outer products of multiple vectors through tensor decomposition to obtain a decomposed grid, and the decomposed grid is determined as a four-dimensional bilateral affine grid.

6. The three-dimensional modeling method according to claim 1, characterized in that: The step of rendering the training view by using a three-dimensional Gaussian splash model based on the adjusted color dimension index to obtain a rendering image includes: Projecting each three-dimensional Gaussian point in the training view into a two-dimensional space to obtain a two-dimensional Gaussian distribution of each three-dimensional Gaussian point; Traversing each of the two-dimensional Gaussian distributions, performing rasterization processing based on the coordinates of the two-dimensional Gaussian distribution, the adjusted color dimension index, and the rendering attribute data of the training view, to obtain a color contribution value of the two-dimensional Gaussian distribution to each pixel; Traversing each pixel, and superimposing the color contribution value of each two-dimensional Gaussian distribution to the pixel to obtain the pixel color; Based on the pixel colors of the pixels, the pixels are combined to obtain a rendering.

7. The three-dimensional modeling method according to any one of claims 1 to 6, characterized in that: The step of performing parameter optimization on the four-dimensional bilateral affine grid and the three-dimensional Gaussian splash model to obtain an optimized three-dimensional Gaussian splash model based on the photometric loss between the rendering image and the training view comprises: Traversing each pixel of the training view, inputting the pixel value of the pixel in the rendering image and the pixel value of the pixel in the training view into a preset loss function, and obtaining the photometric loss between the rendering image and the training view; Determining a first gradient of the photometric loss with respect to a model parameter in the three-dimensional Gaussian splatter model, and determining a second gradient of the photometric loss with respect to a model parameter in the four-dimensional bilateral affine grid; Based on the first gradient and the preset optimization algorithm, the model parameters in the three-dimensional Gaussian splash model are updated to obtain an updated three-dimensional Gaussian splash model, and based on the second gradient and the preset optimization algorithm, the model parameters in the four-dimensional bilateral affine grid are updated to obtain an updated four-dimensional bilateral affine grid, until the preset optimization conditions are reached, and the updated three-dimensional Gaussian splash model is determined as the optimized three-dimensional Gaussian splash model.

8. An electronic device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the three-dimensional modeling method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the three-dimensional modeling method according to any one of claims 1 to 7 are implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the steps of the three-dimensional modeling method according to any one of claims 1 to 7 are implemented.

Images