Large-scale geometric deformation method and device based on Gaussian network representation, and storage medium

Abstract

The invention provides a large-scale geometric deformation method and device based on Gaussian network representation and a storage medium, and the method comprises the steps: embedding a Gaussian into each surface of a grid of an editing object, and constructing Gaussian network representation; the method comprises the following steps: taking a multi-view picture as input, extracting a proxy grid, taking the proxy grid as priori to initialize a Gaussian network, and performing training optimization on Gaussian network representation; and executing a deformation operation on the proxy grid, and driving the optimized Gaussian network to deform. According to the method, the deformed high-quality picture can be rendered in real time in a synthetic or real object and scene.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision, and in particular relates to a large-scale geometric deformation method, device and storage medium based on Gaussian network representation. Background Art

[0002] Deformation is of great significance in the computer field. Specifically, in the fields of film, games and animation, 3D deformation is a key technology to achieve realistic animation and special effects. By deforming 3D models, the natural movement and deformation of characters, objects and scenes can be achieved, making them more vivid and realistic; in terms of user experience such as VR, 3D deformation allows users to freely edit and design three-dimensional models. Through deformation operations, users can change the shape, size, posture and topology of the model, thereby achieving personalized shape customization and creative design. In the direction of computer vision and pattern recognition, 3D deformation can be used for shape matching and alignment. By deforming one shape into a shape similar to another, shape matching and alignment can be achieved, which can be used in applications such as target recognition, object tracking and posture estimation.

[0003] Meshes, neural radiation fields, and Gaussian splattering are related technologies that have been applied to 3D deformation, but these methods cannot meet the three requirements of real-time rendering, high-quality rendering, and applicability to large-scale deformation.

[0004] Among them, the representation of the mesh includes editing of shape and texture by editing the 3D mesh, where shape editing includes changing its shape and simulating shape editing according to user-defined conditions. This method can be performed in real time while maintaining geometric details. However, it can only be done on an explicit mesh representation, which hinders its further use in capturing modern representations of geometry and appearance, and it is difficult to render high-quality results using traditional rendering pipelines under new perspectives. The neural radiation field representation learns the mapping relationship between spatial coordinates and target shapes through a neural network to reconstruct highly realistic textures and light and shadow effects, but a high-quality neural radiation field requires a large number of sampling operations, resulting in slow training and inference speeds, and it is difficult to edit and drive intuitively due to the lack of a display agent, making the neural radiation field unable to support large-scale geometric editing and real-time interactive editing. The Gaussian splash representation combines the advantages of explicit and implicit representations, expressing the three-dimensional scene as a combination of multiple three-dimensional Gaussian spherical functions, with the advantages of fast training speed, short inference time and high rendering quality. However, due to its use of discrete Gaussian distribution and lack of explicit topological structure, it is not easy to deform. Summary of the invention

[0005] In view of the shortcomings of the prior art, the present invention proposes a large-scale geometric deformation method, device, and storage medium based on Gaussian mesh representation. The method can render deformed high-quality images in real time in synthetic or real objects and scenes, and supports mesh deformation technologies such as data-driven deformation.

[0006] In order to achieve the above object, the present invention provides a large-scale geometric deformation method based on Gaussian network representation, comprising:

[0007] Embed a Gaussian on each face of the mesh of the edited object to construct a Gaussian network representation;

[0008] Taking multi-view images as input, extracting proxy meshes, using the proxy meshes as priors to initialize Gaussian nets, and training and optimizing the Gaussian net representation;

[0009] Perform a deformation operation on the proxy mesh to drive the optimized Gaussian mesh to deform.

[0010] In some embodiments, during the initial construction of the Gaussian mesh, each Gaussian of the Gaussian mesh is constrained by an offset amplitude for limiting the moving distance of the Gaussian on the attached mesh face and a maximum distance of the Gaussian offset along the normal direction, and the spatial coordinates of the Gaussian are obtained;

[0011] Among them, the spatial coordinates of the Gaussian are the product of the offset amplitude and the maximum distance offset along the normal direction and the sum of the centroid coordinates of the mesh face to which the Gaussian is attached; the maximum distance offset along the normal direction is represented by the radius of the circumscribed circle of the mesh face to which each Gaussian is attached.

[0012] In some embodiments, a multi-view image is used as input to estimate camera parameters and obtain a mask of the image;

[0013] The proxy mesh is reconstructed using the camera parameters and the mask.

[0014] In some embodiments, during the optimization iteration process of the Gaussian mesh representation, a regularized loss term constrained by the mesh patch size is introduced into the loss function, and the Gaussian shape is adjusted according to the size of the attached mesh patch.

[0015] In some embodiments, the regularized loss term adjusts the Gaussian shape according to the circumscribed circle radius of the triangular mesh patch to which the Gaussian is attached. The regularized loss term is expressed as:

[0016] L r =∑max(max(s i )-γR i ,0)

[0017] Among them, L r represents the regular loss term, s iRepresents the three-dimensional scale vector of each Gaussian distribution, R i It represents the radius of the circumscribed circle of the triangular mesh patch where the Gaussian distribution is located. γ is a hyperparameter used to control the influence of adjacent triangular mesh patches on the size of the Gaussian distribution.

[0018] In some embodiments, during the optimization iteration process of the Gaussian mesh representation, Gaussian spheres that do not meet preset rendering conditions are identified, split according to the attached mesh faces, and the newly split mesh faces are updated to the proxy mesh.

[0019] In some embodiments, for the Gaussian sphere that does not meet the preset rendering condition, a new vertex is inserted into the midpoint of each side of the triangular mesh patch to which the Gaussian sphere is attached, so that the single triangular mesh patch is split into four small triangular mesh patches on the curved surface;

[0020] The Gaussian sphere is split into four parts, and each part belongs to a small triangular mesh patch.

[0021] In some embodiments, performing a deformation operation on the proxy mesh to drive the optimized Gaussian mesh to deform includes:

[0022] Deform the proxy mesh, extract the relative displacement of the mesh face vertices before and after deformation and the corresponding deformation gradient, and obtain the mesh deformation information;

[0023] The mesh deformation information is transmitted to the Gaussian attached to the mesh patch;

[0024] Each Gaussian in the Gaussian network is driven to deform using the mesh deformation information to obtain a deformed Gaussian network.

[0025] In some embodiments, a transformation matrix is ​​determined by minimizing a deformation energy function to control the proxy mesh to deform;

[0026] Decomposing the transformation matrix into a rotation matrix and an affine matrix;

[0027] The relative displacement is determined using the coordinates of the center of gravity of the mesh surface before deformation and the coordinates of the vertices of the mesh surface before and after deformation;

[0028] Determine a rotation component using the centroid coordinates of the mesh face before deformation and the rotation matrix;

[0029] Determine an affine component using the centroid coordinates of the mesh face before deformation and the affine matrix;

[0030] The deformation gradient is determined according to the product of the rotation component and the affine component.

[0031] In some embodiments, the deformation energy function is expressed as:

[0032]

[0033] Where N(i) represents the set of 1-ring adjacent vertices of vertex i, v i is the spatial coordinate of the i-th vertex on the mesh patch M before deformation, v i ' is the spatial coordinate of the corresponding vertex on the deformed mesh patch M' corresponding to M, w ij is the cotangent weight, w ij =cotα ij +cotβ ij , where α ij and β ij is the angle in the adjacent triangle opposite to the edge (i, j) on the mesh patch M, and Ti is the transformation matrix;

[0034] The relative displacement is expressed as:

[0035] ΔP=w a (v a '-v a )+w b (v b '-v b )+w c (v c '-v c )

[0036] Among them, ΔP is the relative displacement, and the centroid coordinates of the mesh patch before deformation are (w a , w b , w c ), the vertex coordinates of the mesh patch before deformation are (v a , v b , v c ), the vertex coordinates of the deformed mesh patch are (v a ',v b ',v c ');

[0037] This rotation component is expressed as:

[0038] logR=w a logR a +w b logR b +w c logR c

[0039] Among them, R is the rotation component, and the rotation matrix is ​​(R a , R b , R c );

[0040] The affine component is expressed as:

[0041] S=w a S a +w b S b +w c S c

[0042] Among them, S is the affine component, and the affine matrix is ​​(S a , S b , S c ).

[0043] In some embodiments, each Gaussian of the Gaussian mesh is driven to deform using the mesh deformation information, comprising:

[0044] Using this relative displacement, the spatial coordinates of the deformed Gaussian are determined as:

[0045] μ'=μ+ΔP

[0046] Among them, μ' is the spatial coordinate before deformation, and μ is the spatial coordinate after deformation;

[0047] Using the deformation gradient, the covariance matrix corresponding to the deformed Gaussian is determined as:

[0048] Σ'=TΣT T

[0049] Where T is the deformation gradient;

[0050] Using the rotation component, the spherical harmonic function of the deformed Gaussian is determined as:

[0051] SH'(d,c)=SH(R T d,c)

[0052] Among them, R is the rotation component, d and c are the parameters of the spherical harmonics.

[0053] In addition, in some embodiments, if the edited object is identified as a scene, during the initial construction of the Gaussian network, the scene is decoupled into an object part and a background part, wherein:

[0054] The object part is represented by a Gaussian net;

[0055] The background portion is represented using Gaussian splattering.

[0056] In some embodiments, if the edited object is identified as a scene, during the training and optimization process of the Gaussian network,

[0057] The multi-view image and the mask extracted from the multi-view image are used as supervision, and a Gaussian network method is used for training optimization to obtain a first Gaussian network;

[0058] Taking the multi-view image as supervision, the first Gaussian net is taken as a fixed parameter and spliced ​​with the first Gaussian field to be optimized, and the Gaussian splashing method is used for training optimization. In the process of optimizing the first Gaussian field, every preset number of rounds, the Gaussians in the first Gaussian field whose spacing distance from the first Gaussian net exceeds a preset distance are deleted.

[0059] In some embodiments, if the edited object is identified as a scene, the deformed Gaussian mesh is further spliced ​​with the first Gaussian field represented by the background as Gaussian splash and rendered to obtain a final deformed scene image.

[0060] Another aspect of the present invention is to provide a large-scale geometric deformation device based on Gaussian network representation, which adopts the above method and at least comprises:

[0061] Gaussian network representation module, used to embed a Gaussian on each face of the mesh of the edited object to construct a Gaussian network representation;

[0062] A Gaussian network training and optimization module is used to extract a proxy grid from a multi-view image as input, and use the proxy grid as a prior to initialize the Gaussian network, and train and optimize the Gaussian network representation;

[0063] The Gaussian network deformation driving module is used to perform deformation operations on the proxy mesh and drive the optimized Gaussian network to deform.

[0064] In yet another aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the above-mentioned large-scale geometric deformation method based on Gaussian network representation.

[0065] It can be seen from the above scheme that the advantages of the present invention are:

[0066] The large-scale geometric deformation method based on Gaussian network representation disclosed by the present invention constructs a Gaussian network representation by embedding a Gaussian on each face of the mesh of the edited object; then, a proxy mesh is extracted with a multi-view image as input, and the proxy mesh is used as a priori to initialize the Gaussian network, and the Gaussian network representation is trained and optimized using a patch splitting strategy and a regular constraint term; and the optimized Gaussian network is driven to deform by performing a deformation operation on the proxy mesh, so that the user can drive the three-dimensional Gaussian field to transform by performing an intuitive and interactive deformation on the proxy mesh, and the deformation has extremely high real-time performance. The method can render high-quality deformed images in real time in both synthetic or real objects and scenes, and supports mesh deformation technologies such as data-driven deformation. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1A schematic diagram of the overall process of a large-scale geometric deformation method based on Gaussian network representation provided by an embodiment of the present invention is shown;

[0068] Figure 2 Shows Figure 1 A schematic diagram showing the Gaussian network representation principle of step S1;

[0069] Figure 3 The rendering results of the object at different angles after being deformed by the method of the present invention are shown;

[0070] Figure 4 The deformation results of the method of the present invention in different scenarios are shown;

[0071] Figure 5 The deformation results of the method of the present invention at different grid resolutions are shown;

[0072] Figure 6 The deformation results of different mesh extraction methods are shown;

[0073] Figure 7 shows the rendering result of data-driven deformation;

[0074] Figure 8 A schematic block diagram of a large-scale geometric deformation device based on Gaussian network representation provided by an embodiment of the present invention is shown.

[0075] in:

[0076] 200: Large-scale geometric deformation device based on Gaussian network representation;

[0077] 201: Gaussian network representation module;

[0078] 202: Gaussian network training optimization module;

[0079] 203: Gaussian network deformation driving module. DETAILED DESCRIPTION

[0080] In order to make the above features and effects of the present invention more clearly understood, embodiments are given below and described in detail with reference to the accompanying drawings.

[0081] See also Figure 1 , Figure 1 The overall process diagram of a large-scale geometric deformation method based on Gaussian network representation provided by an embodiment of the present invention is shown. The large-scale geometric deformation method based on Gaussian network representation comprises the following steps:

[0082] Step S1: embed a Gaussian on each face of the mesh of the edited object to construct a Gaussian network representation.

[0083] The Gaussian net representation is a representation method based on the Gaussian splash of the grid, which takes the rough grid of the object part of an edited object as the geometric prior, and embeds Gaussians on the surface of the grid. In some embodiments, a Gaussian is embedded in the center of gravity of each surface of the grid, and the coordinates of each vertex of the grid surface and the coordinates of the center of gravity are recorded. Specifically, the Gaussians in the Gaussian net retain the characteristics of the original Gaussian splash representation of scaling, rotation, occupancy and color, and use the optimizable center of gravity coordinates and the three vertex coordinates of the grid surface to represent the spatial coordinates of the Gaussian.

[0084] Specifically, Figure 2 As shown in , in the process of constructing the Gaussian mesh representation, a normal-guided strategy is used, and an additional offset amplitude t is introduced to limit the moving distance of the Gaussian on the attached mesh face, where t is between -0.5 and 0.5, and a Sigmoid function is used for activation. At the same time, the maximum distance of the Gaussian offset along the normal direction n is limited by the circumscribed circle radius R of the mesh face to which each Gaussian is attached, so that in the initial construction process of the Gaussian mesh, each Gaussian of the Gaussian mesh is constrained by the offset amplitude and the maximum distance of the Gaussian offset along the normal direction to obtain the spatial coordinates of the Gaussian. In order to reduce the error caused by the inaccurate mesh, combined with the barycentric coordinates, the spatial coordinates μ of the Gaussian are the product of the offset amplitude and the maximum distance offset along the normal direction and the barycentric coordinates of the mesh face to which the Gaussian is attached, that is:

[0085] μ=(w a +w b +w c )+tRn

[0086] Among them, the center of gravity coordinate w=(w a , w b , w c ).

[0087] In addition, a patch splitting strategy is also used in the process of constructing the Gaussian mesh representation. Gaussians that do not meet the preset rendering conditions, that is, those Gaussians that affect the rendering quality, are identified, and are split according to the attached mesh patches, and the newly split mesh patches are updated to the mesh. Specifically, for the Gaussian that does not meet the preset rendering conditions, a new vertex is inserted into the midpoint of each side of the triangular mesh patch to which the Gaussian is attached, and a single triangular mesh patch is split into four small triangular mesh patches on the surface; the Gaussian is split into four parts in the same way, and each part belongs to a small triangular mesh patch.

[0088] In this embodiment, if the editing object is an object, the editing object can be directly represented by a Gaussian net through the above process. However, in some embodiments, if the editing object is identified as a scene rather than an object, in this case, a hierarchical representation based on a Gaussian net is adopted. That is, in the initial construction process of the Gaussian net, the scene is decoupled into an object part and a background part, wherein: the object part is represented by a Gaussian net, and the background part is represented by a Gaussian splash, so as to facilitate staged training during the optimization process of step S2, and in step S3, the deformed Gaussian net is spliced ​​with the background Gaussian field during driving and rendered together to complete the editing of the scene.

[0089] In this embodiment, through the above step S1, a Gaussian network expression is created, and the Gaussian network can be further trained and deformed.

[0090] Step S2: Taking the multi-view image as input, extracting the proxy grid, and using the proxy grid as a priori to initialize the Gaussian network, and training and optimizing the Gaussian network representation.

[0091] In this embodiment, for the training and optimization stage of the Gaussian network, firstly, multi-view images (i.e., images taken from multiple perspectives) are selected as model input, and the ColMap camera pose estimation method is used to estimate the camera parameters; at the same time, an unsupervised segmentation method is used to obtain the mask of the object to be deformed, that is, the mask of the image is obtained. Then, the camera parameters and the mask are used to reconstruct the proxy mesh using the mesh reconstruction method.

[0092] Furthermore, during the initialization of the Gaussian network, a Gaussian is embedded in the centroid of each face of the proxy mesh, the vertex coordinates of the Gaussian attachment patch are recorded, the minor axis of the Gaussian ellipsoid is parallel to the mesh face normal, and the remaining parameters are randomly initialized to achieve the initialization of the Gaussian network.

[0093] After initialization, the patch splitting strategy and regular constraints are further used to train and optimize the Gaussian network representation.

[0094] Specifically, during the optimization iteration process of the Gaussian network at regular intervals, a regular loss term constrained by the size of the mesh patch is introduced into the loss function, and the Gaussian shape is adjusted according to the size of the attached mesh patch to ensure that the appropriate Gaussian shape is learned and local continuity is maintained during the deformation process. In some embodiments, the regular loss term adjusts the Gaussian shape according to the radius of the circumscribed circle of the triangular mesh patch to which the Gaussian is attached, and the regular loss term is expressed as:

[0095] L r =∑max(max(s i )-γR i ,0)

[0096] Among them, Lr represents the regular loss term, s i Represents the three-dimensional scale vector of each Gaussian distribution, R i It represents the radius of the circumscribed circle of the triangular mesh patch where the Gaussian distribution is located. γ is a hyperparameter used to control the influence of adjacent triangular mesh patches on the size of the Gaussian distribution.

[0097] In addition, in some embodiments, during the iteration process after the Gaussian mesh is optimized every certain number of times, a patch splitting strategy is further adopted to identify Gaussians that do not meet the preset rendering conditions, split them according to the attached mesh patches, and update the newly split mesh patches to the proxy mesh. In one embodiment, for the Gaussian that does not meet the preset rendering conditions, a new vertex is inserted into the midpoint of each side of the triangular mesh patch to which the Gaussian is attached, and a single triangular mesh patch is split into four small triangular mesh patches on the surface; the Gaussian is split into four parts, and each part belongs to a small triangular mesh patch.

[0098] In addition, as mentioned above, if the edited object is identified as a scene, a hierarchical representation based on a Gaussian net is used to decouple the scene into an object part and a background part, wherein the object part is represented by a Gaussian net and the background part is represented by a Gaussian splash. In this case, in the process of training and optimizing the Gaussian net, it is necessary to use the Gaussian splash and the Gaussian net representation to optimize the object with the Gaussian net, and use the Gaussian net as a priori to optimize the Gaussian splash representation of the background. That is, firstly, the multi-view image and the mask extracted from the multi-view image are used as supervision, and the Gaussian net method mentioned above in step S2 is used for training and optimization to obtain a first Gaussian net. Then, using the multi-view image as supervision, the first Gaussian net is spliced ​​as a fixed parameter with the first Gaussian field to be optimized, and the Gaussian splash method is used for training and optimization. In the process of optimizing the first Gaussian field, every preset number of rounds, the Gaussian in the first Gaussian field whose spacing distance from the first Gaussian net exceeds the preset distance is deleted. Through this process, the hierarchical representation of the Gaussian net is optimized.

[0099] In this embodiment, through the above steps, the optimization training of the Gaussian network is achieved, so that the user can further drive and deform the Gaussian on a large scale.

[0100] Step S3: Perform a deformation operation on the proxy mesh to drive the optimized Gaussian mesh to deform.

[0101] In this embodiment, first, the proxy mesh is deformed, and the relative displacement of the mesh face vertices before and after the deformation and the corresponding deformation gradient are extracted to obtain mesh deformation information. Then, the mesh deformation information is transmitted to the Gaussians attached to the mesh face, and each Gaussian of the Gaussian mesh is driven to deform using the mesh deformation information to obtain the deformed Gaussian mesh.

[0102] Specifically, during the deformation operation of the proxy mesh, after the user drags and deforms the proxy mesh, the deformation energy is optimized according to the proxy mesh before and after the deformation, and a transformation matrix is ​​determined by minimizing the deformation energy function to control the deformation of the proxy mesh. The deformation is represented by minimizing the deformation energy function, which ensures the rigidity of the mesh and prevents local deformation. In some embodiments, the deformation energy function is represented as:

[0103]

[0104] Where N(i) represents the set of 1-ring adjacent vertices of vertex i, v i is the spatial coordinate of the i-th vertex on the mesh patch M before deformation, v i ' is the spatial coordinate of the corresponding vertex on the deformed mesh patch M' corresponding to M, w ij is the cotangent weight, w ij =cotα ij +cotβ ij , where α ij and β ij is the angle in the adjacent triangle opposite to the edge (i, j) on the mesh patch M, and Ti is the transformation matrix.

[0105] In a deformed mesh patch M', each vertex is associated with a transformation matrix, which represents the local transformation of the mesh patch at the vertex before and after deformation. Through the polar decomposition of linear algebra, the transformation matrix can be decomposed into a rotation matrix and an affine matrix. And the high-dimensional Gaussian distribution remains unchanged after rotation and affine transformations. Therefore, the rotation matrix and affine matrix can be simply transferred from the mesh to the Gaussian kernel to change its covariance matrix, and the relative displacement of the mesh patch vertices before and after deformation can be used to change the center position of the Gaussian kernel to complete the deformation of the Gaussian field.

[0106] In one embodiment, the relative displacement ΔP and deformation gradient T of each Gaussian are obtained by barycentric coordinate interpolation. Specifically:

[0107] The relative displacement is determined by using the centroid coordinates of the mesh patch before deformation and the vertex coordinates of the mesh patch before and after deformation; the relative displacement is expressed as:

[0108] ΔP=w a (va '-v a )+w b (v b '-v b )+w c (v c '-v c )

[0109] Among them, ΔP is the relative displacement, and the centroid coordinates of the mesh patch before deformation are (w a , w b , w c ), the vertex coordinates of the mesh patch before deformation are (v a , v b , v c ), the vertex coordinates of the deformed mesh patch are (v a ',v b ',v c ').

[0110] The rotation component is determined using the centroid coordinates of the mesh patch before deformation and the rotation matrix. The rotation component is expressed as:

[0111] logR=w a logR a +w b logR b +w c logR c

[0112] Among them, R is the rotation component, and the rotation matrix is ​​(R a , R b , R c ).

[0113] The affine component is determined using the centroid coordinates of the mesh patch before deformation and the affine matrix. The affine component is expressed as:

[0114] S=w a S a +w b S b +w c S c

[0115] Among them, S is the affine component, and the affine matrix is ​​(S a , S b , S c ).

[0116] Finally, the deformation gradient T=RS can be determined according to the product of the rotation component and the affine component.

[0117] Next, the above mesh deformation information (including relative displacement and deformation gradient) is used to further drive deformation of each Gaussian in the Gaussian network. Specifically, the spatial coordinates, covariance matrix, and input perspective of the spherical harmonic function of the deformed Gaussian are transformed by relative displacement and deformation gradient, while the other properties remain unchanged, thus achieving driven deformation of each Gaussian.

[0118] Specifically, using the relative displacement, the spatial coordinates of the deformed Gaussian are determined as:

[0119] μ'=μ+ΔP

[0120] Among them, μ' is the spatial coordinate before deformation, and μ is the spatial coordinate after deformation.

[0121] Using the deformation gradient, the covariance matrix Σ' corresponding to the deformed Gaussian is determined as:

[0122] Σ'=TΣT T

[0123] Where T is the deformation gradient.

[0124] Using the rotation component, the spherical harmonic function SH'(d,c) of the deformed Gaussian is determined as:

[0125] SH'(d,c)=SH(R T d,c)

[0126] Among them, R is the rotation component, d and c are the parameters of the spherical harmonics.

[0127] Finally, through the above process, the process of transferring the deformation gradient and displacement on the vertices of the deformed mesh to the Gaussian mesh and then deforming the Gaussian mesh is completed.

[0128] In addition, as mentioned above, if the edited object is identified as a scene, a hierarchical representation based on a Gaussian net is used to decouple the scene into an object part and a background part, wherein the object part is represented by a Gaussian net and the background part is represented by a Gaussian splash. At the same time, in the process of training and optimizing the Gaussian net, the representations of Gaussian splash and Gaussian net are combined, the Gaussian net is used to optimize the object, and the Gaussian net is used as a priori to optimize the Gaussian splash representation of the background. In this case, after the Gaussian net-driven deformation is completed, the deformed Gaussian net is further spliced ​​with the first Gaussian field represented by the background with Gaussian splash and rendered to obtain the final deformed scene image.

[0129] The effect of the large-scale geometric deformation method based on Gaussian network representation disclosed in the above embodiment of the present invention is verified as follows.

[0130] like Figure 3As shown in the figure, the rendering results of the new perspective after different deformations of four objects are shown in the present invention. Each row of each object shows a deformation result. The first column shows the deformation process, the rendering situation under the original perspective before and after the deformation, and the second to fourth columns show the rendering results after deformation under the new perspective. It can be seen from the results shown in the figure that the method of the present invention can produce high-quality rendering results without spikes or artifacts.

[0131] like Figure 4 As shown in the figure, the deformation results of the present invention in four scenes are shown. The left side of each scene shows the rendering result before deformation, and the right side shows the rendering result after deformation. It can be seen from the results shown in the figure that the Gaussian network can arbitrarily edit the objects in the mask after expansion, and the deformation object and the background of the original scene are combined to maintain a high-realistic rendering result and there is no obvious sense of separation between the background and the foreground.

[0132] like Figure 5 As shown, the rendering results of the present invention using meshes of different resolutions as proxy meshes are shown. The first column shows the type of object to be deformed. The second, third and fourth columns are the results of meshes with different numbers of mesh faces after deformation: each column has two rows, the first row shows the proxy mesh used by the Gaussian network, and the second row shows the rendering results at a new perspective after deformation.

[0133] like Figure 6 As shown in the figure, the rendering results of the proxy mesh extracted by the present invention using different mesh reconstruction methods are shown. The first column is the original view data image and editing method; the second, third and fourth columns are the results obtained using mesh reconstruction methods based on different principles. Each column has three rows: the first row is the reconstructed proxy mesh display, and the second and third rows are the deformed rendering results under the new view. It can be seen from the results shown in the figure that the present invention can obtain high-quality rendering results after deformation through this proxy mesh regardless of the complexity and reconstruction quality of the proxy mesh, and has extremely high robustness.

[0134] like Figure 7 As shown, the effect display of the present invention on two types of objects in data-driven deformation is demonstrated. For each object, the first row shows the mesh to be deformed and the two meshes after data-driven deformation, and the second row shows the rendered image before deformation and the new perspective synthetic image after data-driven deformation, with two perspective images for each deformation. It can be seen from the graphical results that the present invention can migrate the data-driven method in the field of traditional mesh deformation to Gaussian splashing, which has extremely high application value and expansion value.

[0135] In summary, the large-scale geometric deformation method based on Gaussian mesh representation disclosed by the present invention allows users to intuitively and interactively deform the proxy mesh and then drive the three-dimensional Gaussian field to transform. The deformation of the present invention has extremely high real-time performance. The method can render high-quality deformed images in real time in synthetic or real objects and scenes, and supports mesh deformation technologies such as data-driven deformation.

[0136] In addition, it should be noted that the large-scale geometric deformation method based on Gaussian network representation disclosed in the present invention is not limited to being performed in the order of the above steps S1-S3, and the steps can be reordered, added or deleted. For example, the steps recorded in the present invention can be performed in parallel, sequentially or in different orders, as long as the expected results of the technical solution of the present invention can be achieved, and the present invention is not limited here.

[0137] In addition, corresponding to the aforementioned embodiment of the large-scale geometric deformation method based on Gaussian network representation, the present invention also provides an embodiment of a large-scale geometric deformation device based on Gaussian network representation.

[0138] See also Figure 8 , an embodiment of the present invention provides a large-scale geometric deformation device 200 based on Gaussian network representation, the device at least comprising:

[0139] The Gaussian network representation module 201 is used to embed a Gaussian on each face of the mesh of the edited object to construct a Gaussian network representation.

[0140] The Gaussian network training and optimization module is used to extract the proxy grid from the multi-view image as input, and use the proxy grid as a prior to initialize the Gaussian network, and train and optimize the Gaussian network representation.

[0141] The Gaussian network deformation driving module is used to perform deformation operations on the proxy mesh and drive the optimized Gaussian network to deform.

[0142] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of each module of the described device can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0143] According to an embodiment of the present invention, the present invention also provides a machine-readable medium, which may be a tangible medium, which may contain or store a program for use by an instruction execution system, apparatus or device or for use in conjunction with an instruction execution system, apparatus or device, and the computer program, when executed by a processor, implements the steps of the above-mentioned large-scale geometric deformation method based on Gaussian network representation.

[0144] In addition, the machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the above. More specific examples of machine-readable storage media may include electrical connections based on one or more lines, portable computer disks, hard disks, random access memories (RAM), read-only memories (ROM), erasable programmable read-only memories (EPROM or flash memory), optical fibers, portable compact disk read-only memories (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the above.

[0145] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A large-scale geometric deformation method based on Gaussian network representation, characterized in that: Include: Embed each face of the mesh of the edited object into a Gaussian to construct a Gaussian network representation; Taking multi-view images as input, extracting proxy meshes, using the proxy meshes as priors to initialize Gaussian nets, and training and optimizing the Gaussian net representation; Perform a deformation operation on the proxy mesh to drive the optimized Gaussian mesh to deform.

2. The method according to claim 1, characterized in that During the Gaussian mesh representation construction process, each Gaussian of the Gaussian mesh is constrained by an offset amplitude for limiting the moving distance of the Gaussian on the attached mesh face and a maximum distance of the Gaussian offset along the normal direction, and the spatial coordinates of the Gaussian are obtained; Among them, the spatial coordinates of the Gaussian are the product of the offset amplitude and the maximum distance offset along the normal direction and the sum of the centroid coordinates of the mesh face to which the Gaussian is attached; the maximum distance offset along the normal direction is represented by the radius of the circumscribed circle of the mesh face to which each Gaussian is attached.

3. The method according to claim 1, characterized in that Taking multi-view images as input, estimate camera parameters and obtain the image mask; The proxy mesh is reconstructed using the camera parameters and the mask.

4. The method according to claim 1, characterized in that: During the optimization iteration process of the Gaussian mesh representation, a regularized loss term constrained by the size of the mesh patch is introduced into the loss function, and the Gaussian shape is adjusted according to the size of the attached mesh patch.

5. The method according to claim 4, characterized in that The regularized loss term adjusts the Gaussian shape according to the circumscribed circle radius of the triangular mesh patch to which the Gaussian is attached. The regularized loss term is expressed as: Among them, L r represents the regular loss term, s i Represents the three-dimensional scale vector of each Gaussian distribution, R i It represents the radius of the circumscribed circle of the triangular mesh patch where the Gaussian distribution is located. γ is a hyperparameter used to control the influence of adjacent triangular mesh patches on the size of the Gaussian distribution.

6. The method according to claim 1, characterized in that During the optimization iteration process of the Gaussian mesh representation, Gaussians that do not meet the preset rendering conditions are identified, split according to the attached mesh faces, and the newly split mesh faces are updated to the proxy mesh.

7. The method according to claim 6, characterized in that For the Gaussian that does not meet the preset rendering conditions, a new vertex is inserted into the midpoint of each side of the triangular mesh patch to which the Gaussian is attached, so as to split the single triangular mesh patch into four small triangular mesh patches on the surface; The Gaussian is split into four parts, and each part belongs to a small triangular mesh patch.

8. The method according to claim 1, characterized in that: The step of performing a deformation operation on the proxy mesh to drive the optimized Gaussian mesh to deform includes: Deform the proxy mesh, extract the relative displacement of the mesh vertices before and after deformation and the corresponding deformation gradient, and obtain the mesh deformation information; The mesh deformation information is transmitted to the Gaussian attached to the mesh patch; Each Gaussian in the Gaussian network is driven to deform using the mesh deformation information to obtain a deformed Gaussian network.

9. The method according to claim 8, characterized in that By minimizing the deformation energy function, a transformation matrix is ​​determined to control the deformation of the proxy mesh; Decomposing the transformation matrix into a rotation matrix and an affine matrix; The relative displacement is determined using the coordinates of the center of gravity of the mesh surface before deformation and the coordinates of the vertices of the mesh surface before and after deformation; Determine a rotation component using the centroid coordinates of the mesh face before deformation and the rotation matrix; Determine an affine component using the centroid coordinates of the mesh face before deformation and the affine matrix; Determining the deformation gradient according to the product of the rotation component and the affine component; Among them: The deformation energy function is expressed as: Where N(i) represents the set of 1-ring adjacent vertices of vertex i, v i is the spatial coordinate of the i-th vertex on the mesh patch M before deformation, v i ' is the spatial coordinate of the corresponding vertex on the deformed mesh patch M' corresponding to M, w ij is the cotangent weight, w ij =cotα ij +cotβ ij , where α ij and β ij is the angle in the adjacent triangle opposite to the edge (i, j) on the mesh patch M, and Ti is the transformation matrix; The relative displacement is expressed as: ΔP=w a (v a '-v a )+w b (v b '-v b )+w c (v c '-v c ) Among them, ΔP is the relative displacement, and the centroid coordinates of the mesh patch before deformation are (w a , w b , w c ), the vertex coordinates of the mesh patch before deformation are (v a , v b , v c ), the vertex coordinates of the deformed mesh patch are (v a ',v b ',v c '); This rotation component is expressed as: logR=w a logR a +w b logR b +w c logR c Among them, R is the rotation component, and the rotation matrix is ​​(R a , R b , R c ); The affine component is expressed as: S=w a S a +w b S b +w c S c Among them, S is the affine component, and the affine matrix is ​​(S a , S b , S c ).

10. The method according to claim 9, characterized in that Each Gaussian of the Gaussian network is driven to deform using the mesh deformation information, including: Using this relative displacement, the spatial coordinates of the deformed Gaussian are determined as: μ'=μ+ΔP Among them, μ' is the spatial coordinate before deformation, and μ is the spatial coordinate after deformation; Using the deformation gradient, the covariance matrix corresponding to the deformed Gaussian is determined as: S'=TΣT T Where T is the deformation gradient; Using the rotation component, the spherical harmonic function of the deformed Gaussian is determined as: SH'(d,c)=SH(R T d,c) Among them, R is the rotation component, d and c are the parameters of the spherical harmonics.

11. The method according to claim 1, characterized in that If the edited object is identified as a scene, during the initial construction of the Gaussian network, the scene is decoupled into an object part and a background part, where: The object part is represented by a Gaussian net; The background portion is represented using Gaussian splattering.

12. The method according to claim 11, characterized in that If the edited object is identified as a scene, during the training and optimization process of the Gaussian network, The multi-view image and the mask extracted from the multi-view image are used as supervision, and a Gaussian network method is used for training optimization to obtain a first Gaussian network; Taking the multi-view image as supervision, the first Gaussian net is taken as a fixed parameter and spliced ​​with the first Gaussian field to be optimized, and the Gaussian splashing method is used for training optimization. In the process of optimizing the first Gaussian field, every preset number of rounds, the Gaussians in the first Gaussian field whose spacing distance from the first Gaussian net exceeds a preset distance are deleted.

13. The method according to claim 12, characterized in that If the editing object is identified as a scene, the method further comprises: The deformed Gaussian mesh is spliced ​​with the first Gaussian field represented by the background as Gaussian splash and rendered to obtain the final deformed scene image.

14. A large-scale geometric deformation device based on Gaussian network representation, characterized in that: The method according to any one of claims 1 to 13 is adopted, wherein the device at least comprises: Gaussian network representation module, used to embed a Gaussian on each face of the mesh of the edited object to construct a Gaussian network representation; A Gaussian network training and optimization module is used to extract a proxy grid from a multi-view image as input, and use the proxy grid as a prior to initialize the Gaussian network, and train and optimize the Gaussian network representation; The Gaussian network deformation driving module is used to perform deformation operations on the proxy mesh and drive the optimized Gaussian network to deform.

15. A computer-readable storage medium, characterized in that: A computer program is stored, and when the computer program is executed by a processor, the steps of the large-scale geometric deformation method based on Gaussian network representation described in any one of claims 1 to 13 are implemented.

Images