Three-dimensional mesh style transfer method based on convolutional neural network

Abstract

The application discloses a three-dimensional mesh style transfer method based on a convolutional neural network, and relates to the field of three-dimensional mesh processing.1. Select a suitable neural network as a pre-training model;2. Input a content mesh and a style mesh into the pre-trained neural network, and extract some layers as content features and style features of the three-dimensional mesh;3. Initialize a synthetic mesh as the content mesh;4. Calculate a content loss according to the content features of the content mesh and the synthetic mesh, and calculate a style loss according to the style features of the style mesh and the synthetic mesh;5. Calculate a shape regularization term according to the synthetic mesh itself;6. Update the vertex position of the synthetic mesh by using the loss functions of steps 4 and 5;7. Repeat steps 4 and 6 until a set number of iterations is reached; and obtain a final synthetic mesh after style transfer. The method is simple and effective, does not need too much complex processing, obtains a mesh model with good style transfer effect, and can help enrich mesh data content.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of three-dimensional mesh processing, and in particular to a three-dimensional mesh style transfer method based on a convolutional neural network. BACKGROUND

[0002] Three-dimensional shape representation is a basic research field in computer vision and computer graphics, and three-dimensional mesh, as one of the three-dimensional shape representations, is paid more attention due to containing more abundant model geometric information and has great application prospect. However, due to the complex topological structure and various data structures of the three-dimensional mesh, it is difficult to directly process, which makes the task of editing the existing mesh data to enrich the mesh data content more challenging.

[0003] In the field of images, style transfer of images based on deep learning is a classic research problem. The solution is to use a pre-trained network model to extract the features of the image, automatically apply the style of one image to another image, and use a neural network to modify the content image so that it is close to the style image in style. The success of image style transfer shows that deep learning is a powerful tool for artistic creation. How to extend the simple two-dimensional image style transfer method to the complex three-dimensional mesh style transfer and use the neural network-based method to learn the geometric style of the three-dimensional mesh to obtain three-dimensional mesh data with a specific style. Therefore, a pre-trained neural network that can extract mesh representation features is needed.

[0004] In recent years, many scholars have studied the application of three-dimensional mesh in neural networks for different tasks, such as shape classification, shape segmentation, shape correspondence, and shape retrieval. These methods propose basic processing units and symmetric functions on the mesh according to the complex properties of the mesh, which aggregate the global features of the mesh. The intermediate output results of these networks are also different, and the geometric style of the mesh that can be extracted is also different, which needs to be further explored. At the same time, the geometric style of the mesh is also difficult to define specifically, so it is important to select a suitable neural network as a pre-trained model. SUMMARY

[0005] The present application relates to the field of three-dimensional mesh processing, and in particular to a three-dimensional mesh style transfer method based on a convolutional neural network.

[0006] The present application comprises the following steps:

[0007] 1) Selecting a suitable neural network as a pre-training network model;

[0008] 2) Inputting a content mesh and a style mesh into a pre-training neural network, and extracting some layers in the pre-training neural network as content features and style features of the three-dimensional mesh;

[0009] 3) Initializing a synthesized mesh as the content mesh;

[0010] 4) Calculating a content loss according to the content features of the content mesh and the synthesized mesh, and calculating a style loss according to the style features of the style mesh and the synthesized mesh;

[0011] 5) Calculating a shape regularization term according to the synthesized mesh itself;

[0012] 6) Using the weighted sum of the loss functions of steps 4) to 5) to optimize and update the vertex positions of the synthesized mesh;

[0013] 7) Repeating steps 4) to 6) until a set number of iterations is reached;

[0014] 8) Obtaining the final synthesized mesh after style transfer.

[0015] In step 1), the neural network can be MeshNet, which uses the model learned in the three-dimensional shape classification task as the pre-training model for mesh style transfer; a triangular patch-based convolutional neural network is proposed, which captures and aggregates the spatial and structural features of triangular meshes by treating triangular patches as a processing unit.

[0016] In step 2), the input of a content mesh and a style mesh is the face features of three-dimensional mesh data, and the specific input face features include the center of the face, the direction vectors of the center of the face and the three vertices on the face, the normal vector of the face, and the 1-neighbor patch index of each face;

[0017] For each model, before inputting into the network, the mesh data is simplified to 2000 faces, moved to the geometric center, and normalized to a unit sphere, so that the extracted mesh content representation and style representation have comparable scales.

[0018] The extraction of some layers in the pre-training neural network as content features and style features of the three-dimensional mesh, wherein the content layer selects the result obtained by the spatial description module in the MeshNet network to extract the content features of the corresponding mesh; the style layer selects the features output by the first mesh convolution module after combining the spatial features and structural features in the MeshNet network, extracts the style features of the corresponding mesh, and makes represents the feature obtained after the l-th layer in MeshNet.

[0019] In step 3), the initialization synthesis mesh is a content mesh, which means that the vertex positions of the initialization synthesis mesh are the vertex positions of the content mesh, and the triangular face connection relationship of the initialization synthesis mesh is the triangular face connection relationship of the content mesh.

[0020] In step 4), the content loss is calculated according to the content features of the content mesh and the synthesis mesh, and the content loss function formula is as follows:

[0021]

[0022] Wherein, {lc} represents a set of layers for extracting feature representations used to calculate the content loss from MeshNet.

[0023] In step 4), the style loss is calculated according to the style features of the style mesh and the synthesis mesh, and the style loss function formula is as follows:

[0024]

[0025] Wherein, {ls} represents a set of layers for extracting feature representations used to calculate the style loss from MeshNet; the Gram matrix is a symmetric matrix, The spatial correlation of different filters can be calculated in the output of one layer; let represents the j-th column in represents the i-th column in The i-th element of the Gram matrix of is The inner product of the i-th column and the j-th column of is as follows:

[0026]

[0027] In step 5), a shape regularization term is calculated according to the synthesis mesh itself, and the formula of the shape regularization term is as follows:

[0028]

[0029] Wherein, loss edgeL is an edge length regularizer, which wants to minimize the length of the edge in the predicted mesh; loss normalC is a face normal regularizer, which calculates the consistency of each pair of adjacent face normals in the mesh, and forces the consistency of the adjacent face normals; loss laplacian is a Laplace regularizer; λ e , λ n and λ​​l respectively, are loss edgeL , loss normalC and loss laplacian corresponding weight coefficients, used to help balance the size of each shape regularization term in the smoothing mesh M process.

[0030] In step 6), the loss function weighted sum, that is, the weighted sum of the content loss function, the style loss function and the shape regularization term, is as follows:

[0031]

[0032] Wherein, λ c , λ s and λ r are the weights corresponding to the content loss term, the style loss term and the shape regularization term, respectively, used to help balance the size of the content representation, the style representation and the shape regularization term in the style transfer mesh process.

[0033] In step 6), the style transfer mesh can be calculated by using the neural network gradient back propagation according to the loss function to optimize and update the vertex position of the synthesized mesh, plus its original triangular facet connection relationship, which is the representation of the result mesh.

[0034] In step 7), the iteration number is set to 300.

[0035] In step 8), the synthesized mesh after the final style transfer is obtained, that is, the result mesh with a certain degree of geometric style of the style mesh.

[0036] Compared with the prior art, the present application has the following outstanding advantages:

[0037] 1. The style processing process of the prior art is often very complex, and the present application is simple and effective, and can obtain a mesh model with good style transfer effect without too much complex processing, which can help to enrich the mesh data content and is a good style transfer tool.

[0038] 2. The present application extends the classical two-dimensional image style transfer method to three-dimensional mesh style transfer, and uses a pre-trained MeshNet-based convolutional neural network to extract the geometric style of the three-dimensional mesh.

[0039] 3. The present application proposes to use a shape regularization term to optimize the mesh, so that the result mesh is smoother, the obtained model quality is better, and the visual effect is more beautiful, and fewer mesh faces can be used to obtain a result with good geometric style transfer quality. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1Fig. 1 is a schematic diagram of the overall architecture of the network model for three-dimensional mesh style transfer. The solid arrows represent the forward propagation, and the dashed arrows represent the backward propagation.

[0041] Figure 2 Fig. 2 is a diagram of an input content mesh model.

[0042] Figure 3 Fig. 3 is a diagram of an input style mesh model.

[0043] Figure 4 Fig. 4 is a diagram of the intermediate results in the style transfer optimization process. The top left shows the results of 10 iterations, the top right shows the results of 50 iterations, the bottom left shows the results of 100 iterations, and the bottom right shows the results of 200 iterations.

[0044] Figure 5 Fig. 5 is a diagram of the final result model obtained after processing by the style transfer method. DETAILED DESCRIPTION

[0045] To make the purpose, technical solutions and advantages of the present application clearer and more apparent, the following embodiments will be further described in conjunction with the accompanying drawings.

[0046] Before describing the present embodiment in detail, it should be noted that the three-dimensional mesh style transfer method based on convolutional neural network demonstrated in the present embodiment is an automatic method, and the user only needs to provide an input content mesh model and a style mesh model.

[0047] The present embodiment provides a three-dimensional mesh style transfer method based on convolutional neural network, which comprises the following steps:

[0048] S1. Select a suitable neural network as a pre-trained model.

[0049] The overall architecture of the network model of the present embodiment is shown in Fig. 1. The MeshNet neural network is selected, and the model learned in the three-dimensional shape classification task is used as the pre-trained model for mesh style transfer. A convolutional neural network based on triangular patches is proposed, which captures and aggregates the spatial and structural features of triangular meshes by treating triangular patches as a processing unit. Figure 1

[0050] S2. Input a content mesh and a style mesh into a pre-trained neural network, and extract certain layers as content features and style features of the three-dimensional mesh.

[0051] An input content mesh model C and a style mesh model S are shown in Fig. 2. Figures 2-3 ​The input is the face feature of the three-dimensional mesh data, and the specific input face feature includes the center of the face, the direction vector of the center of the face and three vertices on the face, the normal vector of the face, and the 1-neighborhood face index of each face.

[0052] For each model, the mesh data is pre-processed before being input into the network, which is simplified to 2000 faces, moved to the geometric center, and normalized to a unit sphere, so that the extracted mesh content representation and style representation have comparable scales.

[0053] Some layers in the MeshNet pre-training network are extracted as content features and style features of the three-dimensional mesh, where the content layer selects the result obtained by the spatial description module in the MeshNet network to extract the content features of the corresponding mesh; the style layer selects the features output by the first mesh convolution module after the combination of the spatial features and structural features in the MeshNet network to extract the style features of the corresponding mesh, and let represents the features obtained after the lth layer in MeshNet.

[0054] S3, initialize the synthesized mesh as the content mesh.

[0055] The vertex position of the initialized synthesized mesh M is the vertex position of the content mesh, and the triangular face connection relationship of the synthesized mesh is the triangular face connection relationship of the content mesh.

[0056] S4, calculate the content loss according to the content features of the content mesh and the synthesized mesh, and calculate the style loss according to the style features of the style mesh and the synthesized mesh.

[0057] The content loss is calculated according to the content features of the content mesh and the synthesized mesh, and the content loss function formula is as follows:

[0058]

[0059] Where {lc} represents the set of layers extracted from MeshNet for calculating the feature representation of the content loss.

[0060] The style loss is calculated according to the style features of the style mesh and the synthesized mesh, and the style loss function formula is as follows:

[0061]

[0062] Where {ls} represents the set of layers extracted from MeshNet for calculating the feature representation of the style loss. The Gram matrix is a symmetric matrix, The spatial correlation of different filters can be calculated in the output of a layer. Let represents the jth column in , denotes the i-th column of The i,j-th element of the Gram matrix of The i,j-th element of the Gram matrix of

[0063]

[0064] S5, calculate a shape regularization term according to the synthesized mesh itself.

[0065] The formula of the shape regularization term is as follows:

[0066]

[0067] Where, loss edgeL is an edge length regularizer, which wants to minimize the length of edges in the predicted mesh; loss normalC is a face normal regularizer, which calculates the consistency of each pair of adjacent face normals in the mesh, and forces the consistency of adjacent face normals; loss laplacian is a Laplace regularization term. λ e , λ n and λ l are the weight coefficients corresponding to loss edgeL , loss normalC and loss laplacian , which help to balance the size of each shape regularization term in the smoothing mesh M process.

[0068] S6, the style transfer network optimizes the vertex positions of the synthesized mesh by using the weighted sum of the loss functions of S4-S5.

[0069] The weighted sum of the loss function, that is, the weighted sum of the above content loss function, style loss function and shape regularization term, is as follows:

[0070]

[0071] Where, λ c , λ s and λ r are the weights corresponding to the content loss term, the style loss term and the shape regularization term, which help to balance the size of the content representation, the style representation and the shape regularization term in the style transfer mesh process.

[0072] The style transfer network will use neural network gradient back propagation to constantly optimize and update the vertex positions of the synthesized mesh according to the loss function, plus its original triangular face connection relationship, which is the desired result mesh representation of the present application.

[0073] ​S7, repeating steps S4-S6 until the set number of iterations is reached.

[0074] The number of iterations set in the embodiment is 300. After reaching the set number of iterations, the result grid obtained at this time is the synthesized result grid. The intermediate results in the style transfer optimization process are shown in Figure 4 , which shows the intermediate results of 10 iterations, 50 iterations, 100 iterations, and 200 iterations.

[0075] S8, obtaining the final synthesized grid after style transfer.

[0076] As shown in Figure 5 , the final synthesized grid after style transfer is obtained, that is, the result grid with a certain degree of geometric style of the style grid.

Claims

1. A 3D mesh style transfer method based on convolutional neural networks, characterized in that... Includes the following steps: 1) Select a neural network as the pre-trained network model; The neural network is MeshNet, which uses the model learned from the 3D shape classification task as a pre-trained network model for mesh style transfer; a convolutional neural network based on triangular facets is proposed, which captures and aggregates the spatial and structural features of triangular meshes by treating triangular facets as processing units. 2) Input a content grid and a style grid into a pre-trained neural network, and extract certain layers in the pre-trained neural network as the content feature and style feature representation of the 3D grid; 3) Initialize the composite mesh, which means that the vertex positions of the composite mesh are the same as the vertex positions of the content mesh, and the triangle facet connection relationships of the composite mesh are the same as the triangle facet connection relationships of the content mesh; 4) Calculate the content loss based on the content features of the content grid and the composite grid, and calculate the style loss based on the style features of the style grid and the composite grid; 5) Calculate a shape regularization term based on the synthesized mesh itself. The formula for the shape regularization term is as follows: in, This is an edge length regularizer that minimizes the length of edges in the predicted grid. For surface normal regularizers, calculate the consistency of normals for each pair of adjacent surfaces in the mesh and force the consistency of normals between adjacent surfaces; It is a Laplace regularizer; , and They are respectively , and The corresponding weighting coefficients are used to help balance the size of each shape regularization term in the smoothing mesh M process; 6) The style transfer mesh uses a weighted sum of loss functions to optimize and update the vertex positions of the synthesized mesh; The weighted sum of the loss functions is the weighted sum of the content loss function, style loss function, and shape regularization term, and its formula is shown below: in, , and These are the weights corresponding to the content loss term, style loss term, and shape regularization term, respectively, which help balance the size of the content representation, style representation, and shape regularization term in the style transfer mesh process; The style transfer mesh is backpropagated using neural network gradient calculation based on the loss function to optimize and update the vertex positions of the synthesized mesh; 7) Repeat steps 4) to 6) until the set number of iterations is reached; 8) Obtain the final style-transferred composite mesh.

2. The 3D mesh style transfer method based on convolutional neural networks according to claim 1, characterized in that... In step 2), the input is a content mesh and a style mesh. The input is the surface features of the 3D mesh data. Specifically, the input surface features include the center of the surface, the direction vectors of the center of the surface and the three vertices on the surface, the normal vector of the surface, and the 1-neighborhood patch index of each surface. For the model, before inputting it into the network, the mesh data is simplified to 2000 faces, moved to the geometric center, and normalized to a unit sphere, so that the extracted mesh content representation and style representation have a comparable scale.

3. The 3D mesh style transfer method based on convolutional neural networks according to claim 1, characterized in that... In step 2), certain layers in the pre-trained neural network are extracted as content and style feature representations of the 3D mesh, wherein the content layer selects the results obtained from the spatial description module in the MeshNet network to extract the content features of the corresponding mesh; The style layer selects spatial and structural features from the MeshNet network, combines them, and then passes them through the first grid convolutional module to extract the style features of the corresponding grid. This represents the features obtained after passing through the l-th layer of MeshNet.

4. The 3D mesh style transfer method based on convolutional neural networks according to claim 1, characterized in that... In step 4), the content loss is calculated based on the content features of the content grid and the composite grid. The content loss function formula is as follows: Here, {lc} represents the set of layers from which feature representations for calculating content loss are extracted from MeshNet.

Images