3D mesh reconstruction method and system based on depth 3D deformable model

Abstract

The invention provides a 3D mesh reconstruction method and system based on a depth 3D deformable model, and relates to the technical field of 3D mesh reconstruction. Comprising the following steps: inputting 3D grid data into an encoder of a depth 3D deformable model, and encoding the 3D grid data by utilizing leapfrogging part attention spiral convolution to obtain a feature vector; inputting the feature vectors into a decoder of a depth 3D deformable model, respectively performing attention-based neighborhood selection and aggregation weight decoupling non-local spatial feature aggregation, performing spiral neighborhood-based spatial feature aggregation, and performing spatial feature aggregation again based on multi-head attention to obtain an aggregation result; and inputting an aggregation result into the partial attention spiral convolution to obtain a reconstructed 3D grid. According to the method, the lightweight of the model is realized by using the leapfrog spiral convolution, the proposed feature aggregation method only learns the weight aggregation matrix in the training, and no extra cost is needed in the reasoning stage.

Description

Technical Field

[0001] The present invention belongs to the technical field of 3D mesh reconstruction, and in particular to a 3D mesh reconstruction method and system based on a deep 3D deformable model. Background Art

[0002] The success of deep learning models in 2D vision has sparked great interest in extending their application to 3D vision. Deep 3D deformable models apply deep learning to 3D mesh data and have been successfully applied to tasks such as 3D mesh reconstruction. 3D mesh reconstruction can extract features of complex 3D mesh data in the real world and perform feature compression. It is a basic 3D vision task for achieving upstream tasks such as 3D mesh data classification, deformation, and generation. Therefore, the study of using deep 3D deformable models for 3DZ mesh reconstruction has certain academic and practical value.

[0003] The inventors found that due to the irregular connection relationship of 3D grid data, a method of extending convolution to 3D grids uses 3D voxels or 2D views to convert the original data, and then uses conventional convolution and pooling to operate. This method inevitably loses information and affects the accuracy of the model.

[0004] Another type of method directly designs convolution and pooling operators for 3D grid data and achieves the best results available. A key issue with this type of method is how to aggregate features. Due to the irregular sampling and connection in 3D grids, the pooling operator with local sharing and translation invariance properties in neural networks cannot be directly extended to grid convolution. To solve this problem, the use of learnable feature aggregation is currently the most advanced method, but the problem with this method is that the conjugation of neighborhood selection and aggregation weight selection for feature aggregation affects its performance. Summary of the invention

[0005] In order to overcome the shortcomings of the above-mentioned prior art, the present invention provides a 3D mesh reconstruction method and system based on a deep 3D deformable model, proposes a non-local spatial feature aggregation algorithm based on attention neighborhood selection and aggregation weight decoupling, and a local spatial feature aggregation algorithm based on spiral neighborhood, and the two can be combined to realize feature aggregation of multi-head attention. The feature aggregation method only learns the weight aggregation matrix during training and does not require additional cost in the reasoning stage. In addition, for the convolution operation, the proposed skipped spiral convolution and partial attention grid convolution realize the lightweight of the model.

[0006] To achieve the above objectives, one or more embodiments of the present invention provide the following technical solutions:

[0007] A first aspect of the present invention provides a 3D mesh reconstruction method based on a deep 3D deformable model.

[0008] The 3D mesh reconstruction method based on the deep 3D deformable model includes the following steps:

[0009] Get 3D mesh data;

[0010] Input the 3D mesh data into the encoder of the deep 3D deformable model, use multiple skip-step partial attention spiral convolutions to extract features and downsample the 3D mesh data to obtain the feature vector Z;

[0011] The feature vector Z is input into the decoder of the deep 3D deformable model, and non-local spatial feature aggregation based on attention neighborhood selection and aggregation weight decoupling, and spatial feature aggregation based on spiral neighborhood are performed respectively. The results of non-local spatial feature aggregation and spatial feature aggregation are aggregated again based on multi-head attention to finally obtain the aggregation result; the aggregation result is input into the partial attention spiral convolution; the above two processes are iterated multiple times to obtain the reconstructed 3D mesh.

[0012] A second aspect of the present invention provides a 3D mesh reconstruction system based on a deep 3D deformable model.

[0013] 3D mesh reconstruction system based on deep 3D deformable model, including:

[0014] The mesh data acquisition module is configured to: acquire 3D mesh data;

[0015] The encoding module is configured to: input the 3D mesh data into an encoder of the deep 3D deformable model, perform feature extraction and down-sampling on the 3D mesh data using multiple skip-step partial attention spiral convolutions to obtain a feature vector Z;

[0016] The decoding module is configured to: input the feature vector Z into the decoder of the deep 3D deformable model, perform non-local spatial feature aggregation based on attention neighborhood selection and aggregation weight decoupling, and perform spatial feature aggregation based on spiral neighborhood, and perform spatial feature aggregation again based on multi-head attention on the results of non-local spatial feature aggregation and spatial feature aggregation; input the aggregation result into the partial attention spiral convolution; iterate the above two processes multiple times to obtain the reconstructed 3D mesh.

[0017] A third aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the 3D mesh reconstruction method based on a deep 3D deformable model as described in the first aspect of the present invention.

[0018] The fourth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the 3D mesh reconstruction method based on the deep 3D deformable model as described in the first aspect of the present invention are implemented.

[0019] One or more of the above technical solutions have the following beneficial effects:

[0020] The present invention provides a 3D mesh reconstruction method and system based on a deep 3D deformable model. For the pooling operation, a non-local spatial feature aggregation algorithm based on attention neighborhood selection and aggregation weight decoupling and a local spatial feature aggregation algorithm based on spiral neighborhood are proposed. The two can be combined to realize feature aggregation of multi-head attention. The feature aggregation method only learns the weight aggregation matrix during training and does not require additional cost in the inference stage.

[0021] For the convolution operation, the present invention proposes skipped spiral convolution and partial attention spiral convolution, which greatly reduce the number of model parameters and calculation amount, and achieve lightweight model.

[0022] The model structure proposed in the present invention adopts operators that are partially decoupled in space and channels and an attention mechanism to improve the performance of the model. The proposed pooling and convolution modules can be individually embedded in various deep 3D deformable convolutions as a direct replacement.

[0023] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0025] Figure 1 This is a flow chart of the method of the first embodiment.

[0026] Figure 2 An example diagram of 3D mesh data with the same topological structure input for the first embodiment.

[0027] Figure 3 This is an illustration of the skipped spiral convolution of the first embodiment.

[0028] Figure 4 Partial attention grid convolution data processing flow chart for the first embodiment.

[0029] Figure 5 This is a flow chart of the non-local spatial feature aggregation method with decoupling of neighborhood selection and aggregation weight in the first embodiment.

[0030] Figure 6 This is an illustration of the neighborhood selection of spiral aggregation weights in the first embodiment.

[0031] Figure 7 This is a flow chart of the local spatial feature aggregation method based on spiral neighborhood in the first embodiment.

[0032] Figure 8 Flowchart of the method for generating supersampling frames for 3D deformable animations.

[0033] Fig. 9 Schematic diagram for supersampling frames of 3D deformation animation.

[0034] Fig.10 Comparison of reconstruction errors of different spatial feature aggregation algorithms.

[0035] Fig.11 The line graph shows the reconstruction error comparison of different spatial feature aggregation algorithms on the COMA dataset.

[0036] Fig.12 A line chart comparing the reconstruction errors of different spatial feature aggregation algorithms on the D-FAUST dataset.

[0037] Fig.13 The following is a line graph comparing the reconstruction errors of different spatial feature aggregation algorithms on the Freihand dataset. DETAILED DESCRIPTION

[0038] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0039] It should be noted that the terms used herein are for describing specific embodiments only and are not intended to be limiting of exemplary embodiments according to the present invention.

[0040] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0041] Terminology explanation:

[0042] Deep 3D deformable model: Through deep learning methods, the 3D mesh data is encoded into a low-dimensional feature vector through the encoder, and then reconstructed into the original 3D mesh data through the decoder. Real-time deformation of the 3D mesh data can be achieved by operating the low-dimensional feature vector.

[0043] 3D mesh reconstruction: After the deep 3D deformable model, the output 3D mesh data is made as similar as possible to the original input 3D mesh data.

[0044] Attention mechanism: In deep learning, features are aggregated according to importance weights through learnable queries, keys, and values.

[0045] Spiral convolution: The neighborhood of each point in the grid is selected through a spiral line starting from each point, and feature aggregation operations are performed on all points using shared weights.

[0046] Cross entropy: A measure of the difference between two probability distributions.

[0047] Encoder: Encode the original features into a low-dimensional feature vector through a deep neural network.

[0048] Decoder: The low-dimensional feature vector is restored to the original feature through a deep neural network.

[0049] Mesh simplification: An algorithm that reduces the total number of points in 3D mesh data while keeping the original features of the mesh data as much as possible.

[0050] Embodiment 1

[0051] This embodiment discloses a 3D mesh reconstruction method based on a deep 3D deformable model, wherein the deep 3D deformable model mainly includes:

[0052] Skip-step spiral convolution module: Use skip-step spiral convolution to replace spiral convolution and downsampling.

[0053] Partially Attention Spiral Convolution Module: Improve spiral convolution using partially attention spiral convolution.

[0054] A non-local spatial feature aggregation algorithm based on attention-based neighborhood selection and aggregation weight decoupling: Based on the attention method, high-performance non-local spatial feature aggregation is achieved by decoupling neighborhood selection and aggregation weight.

[0055] Local spatial feature aggregation algorithm based on spiral neighborhood: Use spiral neighborhood to select feature aggregation neighborhood, and learn aggregation weights to achieve efficient local spatial feature aggregation.

[0056] Multi-head attention spatial feature aggregation algorithm: The non-local spatial feature aggregation algorithm that decouples attention-based neighborhood selection and aggregation weight and the local spatial feature aggregation algorithm based on spiral neighborhood are fused using the multi-head attention mechanism to jointly perform spatial feature aggregation.

[0057] like Figure 1 As shown, the 3D mesh reconstruction method based on the deep 3D deformable model includes the following steps:

[0058] A 3D mesh reconstruction method based on a deep 3D deformable model, characterized in that it comprises the following steps:

[0059] Get 3D mesh data;

[0060] Input the 3D mesh data into the encoder of the deep 3D deformable model, use multiple skip-step partial attention spiral convolutions to extract features and downsample the 3D mesh data to obtain the feature vector Z;

[0061] The feature vector Z is input into the decoder of the deep 3D deformable model, and non-local spatial feature aggregation based on attention neighborhood selection and aggregation weight decoupling, and spatial feature aggregation based on spiral neighborhood are performed respectively. The results of non-local spatial feature aggregation and spatial feature aggregation are aggregated again based on multi-head attention to finally obtain the aggregation result; the aggregation result is input into the partial attention spiral convolution; the above two processes are iterated multiple times to obtain the reconstructed 3D mesh.

[0062] This embodiment adopts the deep 3D deformable model of the above structure, which has the following technical advantages:

[0063] Skip-step spiral convolution can reduce the computational complexity of the encoder part to 1 / K of the original when the grid simplification factor is K.

[0064] Partial attention spiral convolution can significantly reduce the amount of computation and parameters of the model, and has little impact on the model accuracy and data communication volume.

[0065] A non-local spatial feature aggregation algorithm based on attention-based neighborhood selection and aggregation weight decoupling outperforms previous learnable feature aggregation and fixed-weight feature aggregation methods in both cases using spiral convolution and spectral graph convolution.

[0066] On the 3D reconstruction task, our model outperforms the previous best model with fewer parameters and less than 1 / 5 of the computational effort.

[0067] In some embodiments, the 3D mesh data is sequentially encoded using multiple skip-step partial attention spiral convolutions to obtain a feature vector Z, specifically including:

[0068] First, in the grid before encoder downsampling, take the sampled points for spiral convolution and perform skipped spiral convolution;

[0069] In the skip-step spiral convolution, only part of the channels are convolved to obtain the features after partial convolution; then the channel attention is calculated through the shrink-expansion operation and multiplied with the feature points after partial convolution, and finally a linear layer is used to aggregate the features between channels to complete the partial attention spiral convolution and obtain the feature vector Z;

[0070] In the partial attention spiral convolution, when the input channel and the output channel are the same, a residual connection is added between the input and the output.

[0071] In some embodiments, performing attention-based neighborhood selection and aggregation weight decoupling non-local spatial feature aggregation specifically includes:

[0072] Set the initial values ​​of learnable features for neighborhood selection and aggregation weights for each layer of the network;

[0073] The attention matrix of the feature matrix before upsampling (downsampling) and the feature matrix after upsampling (downsampling) is calculated through a cross attention;

[0074] Take the positions of the points with the largest k values ​​from each column of the attention matrix as the neighborhood selected for each point;

[0075] The selected neighborhood is dot-multiplied with a learnable weight matrix to achieve non-local spatial feature aggregation that decouples neighborhood selection and aggregation weights.

[0076] In some embodiments, attention-based neighborhood selection and aggregation weight decoupling non-local spatial feature aggregation are performed, and the specific formula is:

[0077]

[0078] Among them, l represents the number of layers; i represents the i-th point, and j represents the j-th point; represents weight; express n l The vector composed of dot products, n l is the number of points in the lth layer; and They are the features of the i-th point in the l-th layer and the j-th point in the l+1-th layer used for feature aggregation; Represents the non-local spatial feature aggregation results of the i-th point and the j-th point.

[0079] In some embodiments, performing spatial feature aggregation based on spiral neighborhood specifically includes:

[0080] Get the sequence of counterclockwise spirals with a length of p starting from point i on the lth layer

[0081] Let X l represents the set of all points in the lth layer. If the points in the l+1th layer Helix sequence The points in X l , then add the point to the neighborhood of feature aggregation, thus obtaining a neighborhood selection with aggregation weight

[0082] The neighborhood to be selected Multiply by a weight matrix to achieve spatial feature aggregation based on spiral neighborhood.

[0083] In some embodiments, the specific formula for the spatial feature aggregation based on the spiral neighborhood is:

[0084]

[0085]

[0086] in, The spatial feature aggregation result based on spiral neighborhood; is the weight matrix.

[0087] In some embodiments, the results of non-local spatial feature aggregation and spatial feature aggregation are aggregated again based on multi-head attention, specifically:

[0088]

[0089] Among them, a l is the learnable aggregation weight.

[0090] Next, the technical solution of this embodiment will be further explained in conjunction with the accompanying drawings.

[0091] (A) Fast and accurate deep 3D deformable models based on feature aggregation method with decoupled weight learning and neighborhood selection and partial attention convolution.

[0092] Step 1, encoder;

[0093] The algorithm input is 3D mesh data with the same topological structure, see Figure 2 , where all three grids are 3D grids consisting of 5023 points and 9976 triangular facets, and the points with the same number in different grids correspond to the same position of the topological object. Figure 2 The data set is in R format. 5023×3 The feature vector of the input 3D mesh data is encoded using the encoder to obtain a low-dimensional feature vector Z. The encoder part consists of four skip-step partial attention spiral convolution modules.

[0094] (1) Skip-step spiral convolution

[0095] Due to the irregularity of 3D mesh connections, the pooling operation cannot be directly extended to 3D mesh data. Based on the existing spiral convolution algorithm, we propose a skipped partial attention spiral convolution. First, we extend the skipped convolution to 3D mesh convolution. When the simplification factor is 4, the skipped spiral convolution can reduce the total computational effort to 1 / 4 of the original spiral convolution + pooling operator, and this partial simplification will not cause a loss of accuracy. Figure 3 .

[0096] Figure 3 This is an illustration of skipped spiral convolution. The grid is the grid before the encoder downsampling, the red dots represent the points after downsampling, the red line represents the spiral line of point B1, and the neighborhood of point B1 after downsampling is selected as yellow points (B1-B9). Since the points after downsampling have a partial 1-1 correspondence with the original points, our method takes the points after downsampling (red dots) in the grid before downsampling for spiral convolution.

[0097] (2) Partial Attention Spiral Convolution

[0098] Drawing on the idea of ​​lightweight 2D convolution, we found that the channels of convolution in 3D deformable models are also highly similar and redundant. We only convolve some channels, calculate the channel attention through shrink-expansion operations and multiply it with the feature points after partial convolution, and finally perform feature aggregation between channels through a linear layer. In order to improve training stability, we will add a residual connection when the input channel and output channel are the same. Our lightweight partial attention grid convolution structure can be seen in Figure 4 .

[0099] Step 2, decoder;

[0100] After the feature vector Z is subjected to feature upsampling operation through the non-local spatial feature aggregation based on attention neighborhood selection and weight aggregation decoupling and the multi-head attention mechanism based on spiral neighborhood local spatial feature aggregation, the spatial feature aggregation is then performed through partial attention spiral convolution. The decoder consists of four groups of the above modules and an additional partial attention spiral convolution. After the decoder, the low-dimensional feature vector is decoded into reconstructed 3D mesh data.

[0101] (1) Non-local spatial feature aggregation based on attentional neighborhood selection and aggregation weight decoupling The feature aggregation method of (de)pooling of spatial graph convolution can be written as

[0102] X l+1 =M l X l (1)

[0103] Where X l+1 is the feature of the l+1th layer, M lis the mapping matrix from layer l to layer l+1. This operation can be regarded as a linear layer in a neural network, but due to its large amount of calculation, the pooling matrix Most of them are calculated by grid simplification method, the anti-pooling matrix M l It is a sparse matrix with the minimum loss calculated using barycentric coordinates. However, the upsampling weight aggregation matrix obtained by this method only has the minimum reconstruction error for the original 3D feature input of the fixed template grid data, and is not applicable to other 3D grids of the data set and features extracted by the neural network.

[0104] To this end, we propose a spatial feature aggregation algorithm that decouples neighborhood selection and weight learning to automatically select (de)pooled aggregation neighborhoods and aggregation weights for different tasks and datasets. We first set the learnable feature initial values ​​for neighborhood selection and aggregation weights for each layer of the network. The feature initial value is obtained by normalizing the 3D grid vectors corresponding to each layer of the grid simplification. Padded with zeros to a unified high dimension N≥4, and then added with Gaussian noise with a standard deviation of 0 and a variance of 0.1, where l represents the number of layers. The reason for selecting this vector as the initial value of the feature aggregation feature is that the dot product values ​​of different feature vectors will be limited to between -1 and 1 after standardization, which is beneficial to the stability of our subsequent neighborhood and weight selection decoupling method. The reason for choosing to initialize it as a high-dimensional feature is that high-dimensional features have stronger representation capabilities.

[0105] The selection of the neighborhood of our method can be viewed as the process of taking the top k positions with the largest cross-attention similarity. Let They are the features of the i-th point in the l-th layer and the j-th point in the l+1-th layer for feature aggregation. Calculate the dot product of the two As the similarity, the 0-1 binary mask matrix b is then used to take the top-K points of similarity for weighted aggregation, where

[0106]

[0107] in express n l The vector composed of dot products, n l is the number of points in the lth layer. Then, by learning a weight matrix W l Learn the aggregation weights and get the aggregation weight matrix S l The value satisfies

[0108]

[0109] The weight All values ​​in are initialized to 1 / K, so that the sum of the weights of the largest k points at the beginning of training is 1, making the training more stable. Through this method, d is used for neighborhood selection and w is used for weight aggregation, which realizes the decoupling of neighborhood selection and weight learning. Our feature aggregation method for decoupling neighborhood selection and weight learning is shown in Figure 5 .

[0110] Figure 5 :A feature aggregation method that decouples neighborhood selection and weight learning. First, a cross-attention method is used to calculate the feature matrix K before upsampling. l+1 After upsampling, Q l The attention matrix of the feature matrix obtains the largest k values ​​through a 0-1 binary mask matrix b, and then multiplies it with a learnable weight matrix W, thereby realizing feature aggregation that decouples neighborhood selection and weight learning.

[0111] (2) Local spatial feature aggregation algorithm based on spiral neighborhood

[0112] The advantage of the spatial feature aggregation algorithm based on attention-based neighborhood selection and aggregation weight decoupling is that, given the prior of the feature vector direction, the closer the point is, the better the neighborhood and weight should be. However, the disadvantage is that the attention matrix calculation has O(n 2 ) complexity. When the number of points in a 3D deformable model is large, the training speed is slow. Therefore, we propose a heuristic feature aggregation method. This method selects the aggregation neighborhood through a spiral line. The priori is that the points closer to the front of the spiral line starting from any point should be the neighbors of the point. By selecting a fixed neighborhood, the model only needs to learn the aggregation weights of the top-K neighbors. The amount of weight aggregation calculation is reduced to O(kn), the training speed is fast, and for the grid reconstruction task, it has a performance close to that of the spatial feature aggregation algorithm that decouples neighborhood selection and weight learning.

[0113] make is a sequence of counterclockwise spirals of length p starting from point i on the lth layer, denoted by

[0114]

[0115] Where k-ring represents The k-ring neighborhood of for The traversal order of each ring neighborhood is obtained by the spiral. Let X l Represents the set of all points in the lth layer. For the points in the l+1th layer Helix sequence If the point in l , then add the point to the neighborhood of its feature aggregation, and you can get a neighborhood selection with aggregation weight in

[0116]

[0117] Neighborhood selection for spiral aggregation weights is implemented in Figure 6 .

[0118] Figure 6 : Neighborhood selection of spiral aggregation weight. The grid is the grid after upsampling, the red point is any point in the grid after upsampling, the blue point represents the point before upsampling, and the red line represents the selection of the spiral line. As shown in the figure, the first 6 pre-upsampled points (A1-A6) that the upsampled point x passes through after the counterclockwise spiral line traverses will be used as the first 6 neighborhoods of its feature aggregation.

[0119] Our feature aggregation method based on the local spatial feature aggregation algorithm of the spiral neighborhood can be written as

[0120]

[0121] What is different from the previous method is that is a non-learnable mask that allows feature aggregation to aggregate only K neighborhoods. Demonstration of local spatial feature aggregation algorithm based on spiral neighborhood Figure 7 .

[0122]

[0123] (3) Multi-head attention spatial feature aggregation

[0124] Combining our attention-based neighborhood selection and aggregation weight decoupling spatial feature aggregation algorithm with the rotation neighborhood-based local spatial feature aggregation algorithm, we can obtain our multi-head attention spatial feature aggregation algorithm.

[0125]

[0126] where a l is a learnable aggregation weight that determines the importance ratio of the two algorithms. This method is equivalent to a multi-head attention mechanism that integrates the advantages of (1) the non-local spatial feature aggregation algorithm based on attention-based neighborhood selection and aggregation weight decoupling and (2) the local spatial feature aggregation algorithm based on spiral neighborhood in non-local feature aggregation and local feature extraction.

[0127] (II) 3D deformation animation frame super sampling generation

[0128] Based on the 3D mesh reconstruction method based on the deep 3D deformable model of this embodiment, the application of the method in the 3D deformation animation frame super sampling generation method is provided below. Specifically, the following steps are included:

[0129] Get the start frame and end frame that need to be supersampled;

[0130] Input the start frame and the end frame into the encoder of the deep 3D deformable model respectively to obtain the feature vectors of the start frame and the end frame;

[0131] Linearly interpolate the feature vectors of the start frame and the end frame to obtain multiple intermediate feature vectors;

[0132] The feature vectors of the start frame and the end frame, and multiple intermediate feature vectors are respectively input into the decoder of the deep 3D deformable model to obtain the corresponding reconstructed start frame, the reconstructed end frame and multiple reconstructed intermediate frames.

[0133] The specific workflow of this application is as follows:

[0134] Using the 3D mesh data with fixed topology as a training set, the model in the above embodiment is trained;

[0135] Mesh Reconstruction: Run the model on 3D mesh data of the same topology to reconstruct the mesh.

[0136] 3D deformation animation frame super-sampling generation: Run the model encoder part of the present invention on the start frame and the end frame that need to be super-sampled to obtain the feature vectors of the start frame and the end frame, perform linear interpolation on the feature vectors of the two, and then run the model decoder part of the present invention to achieve super-sampling generation of 3D deformation animation frames.

[0137] like Figure 8 and Fig. 9 As shown, the model in this embodiment is applied to the super-sampling generation of 3D deformation animation frames. To solve the problem of insufficient sampling frames that may exist when using a motion capture system to create 3D deformation animation, our model is used to use an encoder to calculate the feature vectors Z1 and Z2 of the two frames before and after the interpolation. n The frame that needs to be interpolated uses linear interpolation to calculate its feature vector Z i , the formula is

[0138] Z i =αZ1+(1-α)Z N (9)

[0139] If it is assumed that the number of interpolated frames between the previous and next frames is m, then α=i / m, i=1, 2,…, m-1.

[0140] The result of supersampling 4 frames between the previous and next frames is shown in Fig. 9 The leftmost frame is the start frame, and the rightmost frame is the end frame. It can be seen that even when there is a huge difference between the start frame and the end frame, our method can still achieve smooth supersampling.

[0141] (III) Experiment

[0142] ①Dataset and training settings:

[0143] COMA [Generating 3D faces using Convolutional Mesh Autoencoders] is a dataset containing 20K+ registered face shapes. It contains 12 different expression sequences of 12 people, and each 3D mesh shape contains 5023 vertices. We randomly select 90% of the data, a total of 18415 samples, as the training set, and 10% of the data, a total of 2050 samples, as the test set.

[0144] The DFAUST [Dynamic FAUST: registering human bodies in motion] dataset is a dataset containing 40K+ human shapes. It has more than 100 motion sequences, covering 10 different identities, and each human body's 3D mesh contains 6890 vertices. In the 3D reconstruction task, with the same setting as [Neural 3D Morphable Models: Spiral Convolutional Networks for 3D Shape Representation Learning and Generation], we randomly split the dataset into 34,5K for training, 500 for validation, and 5000 for testing.

[0145] FreiHAND [FreiHAND: a dataset for markerless capture of hand pose and shape from single RGB images] is a dataset containing 32,560 training samples and 3960 test samples for hand shape and pose estimation, where each 3D mesh shape contains 778 vertices. On the 3D reconstruction task, we randomly select 1000 shapes from the training samples as our validation set, the other 31,560 samples are our training set, and 3960 test samples are our test set.

[0146] We use the Adam optimization algorithm for training, with the learning rate set to 0.001, the learning rate decay set to 0.99, and the batch size set to 16. We train for 300 epochs and train and test all the codes on a single NVIDIA RTX 4090.

[0147] ②3D reconstruction:

[0148] We conduct experiments on the coma, d-faust and freihand datasets to illustrate the effectiveness of our feature aggregation algorithm in 3D reconstruction tasks. The number of neighbors of our spiral convolution is set to 9, the dilation coefficient is set to 1, the encoder output channels in our encoder-decoder structure are [32,32,32,64], the encoder output channels are [64,32,32,32,3], and the e lu activation function is used. The following models only perform weight aggregation in the decoder. We select the neighborhood size of weight aggregation from 4 to 24, w is set to 0.2, and the number of aggregated neighbors of the two heads is 1 / 2 of the total number of aggregated neighbors. Our algorithm is compared with the most advanced method on the coma dataset. Our algorithm selects 1 / 4 of the channels for partial convolution, the coefficient of channel attention compression is set to 1.2, and the learnable neighborhood selection in the decoder is 8. It can be seen that our method is superior to the existing reconstruction methods in terms of both parameter quantity and computational complexity. Our multi-head attention decoupled feature aggregation and lightweight convolution operator play a role. Compared with the most advanced adaptive spirral algorithm, it achieves better results with fewer parameters and less than 1 / 5 of the computational complexity. The results are shown in Table 1.

[0149] Table 1: Experimental results of 3D reconstruction on COMA dataset.

[0150] Model Error Mean and Mean Median Error Total parameters Total calculation amount Spiralnet++ 0.423+0.534mm 0.236mm 175.2k 92.3m Neural3DMM 0.431+0.531mm 0.245mm 177.4k 92.5m adaptivespiral 0.389+0.521mm 0.214mm 194.8k 116.0m Our 0.384+0.512mm 0.201mm 150.5k 20.7m

[0151] The number of parameters and the amount of computation in Table 1 are divided into the number of parameters and the amount of computation for convolution and spatial feature aggregation.

[0152] This embodiment can be used for 3D mesh reconstruction of various fixed topology 3D data. Fig.10 This is a comparison of the quantization errors between the graph data spatial feature aggregation method proposed in this patent and the LFA [Learning Feature Aggregation for Deep 3D Morphing Modes] method. The neighborhoods for weighted aggregation are all 20. DLFA, SLFA, and MALFA refer to the (1) non-local spatial feature aggregation based on attention-based neighborhood selection and aggregation weight decoupling mentioned in the decoder of step 2, (2) local spatial feature aggregation algorithm based on spiral neighborhood, and (3) multi-head attention spatial feature aggregation.

[0153] Fig.10 The comparison of reconstruction errors of different spatial feature aggregation algorithms is shown, where red indicates a larger error and blue indicates a smaller error.

[0154] Figure 11-13The comparison of reconstruction errors of different spatial feature aggregation algorithms on the above different datasets is shown. The horizontal axis represents the average number of aggregated neighbors of the decoder, and the vertical axis represents the error of the algorithm.

[0155] ③Identity Identification:

[0156] The COMA dataset is selected to test the accuracy of our method on the identity recognition task. Our algorithm selects 1 / 2 of the channels for partial convolution, the channel attention compression factor is set to 2, and the cross entropy loss is used. For the identity recognition task, we select a 4-stage encoder with an output channel of [4,4,4,8] connected to a linear layer FC(12). The results of the identity recognition task are shown in Table 2. It can be seen that through skip-step spiral convolution and partial attention spiral convolution, our method achieves better results than the adaptive ivesp i ra l model with less than one-tenth of the number of parameters.

[0157] Table 2: Comparison of identity recognition algorithms

[0158] Model Recognition accuracy Parameter quantity Calculation amount Flops Spiralnet++ 95.6％ 4.3k 874.3k adaptivespiral 97.0％ 7.9k 1745.9k Our 99.0％ 2.4k 169.5k

[0159] Embodiment 2

[0160] This embodiment discloses a 3D mesh reconstruction system based on a deep 3D deformable model.

[0161] 3D mesh reconstruction system based on deep 3D deformable model, including:

[0162] The mesh data acquisition module is configured to: acquire 3D mesh data;

[0163] The encoding module is configured to: input the 3D mesh data into an encoder of the deep 3D deformable model, perform feature extraction and down-sampling on the 3D mesh data using multiple skip-step partial attention spiral convolutions to obtain a feature vector Z;

[0164] The decoding module is configured to: input the feature vector Z into the decoder of the deep 3D deformable model, perform non-local spatial feature aggregation based on attention neighborhood selection and aggregation weight decoupling, and perform spatial feature aggregation based on spiral neighborhood, and perform spatial feature aggregation again based on multi-head attention on the results of non-local spatial feature aggregation and spatial feature aggregation; input the aggregation result into the partial attention spiral convolution; iterate the above two processes multiple times to obtain the reconstructed 3D mesh.

[0165] Embodiment 3

[0166] The purpose of this embodiment is to provide a computer-readable storage medium.

[0167] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the 3D mesh reconstruction method based on a deep 3D deformable model as described in Embodiment 1 of the present disclosure.

[0168] Embodiment 4

[0169] The purpose of this embodiment is to provide an electronic device.

[0170] An electronic device includes a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, the steps in the 3D mesh reconstruction method based on a deep 3D deformable model as described in Example 1 of the present disclosure are implemented.

[0171] The steps involved in the apparatuses of the above embodiments 2, 3 and 4 correspond to the method embodiment 1, and the specific implementation methods can refer to the relevant description part of embodiment 1. The term "computer-readable storage medium" should be understood as a single medium or multiple media including one or more instruction sets; it should also be understood to include any medium that can store, encode or carry an instruction set for execution by a processor and enable the processor to execute any method in the present invention.

[0172] Those skilled in the art should understand that the modules or steps of the present invention described above can be implemented by a general-purpose computer device, or alternatively, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. The present invention is not limited to any specific combination of hardware and software.

[0173] Although the above describes the specific implementation mode of the present invention in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A 3D mesh reconstruction method based on a deep 3D deformable model, characterized in that: The following steps are involved: Get 3D mesh data; Input the 3D mesh data into the encoder of the deep 3D deformable model, use multiple skip-step partial attention spiral convolutions to extract features and downsample the 3D mesh data to obtain the feature vector Z; The feature vector Z is input into the decoder of the deep 3D deformable model, and non-local spatial feature aggregation based on attention neighborhood selection and aggregation weight decoupling, and spatial feature aggregation based on spiral neighborhood are performed respectively. The results of non-local spatial feature aggregation and spatial feature aggregation are aggregated again based on multi-head attention to finally obtain the aggregation result; the aggregation result is input into the partial attention spiral convolution; the above two processes are iterated multiple times to obtain the reconstructed 3D mesh.

2. The 3D mesh reconstruction method based on a deep 3D deformable model according to claim 1, characterized in that: The 3D grid data is encoded sequentially using multiple skip-step partial attention spiral convolutions to obtain the feature vector Z, which includes: First, in the grid before encoder downsampling, take the sampled points for spiral convolution and perform skipped spiral convolution; In the skip-step spiral convolution, only part of the channels are convolved to obtain the features after partial convolution; then the channel attention is calculated through the shrink-expansion operation and multiplied with the feature points after partial convolution, and finally a linear layer is used to aggregate the features between channels to complete the partial attention spiral convolution and obtain the feature vector Z; In the partial attention spiral convolution, when the input channel and the output channel are the same, a residual connection is added between the input and the output.

3. The 3D mesh reconstruction method based on a deep 3D deformable model according to claim 1, characterized in that: Perform attention-based neighborhood selection and aggregation weight decoupling for non-local spatial feature aggregation, including: Set the initial values ​​of learnable features for neighborhood selection and aggregation weights for each layer of the network; Calculate the attention matrix of the feature matrix before upsampling and the feature matrix after upsampling through a cross attention; Take the positions of the points with the largest k values ​​from each column of the attention matrix as the neighborhood selected for each point; The selected neighborhood is dot-multiplied with a learnable weight matrix to achieve non-local spatial feature aggregation that decouples neighborhood selection and aggregation weights.

4. The 3D mesh reconstruction method based on a deep 3D deformable model according to claim 1, characterized in that: Perform non-local spatial feature aggregation based on attention-based neighborhood selection and aggregation weight decoupling. The specific formula is: Among them, l represents the number of layers; i represents the i-th point, and j represents the j-th point; represents weight; express and The vector composed of dot products, n l is the number of points in the lth layer; and They are the features of the i-th point in the l-th layer and the j-th point in the l+1-th layer used for feature aggregation; Represents the non-local spatial feature aggregation results of the i-th point and the j-th point.

5. The 3D mesh reconstruction method based on a deep 3D deformable model according to claim 4, characterized in that: Perform spatial feature aggregation based on spiral neighborhood, including: Get the sequence of counterclockwise spirals with a length of p starting from point i on the lth layer Let X l represents the set of all points in the lth layer. If the points in the l+1th layer Helix sequence The points in X l , then add the point to the neighborhood of feature aggregation, thus obtaining a neighborhood selection with aggregation weight The neighborhood to be selected Multiply by a weight matrix to achieve spatial feature aggregation based on spiral neighborhood.

6. The 3D mesh reconstruction method based on a deep 3D deformable model according to claim 5, characterized in that: The specific formula for the spatial feature aggregation based on the spiral neighborhood is: in, The spatial feature aggregation result based on spiral neighborhood; is the weight matrix.

7. The 3D mesh reconstruction method based on a deep 3D deformable model according to claim 6, characterized in that: The results of non-local spatial feature aggregation and spatial feature aggregation are aggregated again based on multi-head attention, specifically: Among them, a l is the learnable aggregation weight.

8. A 3D mesh reconstruction system based on a deep 3D deformable model, characterized in that: include: The mesh data acquisition module is configured to: acquire 3D mesh data; The encoding module is configured to: input the 3D mesh data into an encoder of the deep 3D deformable model, perform feature extraction and down-sampling on the 3D mesh data using multiple skip-step partial attention spiral convolutions to obtain a feature vector Z; A decoding module is configured to: input the feature vector Z into a decoder of a deep 3D deformable model, perform non-local spatial feature aggregation based on attention-based neighborhood selection and aggregation weight decoupling, and perform spatial feature aggregation based on spiral neighborhood, and perform spatial feature aggregation again based on multi-head attention on the results of non-local spatial feature aggregation and spatial feature aggregation; The aggregation result is input into the partial attention spiral convolution; the above two processes are iterated multiple times to obtain the reconstructed 3D mesh.

9. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the 3D mesh reconstruction method based on a deep 3D deformable model as described in any one of claims 1 to 7 are implemented.

10. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the 3D mesh reconstruction method based on the deep 3D deformable model as described in any one of claims 1 to 7 are implemented.

Images