A 3D model feature extraction method based on mesh subdivision structure

Abstract

The present invention provides a method for extracting features of a three-dimensional model based on a grid subdivision structure, comprising: determining a grid representation of a three-dimensional model of features to be extracted; inputting the grid of the three-dimensional model into a feature extraction module, outputting a target grid representation after feature extraction, and a target feature vector of each facet in the target grid representation; wherein the grid representation is composed of a plurality of triangular facets or a plurality of quadrilateral facets, and is used to depict the three-dimensional shape of a physical object or scene, and the feature extraction module comprises a regridding module and a grid up-and-down sampling module connected in sequence. The method provided by the present invention can extract features that can better describe a wide range of features, and is applied to grid classification, segmentation, retrieval, shape correspondence, etc., and can improve accuracy and robustness.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional mesh technology, and in particular to a three-dimensional model feature extraction method based on mesh subdivision structure. Background Art

[0002] Mesh is a common way to describe three-dimensional shapes, and is widely used in modeling, rendering, animation, 3D printing, etc. Usually a mesh defines the vertices in three-dimensional space, and the connections between the vertices define the faces. The most commonly used meshes are triangular meshes and quadrilateral meshes: all faces in a triangular mesh are triangles, and all faces in a quadrilateral mesh are quadrilaterals.

[0003] Compared with traditional methods, deep neural networks have made significant progress in understanding, analyzing, and generating data formats such as images, natural language processing, and point clouds. However, due to the complexity and irregularity of grid data, the application of deep neural networks is still relatively small.

[0004] Existing 3D mesh feature extraction methods usually use machine learning to train 3D mesh models. During the model training process, the existing 3D mesh model needs to be subjected to feature extraction with different rules to obtain mesh features of different resolutions, thereby achieving local feature aggregation or global feature propagation, which is equivalent to obtaining features of different scales in 2D image processing. Figure 1 Similarly, the final overall feature vector of the image is obtained by fusing feature maps of different scales. The three-dimensional mesh model also needs to extract meshes of different resolutions, and then determine the final features of each face and the overall features of the mesh based on the mesh features of these different resolutions.

[0005] Among them, establishing a multi-resolution hierarchical representation for three-dimensional data and being able to extract features from fine to coarse and from local to global is one of the reasons why neural networks can be effectively applied to regular data such as images and sequences. However, general grids lack this regular hierarchical structure. Previous work has stored features on edges, based on grid simplification, and obtained a coarse-grained representation by continuously and dynamically deleting edges, but this method deletes details unevenly and is slow.

[0006] Therefore, how to avoid the difficulty in obtaining reasonable feature representation due to the lack of multi-resolution hierarchical representation of three-dimensional data in existing three-dimensional mesh feature extraction methods is still an urgent problem to be solved by technical personnel in this field. Summary of the invention

[0007] The present invention provides a three-dimensional model feature extraction method based on a grid subdivision structure, which is used to solve the problem that the existing three-dimensional grid feature extraction method is difficult to obtain a reasonable feature representation due to the lack of a multi-resolution hierarchical representation of three-dimensional data. By constructing a re-gridding module and a grid up-and-down sampling module connected in sequence in a feature extraction module, a new multi-resolution hierarchical method is provided. By means of surface subdivision, a subdivision structure is first added to re-grid the original grid model, and then the re-grid model is up-and-down sampled to obtain a feature vector of each facet.

[0008] The present invention provides a three-dimensional model feature extraction method based on a grid subdivision structure, comprising:

[0009] Determining a mesh representation of a three-dimensional model of features to be extracted;

[0010] Inputting the mesh of the three-dimensional model into a feature extraction module, outputting a target mesh representation after feature extraction, and a target feature vector of each facet in the target mesh representation;

[0011] The mesh representation is composed of a plurality of triangular facets or a plurality of quadrilateral facets, and is used to depict the three-dimensional shape of a physical object or scene. The feature extraction module includes a re-gridding module and a mesh up-and-down sampling module connected in sequence.

[0012] According to a three-dimensional model feature extraction method based on a grid subdivision structure provided by the present invention, the feature extraction module includes a regridding module and a grid up-and-down sampling module connected in sequence, specifically including:

[0013] The regridding module regrids the grid of the input three-dimensional model to obtain a regrid model with a subdivision structure and outputs it to the grid up- and down-sampling module;

[0014] The grid up-and-down sampling module performs several down-sampling and several up-sampling on the regrid model to obtain a target grid representation after feature extraction and a target feature vector of each facet in the target grid representation.

[0015] According to a three-dimensional model feature extraction method based on a mesh subdivision structure provided by the present invention, the mesh of the input original three-dimensional model is re-meshed to obtain a re-meshed model with a subdivision structure, which specifically includes:

[0016] Simplifying the input original mesh of the three-dimensional model to obtain a simplified mesh, and establishing a mapping between the simplified mesh and the original mesh of the three-dimensional model;

[0017] Subdividing the simplified grid according to a preset subdivision rule to obtain a grid model with a subdivision structure;

[0018] Based on the mapping, the mesh model is mapped back to the original mesh of the three-dimensional model to obtain a re-meshed model of the original mesh of the three-dimensional model with a subdivision structure.

[0019] According to a 3D model feature extraction method based on a mesh subdivision structure provided by the present invention, the re-mesh model is downsampled and upsampled several times to obtain a target mesh representation after feature extraction, and a target feature vector of each facet in the target mesh representation, specifically comprising:

[0020] The m-th sampling layer of the grid up and down sampling module samples the input m-1-th sampling layer feature vector and the m-1-th sampling layer grid model of each face patch according to a preset rule, and outputs the m-th sampling layer feature vector and the m-th sampling layer grid model of each face patch to the m+1-th sampling layer, the grid up and down sampling module outputs the L-th layer feature vector, the m-th sampling layer is an up sampling layer or a down sampling layer, and the preset rule corresponding to the m-th sampling layer is a preset up sampling rule or a preset down sampling rule;

[0021] Among them, m=1,2,…,L, L is the total number of sampling layers in the grid up and down sampling modules, the zeroth sampling layer feature vector is composed of the patch shape description and patch pose description of the re-grid model, and the zeroth sampling layer grid model is the re-grid model.

[0022] According to a 3D model feature extraction method based on a grid subdivision structure provided by the present invention, the m-th sampling layer samples the input m-1 sampling layer feature vector and the m-1 sampling layer grid model of each face patch according to a preset rule, and outputs the m-th sampling layer feature vector and the m-th sampling layer grid model of each face patch, specifically comprising:

[0023] If the mth sampling layer is an upsampling layer,

[0024] Then, each facet of the input m-th sampling layer is split into four faces according to a preset subdivision rule to obtain an m-th sampling layer grid model; based on the m-1-th sampling layer feature vector of each input facet, the m-th sampling layer feature vector of the four faces is determined according to a preset propagation rule;

[0025] If the mth sampling layer is a downsampling layer,

[0026] Then, four patches in the same subdivision structure in the grid model of the m-th sampling layer are input to obtain the m-th sampling layer grid model; and the m-th sampling layer feature vectors of the four patches are determined according to a preset fusion rule based on the m-1-th sampling layer feature vectors of each input patch.

[0027] According to a three-dimensional model feature extraction method based on a mesh subdivision structure provided by the present invention, if the mesh is a triangular mesh, the preset subdivision rule is to use a Loop subdivision rule;

[0028] If the grid is a quadrilateral grid, the preset subdivision rule is to use the Catmull-Clerk subdivision rule.

[0029] According to a three-dimensional model feature extraction method based on a grid subdivision structure provided by the present invention, the preset fusion rule is to take the average, maximum or minimum value of each dimension of the feature vectors of four facets.

[0030] According to a three-dimensional model feature extraction method based on a grid subdivision structure provided by the present invention, the preset propagation rule is nearest neighbor upsampling or bilinear upsampling.

[0031] The present invention also provides a multi-resolution deep neural network for three-dimensional grids, comprising the feature extraction module and the specific task module in any one of the above three-dimensional model feature extraction methods based on grid subdivision structure, characterized in that:

[0032] During training,

[0033] The feature extraction module extracts the sample feature vector of each facet of the input sample three-dimensional mesh model and outputs it to the specific task module;

[0034] The specific task module processes the sample feature vector according to specific rules to obtain a prediction result;

[0035] A loss function is constructed based on the prediction results and the facet labels with a preset learning objective; wherein the preset learning objective is mesh classification or triangular mesh model physical type recognition.

[0036] The present invention also provides a three-dimensional model feature extraction device based on a grid subdivision structure, comprising:

[0037] An input unit for determining a mesh representation of a three-dimensional model of features to be extracted;

[0038] An output unit, used for inputting the mesh of the three-dimensional model into a feature extraction module, and outputting a target mesh representation after feature extraction, and a target feature vector of each facet in the target mesh representation;

[0039] The mesh representation is composed of a plurality of triangular facets or a plurality of quadrilateral facets, and is used to depict the three-dimensional shape of a physical object or scene. The feature extraction module includes a re-gridding module and a mesh up-and-down sampling module connected in sequence.

[0040] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned three-dimensional model feature extraction methods based on a grid subdivision structure are implemented.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned three-dimensional model feature extraction methods based on a grid subdivision structure.

[0042] The three-dimensional model feature extraction method based on the grid subdivision structure provided by the present invention determines the grid representation of the three-dimensional model of the feature to be extracted, inputs the grid of the three-dimensional model into the feature extraction module, outputs the grid representation after feature extraction and the feature vector of each face of the grid, and the re-gridding module is used to add the subdivision structure to the grid of the original three-dimensional model by preset surface subdivision rules. Since the re-gridding module and the grid up-and-down sampling module connected in sequence are constructed in the feature extraction module, a new multi-resolution hierarchical method is provided. By means of surface subdivision, the subdivision structure is first added to re-grid the original grid model, and then the re-gridding model is up-and-down sampled to obtain different resolution features of each face, which can support traditional machine learning and deep learning, and is applied to three-dimensional model classification, segmentation, retrieval, shape correspondence, etc., and can improve the high accuracy and robustness of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0044] Figure 1 A schematic diagram of a flow chart of a three-dimensional model feature extraction method based on a grid subdivision structure provided by the present invention;

[0045] Figure 2 An example diagram of the regridding result and downsampling process provided by the present invention;

[0046] Figure 3 An example diagram of a grid of a subdivision structure provided by the present invention;

[0047] Figure 4 A schematic diagram of the up and down sampling calculation principle of the three-dimensional grid provided by the present invention;

[0048] Figure 5A schematic diagram of the structure of a three-dimensional model feature extraction device based on a grid subdivision structure provided by the present invention;

[0049] Figure 6 A schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0050] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0051] The existing 3D mesh feature extraction methods lack a hierarchical representation of 3D data with multiple resolutions, which makes it difficult to obtain a reasonable feature representation. Figure 1-Figure 4 The present invention describes a three-dimensional model feature extraction method based on a grid subdivision structure. Figure 1 The present invention provides a three-dimensional model feature extraction method based on a grid subdivision structure, such as Figure 1 As shown, the method includes:

[0052] Step 110: determine a mesh representation of the three-dimensional model of the feature to be extracted.

[0053] Specifically, first determine the grid representation of the 3D model of the feature to be extracted. The 3D model can be a variety of physical objects or scenes, such as human and animal models, clothing and fabric models, human organ models, furniture models, industrial product models, film and television special effects models, sculpture and architectural models, indoor scene models, etc.

[0054] The mesh representation of a three-dimensional model is composed of polygonal facets, and the mesh representation referred to in the present invention specifically refers to a triangular mesh or a quadrilateral mesh. There are many ways to determine the triangular mesh of a three-dimensional model, such as using modeling software to manually construct a triangular mesh or automatically generate a triangular mesh according to a certain algorithm, or a three-dimensional point cloud constructed based on multi-viewpoint images of an object or scene collected in various directions, and then meshing it, or a three-dimensional point cloud of an object or scene obtained based on omnidirectional radar scanning, and then meshing it, etc., which are not specifically limited here. The three-dimensional model feature extraction method provided by the present invention extracts feature vectors for the three-dimensional model based on the mesh of the original three-dimensional model determined in the current step 110.

[0055] Step 120, inputting the mesh of the three-dimensional model into a feature extraction module, outputting a target mesh representation after feature extraction, and a target feature vector of each facet in the target mesh representation;

[0056] The mesh representation is composed of a plurality of triangular facets or a plurality of quadrilateral facets, and is used to depict the three-dimensional shape of a physical object or scene. The feature extraction module includes a re-gridding module and a mesh up-and-down sampling module connected in sequence.

[0057] The output feature vector is explained here. It is obtained after several downsampling or upsampling in the feature extraction module. Each downsampling can aggregate the input feature vector, and each upsampling can propagate the feature vector. Therefore, the feature vector output by the feature extraction module is an aggregation of a wide range of input feature vectors, and can describe the features of input patches in a wide range.

[0058] Among them, the output feature vector can support traditional machine learning methods, such as support vector machines, etc. Specifically, the output feature vector is used as input data to train a machine learning model, and the learning goal of the machine learning model can be to solve tasks such as three-dimensional model classification, three-dimensional model segmentation, three-dimensional model retrieval, and three-dimensional model shape correspondence. It can also be applied to deep learning. Specifically, one or more of the feature extraction modules are used as a network layer of a deep neural network, which is used to take the deep features of each face as input, and output the feature vector output after passing through the feature extraction module as the input of the next layer of the deep neural network.

[0059] The three-dimensional model feature extraction method based on the grid subdivision structure provided by the present invention determines the grid representation of the three-dimensional model of the feature to be extracted, inputs the grid of the three-dimensional model into the feature extraction module, outputs the grid representation after feature extraction and the feature vector of each face of the grid, and the re-gridding module is used to add the subdivision structure to the grid of the original three-dimensional model by preset surface subdivision rules. Since the re-gridding module and the grid up-and-down sampling module connected in sequence are constructed in the feature extraction module, a new multi-resolution hierarchical method is provided. By means of surface subdivision, the subdivision structure is first added to re-grid the original grid model, and then the re-gridding model is up-and-down sampled to obtain different resolution features of each face, which can support traditional machine learning and deep learning, and is applied to three-dimensional model classification, segmentation, retrieval, shape correspondence, etc., and can improve the high accuracy and robustness of the system.

[0060] Based on the above embodiment, in the method, the feature extraction module includes a regridding module and a grid up-and-down sampling module connected in sequence, specifically including:

[0061] The regridding module regrids the grid of the input three-dimensional model to obtain a regrid model with a subdivision structure and outputs it to the grid up- and down-sampling module;

[0062] The grid up-and-down sampling module performs several down-sampling and several up-sampling on the regrid model to obtain a target grid representation after feature extraction and a target feature vector of each facet in the target grid representation.

[0063] Specifically, the re-gridding module and the grid up-sampling and down-sampling module are combined to extract different resolution features of each facet in the original three-dimensional grid.

[0064] Based on the above embodiment, in the method, re-gridding the mesh of the input original three-dimensional model to obtain a re-grid model with a subdivision structure specifically includes:

[0065] Simplifying the input original mesh of the three-dimensional model to obtain a simplified mesh, and establishing a mapping between the simplified mesh and the original mesh of the three-dimensional model;

[0066] Subdividing the simplified grid according to a preset subdivision rule to obtain a grid model with a subdivision structure;

[0067] Based on the mapping, the mesh model is mapped back to the original mesh of the three-dimensional model to obtain a re-meshed model of the original mesh of the three-dimensional model with a subdivision structure.

[0068] Specifically, Figure 2 The following are examples of the regridding results and downsampling process provided by the present invention, such as Figure 2 As shown, the mesh of the input three-dimensional model is simplified first using a self-parameterization method, and a parametric mapping between the simplified mesh and the original mesh is established. There are many ways to establish a mapping. Preferably, a multi-resolution adaptive parameterized MAPS method is used, which simplifies by deleting vertices, and a multi-layer conformal mapping is established between the simplified mesh and the original mesh. After the simplified mesh is subdivided and mapped back to the original three-dimensional mesh, the re-meshing result of the original three-dimensional mesh can be obtained. At this time, the connection relationship of the nodes of the mesh conforms to the surface subdivision rule. It should be noted here that the facets in the original mesh model can be triangles or quadrilaterals. If the input original three-dimensional mesh is a triangular mesh, the subdivision rule used is Loop subdivision. If the input original three-dimensional mesh is a quadrilateral mesh, the subdivision rule used is Catmull-Clerk subdivision. The following descriptions all take triangular facets as an example, and the situation of quadrilateral meshes is similar to that of triangular facets.

[0069] Figure 3 An example diagram of a grid of a subdivision structure provided by the present invention, such as Figure 3As shown in the figure, both sides are 3D mesh models of the same rabbit. The 3D mesh model on the left is obtained by adding a subdivision structure to the 3D mesh model on the right. Both mesh models extract a same large patch at the same position. Four small patches are filled into the large patch on the left relative to the same patch on the right. This is an example of adding a subdivision structure. If the mesh has a subdivision structure, adjacent patches can be merged into a large patch to obtain a coarser-grained, low-resolution mesh. At the same time, a low-resolution mesh can also be obtained by splitting the patches to obtain a high-resolution mesh.

[0070] Based on the above embodiment, in the method, downsampling and upsampling the re-grid model several times to obtain a target grid representation after feature extraction, and a target feature vector of each facet in the target grid representation, specifically includes:

[0071] The m-th sampling layer of the grid up and down sampling module samples the input m-1-th sampling layer feature vector and the m-1-th sampling layer grid model of each face patch according to a preset rule, and outputs the m-th sampling layer feature vector and the m-th sampling layer grid model of each face patch to the m+1-th sampling layer, the grid up and down sampling module outputs the L-th layer feature vector, the m-th sampling layer is an up sampling layer or a down sampling layer, and the preset rule corresponding to the m-th sampling layer is a preset up sampling rule or a preset down sampling rule;

[0072] Among them, m=1,2,…,L, L is the total number of sampling layers in the grid up and down sampling modules, the zeroth sampling layer feature vector is composed of the patch shape description and patch pose description of the re-grid model, and the zeroth sampling layer grid model is the re-grid model.

[0073] Specifically, the grid up and down sampling module includes multiple sampling layers, and their connection relationship is in a cascade form, that is, they are connected in sequence. Each sampling layer can be an up-sampling layer or a down-sampling layer. The arrangement and combination of the up-sampling layers and the down-sampling layers in the grid up and down sampling module and the number of layers are determined according to the requirements of feature extraction.

[0074] Based on the above embodiment, in the method, the m-th sampling layer samples the input m-1-th sampling layer feature vector and the m-1-th sampling layer grid model of each face patch according to a preset rule, and outputs the m-th sampling layer feature vector and the m-th sampling layer grid model of each face patch, specifically including:

[0075] If the mth sampling layer is an upsampling layer,

[0076] Then, each facet of the input m-th sampling layer is split into four faces according to a preset subdivision rule to obtain an m-th sampling layer grid model; based on the m-1-th sampling layer feature vector of each input facet, the m-th sampling layer feature vector of the four faces is determined according to a preset propagation rule;

[0077] If the mth sampling layer is a downsampling layer,

[0078] Then, four patches in the same subdivision structure in the grid model of the m-th sampling layer are input to obtain the m-th sampling layer grid model; and the m-th sampling layer feature vectors of the four patches are determined according to a preset fusion rule based on the m-1-th sampling layer feature vectors of each input patch.

[0079] Specifically, the above content specifically defines the specific operations of each sampling layer, including how to process the grid model and the corresponding feature vectors of each facet. The downsampling layer performs downsampling according to the preset downsampling rules, specifically including: merging the four faces in the same subdivision structure in the current input grid model to obtain an output grid, and the feature vector of the input grid is fused based on the feature vectors of the four faces according to the preset fusion rules; the upsampling layer performs upsampling according to the preset upsampling rules, specifically including: splitting each facet in the current input grid model into four faces according to the preset subdivision rules, and obtaining the feature vectors of the four faces based on the feature vectors of the input faces according to the preset propagation rules.

[0080] Based on the above embodiment, in this method, if the mesh is a triangular mesh, the preset subdivision rule is to use a Loop subdivision rule;

[0081] If the grid is a quadrilateral grid, the preset subdivision rule is to use the Catmull-Clerk subdivision rule.

[0082] Specifically, if the processing object is a triangular mesh, it uses the Loop subdivision rule to split a triangular face into four triangular facets; if the processing object is a quadrilateral mesh, it uses the Catmull-Clerk subdivision rule to split a quadrilateral face into four quadrilateral facets.

[0083] Based on the above embodiment, in this method, the preset fusion rule is to take the mean, maximum value or minimum value of each dimension of the feature vectors of the four patches.

[0084] Specifically, the preset fusion rule is to take the mean, maximum or minimum of each dimension of the feature vector. The above three fusion rules belong to the fusion method with the least amount of calculation, and generate the features of the fused low-resolution patch in the simplest way while retaining the feature information of the original patch. Figure 4 The schematic diagram of the up-and-down sampling calculation of the three-dimensional grid provided by the present invention is as follows: Figure 4 As shown, the first figure from the left shows the downsampling process, which combines the patches containing subdivision structures into a larger patch with lower resolution ( Figure 4 In the figure, the large triangle containing four small triangles is integrated into the final large triangle. The feature vector of the fused large patch is obtained by fusing the feature vectors of the four small triangle patches contained in the subdivision structure. That is, assuming that the feature vectors of the four small patches are e1, e2, e3, and e4, and the dimension of the feature is d; assuming that the feature vector of the fused large patch is E, then its feature dimension is also d, and the fusion rule of taking the mean of each dimension is:

[0085]

[0086] Similarly, the fusion rule for taking the maximum value in each dimension is,

[0087] E i =max{e 1,i , e 2,i , e 3,i , e 4,i}

[0088] The fusion rule for taking the minimum value of each dimension is:

[0089] E i =min{e 1,i , e 2,i , e 3,i , e 4,i}

[0090] Based on the above embodiment, in the method, the preset propagation rule is nearest neighbor upsampling or bilinear upsampling.

[0091] Specifically, if nearest neighbor upsampling is used, the feature vectors of the four facets obtained by the split are all feature vectors of the input facet; if bilinear upsampling is used, the feature vector of the facet after the split is obtained by interpolating the feature vectors of the facet before the split and its adjacent facets, and the interpolation ratio is the inverse of the distance from the center of the facet before the split to the facet after the split. When upsampling, the features of the low-resolution mesh facet are propagated to the corresponding high-resolution mesh facet. The propagation methods include nearest neighbor upsampling, bilinear upsampling, etc. Nearest neighbor upsampling is as follows Figure 4 The second "nearest neighbor upsampling" from the left in the figure shows the process, where each low-resolution patch feature is directly propagated to the corresponding high-resolution patch; bilinear upsampling is shown in Figure 4 In the second "bilinear upsampling" display process from the left, the high-resolution features are interpolated from adjacent low-resolution patches, and the interpolation ratio is inversely proportional to the distance from the high-resolution patch to the low-resolution patch.

[0092] The method provided by the present invention mainly proposes an improvement on the hierarchical feature extraction method of different resolutions of three-dimensional grids. First, the input grid is preprocessed to make it have a subdivision structure. Specifically, the grid of the input three-dimensional model is simplified first using a self-parameterization method, a parameterized mapping between the simplified grid and the original grid is established, and the grid of the input three-dimensional model is re-gridded so that the connection relationship of the grid nodes conforms to the surface subdivision rule. Specifically, if the input is a triangular grid, it is required to conform to the Loop subdivision rule, and if the input is a quadrilateral grid, it is required to conform to the Catmull-Clerk subdivision rule. In this network, the features of the grid are stored on the facets. For the mesh that has been re-gridded, its facet input features include two types: shape features and posture features. Shape features include the area of ​​the facet, the size of the inner angle of the facet, and the curvature of the facet vertices; posture features include the center coordinates of the facet and the normal direction of the facet. Based on the subdivision connection structure properties, every four facets can be merged into one facet, so regular downsampling and upsampling can be performed. When downsampling, find the corresponding low-resolution mesh patches after merging multiple high-resolution mesh patches according to the subdivision connection structure, aggregate the features of the high-resolution mesh and pass them to the low-resolution mesh. The feature aggregation methods include calculating the mean, maximum value, minimum value, etc.; when upsampling, propagate the features of the low-resolution mesh patches to the corresponding high-resolution mesh patches. The propagation methods include nearest neighbor upsampling, bilinear upsampling, etc.

[0093] Based on the above, a multi-resolution mesh neural network is constructed to learn the features of three-dimensional mesh patches. The network contains downsampling layers and upsampling layers, which respectively reduce or increase the mesh resolution, thereby achieving local feature aggregation or global feature propagation.

[0094] Based on the above embodiments, the present invention further provides a multi-resolution deep neural network for three-dimensional grids, including the feature extraction module and the specific task module as described in any of the above embodiments,

[0095] During training,

[0096] The feature extraction module extracts the sample feature vector of each facet of the input sample three-dimensional mesh model and outputs it to the specific task module;

[0097] The specific task module processes the sample feature vector according to specific rules to obtain a prediction result;

[0098] A loss function is constructed based on the prediction results and the facet labels with a preset learning objective; wherein the preset learning objective is mesh classification or triangular mesh model physical type recognition.

[0099] Specifically, for example, if the preset learning goal is a human body mesh segmentation method, then the human body mesh segmentation method includes the following steps: determining the original three-dimensional mesh of the human body to be tested; inputting the original three-dimensional mesh into a segmentation model, and outputting the areas to which each facet in the original three-dimensional mesh belongs, and the areas include the head, limbs and torso; wherein the segmentation model is trained based on the original three-dimensional mesh of the sample and the corresponding human body area labels, and the segmentation model includes a re-gridding module such as the one in any of the above embodiments, a mesh up and down sampling module and a classification module such as the one in any of the above embodiments, which are connected in sequence, and the re-gridding module is used to add a subdivision structure to the original three-dimensional mesh through a preset surface subdivision rule. In the object category recognition method, during the classification model training process, the feature extraction module performs convolution calculation on the features of each facet in the input sample original three-dimensional grid according to preset rules, obtains the feature vector of each facet and outputs it to the feature fusion module; the feature fusion module fuses the feature vector of each facet according to preset fusion rules, obtains the overall feature vector of the sample original three-dimensional grid and outputs it to the classification fully connected layer; the classification fully connected layer determines the predicted object type corresponding to the sample original three-dimensional grid based on the overall feature vector; wherein the predicted object type is used to construct the loss function during the classification model training, and convolution calculation is performed on any facet according to preset rules to aggregate the neighborhood facet features of any facet to extract the local features of any facet. Furthermore, the segmentation model includes a regridding module, a grid up and down sampling module and a classification module connected in sequence, specifically including: during the training process of the segmentation model, the regridding module regrids the original three-dimensional grid of the input sample to obtain a regrid model including a subdivision structure and outputs it to the grid up and down sampling module; the grid up and down sampling module performs a number of downsampling and upsampling in any combination on the regrid model in sequence to obtain a target feature vector for each face patch and outputs it to the classification module; the classification module determines the prediction area to which each face patch belongs based on the target feature vector of each face patch using a fully connected layer with shared parameters; wherein the prediction area is used to construct a loss function during the training of the segmentation model.

[0100] The following is a description of a three-dimensional model feature extraction device based on a grid subdivision structure provided by the present invention. The three-dimensional model feature extraction device described below and the three-dimensional model feature extraction method described above can be referenced to each other.

[0101] Figure 5 A schematic diagram of a three-dimensional model feature extraction device based on a grid subdivision structure provided by the present invention is shown in FIG. Figure 5 As shown, the device includes an input unit 510 and an output unit 520, wherein:

[0102] The input unit 510 is used to determine the grid representation of the three-dimensional model of the feature to be extracted;

[0103] The output unit 520 is used to input the mesh of the three-dimensional model into the feature extraction module, and output the target mesh representation after feature extraction, and the target feature vector of each facet in the target mesh representation;

[0104] The mesh representation is composed of a plurality of triangular facets or a plurality of quadrilateral facets, and is used to depict the three-dimensional shape of a physical object or scene. The feature extraction module includes a re-gridding module and a mesh up-and-down sampling module connected in sequence.

[0105] The device provided by the present invention determines the mesh representation of the three-dimensional model of the feature to be extracted, inputs the mesh of the three-dimensional model into the feature extraction module, outputs the mesh representation after feature extraction and the feature vector of each face of the mesh, and the re-gridding module is used to add a subdivision structure to the mesh of the original three-dimensional model by preset surface subdivision rules. Since the re-gridding module and the grid up-and-down sampling module connected in sequence are constructed in the feature extraction module, a new multi-resolution hierarchical method is provided. By means of surface subdivision, the subdivision structure is first added to re-grid the original mesh model, and then the re-gridding model is up-and-down sampled to obtain different resolution features of each face, which can support traditional machine learning and deep learning, and is applied to three-dimensional model classification, segmentation, retrieval, shape correspondence, etc., and can improve the high accuracy and robustness of the system.

[0106] Based on the above embodiment, in the device, the feature extraction module includes a re-gridding module and a grid up-and-down sampling module connected in sequence, specifically including:

[0107] The regridding module regrids the grid of the input three-dimensional model to obtain a regrid model with a subdivision structure and outputs it to the grid up- and down-sampling module;

[0108] The grid up-and-down sampling module performs several down-sampling and several up-sampling on the regrid model to obtain a target grid representation after feature extraction and a target feature vector of each facet in the target grid representation.

[0109] Based on the above embodiment, in the device, re-gridding the mesh of the input original three-dimensional model to obtain a re-grid model with a subdivision structure specifically includes:

[0110] Simplifying the input original mesh of the three-dimensional model to obtain a simplified mesh, and establishing a mapping between the simplified mesh and the original mesh of the three-dimensional model;

[0111] Subdividing the simplified grid according to a preset subdivision rule to obtain a grid model with a subdivision structure;

[0112] Based on the mapping, the mesh model is mapped back to the original mesh of the three-dimensional model to obtain a re-meshed model of the original mesh of the three-dimensional model with a subdivision structure.

[0113] Based on the above embodiment, in the device, the re-grid model is down-sampled and up-sampled several times to obtain a target grid representation after feature extraction, and a target feature vector of each facet in the target grid representation, specifically including:

[0114] The m-th sampling layer of the grid up and down sampling module samples the input m-1-th sampling layer feature vector and the m-1-th sampling layer grid model of each face patch according to a preset rule, and outputs the m-th sampling layer feature vector and the m-th sampling layer grid model of each face patch to the m+1-th sampling layer, the grid up and down sampling module outputs the L-th layer feature vector, the m-th sampling layer is an up sampling layer or a down sampling layer, and the preset rule corresponding to the m-th sampling layer is a preset up sampling rule or a preset down sampling rule;

[0115] Among them, m=1,2,…,L, L is the total number of sampling layers in the grid up and down sampling modules, the zeroth sampling layer feature vector is composed of the patch shape description and patch pose description of the re-grid model, and the zeroth sampling layer grid model is the re-grid model.

[0116] Based on the above embodiment, in the device, the m-th sampling layer samples the input m-1 sampling layer feature vector and m-1 sampling layer grid model of each face patch according to a preset rule, and outputs the m-th sampling layer feature vector and m-th sampling layer grid model of each face patch, specifically including:

[0117] If the mth sampling layer is an upsampling layer,

[0118] Then, each facet of the input m-th sampling layer is split into four faces according to a preset subdivision rule to obtain an m-th sampling layer grid model; based on the m-1-th sampling layer feature vector of each input facet, the m-th sampling layer feature vector of the four faces is determined according to a preset propagation rule;

[0119] If the mth sampling layer is a downsampling layer,

[0120] Then, four patches in the same subdivision structure in the grid model of the m-th sampling layer are input to obtain the m-th sampling layer grid model; and the m-th sampling layer feature vectors of the four patches are determined according to a preset fusion rule based on the m-1-th sampling layer feature vectors of each input patch.

[0121] Based on the above embodiment, in the device, if the mesh is a triangular mesh, the preset subdivision rule is to use a Loop subdivision rule;

[0122] If the grid is a quadrilateral grid, the preset subdivision rule is to use the Catmull-Clerk subdivision rule.

[0123] Based on the above embodiment, in the device, the preset fusion rule is to take the average, maximum value or minimum value of each dimension of the feature vectors of the four patches.

[0124] Based on the above embodiment, in the device, the preset propagation rule is nearest neighbor upsampling or bilinear upsampling.

[0125] Figure 6 A schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute a three-dimensional model feature extraction method based on a mesh subdivision structure, the method comprising: given a mesh representation of a three-dimensional model of features to be extracted, inputting the mesh into a feature extraction module, and outputting feature vectors of each facet of the mesh; wherein the feature extraction module comprises a re-gridding module and a mesh up-and-down sampling module connected in sequence. .

[0126] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0127] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the three-dimensional model feature extraction method based on the grid subdivision structure provided by the above-mentioned methods, and the method includes: given a grid representation of a three-dimensional model of features to be extracted, the grid is input into a feature extraction module, and the feature vectors of each facet of the grid are output; wherein the feature extraction module includes a re-gridding module and a grid up and down sampling module connected in sequence.

[0128] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the three-dimensional model feature extraction method based on a grid subdivision structure provided by the above-mentioned methods, the method comprising: given a grid representation of a three-dimensional model of features to be extracted, inputting the grid into a feature extraction module, and outputting feature vectors of each facet of the grid; wherein the feature extraction module comprises a re-gridding module and a grid up and down sampling module connected in sequence.

[0129] The server embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0130] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional model feature extraction method based on a grid subdivision structure, characterized in that: include: Determining a mesh representation of a three-dimensional model of features to be extracted; Inputting the mesh of the three-dimensional model into a feature extraction module, outputting a target mesh representation after feature extraction, and a target feature vector of each facet in the target mesh representation; Wherein, the mesh representation is composed of a plurality of triangular facets or a plurality of quadrilateral facets, and is used to depict the three-dimensional shape of a physical object or scene, and the feature extraction module comprises a re-gridding module and a mesh up-and-down sampling module connected in sequence; The feature extraction module includes a regridding module and a grid up-and-down sampling module connected in sequence, specifically including: The regridding module regrids the grid of the input three-dimensional model to obtain a regrid model with a subdivision structure and outputs it to the grid up- and down-sampling module; The grid up- and down-sampling module performs several down-sampling and several up-sampling on the regrid model to obtain a target grid representation after feature extraction and a target feature vector of each facet in the target grid representation; The regrid model is subjected to several downsampling and several upsampling to obtain a target grid representation after feature extraction, and a target feature vector of each facet in the target grid representation, specifically including: The m-th sampling layer of the grid up and down sampling module samples the input m-1-th sampling layer feature vector and the m-1-th sampling layer grid model of each face patch according to a preset rule, and outputs the m-th sampling layer feature vector and the m-th sampling layer grid model of each face patch to the m+1-th sampling layer, the grid up and down sampling module outputs the L-th layer feature vector, the m-th sampling layer is an up sampling layer or a down sampling layer, and the preset rule corresponding to the m-th sampling layer is a preset up sampling rule or a preset down sampling rule; Wherein, m=1,2,…,L, L is the total number of sampling layers in the grid up and down sampling modules, the zeroth sampling layer feature vector is composed of the patch shape description and patch pose description of the re-grid model, and the zeroth sampling layer grid model is the re-grid model.

2. The three-dimensional model feature extraction method based on grid subdivision structure according to claim 1, characterized in that: The step of re-gridding the mesh of the input original three-dimensional model to obtain a re-grid model with a subdivision structure specifically includes: Simplifying the input original mesh of the three-dimensional model to obtain a simplified mesh, and establishing a mapping between the simplified mesh and the original mesh of the three-dimensional model; Subdividing the simplified grid according to a preset subdivision rule to obtain a grid model with a subdivision structure; Based on the mapping, the mesh model is mapped back to the original mesh of the three-dimensional model to obtain a re-meshed model of the original mesh of the three-dimensional model with a subdivision structure.

3. The 3D model feature extraction method based on a grid subdivision structure according to claim 1, wherein the m-th sampling layer samples the input m-1 sampling layer feature vector and the m-1 sampling layer grid model of each face patch according to a preset rule, and outputs the m-th sampling layer feature vector and the m-th sampling layer grid model of each face patch, specifically comprising: If the mth sampling layer is an upsampling layer, Then, each facet of the input m-th sampling layer is split into four faces according to a preset subdivision rule to obtain an m-th sampling layer grid model; based on the m-1-th sampling layer feature vector of each input facet, the m-th sampling layer feature vector of the four faces is determined according to a preset propagation rule; If the mth sampling layer is a downsampling layer, Then, four patches in the same subdivision structure in the grid model of the m-th sampling layer are input to obtain the m-th sampling layer grid model; and the m-th sampling layer feature vectors of the four patches are determined according to a preset fusion rule based on the m-1-th sampling layer feature vectors of each input patch.

4. The three-dimensional model feature extraction method based on grid subdivision structure according to claim 3 is characterized in that: If the mesh is a triangular mesh, the preset subdivision rule is to use the Loop subdivision rule; If the grid is a quadrilateral grid, the preset subdivision rule is to use the Catmull-Clerk subdivision rule.

5. The three-dimensional model feature extraction method based on grid subdivision structure according to claim 3 is characterized in that: The preset fusion rule is to take the mean, maximum or minimum value of each dimension of the feature vectors of the four patches.

6. The three-dimensional model feature extraction method based on grid subdivision structure according to claim 3 is characterized in that: The preset propagation rule is nearest neighbor upsampling or bilinear upsampling.

7. A three-dimensional model feature extraction method based on a grid subdivision structure according to claim 1, characterized in that: Constructing a multi-resolution deep neural network for learning three-dimensional grids, wherein the multi-resolution deep neural network includes the feature extraction module and the specific task module; During training, The feature extraction module extracts the sample feature vector of each facet of the input sample three-dimensional mesh model and outputs it to the specific task module; The specific task module processes the sample feature vector according to specific rules to obtain a prediction result; A loss function is constructed based on the prediction results and the facet labels with a preset learning objective; wherein the preset learning objective is mesh classification or triangular mesh model physical type recognition.

8. A three-dimensional model feature extraction device based on a grid subdivision structure, characterized in that: include: An input unit for determining a mesh representation of a three-dimensional model of features to be extracted; An output unit, used for inputting the mesh of the three-dimensional model into a feature extraction module, and outputting a target mesh representation after feature extraction, and a target feature vector of each facet in the target mesh representation; Wherein, the mesh representation is composed of a plurality of triangular facets or a plurality of quadrilateral facets, and is used to depict the three-dimensional shape of a physical object or scene, and the feature extraction module comprises a re-gridding module and a mesh up-and-down sampling module connected in sequence; The feature extraction module includes a regridding module and a grid up-and-down sampling module connected in sequence, specifically including: The regridding module regrids the grid of the input three-dimensional model to obtain a regrid model with a subdivision structure and outputs it to the grid up- and down-sampling module; The grid up- and down-sampling module performs several down-sampling and several up-sampling on the regrid model to obtain a target grid representation after feature extraction and a target feature vector of each facet in the target grid representation; The regrid model is subjected to several downsampling and several upsampling to obtain a target grid representation after feature extraction, and a target feature vector of each facet in the target grid representation, specifically including: The m-th sampling layer of the grid up and down sampling module samples the input m-1-th sampling layer feature vector and the m-1-th sampling layer grid model of each face patch according to a preset rule, and outputs the m-th sampling layer feature vector and the m-th sampling layer grid model of each face patch to the m+1-th sampling layer, the grid up and down sampling module outputs the L-th layer feature vector, the m-th sampling layer is an up sampling layer or a down sampling layer, and the preset rule corresponding to the m-th sampling layer is a preset up sampling rule or a preset down sampling rule; Wherein, m=1,2,…,L, L is the total number of sampling layers in the grid up and down sampling modules, the zeroth sampling layer feature vector is composed of the patch shape description and patch pose description of the re-grid model, and the zeroth sampling layer grid model is the re-grid model.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the three-dimensional model feature extraction method based on a grid subdivision structure are implemented as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional model feature extraction method based on a grid subdivision structure as described in any one of claims 1 to 7 are implemented.

Images