A 3D model feature extraction method using patch convolution

Abstract

The present invention provides a method for extracting features of a three-dimensional model using facet convolution, comprising: determining a triangular mesh representation of a three-dimensional model of features to be extracted; wherein the triangular mesh representation is a three-dimensional shape composed of multiple triangular facets for depicting an entity or scene; inputting the triangular mesh of the three-dimensional model into a feature extraction module, and outputting a feature vector of each facet of the triangular mesh; wherein the feature extraction module is composed of one or more facet convolution layers, and the facet convolution layers perform convolution operations on each facet in the input triangular mesh according to preset rules. In the method provided by the present invention, the extracted features have permutation invariance, can support traditional machine learning and deep learning, and are applied to three-dimensional model classification, segmentation, retrieval, and shape correspondence, etc., which can improve accuracy and robustness.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional models, and in particular to a three-dimensional model feature extraction method using facet convolution. Background Art

[0002] Mesh is a common way to describe the shape of a 3D model, and is widely used in modeling, rendering, animation, 3D printing, etc. Usually a mesh defines the vertices in a 3D space, and the faces defined by the connection relationship between the vertices. The most commonly used meshes are triangular meshes and quadrilateral meshes: all the faces in a triangular mesh are triangles, and all the faces in a quadrilateral mesh are quadrilaterals.

[0003] Deep neural networks have made significant progress compared to traditional methods in understanding, analyzing, and generating data formats such as images, natural language processing, and point clouds.

[0004] Existing feature extraction methods based on triangular meshes usually use machine learning. In a two-dimensional feature extraction scheme based on a convolutional neural network, the neural network extracts features from the image through image convolution operations. However, for three-dimensional models represented by triangular meshes, because the triangular mesh lacks a regular structure like an image, it is difficult to apply the traditional image method convolution, resulting in the feature extraction of three-dimensional shapes represented by triangular meshes being unable to be achieved through machine learning. Previously, there have been methods that have attempted to design convolution operations for the nodes and edges of the mesh, such as the MeshCNN proposed by Hanocka et al. However, these convolutions can only support one convolution kernel size and cannot support holes, so they lack flexibility and have limited receptive fields.

[0005] Therefore, how to avoid the problem that the convolution operation is difficult to implement in the feature extraction of existing triangular meshes, which leads to the difficulty in designing deep neural networks, the difficulty in applying machine learning methods, and the difficulty in improving the effects of related systems, remains a problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0006] The present invention provides a three-dimensional model feature extraction method using facet convolution, which is used to solve the problem that existing deep learning methods are difficult to realize learning and representation of irregular triangular data. Through one or more facet convolution layers, faces in a triangular mesh are convolved according to preset rules, and a convolution operation rule for irregular graphics such as triangular meshes is provided. The features of the triangular mesh can be extracted, the method has order invariance, and can support traditional machine learning methods and deep learning methods, so that effective deep convolutional neural networks in the image field can be applied to three-dimensional models represented by triangular meshes.

[0007] The present invention provides a three-dimensional model feature extraction method using patch convolution, comprising:

[0008] Determine a triangular mesh representation of a three-dimensional model of features to be extracted; wherein the triangular mesh representation is a three-dimensional shape composed of a plurality of triangular facets for depicting an entity or scene;

[0009] Inputting the triangular mesh of the three-dimensional model into a feature extraction module, and outputting a feature vector of each facet of the triangular mesh;

[0010] The feature extraction module is composed of one or more facet convolution layers, and the facet convolution layer performs a convolution operation on each facet in the input triangular mesh according to a preset rule.

[0011] According to a three-dimensional model feature extraction method using patch convolution provided by the present invention, the feature extraction module is composed of one or more patch convolution layers, specifically including:

[0012] If the feature extraction module is composed of a patch convolution layer, the patch convolution layer performs a convolution operation on the input triangular mesh according to a preset rule based on the shape description and the posture description of each patch, and outputs a feature vector of each patch of the triangular mesh;

[0013] If the feature extraction module is composed of multiple patch convolution layers, the m-th layer patch convolution layer will perform a convolution operation on the m-1th layer feature vector of each input patch according to the preset rule, and output the m-th layer feature vector of each patch to the m+1th layer patch convolution layer, and the feature extraction module outputs the L-th layer feature vector; wherein m=1, 2, ..., L, L is the total number of layers in the multi-layer patch convolution layer, and the zeroth layer feature vector is composed of the patch shape description and the patch pose description of the triangular mesh.

[0014] According to a three-dimensional model feature extraction method using facet convolution provided by the present invention, the feature vector of the m-1th layer of each facet input is convolved according to the preset rule to output the feature vector of the mth layer of each facet, specifically comprising:

[0015] Traverse each face in the input triangular mesh, and calculate the m-th layer feature vector of any face as the center face in the traversal order and output it;

[0016] Wherein, for any central patch, a set of computational patches participating in the convolution is determined based on a preset size and a preset hole of the convolution kernel;

[0017] The calculation patch set is convolved in a patch order-invariant convolution manner.

[0018] According to a three-dimensional model feature extraction method using patch convolution provided by the present invention, the set of computational patches involved in the convolution is determined based on a preset size and a preset hole of the convolution kernel, specifically comprising:

[0019] If the convolution kernel size k and the hole d both satisfy k = 3 and d = 1, the neighborhood range of the center patch is its three adjacent patches;

[0020] If the convolution kernel size k and the hole d both satisfy k>3 and d>1, the maximum distance from the center patch to the center patch in the neighborhood is set to The interval distance between adjacent facets in the neighborhood is determined by a preset measurement rule, and the facets included in the neighborhood range are determined based on the maximum distance and the interval distance.

[0021] According to a three-dimensional model feature extraction method using patch convolution provided by the present invention, the preset measurement rule is to measure a zigzag route obtained by extending counterclockwise with a zigzag pace and clockwise with a zag pace.

[0022] According to a three-dimensional model feature extraction method using facet convolution provided by the present invention, the convolution operation is performed on the calculation facet set in a facet order-invariant convolution manner, specifically comprising:

[0023] Sorting the computational facet set of the central facet in depth-first order;

[0024] For the m-1th layer feature vector of any input central face patch, the sum of the m-1th layer feature vector differences between the central face patch and the surrounding face patches is determined based on the sorting as the m-th layer central difference term, and the sum of the m-1th layer feature vector differences between the surrounding face patches before and after the sorting is determined based on the sorting as the m-th layer annular difference term;

[0025] The m-th layer feature vector of any central face patch is determined based on the m-th layer center difference term, the m-th layer annular difference term, the m-1-th layer feature vector of any central face patch and the m-1-th layer feature vector corresponding to the surrounding face patches.

[0026] According to a three-dimensional model feature extraction method using facet convolution provided by the present invention, the sum of the m-1th layer feature vector differences corresponding to the central facet and the surrounding facets determined based on the sorting is the m-th layer center difference term, and the sum of the m-1th layer feature vector differences corresponding to the surrounding facets before and after the sorting is determined based on the sorting as the m-th layer annular difference term, specifically comprising:

[0027] The m-th layer center difference of any center patch i is determined by the following formula and the mth layer annular difference term of any central patch i

[0028]

[0029] in, is the m-1th layer feature vector of any central facet i, and the m-1th layer feature vectors corresponding to the surrounding faces of any central facet i after being arranged in depth-first order are respectively n is the total number of surrounding patches, abs(·) is the absolute value function;

[0030] Correspondingly, determining the m-th layer feature vector of any central face patch based on the m-th layer center difference term, the m-th layer annular difference term, the m-1-th layer feature vector of any central face patch, and the m-1-th layer feature vector corresponding to the surrounding face patches specifically includes:

[0031] The m-1th layer feature vector of any central patch i is calculated by the following formula after the m-1th layer feature vector passes through the mth layer patch convolution layer and the output m-th layer feature vector conv(f i ):

[0032]

[0033] in, is the m-th layer center difference term of any center patch i, is the m-th layer annular difference term of any central facet i, and w0, w1, w2 and w3 are all preset weight matrices.

[0034] The present invention also provides a deep convolutional neural network for three-dimensional meshes, comprising the feature extraction module and the specific task module in any of the three-dimensional model feature extraction methods using patch convolution as described above,

[0035] During training,

[0036] The feature extraction module extracts the sample feature vector of each facet of the input sample triangular mesh model and outputs it to the specific task module;

[0037] The specific task module processes the sample feature vector according to specific rules to obtain a prediction result;

[0038] A loss function is constructed based on the prediction results and the facet labels with a preset learning objective; wherein the preset learning objective is mesh classification or triangular mesh model physical type recognition.

[0039] The present invention also provides a three-dimensional model feature extraction device using facet convolution, comprising:

[0040] An input unit, used to determine a triangular mesh representation of a three-dimensional model of features to be extracted; wherein the triangular mesh representation is a three-dimensional shape composed of a plurality of triangular facets for describing an entity or scene;

[0041] An output unit, used for inputting the triangular mesh of the three-dimensional model into a feature extraction module, and outputting a feature vector of each facet of the triangular mesh;

[0042] The feature extraction module is composed of one or more facet convolution layers, and the facet convolution layer performs a convolution operation on each facet in the input triangular mesh according to a preset rule.

[0043] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the above-mentioned three-dimensional model feature extraction methods using facet convolution are implemented.

[0044] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-mentioned three-dimensional model feature extraction methods using facet convolution.

[0045] The present invention provides a three-dimensional model feature extraction method using facet convolution, which determines the triangular mesh representation of the three-dimensional model of the feature to be extracted; wherein the triangular mesh representation is composed of multiple triangular facets for depicting the three-dimensional shape of the entity or scene; the triangular mesh of the three-dimensional model is input into the feature extraction module, and the feature vector of each facet of the triangular mesh is output; wherein the feature extraction module is composed of one or more facet convolution layers. Since the facet convolution layer performs convolution operation on each facet in the input triangular mesh according to a preset rule, a convolution operation rule for an irregular data structure such as a triangular mesh is provided, and the features of the three-dimensional model can be extracted for the three-dimensional model to be applied to specific applications such as three-dimensional model classification, segmentation, retrieval, shape correspondence, etc. By setting the convolution operation mode of the triangular mesh, a process similar to image convolution is realized, and the previous method cannot provide a regular and flexible triangular mesh convolution mode, which can support traditional machine learning and deep learning, and is applied to three-dimensional model classification, segmentation, retrieval and shape correspondence, etc., and can improve the high accuracy and robustness of the learning system. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 A schematic diagram of a process of extracting features of a three-dimensional model using facet convolution provided by the present invention;

[0048] Figure 2 An example diagram of a patch convolution for a specific operation of the convolution operation provided by the present invention;

[0049] Figure 3 An example diagram of the convolution operation process provided by the present invention;

[0050] Figure 4 A schematic diagram explaining the principle of repeated access provided by the present invention;

[0051] Figure 5 An example diagram of convolution operation under complex convolution kernel size and hole combination provided by the present invention;

[0052] Figure 6 A schematic diagram of a zigzag cavity provided by the present invention;

[0053] Figure 7 A schematic diagram of the structure of a three-dimensional model feature extraction device using facet convolution provided by the present invention;

[0054] Figure 8 A schematic diagram of the physical structure of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0055] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0056] Since existing feature extraction methods suitable for deep learning are difficult to achieve irregular three-dimensional data learning and representation, the present invention.

[0057] Combine the following Figure 1-Figure 6 A three-dimensional model feature extraction method using facet convolution of the present invention is described. Figure 1 A schematic diagram of a process flow of a three-dimensional model feature extraction method using facet convolution provided by the present invention, such as Figure 1 As shown, the method includes:

[0058] Step 110, determining a triangular mesh representation of the three-dimensional model of the feature to be extracted; wherein the triangular mesh representation is a three-dimensional shape composed of a plurality of triangular facets for describing an entity or a scene.

[0059] Specifically, firstly, the input three-dimensional model is received, and then the three-dimensional model is described and expressed in the form of a triangular mesh; wherein the three-dimensional model can be a variety of objects or scenes, such as human animal models, clothing and fabric models, human organ models, furniture models, industrial product models, film and television special effects models, sculpture and architectural models, indoor scene models, etc. There are many ways to determine the triangular mesh of the three-dimensional model, such as using modeling software to manually construct or automatically generate a triangular mesh according to a certain algorithm, or a three-dimensional point cloud constructed based on multi-viewpoint images of objects or scenes collected in various directions, and then meshing, or a three-dimensional point cloud of objects or scenes obtained based on omnidirectional radar scanning, and then meshing, etc., which are not specifically limited here. The three-dimensional model feature extraction method using patch convolution provided by the present invention is to extract feature vectors from each triangular patch in the mesh based on the triangular mesh of the three-dimensional model determined in the current step 110.

[0060] Step 120, inputting the triangular mesh of the three-dimensional model into a feature extraction module, and outputting a feature vector of each facet of the triangular mesh;

[0061] The feature extraction module is composed of one or more facet convolution layers, and the facet convolution layer performs a convolution operation on each facet in the input triangular mesh according to a preset rule.

[0062] The output feature vector is explained here. The feature vector of each facet obtained by the feature extraction module describes the geometric or semantic information of the facet and the faces around it. Its specific physical meaning is determined by the preset weight matrix of the facet convolution layer. The preset weight of the facet convolution layer can be specified manually or obtained through machine learning according to the learning objectives.

[0063] Among them, the output feature vector can support traditional machine learning methods, such as support vector machines, etc. Specifically, the output feature vector is used as input data to train a machine learning model. The learning goal of the machine learning model can be to solve tasks such as three-dimensional model classification, three-dimensional model segmentation, three-dimensional model retrieval, and three-dimensional model shape correspondence, and can also be applied to deep learning. Specifically, one or more of the feature extraction modules are used as network layers of a deep neural network, wherein one of the feature extraction modules can include a patch convolution layer or multiple patch convolution layers. For any patch convolution layer, the operation mechanism is to convolve the feature vector of each patch input into any patch convolution layer according to a preset rule and output it. It should be noted here that when the feature extraction module includes multiple patch convolution layers, the multiple patch convolution layers are in a cascade form, that is, they are connected in sequence. The feature vector of each patch output by the previous layer will be used as the input of the next layer, and the next layer will convolve the feature vector of each patch output by the previous layer according to the preset rule to obtain the feature vector of each patch and then output it to the next layer of patch convolution layer.

[0064] The present invention provides a three-dimensional model feature extraction method using facet convolution, which determines the triangular mesh representation of the three-dimensional model of the feature to be extracted; wherein the triangular mesh representation is composed of multiple triangular facets for depicting the three-dimensional shape of the entity or scene; the triangular mesh of the three-dimensional model is input into the feature extraction module, and the feature vector of each facet of the triangular mesh is output; wherein the feature extraction module is composed of one or more facet convolution layers. Since the facet convolution layer performs convolution operation on each facet in the input triangular mesh according to a preset rule, a convolution operation rule for an irregular data structure such as a triangular mesh is provided, and the features of the three-dimensional model can be extracted for the three-dimensional model to be applied to specific applications such as three-dimensional model classification, segmentation, retrieval, shape correspondence, etc. By setting the convolution operation mode of the triangular mesh, a process similar to image convolution is realized, and the previous method cannot provide a regular and flexible triangular mesh convolution mode, which can support traditional machine learning and deep learning, and is applied to three-dimensional model classification, segmentation, retrieval and shape correspondence, etc., and can improve the high accuracy and robustness of the learning system.

[0065] Based on the above embodiment, in this method, the feature extraction module is composed of one or more patch convolution layers, specifically including:

[0066] If the feature extraction module is composed of a patch convolution layer, the patch convolution layer performs a convolution operation on the input triangular mesh according to a preset rule based on the shape description and the posture description of each patch, and outputs a feature vector of each patch of the triangular mesh;

[0067] If the feature extraction module is composed of multiple patch convolution layers, the m-th layer patch convolution layer will perform a convolution operation on the m-1th layer feature vector of each input patch according to the preset rule, and output the m-th layer feature vector of each patch to the m+1th layer patch convolution layer, and the feature extraction module outputs the L-th layer feature vector; wherein m=1, 2, ..., L, L is the total number of layers in the multi-layer patch convolution layer, and the zeroth layer feature vector is composed of the patch shape description and the patch pose description of the triangular mesh.

[0068] Specifically, the specific parameter types in the shape description and posture description of the face are further limited here. The shape description includes the face product, the size of each internal angle of the face and the curvature term of each vertex of the face. The posture description includes the center coordinates of the face and the normal direction of the face. The extraction of the shape description and posture description of each face is to splice the various parameters into a vector representation.

[0069] If the feature extraction module is composed of a patch convolution layer, the patch convolution layer extracts the shape description and posture description of each patch in the input triangular mesh, performs a convolution operation according to a preset rule, and outputs a feature vector of each patch in the triangular mesh;

[0070] If the feature extraction module is composed of multiple facet convolution layers, the first layer input is the facets in the triangular mesh, extracts the shape description and posture description of each facet, performs convolution operations according to preset rules, and outputs the first layer feature vector of each facet. The second layer of facet convolution layer takes the first layer feature vector as input, performs convolution operations according to preset rules, and outputs the second layer feature vector of each facet, and so on. The feature vector of each facet output by the last layer is the feature vector of each facet of the triangular mesh after passing through the feature extraction module.

[0071] Based on the above embodiment, in the method, performing a convolution operation on the input m-1th layer feature vector of each face patch according to the preset rule to output the mth layer feature vector of each face patch specifically includes:

[0072] Traverse each face in the input triangular mesh, and calculate the m-th layer feature vector of any face as the center face in the traversal order and output it;

[0073] Wherein, for any central patch, a set of computational patches participating in the convolution is determined based on a preset size and a preset hole of the convolution kernel;

[0074] The calculation patch set is convolved in a patch order-invariant convolution manner.

[0075] Specifically, for the to-be-processed triangular mesh model of the input m-th layer patch convolution layer and the corresponding m-1-th layer feature vector of each patch, each patch in the to-be-processed triangular mesh model is traversed in a preset order rule, wherein the preset order rule can be a Z-shaped traversal performed row by row, or an H-shaped traversal performed column by column. No specific limitation is made here, and it is only required that each patch in the to-be-processed mesh model is visited, and then any patch is calculated in turn in the traversal order with any patch as the center patch. The m-th layer feature vector is obtained and output. It should be noted here that each facet in the mesh model to be processed will have a convolution value calculated for it, and the operation of the convolution operation is that when the convolution value of facet A needs to be calculated, facet A needs to be taken as the center facet, and other faces around facet A that can participate in the convolution operation are obtained, and their m-1-th layer feature vectors are convoluted according to preset rules. The "taking any facet as the center facet" means that the convolution value of any facet is calculated by taking the any facet as the center facet. The two parameters of the convolution kernel size and the hole can be combined to determine the set of calculation facets participating in the convolution. After the set of calculation facets that need to participate in the convolution operation is determined, the set of calculation facets is convoluted in a facet order-invariant convolution manner. It should be noted here that the convolution operation provided by the present invention is a convolution operation for triangular mesh facets, and only works in the geometric relationship of triangular mesh facets.

[0076] Based on the above embodiment, in the method, determining the set of computational patches participating in the convolution based on the preset size and preset hole of the convolution kernel specifically includes:

[0077] If the convolution kernel size k and the hole d both satisfy k = 3 and d = 1, the neighborhood range of the center patch is its three adjacent patches;

[0078] If the convolution kernel size k and the hole d both satisfy k>3 and d>1, the maximum distance from the center patch to the center patch in the neighborhood is set to The interval distance between adjacent facets in the neighborhood is determined by a preset measurement rule, and the facets included in the neighborhood range are determined based on the maximum distance and the interval distance.

[0079] Specifically, given a triangular mesh, the convolution mode is defined based on the patches and their neighborhoods. Since each triangular patch has only three adjacent patches, after the neighborhood range is specified, the number of patches participating in the convolution is fixed. Therefore, when performing convolution on the patches, the convolution kernel can maintain the same shape on any patch.

[0080] When the simplest convolution kernel size is 3, the neighborhood of the central patch is its three adjacent patches. Figure 2An example diagram of a triangular convolution for a specific operation of the convolution operation provided by the present invention is shown in FIG. Figure 2 As shown in the figure on the left, the center patch in the first figure is a small triangle patch with a diagonal grid in the middle of the large triangle, and the three small triangle patches with diagonal lines around it are the neighborhood of the center patch, where k is the size of the convolution kernel, and d is the number of holes in the convolution kernel. Figure 3 The convolution operation process example diagram provided by the present invention is as follows: Figure 3 As shown in, for reference, the image convolution with a kernel size of 3 and a hole size of 1 is as follows Figure 3 As shown in the leftmost figure.

[0081] Set a larger convolution kernel size, in this case k>3, and set the maximum distance from the patch in the neighborhood to the center patch to Then the number of patches in the convolution kernel is like Figure 2 As shown in the second figure from the left in the figure, as a reference, the image convolution with a larger convolution kernel size is as follows Figure 3 As shown in the second figure from the left, Figure 3 The nine patches filled with diagonal lines surrounding the central patch filled with diagonal squares in the second figure from the left are the neighborhood.

[0082] The program can use depth-first search to find patches within the distance range. The above assumes that each patch will not be visited repeatedly, but in reality, repeated patches may be visited. In this case, each visited patch is considered a different patch and added to the convolution process. Figure 4 The schematic diagram for explaining the repeated access provided by the present invention is as follows: Figure 4 As shown, in each figure, the patches filled with diagonal lines are visited only once, and the patches filled with vertical squares are visited twice, because these patches are visited separately through the patches filled with diagonal lines on both sides during the depth-first search.

[0083] The above are all convolutions with a hole size of 1. When the hole is larger, that is, d>1, it means that the distance between adjacent convolution patches is small, and the empty patches in the middle do not participate in the convolution operation, such as Figure 2 As shown in the third figure from the left in the figure, the unfilled patches do not participate in the convolution operation, and the patches filled with diagonal lines are the neighbors of the center patch. For reference, the image convolution with a hole is set as Figure 3 As shown in the third figure from the left in the figure, it is worth noting that setting the hole will not change the number of patches in the neighborhood. The farthest patch in the neighborhood is 1.33 meters away from the center patch.

[0084] Figure 5 This is an example diagram of convolution operation under complex convolution kernel size and hole combination provided by the present invention, such as Figure 5As shown, two of the figures show more complex combinations of convolution kernel sizes and holes.

[0085] Based on the above embodiment, in the method, the preset measurement rule is to measure the zigzag route obtained by extending the route in a counterclockwise direction using a zig pace and in a clockwise direction using a zag pace.

[0086] Specifically, the distances between neighboring patches in the neighborhood are measured along a zigzag path. Figure 6 The schematic diagram of the zigzag cavity provided by the present invention is as follows: Figure 6 As shown in the figure, the small triangle filled with oblique squares is the center patch, the small triangle filled with oblique lines is the neighbor patch, and the small triangle filled with vertical lines is the previous patch of the neighbor patch on the zigzag route. The zigzag route calculation process includes two steps, zig and zag, which extend in counterclockwise and clockwise directions respectively. The total number of zig and zag is the zigzag distance.

[0087] Based on the above embodiment, in the method, performing a convolution operation on the calculation patch set in a patch order-invariant convolution manner specifically includes:

[0088] Sorting the computational facet set of the central facet in depth-first order;

[0089] For the m-1th layer feature vector of any input central face patch, the sum of the m-1th layer feature vector differences between the central face patch and the surrounding face patches is determined based on the sorting as the m-th layer central difference term, and the sum of the m-1th layer feature vector differences between the surrounding face patches before and after the sorting is determined based on the sorting as the m-th layer annular difference term;

[0090] The m-th layer feature vector of any central face patch is determined based on the m-th layer center difference term, the m-th layer annular difference term, the m-1-th layer feature vector of any central face patch and the m-1-th layer feature vector corresponding to the surrounding face patches.

[0091] Specifically, after determining all the patches in the neighborhood of any central patch that participates in the convolution operation of the mth layer patch convolution layer, the items included in the convolution operation are set, wherein, in order to extract the difference between the m-1th layer feature vectors between any central patch and all other patches in the neighborhood, the m-1th layer center difference item is set, and in order to extract the difference between the m-1th layer feature vectors between all other patches in the neighborhood, the m-1th layer annular difference item is set. The reason why it is called annular difference item is that the surrounding patches of any central patch are first sorted in depth-first order, and then based on the sorting, the sum of the differences of the m-1th layer feature vectors corresponding to the surrounding patches before and after the sorting is determined as the annular difference item. Finally, the convolution result of any central patch, the m-1th layer center difference item, the m-1th layer annular difference item, the m-1th layer feature vector of any central patch, and the m-1th layer feature vector corresponding to the surrounding patches are determined. Since the convolution kernel size and the hole determine the convolution mode, that is, the set of faces participating in the convolution, and further, there is no order of faces in the triangular mesh, a convolution calculation method with unchanged face order is proposed.

[0092] Based on the above embodiment, in the method, the sum of the m-1th layer feature vector differences corresponding to the central face patch and the surrounding face patch based on the sorting is determined as the m-th layer center difference term, and the sum of the m-1th layer feature vector differences corresponding to the surrounding face patch before and after the sorting is determined based on the sorting as the m-th layer annular difference term, specifically includes:

[0093] The m-th layer center difference of any center patch i is determined by the following formula and the mth layer annular difference term of any central patch i

[0094]

[0095] in, is the m-1th layer feature vector of any central facet i, and the m-1th layer feature vectors corresponding to the surrounding faces of any central facet i after being arranged in depth-first order are respectively n is the total number of surrounding patches, abs(·) is the absolute value function;

[0096] Correspondingly, determining the m-th layer feature vector of any central face patch based on the m-th layer center difference term, the m-th layer annular difference term, the m-1-th layer feature vector of any central face patch, and the m-1-th layer feature vector corresponding to the surrounding face patches specifically includes:

[0097] The m-1th layer feature vector of any central patch i is calculated by the following formula after the m-1th layer feature vector passes through the mth layer patch convolution layer and the output m-th layer feature vector conv(fi ):

[0098]

[0099] in, is the m-th layer center difference term of any center patch i, is the m-th layer annular difference term of any central facet i, and w0, w1, w2 and w3 are all preset weight matrices.

[0100] Specifically, if the m-1th layer feature vector of any central patch i is The surrounding faces are arranged in depth-first order, and their m-1th layer feature vectors are n is the total number of neighboring patches. In particular, let The calculation formula contains four order-independent terms, which are the m-1th layer feature vector of any central patch The sum of the features of the neighboring patches The sum of the differences between the features of the center patch and the neighboring patches The sum of the differences between the anterior and posterior terms after the neighboring patches are arranged into a ring These four items are independent of the specific order of the patches; the convolution kernel contains 4 preset weight matrices, namely w0, w1, w2, and w3. The convolution calculation result is the weighted sum of the above four items and the parameters. The overall formula is:

[0101]

[0102] Since the central difference term and the annular difference term are independent of the order of the input facets, such a calculation method makes the output feature vector independent of the order of the input facets.

[0103] Here is an explanation of how to set the preset parameter matrix. The parameter matrix can be set manually. For example, if w0≠0, w1=w2=w3=0 is set, then after passing through a layer of patch convolution layers, the output feature vector only extracts the input features of the center patch. If w1≠0, w0=w0=w0=0 is set, then after passing through a layer of patch convolution layers, the output feature vector extracts a smooth representation of the input features of the center patch and the surrounding patches. The parameter matrix can also be learned through machine learning. After setting the learning goal, such as classifying the three-dimensional model, the parameter matrix can be obtained through a large amount of three-dimensional model data and category annotations, and the loss function is reversely conducted. At this time, the convolution output features represent the three-dimensional model category. If the three-dimensional model is segmented, the output features represent the part of the three-dimensional model to which this patch belongs.

[0104] Based on this convolution, a convolutional neural network on a triangular mesh can be constructed. The neural network may contain multiple facet convolution layers. By aggregating neighborhood features and extracting local features, the learning of the features of the triangular mesh facets of the three-dimensional model or the entire three-dimensional model can be realized. When the convolutional neural network is applied to the classification of three-dimensional models, the neural network will include multiple facet convolution layers, global feature pooling layers and classification fully connected layers. The training goal of the corresponding neural network is to obtain the category of the object corresponding to the three-dimensional model given a three-dimensional model represented by a triangular mesh, such as distinguishing a furniture model as a table, chair or sofa. Specifically, the neural network may include multiple convolutional layers, and the convolutional layers perform the above convolution operations to achieve the aggregation and transformation of the input features. The features of the multiple facets finally obtained can obtain the feature vector of the entire mesh by taking the mean of all facets for each feature dimension, and then obtain the category of the input mesh through the fully connected layer.

[0105] Based on the above embodiments, the present invention further provides a deep convolutional neural network for three-dimensional grids, comprising the feature extraction module and the specific task module as described in any of the above embodiments,

[0106] During training,

[0107] The feature extraction module extracts the sample feature vector of each facet of the input sample triangular mesh model and outputs it to the specific task module;

[0108] The specific task module processes the sample feature vector according to specific rules to obtain a prediction result;

[0109] A loss function is constructed based on the prediction results and the facet labels with a preset learning objective; wherein the preset learning objective is mesh classification or triangular mesh model physical type recognition.

[0110] Specifically, for example, if the preset learning goal is an object category recognition method, then the object category recognition method includes the following steps: determining the original three-dimensional mesh of the object to be recognized; inputting the original three-dimensional mesh into a classification model, and outputting the type of the object to be recognized; wherein the classification model is trained based on the original three-dimensional mesh of the sample and the corresponding object type label, and the classification model includes a feature extraction module, a feature fusion module, and a classification fully connected layer as described in any of the above embodiments, which are connected in sequence, and the feature extraction module performs a convolution operation on the faces in the mesh according to preset rules. In the object category recognition method, during the classification model training process, the feature extraction module performs convolution calculation on the features of each facet in the input sample original three-dimensional grid according to preset rules, obtains the feature vector of each facet and outputs it to the feature fusion module; the feature fusion module fuses the feature vector of each facet according to preset fusion rules, obtains the overall feature vector of the sample original three-dimensional grid and outputs it to the classification fully connected layer; the classification fully connected layer determines the predicted object type corresponding to the sample original three-dimensional grid based on the overall feature vector; wherein the predicted object type is used to construct the loss function during the classification model training, and convolution calculation is performed on any facet according to preset rules to aggregate the neighborhood facet features of any facet to extract the local features of any facet.

[0111] The following is a description of the three-dimensional model feature extraction device using patch convolution provided by the present invention. The three-dimensional model feature extraction device using patch convolution described below and the three-dimensional model feature extraction method using patch convolution described above can be referenced to each other.

[0112] Figure 7 A schematic diagram of the structure of a three-dimensional model feature extraction device using facet convolution provided by the present invention, such as Figure 7 As shown, the device includes an input unit 710 and an output unit 720, wherein:

[0113] The input unit 710 is used to determine a triangular mesh representation of a three-dimensional model of features to be extracted; wherein the triangular mesh representation is a three-dimensional shape composed of a plurality of triangular facets for describing an entity or scene;

[0114] The output unit 720 is used to input the triangular mesh of the three-dimensional model into a feature extraction module, and output a feature vector of each facet of the triangular mesh;

[0115] The feature extraction module is composed of one or more facet convolution layers, and the facet convolution layer performs a convolution operation on each facet in the input triangular mesh according to a preset rule.

[0116] The device provided by the present invention determines the triangular mesh representation of the three-dimensional model of the feature to be extracted; wherein the triangular mesh representation is composed of multiple triangular facets for depicting the three-dimensional shape of the entity or scene; the triangular mesh of the three-dimensional model is input into the feature extraction module, and the feature vector of each facet of the triangular mesh is output; wherein the feature extraction module is composed of one or more facet convolution layers. Since the facet convolution layer performs convolution operation on each facet in the input triangular mesh according to a preset rule, a convolution operation rule for an irregular data structure such as a triangular mesh is provided, and the features of the three-dimensional model can be extracted for the three-dimensional model to be applied to specific applications such as three-dimensional model classification, segmentation, retrieval, shape correspondence, etc. By setting the convolution operation mode of the triangular mesh, a process similar to image convolution is realized, and the previous method cannot provide a regular and flexible triangular mesh convolution mode, which can support traditional machine learning and deep learning, and is applied to three-dimensional model classification, segmentation, retrieval and shape correspondence, etc., and can improve the high accuracy and robustness of the learning system.

[0117] Based on the above embodiment, in the device, the feature extraction module is composed of one or more patch convolution layers, specifically:

[0118] If the feature extraction module is composed of a patch convolution layer, the patch convolution layer performs a convolution operation on the input triangular mesh according to a preset rule based on the shape description and the posture description of each patch, and outputs a feature vector of each patch of the triangular mesh;

[0119] If the feature extraction module is composed of multiple patch convolution layers, the m-th layer patch convolution layer will convolve the m-1th layer feature vector of each input patch according to the preset rule, and output the m-th layer feature vector of each patch to the m+1th layer patch convolution layer, and the feature extraction module outputs the L-th layer feature vector; wherein m=1, 2, ..., L, L is the total number of layers in the multi-layer patch convolution layer, and the zeroth layer feature vector is composed of the patch shape description and the patch pose description of the triangular mesh.

[0120] Based on the above embodiment, in the device, performing a convolution operation on the input m-1th layer feature vector of each face patch according to the preset rule to output the mth layer feature vector of each face patch specifically includes:

[0121] Traverse each face in the input triangular mesh, and calculate the m-th layer feature vector of any face as the center face in the traversal order and output it;

[0122] Wherein, for any central patch, a set of computational patches participating in the convolution is determined based on a preset size and a preset hole of the convolution kernel;

[0123] The calculation patch set is convolved in a patch order-invariant convolution manner.

[0124] Based on the above embodiment, in the device, determining the set of computational patches participating in the convolution based on the preset size and preset hole of the convolution kernel specifically includes:

[0125] If the convolution kernel size k and the hole d both satisfy k = 3 and d = 1, the neighborhood range of the center patch is its three adjacent patches;

[0126] If the convolution kernel size k and the hole d both satisfy k>3 and d>1, the maximum distance from the center patch to the center patch in the neighborhood is set to The interval distance between adjacent facets in the neighborhood is determined by a preset measurement rule, and the facets included in the neighborhood range are determined based on the maximum distance and the interval distance.

[0127] Based on the above embodiment, in the device, the preset measurement rule is to measure the zigzag route obtained by extending the route counterclockwise with a zig pace and clockwise with a zag pace.

[0128] Based on the above embodiment, in the device, the convolution operation is performed on the calculation patch set in a patch order-invariant convolution manner, specifically including:

[0129] Sorting the computational facet set of the central facet in depth-first order;

[0130] For the m-1th layer feature vector of any input central face patch, the sum of the m-1th layer feature vector differences between the central face patch and the surrounding face patches is determined based on the sorting as the m-th layer central difference term, and the sum of the m-1th layer feature vector differences between the surrounding face patches before and after the sorting is determined based on the sorting as the m-th layer annular difference term;

[0131] The m-th layer feature vector of any central face patch is determined based on the m-th layer center difference term, the m-th layer annular difference term, the m-1-th layer feature vector of any central face patch and the m-1-th layer feature vector corresponding to the surrounding face patches.

[0132] Based on the above embodiment, in the device, the sum of the m-1th layer feature vector differences corresponding to the central face patch and the surrounding face patch based on the sorting is determined as the m-th layer center difference term, and the sum of the m-1th layer feature vector differences corresponding to the surrounding face patch before and after the sorting is determined based on the sorting as the m-th layer annular difference term, specifically includes:

[0133] The m-th layer center difference of any center patch i is determined by the following formula and the mth layer annular difference term of any central patch i

[0134]

[0135] in, is the m-1th layer feature vector of any central facet i, and the m-1th layer feature vectors corresponding to the surrounding faces of any central facet i after being arranged in depth-first order are respectively n is the total number of surrounding patches, abs(·) is the absolute value function;

[0136] Correspondingly, determining the m-th layer feature vector of any central face patch based on the m-th layer center difference term, the m-th layer annular difference term, the m-1-th layer feature vector of any central face patch, and the m-1-th layer feature vector corresponding to the surrounding face patches specifically includes:

[0137] The m-1th layer feature vector of any central patch i is calculated by the following formula after the m-1th layer feature vector passes through the mth layer patch convolution layer and the output m-th layer feature vector conv(f i ):

[0138]

[0139] in, is the m-th layer center difference term of any center patch i, is the m-th layer annular difference term of any central facet i, and w0, w1, w2 and w3 are all preset weight matrices.

[0140] Figure 8 A schematic diagram of the physical structure of an electronic device provided by the present invention, such as Figure 8 As shown, the electronic device may include: a processor 810, a communication interface 820, a memory 830 and a communication bus 840, wherein the processor 810, the communication interface 820 and the memory 830 communicate with each other through the communication bus 840. The processor 810 may call the logic instructions in the memory 830 to execute a three-dimensional model feature extraction method using facet convolution, the method comprising: determining a triangular mesh representation of a three-dimensional model of features to be extracted; wherein the triangular mesh representation is a three-dimensional shape composed of a plurality of triangular facets for describing an entity or scene; inputting the triangular mesh of the three-dimensional model into a feature extraction module, and outputting a feature vector of each facet of the triangular mesh; wherein the feature extraction module is composed of one or more facet convolution layers, and the facet convolution layer performs a convolution operation on each facet in the input triangular mesh according to a preset rule.

[0141] In addition, the logic instructions in the above-mentioned memory 830 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0142] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the three-dimensional model feature extraction method using facet convolution provided by the above-mentioned methods, and the method includes: determining a triangular mesh representation of a three-dimensional model of features to be extracted; wherein the triangular mesh representation is a three-dimensional shape composed of multiple triangular faces for depicting an entity or scene; inputting the triangular mesh of the three-dimensional model into a feature extraction module, and outputting a feature vector of each facet of the triangular mesh; wherein the feature extraction module is composed of one or more facet convolution layers, and the facet convolution layers perform convolution operations on each facet in the input triangular mesh according to preset rules.

[0143] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the three-dimensional model feature extraction method using facet convolution provided by the above-mentioned methods, the method comprising: determining a triangular mesh representation of a three-dimensional model of features to be extracted; wherein the triangular mesh representation is composed of a plurality of triangular facets for depicting a three-dimensional shape of an entity or scene; inputting the triangular mesh of the three-dimensional model into a feature extraction module, and outputting a feature vector of each facet of the triangular mesh; wherein the feature extraction module is composed of one or more facet convolution layers, and the facet convolution layers perform convolution operations on each facet in the input triangular mesh according to preset rules.

[0144] The server embodiment described above is merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, i.e., they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0145] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0146] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A three-dimensional model feature extraction method using patch convolution, characterized in that: include: Determine a triangular mesh representation of a three-dimensional model of features to be extracted; wherein the triangular mesh representation is a three-dimensional shape composed of a plurality of triangular facets for depicting an entity or scene; Inputting the triangular mesh of the three-dimensional model into a feature extraction module, and outputting a feature vector of each facet of the triangular mesh; The feature extraction module is composed of one or more facet convolution layers, and the facet convolution layer performs a convolution operation on each facet in the input triangular mesh according to a preset rule; The feature extraction module is composed of one or more patch convolution layers, specifically including: If the feature extraction module is composed of a patch convolution layer, the patch convolution layer performs a convolution operation on the input triangular mesh based on the shape description and the posture description of each patch according to a preset rule, and outputs a feature vector of each patch of the triangular mesh; If the feature extraction module is composed of multiple patch convolution layers, the m-th patch convolution layer performs a convolution operation on the m-1th layer feature vector of each input patch according to the preset rule, and outputs the m-th layer feature vector of each patch to the m+1th layer patch convolution layer, and the feature extraction module outputs the L-th layer feature vector; wherein m=1,2,…,L, L is the total number of layers in the multiple patch convolution layers, and the zeroth layer feature vector is composed of the patch shape description and patch pose description of the triangular mesh; The step of performing a convolution operation on the inputted m-1th layer feature vector of each face patch according to the preset rule and outputting the mth layer feature vector of each face patch specifically includes: Traverse each face in the input triangular mesh, and calculate the m-th layer feature vector of any face as the center face in the traversal order and output it; Wherein, for any central patch, a set of computational patches participating in the convolution is determined based on a preset size and a preset hole of the convolution kernel; The calculation patch set is convolved in a patch order-invariant convolution manner.

2. The three-dimensional model feature extraction method using patch convolution according to claim 1, characterized in that: The step of determining a set of computational patches participating in the convolution based on a preset size and a preset hole of the convolution kernel specifically includes: If the convolution kernel size k and the hole d both satisfy k=3 and d=1, the neighborhood range of the center patch is its three adjacent patches; If the convolution kernel size k and the hole d both satisfy k>3 and d>1, the maximum distance from the center patch to the center patch in the neighborhood is set to , determine the interval distance between adjacent facets in the neighborhood using a preset measurement rule, and determine the facets included in the neighborhood range based on the maximum distance and the interval distance.

3. The three-dimensional model feature extraction method using patch convolution according to claim 2, characterized in that: The preset measurement rule is to measure the zigzag route obtained by extending the route in a counterclockwise direction using a zig pace and in a clockwise direction using a zag pace.

4. The three-dimensional model feature extraction method using patch convolution according to claim 1, characterized in that: The convolution operation is performed on the calculation face set in a face order-invariant convolution manner, specifically comprising: Sorting the computational facet set of the central facet in depth-first order; For the m-1th layer feature vector of any input central face patch, the sum of the m-1th layer feature vector differences between the central face patch and the surrounding face patches is determined based on the sorting as the m-th layer central difference term, and the sum of the m-1th layer feature vector differences between the surrounding face patches before and after the sorting is determined based on the sorting as the m-th layer annular difference term; The m-th layer feature vector of any central face patch is determined based on the m-th layer center difference term, the m-th layer annular difference term, the m-1-th layer feature vector of any central face patch and the m-1-th layer feature vector corresponding to the surrounding face patches.

5. The three-dimensional model feature extraction method using patch convolution according to claim 4, characterized in that: The step of determining the sum of the m-1th layer feature vector differences between the central face patch and the surrounding face patches based on the sorting as the m-th layer central difference term, and determining the sum of the m-1th layer feature vector differences between the surrounding face patches before and after the sorting as the m-th layer annular difference term based on the sorting specifically includes: The m-th layer center difference of any center patch i is determined by the following formula and the mth layer annular difference term of any central patch i : ； in, is the m-1th layer feature vector of any central facet i, and the m-1th layer feature vectors corresponding to the surrounding faces of any central facet i after being arranged in depth-first order are respectively , is the total number of surrounding patches, , To find the absolute value function; Correspondingly, determining the m-th layer feature vector of any central face patch based on the m-th layer center difference term, the m-th layer annular difference term, the m-1-th layer feature vector of any central face patch, and the m-1-th layer feature vector corresponding to the surrounding face patches specifically includes: The m-1th layer feature vector of any central patch i is calculated by the following formula: : ； in, is the m-th layer center difference term of any center patch i, is the mth layer annular difference term of any central patch i, All are preset weight matrices.

6. A three-dimensional model feature extraction device using patch convolution, characterized in that: include: An input unit, used to determine a triangular mesh representation of a three-dimensional model of features to be extracted; wherein the triangular mesh representation is a three-dimensional shape composed of a plurality of triangular facets for describing an entity or scene; An output unit, used for inputting the triangular mesh of the three-dimensional model into a feature extraction module, and outputting a feature vector of each facet of the triangular mesh; The feature extraction module is composed of one or more facet convolution layers, and the facet convolution layer performs a convolution operation on each facet in the input triangular mesh according to a preset rule; The feature extraction module is composed of one or more patch convolution layers, specifically including: If the feature extraction module is composed of a patch convolution layer, the patch convolution layer performs a convolution operation on the input triangular mesh based on the shape description and the posture description of each patch according to a preset rule, and outputs a feature vector of each patch of the triangular mesh; If the feature extraction module is composed of multiple patch convolution layers, the m-th patch convolution layer performs a convolution operation on the m-1th layer feature vector of each input patch according to the preset rule, and outputs the m-th layer feature vector of each patch to the m+1th layer patch convolution layer, and the feature extraction module outputs the L-th layer feature vector; wherein m=1,2,…,L, L is the total number of layers in the multiple patch convolution layers, and the zeroth layer feature vector is composed of the patch shape description and patch pose description of the triangular mesh; The step of performing a convolution operation on the inputted m-1th layer feature vector of each face patch according to the preset rule and outputting the mth layer feature vector of each face patch specifically includes: Traverse each face in the input triangular mesh, and calculate the m-th layer feature vector of any face as the center face in the traversal order and output it; Wherein, for any central patch, a set of computational patches participating in the convolution is determined based on a preset size and a preset hole of the convolution kernel; The calculation patch set is convolved in a patch order-invariant convolution manner.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the three-dimensional model feature extraction method using patch convolution are implemented as described in any one of claims 1 to 5.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the three-dimensional model feature extraction method using patch convolution as described in any one of claims 1 to 5 are implemented.

Images