Feature map acquisition method and system, three-dimensional shape processing method, terminal and medium
By introducing standardized isovariant capsule networks into standardized isovariant networks, processing local neighborhood logarithmic mapping and parallel transmission of three-dimensional grid data, and generating feature maps, the problem of limited generalization ability in the existing technology is solved, and the problem of realization is realized High generalization and accurate classification on different three-dimensional grid structures and vertices.
Patent Information
- Application Number
- CN202411211527.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-30
- Publication Date
- 2024-11-26
AI Technical Summary
When handling irregular three-dimensional shapes, the existing standard isovariant network abandons strict continuous SO(2) and other variable constraints in order to pursue better representation capabilities, resulting in the generalization ability on different three-dimensional grid structures and vertices. limit.
A feature map acquisition method is designed to obtain three-dimensional grid data, logarithmic mapping and parallel transmission processing of local neighborhoods are performed, and input it into a standard isovariant capsule network to generate feature maps. The network includes an initial pose extractor layer, a standardized and so on-variable residual block layer, a pooling layer, an anti-pooling layer and a category capsule layer to ensure continuous SO(2) and other denaturation.
It realizes high generalization capabilities in different specifications, different three-dimensional grid structures, and different vertex positions, improves the performance and classification accuracy of three-dimensional shape processing, and meets the needs of geometric deep learning.
Smart Images

Figure CN119027594A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of representation learning and three-dimensional shape processing, and in particular to a feature map acquisition method, system, three-dimensional shape processing method, terminal and medium. Background Art
[0002] With the development of deep learning, symmetric and equivariant network structures have achieved great success in practical applications. One of the representative structures is the translation-equivariant planar convolutional neural network in image processing technology. However, extending the planar convolutional neural network from images to irregular three-dimensional shapes (manifolds) faces great challenges. The geometric deep learning that has emerged in recent years aims to extend the planar convolutional neural network to manifolds and design equivariant neural networks on manifolds. Equivariant neural networks that process regular data are usually equivariant to the global symmetry of the data distribution space. For example, images can be regarded as functions on a uniformly sampled two-dimensional grid in Euclidean space, and spherical signals are distributed in homogeneous space. Unlike regular data, generalized manifolds do not have global symmetry, but have local canonical symmetry. Therefore, it is necessary to use the local canonical symmetry of the manifold to design equivariant neural networks to process irregular three-dimensional shapes.
[0003] In response to the above technical requirements, researchers have proposed solutions. For example, Wiersma et al. published an article titled "CNNs on Surfaces using Rotation-Equivariant Features" in the 2020 ACM Transactions on Graphics (ToG) journal. Based on complex irreducible representations, they proposed a convolution filter that is equivariant to gauge changes in the continuous rotation group SO(2), but the representation capability is limited. For example, De Haan et al. published an article titled "Gauge Equivariant Mesh CNNs Anisotropic Convolutions on Geometric Graphs" at the 2021 International Conference on Learning Representations (ICLR) conference. Based on real irreducible representations, they applied anisotropic gauge equivariant convolution kernels to extend graph convolution networks to three-dimensional mesh data. However, due to the use of regular nonlinearity, the entire network only satisfies C N(Rotation on integer multiples of discrete angles 2π / N) is equivariant. For another example, He et al. published an article titled "Gauge Equivariant Transformer" at the 2021 Neural Information Processing Systems (NeurIPS) conference, introducing the attention mechanism into the gauge equivariant network. However, due to the use of regular representation as the feature domain type, the entire network only satisfies C N Equivariation.
[0004] In general, existing canonical equivariant network techniques often abandon the strict continuous SO(2) equivariance constraint in pursuit of better representation capabilities, which limits the generalization ability of the network on different 3D mesh structures and different vertices. Therefore, new methods are needed that have higher representation performance while ensuring continuous SO(2) equivariance. Summary of the invention
[0005] In view of the defects in the prior art, the purpose of the present invention is to provide a feature map acquisition method, system and three-dimensional shape processing method, terminal and medium.
[0006] According to a first aspect of the present invention, a method for acquiring a feature map is provided, comprising:
[0007] Get three-dimensional mesh data;
[0008] Preprocessing the three-dimensional grid data, calculating and obtaining the logarithmic mapping of the local neighborhood of each vertex, and the parallel transmission between the local neighborhood center node and the neighboring nodes;
[0009] The three-dimensional coordinates of each vertex, the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood are input into the canonical equivariant capsule network to generate a feature map.
[0010] Preferably, the canonical equivariant capsule network comprises an initial pose extractor layer, a canonical equivariant residual block layer, a canonical equivariant pooling layer, a canonical equivariant unpooling layer and a canonical equivariant class capsule layer; wherein:
[0011] One or more of the residual block layers form a first combined layer located after the initial pose extractor layer;
[0012] One of the pooling layers and at least one of the residual block layers forms a second combination layer, which is located behind the first combination layer; the second combination layer is stacked one or more times;
[0013] One of the anti-pooling layers and at least one of the residual block layers forms a third combination layer, which is located behind the last stacked second combination layer; the third combination layer is stacked one or more times;
[0014] The category capsule layer is located behind the last third combined layer in the stack.
[0015] Preferably, the input of the initial pose extractor layer is the three-dimensional coordinates of each vertex, the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood, and the output is the initial capsule;
[0016] The input of the residual block layer is the output capsule of the initial pose extractor layer, or the output capsule of the adjacent residual block layer, or the output capsule of the pooling layer, or the output capsule of the unpooling layer, as well as the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood, and the output is a new capsule;
[0017] The input of the pooling layer is the output capsule of the adjacent previous residual block layer, the logarithmic mapping of the local neighborhood, the parallel transmission between the central node and the neighboring nodes in the local neighborhood, and the output is a new capsule;
[0018] The input of the depooling layer is the output capsule of the adjacent previous residual block layer, and the output is a new capsule;
[0019] The input of the category capsule layer is the output capsule of the adjacent previous residual block, the logarithmic mapping of the local neighborhood, the parallel transmission between the central node and the neighboring nodes in the local neighborhood, and the output is the category capsule.
[0020] Preferably, the initial pose extractor comprises a canonical equivariant capsule basic unit;
[0021] The residual block layer includes two canonical equivariant capsule basic units and a residual connection;
[0022] The pooling layer selects representative points by sampling the farthest point, and aggregates non-sampled point capsules to the representative points using a canonical equivariant capsule basic unit;
[0023] The anti-pooling layer sets the features of non-representative points to zero vectors;
[0024] The class capsule layer is a canonical equivariant capsule basic unit.
[0025] Preferably, for each of the gauge equivariant capsule units, for any central node p on the three-dimensional grid,
[0026] The input is: N input capsules The input capsule is the three-dimensional coordinates of each vertex or the initial capsule or the output capsule of the previous gauge equivariant capsule basic unit, as well as the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood;
[0027] The output is: N' output capsules The output capsule serves as the input capsule of the next gauge equivariant capsule basic unit or the final output feature map.
[0028] Preferably, the output capsule It is obtained by the input capsule of point p and its neighborhood points, specifically:
[0029] For any neighbor node q in the neighborhood of the central node p, obtain q on the tangent plane of p relative to the norm ω p Polar coordinate representation (r q ,θ q );
[0030] For the i-th input capsule of neighbor node q Obtain the pose vector f′(q i );
[0031] According to the input capsule of neighbor node q and the output capsule of the central node p The parameter matrix W between ij (r q ,θ q ), get the input capsule Output Capsule Prediction
[0032]
[0033] Parameter matrix W ij (r q ,θ q ) satisfies the constraints of gauge equivariance;
[0034] Input capsule of the i-th neighbor node q The jth output capsule with the central node p The spatial consistency coefficient between and the i-th input capsule of the central node p With the jth output capsule Capsule consistency coefficient Initialized to 0;
[0035] According to the spatial consistency coefficient and capsule consistency coefficient Calculation of spatial coupling coefficient and capsule coupling coefficient Weighted average output pose vector And update the spatial consistency coefficient and capsule consistency coefficient Repeat N rGet the output capsule's pose vector and norm
[0036] Preferably, the spatial consistency coefficient and capsule consistency coefficient Calculation of spatial coupling coefficient and capsule coupling coefficient Weighted average output pose vector And update the spatial consistency coefficient and capsule consistency coefficient Repeat N r Get the output capsule's pose vector and norm include:
[0037] Initialize spatial consistency coefficient Initialize capsule consistency coefficient
[0038] Get the spatial coupling coefficient:
[0039] Get the capsule coupling coefficient:
[0040] Compute the capsule at any point p on a 3D grid Input
[0041]
[0042] Get the capsule The pose vector
[0043]
[0044] Update spatial consistency coefficient:
[0045]
[0046] Update capsule consistency coefficient:
[0047]
[0048] Repeat the above steps N r Second, return the pose vector and norm
[0049] Preferably, the canonical equivariant residual block layer comprises:
[0050] The capsule generated by the initial pose extractor or the previous residual block layer The capsule is generated by two standard equivariant capsule basic units. The capsule and The output capsule is obtained through the residual connection
[0051] Among them, the number of iterations N of the first gauge equivariant capsule basic unit is r , fixed to 1; the number of iterations of the second gauge equivariant capsule basic unit is an adjustable hyperparameter.
[0052] Preferably, the canonical equivariant pooling layer selects representative points by sampling the farthest point, and aggregates non-sampling point capsules onto representative points using canonical equivariant capsule basic units, specifically:
[0053] Use the farthest point sampling method to obtain representative points from the vertex sampling of the triangle mesh;
[0054] Obtain the pooled neighborhood of each representative point according to the principle of the closest geodesic distance The pooling neighborhood The capsule of the non-representative point inside is used as the input capsule;
[0055] The input capsules composed of capsules of non-representative points are aggregated to the representative point p through a canonical equivariant capsule basic unit, and the output capsules after the canonical equivariant pooling operation on the representative point are obtained.
[0056] According to a second aspect of the present invention, there is provided a feature map acquisition system, comprising:
[0057] Data acquisition module: obtain three-dimensional grid data;
[0058] Data processing pre-module: pre-processing the three-dimensional grid data, calculating and obtaining the logarithmic mapping of the local neighborhood of each vertex, and the parallel transmission between the local neighborhood center node and the neighboring nodes;
[0059] The gauge equivariant data processing module inputs the three-dimensional coordinates of each vertex, the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood into the gauge equivariant capsule network to generate a feature map.
[0060] According to a third aspect of the present invention, there is provided a three-dimensional shape processing method, comprising:
[0061] Obtain a feature map by using any one of the feature map acquisition methods or the feature map acquisition system;
[0062] Performing three-dimensional shape processing based on the feature map, wherein the three-dimensional shape processing includes shape segmentation, shape classification or spherical image classification;
[0063] When the three-dimensional shape is processed as shape segmentation, a boundary loss function is calculated based on the generated feature map and the label information carried by the training data. The label information is the category information of each manually labeled vertex. The network parameters of the canonical equivariant capsule network are continuously updated according to the boundary loss function until the network model converges to obtain the shape segmentation result.
[0064] When the three-dimensional shape is processed as shape classification, a boundary loss function is calculated based on the feature map after the global average pooling layer and the label information carried by the training data. The label information is the category information of each manually annotated three-dimensional shape. The network parameters of the canonical equivariant capsule network are continuously updated according to the boundary loss function until the network model converges to obtain the shape classification result.
[0065] When the three-dimensional shape is processed as spherical image classification, the boundary loss function is calculated based on the feature map after the global average pooling layer and the label information carried by the training data. The label information is the category information of each manually annotated spherical image. The network parameters of the canonical equivariant capsule network are continuously updated according to the boundary loss function until the network model converges to obtain the spherical image classification result.
[0066] According to a fourth aspect of the present invention, there is provided a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute any one of the methods or systems described above when executing the program.
[0067] According to a fifth aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, can be used to execute any one of the methods or systems described.
[0068] Compared with the prior art, the embodiments of the present invention have at least one of the following beneficial effects:
[0069] The feature map acquisition method and system provided by the embodiments of the present invention achieve higher performance while ensuring the canonical equivariance of continuous SO(2), thereby ensuring the generalization ability of the network in different canonicals, different three-dimensional grid structures, and different vertex positions; the extracted feature map can be used in the fields of shape classification, shape segmentation, spherical image classification, etc., meeting the current needs of geometric deep learning three-dimensional shape processing.
[0070] The feature map acquisition method and system provided by the embodiment of the present invention, compared with the standard non-canonical equivariant deep neural network method, ensure better performance while having a higher ability to generalize to different three-dimensional grid structures.
[0071] Compared with the existing canonical equivariant neural network, the feature map acquisition method and system provided by the embodiment of the present invention can ensure better performance while theoretically satisfying the canonical variation equivariance in continuous SO(2), thereby ensuring the generalization of the network in different canonical, three-dimensional grid structures, and different vertex positions.
[0072] The feature map acquisition method and system provided by the embodiments of the present invention improve the classification accuracy in spherical image classification tasks compared to existing canonical equivariant neural networks and spherical domain deep neural networks. BRIEF DESCRIPTION OF THE DRAWINGS
[0073] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:
[0074] Figure 1 A flowchart of a method for acquiring a feature map provided by a preferred embodiment of the present invention;
[0075] Figure 2 Schematic diagram of the framework of a canonical equivariant capsule network in a preferred embodiment of the present invention;
[0076] Figure 3 A schematic diagram of the principle of establishing a canonical equivariant capsule network in a preferred embodiment of the present invention;
[0077] Figure 4 This is a flow chart of the standard isovariant capsule basic unit structure in a preferred embodiment of the present invention. DETAILED DESCRIPTION
[0078] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention. The parts not described in detail below can be implemented using existing technologies.
[0079] Figure 1 This is a flow chart of a method for obtaining a feature map provided by a preferred embodiment of the present invention. The method for obtaining a feature map provided by this embodiment may include the following steps:
[0080] S1, obtain three-dimensional grid data.
[0081] S2, preprocesses the three-dimensional grid data, calculates the logarithmic mapping of the local neighborhood of each vertex, and the parallel transmission between the local neighborhood center node and the neighboring nodes.
[0082] S3, input the three-dimensional coordinates of each vertex, the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood into the gauge equivariant capsule network to generate a feature map. The output feature map is the output capsule of the last gauge equivariant capsule basic unit.
[0083] The above-mentioned embodiments ensure the canonical equivariance of continuous SO(2) while achieving higher performance, thereby ensuring the generalization ability of the network in different canonicals, different three-dimensional mesh structures, and different vertex positions. It can be used in the fields of shape classification, shape segmentation, spherical image classification, etc., and meets the current needs of geometric deep learning three-dimensional shape processing.
[0084] In a preferred embodiment of the present invention, the three-dimensional mesh data is preprocessed using the Vector Heat method, which is a method for efficiently computing tangent vectors on a surface or more generally parallel transmission of vector-valued data on a meandering manifold.
[0085] In order to better extract feature maps, a preferred embodiment of the present invention provides a preferred structure based on a canonical equivariant capsule network, specifically, it includes: the canonical equivariant capsule network includes an initial pose extractor layer, a canonical equivariant residual block layer, a canonical equivariant pooling layer, a canonical equivariant unpooling layer and a canonical equivariant class capsule layer. Figure 2 As shown, one or more residual block layers constitute a first combination layer located behind the initial pose extractor layer; a pooling layer and at least one residual block layer form a second combination layer, which is located behind the first combination layer; the second combination layer is stacked one or more times; an inverse pooling layer and at least one residual block layer form a third combination layer, which is located behind the stacked second combination layer; the third combination layer is stacked one or more times; and the category capsule layer is located behind the stacked third combination layer.
[0086] Further, in a preferred embodiment, inputs and outputs of each constitutive layer are provided, specifically: the input of the initial pose extractor layer is the three-dimensional coordinates of each vertex, the logarithmic mapping of the local neighborhood, the parallel transmission between the central node and the neighboring nodes in the local neighborhood, and the output is the initial capsule;
[0087] The input of the residual block layer is the output capsule of the initial pose extractor layer, or the output capsule of the previous residual block layer, or the output capsule of the pooling layer, or the output capsule of the unpooling layer, the logarithmic mapping of the local neighborhood, the parallel transmission between the central node and the neighboring nodes in the local neighborhood, and the output of the new capsule;
[0088] The input of the pooling layer is the output capsule of the previous residual block layer, the logarithmic mapping of the local neighborhood, the parallel transmission between the central node and the neighboring nodes in the local neighborhood, and the output of the new capsule;
[0089] The input of the unpooling layer is the output capsule of the adjacent residual block layer, and the output is a new capsule;
[0090] The input of the category capsule layer is the output capsule of the previous residual block, the logarithmic mapping of the local neighborhood, the parallel transmission between the central node and the neighboring nodes in the local neighborhood, and the output category capsule.
[0091] Each layer in the above embodiments is implemented by a canonical equivariant capsule basic unit. Specifically, in a preferred embodiment, the initial pose extractor includes a canonical equivariant capsule basic unit for mapping the three-dimensional coordinate input to an initial capsule; the residual block layer based on the canonical equivariant capsule includes two canonical equivariant capsule basic units and a residual connection; the canonical equivariant pooling operation selects representative points by sampling the farthest point, and uses the canonical equivariant capsule basic unit to aggregate non-sampled point capsules to the representative points; the canonical equivariant unpooling operation samples zero features from representative points to non-representative points; and the category capsule is generated using a canonical equivariant capsule basic unit.
[0092] Based on the structural composition of the canonical equivariant capsule network in the above embodiment, in a preferred embodiment of the present invention, an execution flow of the entire canonical equivariant capsule network is provided, such as Figure 3 As shown, that is, S3 may include the following steps:
[0093] S31, inputting the three-dimensional coordinates, the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood into the initial pose extractor to obtain an initial capsule;
[0094] S32, based on the initial capsule of S31 and the logarithmic mapping of the local neighborhood, the parallel transmission between the central node and the neighboring nodes in the local neighborhood, inputs into the canonical equivariant residual block layer, the canonical equivariant pooling layer and the unpooling layer to obtain a higher-level canonical equivariant capsule;
[0095] S33, based on the higher-level canonical equivariant capsule obtained in S32 and the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood, obtain a category capsule. The category capsule includes a pose vector and the length of the pose vector That is, the feature map is the category capsule, and the feature map includes the pose vector of the category capsule and the length of the pose vector, where the length of the pose vector is the probability that the pose vector belongs to the category.
[0096] In order to ensure that the acquired feature graph has excellent expression ability while maintaining continuous SO(2) equivariance, a preferred embodiment of the present invention proposes a preferred design scheme of a canonical equivariant capsule basic unit, which is specifically described as follows: For any central node p on a three-dimensional grid, the canonical equivariant capsule unit
[0097] The input is: N input capsules The input capsule is the three-dimensional coordinates of each vertex or the initial capsule or the output capsule of the previous gauge equivariant capsule basic unit, as well as the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood;
[0098] The output is: N′ output capsules The output capsule serves as the input capsule of the next gauge equivariant capsule basic unit or the final output feature map.
[0099] Furthermore, in a preferred embodiment, the output capsule is obtained from the input capsules of point p and its neighborhood, that is, the input capsule of neighbor node q Output capsule for the central node p Prediction And the number of iterations N r , output capsule The pose vector and norm Specifically,
[0100] For any neighbor node q in the neighborhood of the central node p, obtain q on the tangent plane of p relative to the norm ω p Polar coordinate representation (r q ,θ q );
[0101] For the i-th input capsule of neighbor node q Obtain the pose vector f′(q i );
[0102] According to the input capsule of neighbor node q and the output capsule of the central node p The parameter matrix W between ij (r q ,θ q ), get the input capsule Output Capsule Prediction
[0103]
[0104] Parameter matrix W ij (r q ,θ q ) satisfies the constraints of gauge equivariance;
[0105] Input capsule of the i-th neighbor node q The jth output capsule with the central node p The spatial consistency coefficient between and the i-th input capsule of the central node p With the jth output capsule Capsule consistency coefficient Initialized to 0;
[0106] According to the spatial consistency coefficient and capsule consistency coefficient Calculation of spatial coupling coefficient and capsule coupling coefficient Weighted average output pose vector And update the spatial consistency coefficient and capsule consistency coefficient Repeat N r Get the output capsule's pose vector and norm
[0107] In order to ensure that the obtained canonical equivariant capsule basic unit has a high representation capability while maintaining the continuous SO(2) equivariance, in a preferred embodiment, Figure 4 As shown, the construction steps of the canonical equivariant capsule basic unit can be carried out as follows:
[0108] S311, initialize the spatial consistency coefficient and capsule consistency coefficient,
[0109] S312, obtaining a spatial coupling coefficient.
[0110] For the i-th input capsule of all neighboring nodes of the central node p and the j-th output capsule of the central node p The spatial consistency coefficient between them is normalized to obtain the spatial coupling coefficient. Specifically, the i-th input capsule of the neighbor node q The jth output capsule with the central node p The spatial coupling coefficient between for:
[0111]
[0112] S313, obtaining capsule coupling coefficient.
[0113] For all input capsules and the jth output capsule of the central node p Capsule consistency coefficient Normalize and obtain the capsule coupling coefficient. Specifically, the i-th input capsule of the central node p and the jth output capsule The capsule coupling coefficient between for:
[0114]
[0115] S314, calculate the capsule at any point p on the three-dimensional grid Input
[0116] Using the spatial coupling coefficient and capsule coupling coefficient The predictions of the input capsules of the neighboring nodes are weighted averaged to obtain the output capsule Weighted prediction of The specific formula is:
[0117]
[0118] S315, Get Capsule The pose vector
[0119] In the above formula, is the neighborhood of the central node p, After squash(·) nonlinear operation, the output pose vector is obtained Specifically:
[0120]
[0121] Here, ||·|| returns the two-norm of a vector;
[0122] S316, update the spatial consistency coefficient.
[0123] Using the output pose vector and input capsule Output Capsule Prediction Update spatial consistency coefficient and capsule consistency coefficient
[0124] Input capsule according to neighbor node q Output Capsule The prediction and the central node p right Predicted inner product update spatial consistency coefficient The specific formula is:
[0125]
[0126] S317, updating the capsule consistency coefficient.
[0127] Using all neighboring nodes of the central node p Input capsule Output Capsule The weighted average of the predictions and the output capsule pose vector The capsule consistency coefficient is updated by the inner product of It is expressed as:
[0128]
[0129] Repeat steps S311-S317N r Second, return the pose vector and norm
[0130] In another preferred embodiment of the present invention, a preferred solution for implementing S32 is provided, wherein the gauge equivariant residual block comprises two gauge equivariant capsule basic units and a residual connection. Generate capsules through two gauge equivariant capsule basic units The capsule and The output capsule is obtained through residual connection Among them, the number of iterations N of the first gauge equivariant capsule basic unit is r , fixed to 1; the number of iterations of the second gauge equivariant capsule basic unit is an adjustable hyperparameter. Only features of the same type are added together through residual connections. In the residual connection step, when the number of input capsules of the residual block is not equal to the number of capsules of the second capsule layer, a linear transformation matrix is used to linearly transform the input capsule to the same dimension as the output capsule.
[0131] The canonical equivariant pooling layer includes downsampling (selecting representative points) and aggregation (aggregating the features of non-sampled points in the neighborhood to the representative points). In this step, the farthest point sampling algorithm is first used to sample the vertices of the triangle mesh to obtain representative points, and the pooled neighborhood of each point is obtained based on the principle of the closest geodesic distance; then a canonical equivariant capsule basic unit layer is used to aggregate the non-sampled point capsules in the pooled neighborhood to the representative points.
[0132] The canonical equivariant unpooling layer upsamples the low-resolution feature domain after pooling to the original resolution, and the canonical equivariant unpooling operation upsamples the zero feature from the representative point to the non-sampling point.
[0133] Based on the same inventive concept, in another embodiment of the present invention, a three-dimensional shape feature map acquisition system is provided, comprising:
[0134] Data acquisition module: obtain three-dimensional grid data;
[0135] Data processing pre-module: pre-process the 3D grid data, calculate the logarithmic mapping of the local neighborhood of each vertex, and the parallel transmission between the local neighborhood center node and the neighboring nodes;
[0136] Gauge equivariant data processing module: The three-dimensional coordinates of each vertex, the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood are input into the gauge equivariant capsule network to generate a feature map.
[0137] The various modules / units in the above examples of the present invention may specifically refer to the implementation techniques of the corresponding steps of the three-dimensional shape feature map acquisition method in the above embodiments, which will not be described in detail here.
[0138] Based on the same inventive concept, in other embodiments of the present invention, a three-dimensional shape processing method is provided, and three-dimensional shape processing is performed based on the feature map obtained by any three-dimensional shape feature map acquisition method or system. Specifically: the feature map is used for three-dimensional shape processing, and the generated category capsules are used for shape segmentation tasks or for shape classification tasks or spherical image classification tasks after passing through a global average pooling layer.
[0139] When the three-dimensional shape is processed as shape segmentation, the boundary loss function is calculated based on the generated category capsules and the label information carried by the training data. The label information is the category information of each manually labeled vertex. The network parameters of the canonical equivariant capsule network are continuously updated according to the boundary loss function until the network model converges and the shape segmentation result is obtained.
[0140] When the 3D shape is processed as shape classification, the boundary loss function is calculated based on the category capsule after the global average pooling layer and the label information carried by the training data. The label information is the category information of each manually annotated 3D shape. The network parameters of the canonical equivariant capsule network are continuously updated according to the boundary loss function until the network model converges to obtain the shape classification result.
[0141] When the three-dimensional shape is processed as spherical image classification, the boundary loss function is calculated based on the category capsule after the global average pooling layer and the label information carried by the training data. The label information is the category information of each manually annotated spherical image. The network parameters of the canonical equivariant capsule network are continuously updated according to the boundary loss function until the network model converges to obtain the spherical image classification result.
[0142] Network parameters refer to the parameters inside all gauge-equivariant basic units, such as the parameter matrix when generating gauge-equivariant capsules using the spatial capsule dynamic routing algorithm.
[0143] Compared with the standard non-canonical equivariant deep neural network method, the above-mentioned embodiment of the present invention ensures better performance and has a higher ability to generalize to different three-dimensional grid structures.
[0144] The three-dimensional shape processing method based on the canonical equivariant capsule network provided in the above embodiment of the present invention can be applied to practical application scenarios on general manifolds, such as three-dimensional shape classification, three-dimensional shape segmentation, spherical image classification, etc. For better understanding, the technical solution provided by the above embodiment of the present invention is further described in detail below in combination with specific application examples.
[0145] In this specific application example, three-dimensional shape classification is realized. Specifically, the three-dimensional shape classification method based on the canonical equivariant capsule network includes the following four main steps:
[0146] Step 1: Obtain three-dimensional grid data.
[0147] Step 2: Use the Vector Heat method to preprocess the data to obtain the logarithmic mapping and parallel transmission of each point and all points in its geometric neighborhood required for training data and training specification equivariant capsule network.
[0148] Step 3: Input the 3D coordinates of each vertex of the 3D mesh data as the input capsule, the logarithmic mapping of each point and all points in its local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood into the canonical equivariant capsule network to generate a feature map. In this step, the canonical equivariant capsule network specifically includes: an initial pose extractor layer; a residual block layer based on the canonical equivariant capsule basic unit; and a category capsule layer based on the canonical equivariant capsule basic unit.
[0149] Step 4: Use the feature map for 3D shape classification. As a specific application example, the generated feature map includes n class Class capsules, n class Indicates the number of three-dimensional shape classification labels. The 2-norm of the class capsule pose vector is used to calculate the boundary loss function of the network parameters through global average pooling, and then the gradient descent method is used to continuously update the network parameters of the canonical equivariant capsule network according to the boundary loss function of the network parameters until the network model converges.
[0150] Network parameters refer to the parameters inside all gauge-equivariant basic units, such as the parameter matrix when generating gauge-equivariant capsules using the spatial capsule dynamic routing algorithm.
[0151] This example compares two other three-dimensional shape classification methods based on gauge equivariant capsule networks. They are the HSN network proposed in "CNNs on Surfaces using Rotation-Equivariant Features" published in the 2020 ACM Transactions on Graphics (ToG) journal, and the GET network proposed in "Gauge Equivariant Transformer" published in the 2021 Conference on Neural Information Processing Systems (NeurIPS) conference. The specific comparative experimental results are shown in Table 1. The evaluation criteria are the average classification accuracy. The higher the value, the better the effect.
[0152] Table 1: Comparison of classification accuracy of HSN, GET and the method proposed in the embodiment of the present invention (GECN) on the SHREC dataset
[0153] Model enter Accuracy (%) HSN 3D Grid 96.1 GET 3D Grid 99.2 GECN 3D Grid 100
[0154] The experimental results in Table 1 show that on the non-rigid deformable SHREC dataset, the classification accuracy of the 3D shape processing method of this embodiment is significantly higher than that of the other two existing gauge equivariant neural networks. While improving the classification accuracy, this embodiment can theoretically guarantee equivariance to the gauge change in SO(2).
[0155] In another embodiment, the present invention provides a three-dimensional shape segmentation method, which may include the following four main steps:
[0156] Step 1: Obtain three-dimensional grid data.
[0157] Step 2: Use the Vector Heat method to preprocess the data to obtain the logarithmic mapping and parallel transmission of each point and all points in its geometric neighborhood required for training data and training specification equivariant capsule network.
[0158] Step 3: Input the 3D coordinates of each vertex of the 3D mesh data as the input capsule, the logarithmic mapping of each point and all points in its local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood into the canonical equivariant capsule network to generate a feature map. In this step, the canonical equivariant capsule network specifically includes: an initial pose extractor layer; a residual block layer based on the canonical equivariant capsule basic unit; a canonical equivariant pooling and unpooling operation layer; and a category capsule layer based on the canonical equivariant capsule basic unit.
[0159] Step 4: Use the feature map for 3D shape segmentation. As a specific application example, the generated feature domain includes n classClass capsules, n class Indicates the number of category labels for each vertex of a three-dimensional shape. The 2-norm of the category capsule pose vector is used to calculate the boundary loss function of the network parameters, and then the gradient descent method is used to continuously update the network parameters of the canonical equivariant capsule network according to the boundary loss function of the network parameters until the network model converges.
[0160] Network parameters refer to the parameters inside all gauge-equivariant basic units, such as the parameter matrix when generating gauge-equivariant capsules using the spatial capsule dynamic routing algorithm.
[0161] In this specific application example, 3D shape segmentation is implemented, and the HSN network proposed in "CNNs on Surfaces using Rotation-Equivariant Features" published in the 2020 ACM Transactions on Graphics (ToG) journal and the GET network proposed in "Gauge Equivariant Transformer" published in the 2021 Conference on Neural Information Processing Systems (NeurIPS) are compared. The specific comparative experimental results are shown in Table 2. The evaluation criteria are overall accuracy (OA) and mean intersection over union (mIoU). The higher the value, the better the effect.
[0162] Table 2: Comparison of overall accuracy (OA) and mean intersection over union (mIoU) of HSN, GET and the method proposed in the embodiment of the present invention (GECN) on the human segmentation dataset
[0163] Model enter OA (%) mIoU(%) HSN 3D Grid 91.1 85.3 GET 3D Grid 92.6 86.3 GECN 3D Grid 92.8 87.2
[0164] The experimental results in Table 2 show that on the human body segmentation dataset, the overall accuracy and average intersection-over-union ratio of the method proposed in this embodiment are significantly higher than those of the other two existing canonical equivariant neural networks. While improving the classification accuracy, this embodiment can better generalize to different three-dimensional grid structures.
[0165] In another embodiment of the present invention, a spherical image classification method is also provided, which may include the following four main steps:
[0166] Step 1: Obtain the three-dimensional grid data corresponding to the spherical image.
[0167] Step 2: Use the Vector Heat method to preprocess the data to obtain the logarithmic mapping and parallel transmission of each point and all points in its geometric neighborhood required for training data and training specification equivariant capsule network.
[0168] Step 3: The pixel value of each vertex of the spherical image data is used as the input capsule, the logarithmic mapping of each point and all points in its local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood are input into the canonical equivariant capsule network to generate a feature map. In this step, the canonical equivariant capsule network specifically includes: an initial pose extractor layer; a residual block layer based on the canonical equivariant capsule basic unit; a canonical equivariant pooling and unpooling operation layer; and a category capsule layer based on the canonical equivariant capsule basic unit.
[0169] Step 4: Use the feature map for spherical image classification. As a specific application example, the generated feature domain includes n class Class capsules, n class Indicates the number of spherical image classification labels. The boundary loss function of the network parameters is calculated using the 2-norm of the category capsule pose vector through global average pooling, and then the network parameters of the canonical equivariant capsule network are continuously updated according to the boundary loss function of the network parameters using the gradient descent method until the network model converges.
[0170] Network parameters refer to the parameters inside all gauge-equivariant basic units, such as the parameter matrix when generating gauge-equivariant capsules using the spatial capsule dynamic routing algorithm.
[0171] In this specific application example, spherical image classification is implemented, and the HSN network proposed in "CNNs on Surfaces using Rotation-Equivariant Features" published in the 2020 ACM Transactions on Graphics (ToG) journal and the GET network proposed in "Gauge Equivariant Transformer" published in the 2021 Conference on Neural Information Processing Systems (NeurIPS) are compared. The specific comparative experimental results are shown in Tables 3, 4, and 5. The evaluation standard is the average classification accuracy. The higher the value, the better the effect.
[0172] The experimental results in Table 3, Table 4, and Table 5 show that on the data sets Spherical MNIST, Spherical CIFAR-10, and Spherical ModelNet40, the average classification accuracy of the three-dimensional shape processing method of this embodiment is significantly higher than that of the other two existing canonical equivariant neural networks. Compared with the existing canonical equivariant neural networks, this embodiment can be better generalized to spherical domain tasks without principal curvature directions.
[0173] Table 3: Comparison of classification accuracy of HSN, GET and the method proposed in the embodiment of the present invention (GECN) on the Spherical MNIST dataset
[0174] Model Data Domain Accuracy (%) HSN Generalized Manifold 98.8 GET Generalized Manifold 98.8 GECN Generalized Manifold 99.3
[0175] Table 4: Comparison of classification accuracy of HSN, GET and the method proposed in the embodiment of the present invention (GECN) on the Spherical CIFAR-10 dataset
[0176]
[0177]
[0178] Table 5: Comparison of classification accuracy of HSN, GET and the method proposed in the embodiment of the present invention (GECN) on the Spherical ModelNet40 dataset
[0179] Model Data Domain Accuracy (%) HSN Generalized Manifold 85.7 GET Generalized Manifold 89.3 GECN Generalized Manifold 91.5
[0180] Based on the same inventive concept, in other embodiments of the present invention, a terminal is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute the above-mentioned method or to run the above-mentioned system when executing the program.
[0181] Optionally, the memory is used to store programs; the memory may include volatile memory (English: volatile memory), such as random-access memory (English: random-access memory, abbreviated: RAM), such as static random-access memory (English: static random-access memory, abbreviated: SRAM), double data rate synchronous dynamic random access memory (English: Double Data Rate Synchronous Dynamic RandomAccess Memory, abbreviated: DDRSDRAM), etc.; the memory may also include non-volatile memory (English: non-volatile memory), such as flash memory (English: flash memory). The memory is used to store computer programs (such as applications, functional modules, etc. that implement the above method), computer instructions, etc., and the above-mentioned computer programs, computer instructions, etc. can be partitioned and stored in one or more memories.
[0182] The processor is used to execute the computer program stored in the memory to implement the various steps in the method involved in the above embodiment. For details, please refer to the relevant description in the above method embodiment.
[0183] The processor and the memory may be independent structures or integrated structures. When the processor and the memory are independent structures, the memory and the processor may be coupled and connected via a bus.
[0184] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the above method, or run the above system.
[0185] Among them, computer-readable media include computer storage media and communication media, wherein the communication media include any media that facilitates the transmission of computer programs from one place to another. The storage medium can be any available medium that can be accessed by a general or special-purpose computer. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and the storage medium can be located in an ASIC. In addition, the ASIC can be located in a user device. Of course, the processor and the storage medium can also be present in a communication device as discrete components.
[0186] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0187] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0188] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0189] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0190] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A method for acquiring a feature map, characterized in that: include: Get three-dimensional mesh data; Preprocessing the three-dimensional grid data, calculating and obtaining the logarithmic mapping of the local neighborhood of each vertex, and the parallel transmission between the local neighborhood center node and the neighboring nodes; The three-dimensional coordinates of each vertex, the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood are input into the canonical equivariant capsule network to generate a feature map.
2. A method for acquiring a feature map according to claim 1, characterized in that: The canonical equivariant capsule network comprises an initial pose extractor layer, a canonical equivariant residual block layer, a canonical equivariant pooling layer, a canonical equivariant unpooling layer and a canonical equivariant class capsule layer; wherein: One or more of the residual block layers form a first combined layer located after the initial pose extractor layer; One of the pooling layers and at least one of the residual block layers forms a second combination layer, which is located behind the first combination layer; the second combination layer is stacked one or more times; One of the anti-pooling layers and at least one of the residual block layers forms a third combination layer, which is located behind the last stacked second combination layer; the third combination layer is stacked one or more times; The category capsule layer is located behind the last third combined layer in the stack.
3. A method for acquiring a feature map according to claim 2, characterized in that: The input of the initial pose extractor layer is the three-dimensional coordinates of each vertex, the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood, and the output is the initial capsule; The input of the residual block layer is the output capsule of the initial pose extractor layer, or the output capsule of the adjacent residual block layer, or the output capsule of the pooling layer, or the output capsule of the unpooling layer, as well as the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood, and the output is a new capsule; The input of the pooling layer is the output capsule of the adjacent previous residual block layer, the logarithmic mapping of the local neighborhood, the parallel transmission between the central node and the neighboring nodes in the local neighborhood, and the output is a new capsule; The input of the depooling layer is the output capsule of the adjacent previous residual block layer, and the output is a new capsule; The input of the category capsule layer is the output capsule of the adjacent previous residual block, the logarithmic mapping of the local neighborhood, the parallel transmission between the central node and the neighboring nodes in the local neighborhood, and the output is the category capsule.
4. A method for acquiring a feature map according to claim 2, characterized in that: The initial pose extractor includes a canonical equivariant capsule basic unit; The residual block layer includes two canonical equivariant capsule basic units and a residual connection; The pooling layer selects representative points by sampling the farthest point, and aggregates non-sampled point capsules to the representative points using a canonical equivariant capsule basic unit; The anti-pooling layer sets the features of non-representative points to zero vectors; The class capsule layer is a canonical equivariant capsule basic unit.
5. A method for acquiring a feature map according to claim 4, characterized in that: For each of the gauge equivariant capsule units, for any central node p on the three-dimensional grid, The input is: N input capsules The input capsule is the three-dimensional coordinates of each vertex or the initial capsule or the output capsule of the previous gauge equivariant capsule basic unit, as well as the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood; The output is: N' output capsules The output capsule serves as the input capsule of the next gauge equivariant capsule basic unit or the final output feature map.
6. A method for acquiring a feature map according to claim 5, characterized in that: The output capsule It is obtained by the input capsule of point p and its neighborhood points, specifically: For any neighbor node q in the neighborhood of the central node p, obtain q on the tangent plane of p relative to the norm ω p Polar coordinate representation (r q ,θ q ); For the i-th input capsule of neighbor node q Get the pose vector f'(q i ); According to the input capsule of neighbor node q and the output capsule of the central node p The parameter matrix W between ij (r q ,θ q ), get the input capsule Output Capsule Prediction Parameter matrix W ij (r q ,θ q ) satisfies the constraints of gauge equivariance; Input capsule of the i-th neighbor node q The jth output capsule with the central node p The spatial consistency coefficient between and the i-th input capsule of the central node p With the jth output capsule Capsule consistency coefficient Initialized to 0; According to the spatial consistency coefficient and capsule consistency coefficient Calculation of spatial coupling coefficient and capsule coupling coefficient Weighted average output pose vector And update the spatial consistency coefficient and capsule consistency coefficient Repeat N r Get the output capsule's pose vector and norm 7. A method for acquiring a feature map according to claim 6, characterized in that:The spatial consistency coefficient and capsule consistency coefficient Calculation of spatial coupling coefficient and capsule coupling coefficient Weighted average output pose vector And update the spatial consistency coefficient and capsule consistency coefficient Repeat N r Get the output capsule's pose vector and norm include: Initialize spatial consistency coefficient Initialize capsule consistency coefficient Get the spatial coupling coefficient: Get the capsule coupling coefficient: Compute the capsule at any point p on a 3D grid Input Get the capsule The pose vector Update spatial consistency coefficient: Update capsule consistency coefficient: Repeat the above steps N r Second, return the pose vector and norm 8. A method for acquiring a feature map according to claim 4, characterized in that: The canonical equivariant residual block layer comprises: The capsule generated by the initial pose extractor or the previous residual block layer The capsule is generated by two standard equivariant capsule basic units. The capsule and The output capsule is obtained through the residual connection Among them, the number of iterations N of the first gauge equivariant capsule basic unit is r , fixed to 1; the number of iterations of the second gauge equivariant capsule basic unit is an adjustable hyperparameter.
9. A method for acquiring a feature map according to claim 4, characterized in that: The canonical equivariant pooling layer selects representative points by sampling the farthest point, and aggregates non-sampled point capsules to representative points using the canonical equivariant capsule basic unit, specifically: Use the farthest point sampling method to obtain representative points from the vertex sampling of the triangle mesh; Obtain the pooled neighborhood of each representative point according to the principle of the closest geodesic distance The pooling neighborhood The capsule of the non-representative point inside is used as the input capsule; The input capsules composed of capsules of non-representative points are aggregated to the representative point p through a canonical equivariant capsule basic unit, and the output capsules after the canonical equivariant pooling operation on the representative point are obtained.
10. A feature map acquisition system, characterized in that: include: Data acquisition module: obtain three-dimensional grid data; Data processing pre-module: pre-processing the three-dimensional grid data, calculating and obtaining the logarithmic mapping of the local neighborhood of each vertex, and the parallel transmission between the local neighborhood center node and the neighboring nodes; The gauge equivariant data processing module inputs the three-dimensional coordinates of each vertex, the logarithmic mapping of the local neighborhood, and the parallel transmission between the central node and the neighboring nodes in the local neighborhood into the gauge equivariant capsule network to generate a feature map.
11. A three-dimensional shape processing method, characterized in that: include: The feature map is obtained by using the feature map acquisition method according to any one of claims 1 to 9 or the feature map acquisition system according to claim 10; Performing three-dimensional shape processing based on the feature map, wherein the three-dimensional shape processing includes shape segmentation, shape classification or spherical image classification; When the three-dimensional shape is processed as shape segmentation, a boundary loss function is calculated based on the generated feature map and the label information carried by the training data. The label information is the category information of each manually labeled vertex. The network parameters of the canonical equivariant capsule network are continuously updated according to the boundary loss function until the network model converges to obtain the shape segmentation result. When the three-dimensional shape is processed as shape classification, a boundary loss function is calculated based on the feature map after the global average pooling layer and the label information carried by the training data. The label information is the category information of each manually annotated three-dimensional shape. The network parameters of the canonical equivariant capsule network are continuously updated according to the boundary loss function until the network model converges to obtain the shape classification result. When the three-dimensional shape is processed as spherical image classification, the boundary loss function is calculated based on the feature map after the global average pooling layer and the label information carried by the training data. The label information is the category information of each manually annotated spherical image. The network parameters of the canonical equivariant capsule network are continuously updated according to the boundary loss function until the network model converges to obtain the spherical image classification result.
12. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to execute the method described in any one of claims 1 to 9 and 11 or the system described in claim 10.
13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method described in any one of claims 1 to 9 and 11 or the system described in claim 10.
