Unsupervised Joint Shape Summarization and Instance Segmentation Based on Shared Membership
By using an unsupervised method that shares a membership matrix, shape summarization and instance segmentation are jointly trained, solving the accuracy problem of part-level 3D shape representation under unsupervised conditions and achieving efficient shape summarization and instance segmentation.
Patent Information
- Application Number
- CN202410615152.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-05-17
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-05-17
AI Technical Summary
Existing unsupervised methods struggle to accurately perform shape summarization and instance segmentation tasks in part-level 3D shape representation under unlabeled conditions, easily leading to oversegmentation or undersegmentation. Furthermore, existing methods neglect the physical meaning of the parts.
An unsupervised joint shape summarization and instance segmentation method based on a shared membership matrix is adopted. A shared membership matrix is generated by a feature encoder and a membership decoder. The softmax operation is combined to generate task-specific coefficient and probability matrices, thereby achieving joint training of shape summarization and instance segmentation.
It enables efficient training and deployment of shape summarization and instance segmentation under unsupervised conditions, improves the accuracy and consistency of the task, avoids over- or under-segmentation, and enhances the physical shape representation of parts.
Smart Images

Figure CN118447068B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of deep learning in artificial intelligence, and in particular to an unsupervised method for joint shape summarization and instance segmentation of point clouds based on shared membership relationships. Background Technology
[0002] Part-level 3D shape representation is crucial for shape reasoning and understanding. The two key sub-tasks are: (i) shape summarization, reconstructing the surface of an object part from primitives; and (ii) instance segmentation, segmenting the point cloud into part instances.
[0003] When part labels are difficult to obtain, current methods, when performing unsupervised shape summarization alone, represent a single complex part instance as multiple meaningless cuboids or replace multiple independent parts with a single large cuboid. This shape representation simply reconstructs the input point cloud using a series of cuboids without physical meaning, losing the true physical meaning of the part. When performing unsupervised instance segmentation alone, the segmentation results generated by clustering and other methods based on predefined similarity ignore the complex geometry of the real part, which easily leads to oversegmentation or undersegmentation. Summary of the Invention
[0004] (a) Technical problems to be solved
[0005] To address the aforementioned problems in existing technologies, this invention provides an unsupervised method based on a shared membership matrix for joint shape summarization and instance segmentation, enabling the two tasks to mutually reinforce each other without requiring any supervision information, thus facilitating training and deployment.
[0006] (II) Technical Solution
[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: an unsupervised joint shape summarization and instance segmentation based on shared membership, comprising the following steps:
[0008] S0: Obtain several existing undirected point clouds as a training set, without any label information or prior knowledge;
[0009] S1: Transfer the point cloud X from the training set pc =[x1,…x N The input feature encoder extracts the feature F for each point. pc =[f1 pc ,…f N pc ], N represents the number of points in the point cloud;
[0010] S2: The features F of the points are processed by a multilayer perceptron. pc Mapping to the membership decoder, we obtain the membership matrix of N points shared by the tasks of M parts. M is a hyperparameter given during method initialization. It is the set of real numbers;
[0011] S3: Obtain task-specific membership relationships;
[0012] S31: Perform softmax on the membership matrix J row by row and transpose it to obtain the linear weighted coefficient matrix W specific to the shape summarization task;
[0013] S32: Perform softmax on the membership matrix J column by column to obtain the assignment probability matrix P specific to the instance segmentation task;
[0014] S4: Generate shape summary and instance segmentation of the point cloud;
[0015] S41: Shape summary, based on the feature matrix F of the points. pc The feature F of M parts is obtained by matrix multiplication of the coefficient matrix W. part =[f1 part ,…f M part That is, the part features are represented as a linear combination of point features, and then the shape decoder is used to extract them from F. part Decode the geometric parameters of the primitives corresponding to the shape summary of each part, and generate the corresponding shape summary through the parametric equations of the primitives;
[0016] S42: Instance segmentation. Perform the argmax operation on the allocation probability of each point in the allocation probability matrix P with respect to the M parts to obtain the part label corresponding to each point, and generate instance segmentation based on the label.
[0017] S5: Use the point cloud in the training set to train the feature encoder, membership decoder and shape decoder in S1-S4. After the training reaches the specified number of iterations, the trained feature encoder, trained membership decoder and trained shape decoder are obtained.
[0018] S6: For a new point cloud, the methods of S1-S4 are used, and the trained feature encoder, the trained membership decoder and the trained shape decoder are utilized to finally obtain the shape summary and instance segmentation corresponding to the new point cloud.
[0019] Specifically, the steps for obtaining the membership matrix J in S2 are as follows:
[0020] J :,i =Linear(MLPs(f i pc ))
[0021] Where MLPs(·) represents a general multilayer perceptron, Linear(·) represents a linear mapping of feature vectors, and f i pc J represents the feature of the i-th point in the point cloud. :,i Let represent the membership relationship of the i-th point with respect to the M parts, and be the i-th column in the membership matrix J.
[0022] Specifically, the steps for obtaining the coefficient matrix W specific to the shape summarization task in S31 are as follows: Perform softmax on each row of the membership matrix J and transpose it to obtain the coefficient matrix W specific to the shape summarization task.
[0023] W = (softmax) row (J)) T
[0024] softmax row (·) indicates that softmax is applied to each row of the matrix.
[0025] Specifically, the steps for obtaining the probability matrix P specific to the instance segmentation task in S32 are as follows: Perform softmax on each column of the membership matrix J to obtain the probability matrix P specific to the instance segmentation task:
[0026] P = softmax col (J)
[0027] softmax col (·) indicates that softmax is applied to each column of the matrix.
[0028] Specifically, the total loss function in S5 is:
[0029] Define the Euclidean distance d(x) from a point to a primitive. i ,s j ):
[0030]
[0031] Where s j Let y represent the component reconstructed from the j-th primitive. k j This represents the k-th point among the K points uniformly sampled from the surface of the part reconstructed from the j-th primitive.
[0032] Define the 2-norm coverage distance c of the primitive. c (s j ):
[0033]
[0034] Define the 1-norm coverage distance c of the primitive.f (s j ):
[0035]
[0036] Based on the three distances mentioned above, the loss function for training is defined as follows:
[0037] 5.1 Reconstruction Loss Function L recon It consists of two parts:
[0038] L recon =0.5*L pc2pr +0.5*L pr2pc
[0039] Where L pc2pr and L pr2pc Let represent the reconstruction loss from point cloud to primitive and the reconstruction loss from primitive to point cloud, respectively:
[0040]
[0041]
[0042] 5.2 Anti-anchoring loss function L hd :
[0043]
[0044] 5.3 Loss Function L of Antitrivial Solution wd :
[0045]
[0046] Where, p ji This represents the element in the j-th row and i-th column of the probability matrix P for the instance segmentation task.
[0047] 5.4 Compact Loss L c :
[0048]
[0049] 5.5 There is a loss L e :
[0050]
[0051] in:
[0052]
[0053] Where, δ j This represents the probability that the part reconstructed from the j-th primitive in the predicted shape summary is represented. Let represent the pseudo-label obtained from the statistics of the instance segmentation probability matrix P, and let Ι(·) represent the indicator function.
[0054] The total loss function L used for training is shown below:
[0055] L = L recon +λ1L hd +λ2L wd +λ3L c +λ4L e
[0056] Where λ1, λ2, λ3 and λ4 represent L respectively. recon L hd L wd L c and L e The coefficient.
[0057] Compared with the prior art, the present invention has at least the following advantages:
[0058] The method of this invention promotes shape summarization and instance segmentation by combining shared membership relationships. In this way, the method requires no prior knowledge, such as the object type of the point cloud or the number of parts it contains; it also requires no supervision information, such as estimating the normal information of the point cloud using algorithms to identify instances by leveraging changes in normals, or reconstructing the input based on binary classification using occupancy function labels; furthermore, it eliminates the need to design separate branch networks for the two tasks, making the method of this invention easy to train and deploy.
[0059] The method of this invention enables shape summarization and instance segmentation to complement and enhance each other. As shown in Tables 1 and 2, the method of this invention performs better than performing either task alone in both tasks. Attached Figure Description
[0060] To more clearly illustrate the technical solutions in the embodiments of the present invention and the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0061] Figure 1 This is a simplified flowchart of the method of the present invention.
[0062] Figure 2 This represents the transformation between the shared membership matrix and the parameter matrix specific to shape summary and instance segmentation.
[0063] Figure 3 This describes the workflow of the shape decoder.
[0064] Figure 4 The first line is the input, the second line is the instance segmentation, and the third line is the shape summary based on deformable hyperquadratic surfaces.
[0065] Figure 5 This is a specific implementation example of the method of the present invention. Detailed Implementation
[0066] The present invention will now be described in further detail.
[0067] The method of this invention uses a single membership matrix J to achieve joint shape summarization and instance segmentation. This invention employs an autoencoder, such as... Figure 1 As shown, a general point cloud feature encoder is used to extract the features of each point and construct the point cloud feature matrix F. pc Then, a membership decoder composed of multilayer perceptrons is used to extract data from F. pc Obtain the membership matrix J.
[0068] For instance segmentation, softmax is performed column-by-column on the membership matrix J to obtain a probability matrix P specific to the instance segmentation task: P = softmax. col (J). For shape summarization, perform softmax row-by-row on the membership matrix J and transpose to obtain a linear weighted coefficient matrix W specific to the shape summarization task: W = (softmax ... row (J)) T Then through the feature matrix F of the points pc The feature F of M parts is obtained by matrix multiplication of the coefficient matrix W. part =[f1 part ,…f M part ],like Figure 1 and Figure 5 As shown. Following this, a shape decoder composed of a multilayer perceptron is used to predict the parameters of the primitives to reconstruct the corresponding parts, as shown. Figure 3 As shown. Each parameter type corresponds to an independent sub-decoder, including size, shape, rotation, and translation. The input to the translation sub-decoder, in addition to the part feature F, is... part In addition, it also includes the product of the coordinate matrix X of the point and the coefficient matrix W, i.e., WX, as shown below. Figure 3 As shown.
[0069] This invention achieves joint shape summarization and instance segmentation through a single membership matrix J, which mutually promote each other. The specific execution flow is as follows:
[0070]
[0071] Where Encoder represents the feature encoder, and Decoder represents the feature encoder. membershipDecoder represents the membership relationship. shape This represents a shape decoder.
[0072] The present invention will now be further described in conjunction with the accompanying drawings and specific implementation methods, such as... Figure 5 As shown, unsupervised joint shape summarization and instance segmentation based on shared membership relationships include the following steps:
[0073] S0: Obtain several existing undirected point clouds as a training set, without any label information or prior knowledge.
[0074] S1: Transfer the point cloud X from the training set pc =[x1,…x N The feature encoder of PointNet++ is used to extract the feature F of each point. pc =[f1 pc ,…f N pc ], N represents the number of points in the point cloud, i.e., the length of the point cloud.
[0075] S2: The membership decoder, composed of a three-layer sensing mechanism, extracts the features of the points. Fp c is mapped to the membership decoder to obtain the membership matrix of N points with respect to M parts. M = 16, The set is the real number set. The membership decoder is specifically implemented as follows: Linear, BatchNorm, ReLU, Linear, BatchNorm, ReLU, and Linear; Linear represents the linear mapping of feature vectors, BatchNorm represents batch normalization, and ReLU represents the linear rectified function in the nonlinear activation function.
[0076] S3: Get task-specific membership relationships.
[0077] S31: Perform softmax on the membership matrix J row by row and transpose it to obtain the linear weighted coefficient matrix W specific to the shape summarization task.
[0078] S32: Perform softmax on each column of the membership matrix J to obtain the assignment probability matrix P specific to the instance segmentation task.
[0079] S4: Generate shape summary and instance segmentation of the point cloud.
[0080] S41: Shape summary, based on the feature matrix F of the points. pc Matrix multiplication of the coefficient matrix W, F part =F pc W, obtain the features of M parts. part =[f1 part ,…fM part That is, the part features are represented as a linear combination of point features, and then the part features F are analyzed by a shape decoder. part The geometric parameters of the primitives corresponding to the shape summary of each part are decoded, and the corresponding shape summary is generated through the parametric equations of the primitives.
[0081] S42: Instance segmentation. Perform the argmax operation on the probability distribution probability of each point in the probability matrix P with respect to the M parts to obtain the part label corresponding to each point, and generate instance segmentation based on the label.
[0082] S5: Use the point cloud in the training set to train the feature encoder, membership decoder and shape decoder in S1-S4. After training reaches the specified 500 iterations, the trained feature encoder, trained membership decoder and trained shape decoder are obtained.
[0083] S6: For a new point cloud, the methods of S1-S4 are used, and the trained feature encoder, the trained membership decoder and the trained shape decoder are utilized to finally obtain the shape summary and instance segmentation corresponding to the new point cloud.
[0084] Specifically, the steps for obtaining the membership matrix J in S2 are as follows:
[0085] J :,i =Linear(MLPs(f i pc ))
[0086] Where MLPs(·) denotes a general multilayer perceptron, Linear(·) denotes a linear mapping of feature vectors, and f i pc J represents the feature of the i-th point in the point cloud. :,i This represents the membership relationship of the i-th column in the membership matrix J, i.e., the membership relationship of the i-th point with respect to the M parts.
[0087] The specific steps for obtaining the shape-summarizing task-specific coefficient matrix W in step S31 are as follows: Perform softmax on each row of the membership matrix J and transpose it to obtain the shape-summarizing task-specific coefficient matrix W.
[0088] W = (softmax) row (J)) T
[0089] softmax row (·) indicates that softmax is applied to each row of the matrix.
[0090] The specific steps for obtaining the instance segmentation-specific probability matrix P in step S32 are as follows: Perform softmax on each column of the membership matrix J to obtain the instance segmentation-specific probability matrix P:
[0091] P = softmax col (J)
[0092] softmax col (·) indicates that softmax is applied to each column of the matrix.
[0093] Specifically, the total loss function in S5 is:
[0094] Define the Euclidean distance from a point to a primitive:
[0095]
[0096] Where s j Let y represent the component reconstructed from the j-th primitive. k j This represents the k-th point among the K points uniformly sampled from the surface of the part reconstructed from the j-th primitive.
[0097] Define the 2-norm coverage distance of the primitive:
[0098]
[0099] Define the 1-norm coverage distance of the primitive:
[0100]
[0101] Based on the three distances mentioned above, the loss function for training is defined as follows:
[0102] Reconstruction loss function L recon It consists of two parts:
[0103] L recon =0.5*L pc2pr +0.5*L pr2pc
[0104] Where L pc2pr and L pr2Lpc Let represent the reconstruction loss from point cloud to primitive and the reconstruction loss from primitive to point cloud, respectively:
[0105]
[0106]
[0107] Anti-anchoring loss function L hd :
[0108]
[0109] The loss function L of the antitrivial solution wd :
[0110]
[0111] Where, p ji This represents the element in the j-th row and i-th column of the probability matrix P for the instance segmentation task:
[0112] Compact loss:
[0113]
[0114] Losses exist:
[0115]
[0116] in:
[0117]
[0118] Where, δ j This represents the probability that the part reconstructed from the j-th primitive in the predicted shape summary is represented. Let represent the pseudo-label obtained from the statistics of the instance segmentation probability matrix P, and let Ι(·) represent the indicator function.
[0119] The total loss function used for training is shown below:
[0120] L = L recon +λ1L hd +λ2L wd +λ3L c +λ4L e
[0121] Where λ1, λ2, λ3, and λ4 represent coefficients, with values of 1, 0.3, 0.1, and 0.01, respectively. Phase P1 training lasts for 100 epochs, and Phase P2 training lasts for 400 epochs.
[0122] This invention uses airplane, chair, and table data from the ShapeNet dataset, as well as the Animal dataset, as experimental data. There are a total of four shape categories, containing 3640 airplane point clouds, 5929 chair point clouds, 7555 table point clouds, and 129 animal point clouds. The training set to test set ratio is 4:1, eliminating the need for independent training for each category. 256 points are uniformly sampled from the surface of each primitive, with a large primitive number M = 16.
[0123] This invention selects a variable hyperquadratic surface as the primitive input point cloud for reconstruction. A hyperquadratic surface is a set of parameterized shapes that can model different objects. Its parameter expression is:
[0124]
[0125] Where a1, a2, a3 are dimension parameters, ∈1, ∈2 are shape parameters, and η, ω represent the latitude and longitude angles of the surface coordinate point X in the spherical coordinate system, respectively. When the five parameters of the hyperquadratic surface take different values, closed surfaces of various shapes can be represented.
[0126] Hyperquadratic surface deformation—tapering: After tapering a standard hyperquadratic surface along the Z-axis, the surface coordinates have the following relationship:
[0127]
[0128] Among them, f x (z) and f y (z) is the coneification function, k x and k y This is the taper coefficient.
[0129] Hyperquadratic surface deformation—bending. The direction of the bending plane is defined by angle α. After bending a standard hyperquadratic surface along the Z-axis, the surface coordinates have the following relationship:
[0130]
[0131] Where γ represents the bending angle, calculated from the curvature parameter b>0 and the Z-axis coordinate. To ensure gradient propagation, the parameters are constrained as follows when optimizing the deformable hyperquadratic surface: the translation parameter is constrained within the range of [-1, 1]; the rotation parameter is constrained within the range of... The dimensional parameters are constrained within the range of [0.02, 0.82]; the shape parameters are constrained within the range of [0.2, 1.7]; the taper coefficient is constrained within the range of [-0.9, 0.9]; the curvature parameters are constrained within the range of [0.01, 0.75]; and the angle parameters are constrained within the range of [0.02, 0.82].
[0132] Since matrix multiplication does not satisfy the commutative law, the order of surface deformation cannot be interchanged and must be performed in a pre-defined order. The definition order chosen in this invention is: translation (rotation (bending (tapering (standard hyperquadratic surface)))).
[0133] Figure 4 The second row shows the predicted instance-level segmentation of the present invention and the shape summary based on deformable hyperquadratic surfaces (the third row), demonstrating that the present invention achieves good results.
[0134] Table 1 shows the comparison between the present invention and two baseline methods on the unsupervised shape summarization task. Bold text indicates the highest reconstruction accuracy. Obviously, the method of the present invention is superior to the baseline methods.
[0135] Tables 2 and 3 show the comparison between the present invention and two baseline methods in the unsupervised instance segmentation task. The bold text indicates the highest segmentation accuracy. Obviously, the method of the present invention is superior to the baseline method.
[0136] As can be seen from Tables 1, 2 and 3, this invention achieves mutual promotion of two tasks by combining shape summarization and instance segmentation through a shared membership matrix, without requiring any supervision information, which facilitates training and deployment.
[0137] Table 1. Quantitative comparison of shape summaries (bevel distance, the smaller the better).
[0138]
[0139] Table 2. Quantitative comparison of instance segmentation (Intersection over Union ratio, the higher the better).
[0140]
[0141] Table 3. Quantitative comparison of instance segmentation (Davidson-Bolding index, the smaller the better).
[0142]
Claims
1. An unsupervised joint shape summarization and instance segmentation method based on shared membership, characterized in that, Includes the following steps: S0: Obtain several existing undirected point clouds as a training set, without any label information or prior knowledge; S1: Transfer the point cloud X from the training set pc =[x1,…x N The input feature encoder extracts the feature F for each point. pc =[f1 pc ,…f N pc ], N represents the number of points in the point cloud; S2: The features F of the points are processed by a multilayer perceptron. pc Mapping to the membership decoder, we obtain the membership matrix of N points shared by the tasks of M parts. M is a hyperparameter given during method initialization. It is the set of real numbers; S3: Obtain task-specific membership relationships; S31: Perform softmax on the membership matrix J row by row and transpose it to obtain the linear weighted coefficient matrix W specific to the shape summarization task; S32: Perform softmax on the membership matrix J column by column to obtain the assignment probability matrix P specific to the instance segmentation task; S4: Generate shape summary and instance segmentation of the point cloud; S41: Shape summary, based on the feature matrix F of the points. pc The feature F of M parts is obtained by matrix multiplication of the coefficient matrix W. part =[f1 part ,…f M part That is, the part features are represented as a linear combination of point features, and then the shape decoder is used to extract them from F. part Decode the geometric parameters of the primitives corresponding to the shape summary of each part, and generate the corresponding shape summary through the parametric equations of the primitives; S42: Instance segmentation. Perform the argmax operation on the allocation probability of each point in the allocation probability matrix P with respect to the M parts to obtain the part label corresponding to each point, and generate instance segmentation based on the label. S5: Use the point cloud in the training set to train the feature encoder, membership decoder and shape decoder in S1-S4. After the training reaches the specified number of iterations, the trained feature encoder, trained membership decoder and trained shape decoder are obtained. S6: For a new point cloud, the methods of S1-S4 are used, and the trained feature encoder, the trained membership decoder and the trained shape decoder are utilized to finally obtain the shape summary and instance segmentation corresponding to the new point cloud.
2. The unsupervised joint shape summarization and instance segmentation based on shared membership as described in claim 1, characterized in that: The specific steps for obtaining the membership matrix J in S2 are as follows: J :,i =Linear(MLPs(f i pc )) Where MLPs(·) represents a general multilayer perceptron, Linear(·) represents a linear mapping of feature vectors, and f i pc J represents the feature of the i-th point in the point cloud. :,i Let represent the membership relationship of the i-th point with respect to the M parts, and represent the i-th column in the membership matrix J.
3. The unsupervised joint shape summarization and instance segmentation based on shared membership as described in claim 1 or 2, characterized in that: The specific steps for obtaining the shape-summarizing task-specific coefficient matrix W in step S31 are as follows: Perform softmax on each row of the membership matrix J and transpose it to obtain the shape-summarizing task-specific coefficient matrix W. W=(softmax row (J)) T softmax row (·) indicates that softmax is applied to each row of the matrix.
4. An unsupervised joint shape summarization and instance segmentation based on shared membership as described in claim 1 or 2, characterized in that: The specific steps for obtaining the instance segmentation-specific probability matrix P in step S32 are as follows: Perform softmax on each column of the membership matrix J to obtain the instance segmentation-specific probability matrix P: P=softmax col (J) softmax col (·) indicates that softmax is applied to each column of the matrix.
5. The unsupervised joint shape summarization and instance segmentation based on shared membership as described in claim 1, characterized in that: the total loss function in S5 is: Define the Euclidean distance d(x) from a point to a primitive. i ,s j ): Where s j Let y represent the component reconstructed from the j-th primitive. k j This represents the k-th point among the K points uniformly sampled from the surface of the part reconstructed from the j-th primitive. Define the 2-norm coverage distance c of the primitive. c (s j ): Define the 1-norm coverage distance c of the primitive. f (s j ): Based on the three distances mentioned above, the loss function for training is defined as follows: 5.1 Reconstruction Loss Function L recon It consists of two parts: L recon <0.5*L pc2pr +0.5*L pr2pc Where L pc2pr and L pr2pc Let represent the reconstruction loss from point cloud to primitive and the reconstruction loss from primitive to point cloud, respectively: 5.2 Anti-anchoring loss function L hd : 5.3 Loss Function L of Antitrivial Solution wd : in, p ji This represents the element in the j-th row and i-th column of the probability matrix P for the instance segmentation task. 5.4 Compact Loss L c : 5.5 There is a loss L e : in: Where, δ j This represents the probability that the part reconstructed from the j-th primitive in the predicted shape summary is represented. Let represent the pseudo-label obtained from the statistics of the instance segmentation probability matrix P, and let Ι(·) denote the indicator function. The total loss function L used for training is shown below: L=L recon +λ1L hd +λ2L wd +λ3L c +λ4L e Where λ1, λ2, λ3 and λ4 represent L respectively. recon L hd L wd L c and L e The coefficient.
