Auto-encoder-based semantic consistent key point extraction method
Through a network composed of autoencoder and loss function, self-supervised training of semantic consistent key point extraction method solves the annotation dependence and noise robustness of key point extraction in the prior art, and realizes semantic consistency and robustness on 3D objects.
Patent Information
- Application Number
- CN202510538332.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-27
- Publication Date
- 2025-08-01
AI Technical Summary
In the extraction of key points of 3D objects, there are problems in the problem that supervised learning methods rely on large-scale manual annotation data sets, noise and data loss affect the quality of labeling, and unsupervised learning methods are difficult to determine specific key points in a specific 3D semantic region, and are not robust enough for point cloud rotation and noise interference.
The semantic consistent key point extraction method based on the autoencoder is adopted, and the key point extraction is self-supervised through a network composed of encoder and decoder, combining loss functions such as fidelity, coverage, resolution, shape and volume to ensure the semantic consistency and robustness of key point extraction.
Semantically consistent key point extraction is realized, which can withstand interference from point cloud noise and pose changes, and improve the accuracy and generalization ability of key point extraction.
Smart Images
Figure CN120411544A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of key point extraction, and specifically, to a semantic-consistent key point extraction method based on an autoencoder. Background Art
[0002] Using a key point set to represent a 3D object is a fundamental step in various geometric reasoning tasks, including pose estimation, action recognition, object tracking, shape registration, deformation, retrieval, and reconstruction. As the primary processing step, key points must be reliably extracted from the point cloud of the object because any error will have a negative impact on subsequent high-level tasks.
[0003] Key point extraction was initially defined as a supervised learning task: given a dataset of manually annotated key points of a point cloud, the model predicts the positions of the key points based on the input point cloud. Although these methods have achieved excellent results on supervised datasets, they also expose the limitations of supervised learning methods. The main problem is that a large-scale dataset containing accurate annotations for each object is required. Constructing such a dataset is very difficult because manually annotating 3D key points is both complex and time-consuming. In addition, noise or data missing in the point cloud will affect the annotation quality, and it may be difficult for annotators to accurately locate key points for highly symmetric or smooth-surfaced objects.
[0004] In view of these limitations, recent research methods have begun to turn to unsupervised learning to avoid the need for manual annotation. Among them, self-supervised learning methods obtain a large number of annotations by defining proxy tasks, which can be directly generated during the training process, such as geometric transformation, canonical mapping, reconstruction, etc. Unsupervised learning methods implicitly determine key points by analyzing the geometric characteristics of the object, such as through methods like point clustering and part contrast.
[0005] These shifts in learning paradigms have significantly improved the generalization ability of key point extraction, but they also bring some problems. In the absence of manual annotation, it is difficult to determine specific key points within a specific 3D semantic region if there are intra-class shape differences. For example Figure 1 for the airplane in , although the point cloud may undergo various changes, the ultimately desired key points always remain near the nose. In addition, the key point extraction process must be able to withstand common interferences acting on the point cloud, such as in the case of point cloud rotation, noise, or downsampling (as shown in Figure 1 ), and still maintain the accuracy of key point extraction. Summary of the Invention
[0006] In view of the above deficiencies in the prior art, the present invention provides a method for extracting semantically consistent key points based on an autoencoder, which can self-supervise the training of the network for extracting semantically consistent key points, ensure semantic consistency, and be robust to interference factors such as the pose change of the target object and point cloud noise.
[0007] To achieve the above object, the present invention provides a method for extracting semantically consistent key points based on an autoencoder, including the following steps:
[0008] Step 1, in the encoder, based on the original point cloud of the target, extract semantic key points, connect all pairs of semantic key points to construct a skeleton containing several connecting edges, and at the same time, the encoder outputs the edge activation intensity of each connecting edge;
[0009] Step 2, in the decoder, generate a sub-point cloud based on each connecting edge, and merge all the sub-point clouds according to the edge activation intensity to generate a reconstructed point cloud;
[0010] Step 3, construct a loss function, which is always composed of weighted fidelity loss, coverage loss, separation loss, shape loss, volume loss, and overlap loss;
[0011] Step 4, with the goal of minimizing the loss function, self-supervise the training of the encoder and the decoder, and perform semantic key point extraction based on the trained encoder.
[0012] Compared with the prior art, the present invention has the following beneficial technical effects:
[0013] The present invention jointly forms a network for extracting semantically consistent key points by an encoder and a decoder. The encoder is responsible for encoding the input object point cloud into a series of key points and a skeleton composed of the connections between these key points. The decoder reconstructs the point cloud of the object according to the skeleton and self-supervises the training of the network with the goal that the reconstructed object point cloud is as close as possible to the original input point cloud. In addition, by designing the loss function, the semantic consistency of the key points is ensured, and the method is robust to interference factors such as the pose change of the object and point cloud noise. BRIEF DESCRIPTION OF THE DRAWINGS
[0014] In order to more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art. Obviously, the following drawings are only some embodiments of the present invention. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained based on the structures shown in these drawings.
[0015] Figure 1 It is an exemplary diagram of the robustness of point cloud semantic key extraction in an embodiment of the present invention;
[0016] Figure 2 This is the flowchart of the semantic - consistent key - point extraction method based on the auto - encoder in the embodiments of the present invention;
[0017] Figure 3 This is the schematic diagram of the encoder in the embodiments of the present invention;
[0018] Figure 4 This is the schematic diagram of the decoder in the embodiments of the present invention.
[0019] The realization, functional characteristics and advantages of the object of the present invention will be further described in conjunction with the embodiments with reference to the accompanying drawings. Detailed implementation manners
[0020] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without making creative efforts belong to the scope of protection of the present invention.
[0021] In addition, in the present invention, descriptions such as "first" and "second" are only for descriptive purposes, and cannot be understood as indicating or implying their relative importance or implicitly indicating the quantity of the indicated technical features. Thus, features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of the present invention, "a plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.
[0022] In addition, the technical solutions between various embodiments of the present invention can be combined with each other, but it must be based on the fact that those of ordinary skill in the art can implement them. When the combination of technical solutions results in contradictions or cannot be implemented, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.
[0023] This embodiment discloses a semantic - consistent key - point extraction method based on an auto - encoder, which forms a key - point extraction network capable of self - supervised training through an encoder and a decoder. In the key - point extraction network, the encoder first generates an ordered list containing k semantic key - points That is, the set of semantic key points. By connecting all pairs of semantic key points, the skeleton of the input point cloud is constructed, which contains M = k(k - 1) / 2 connecting edges. Then, by uniformly sampling each connecting edge of the skeleton and applying a series of learned local offset vectors, these connecting edges are further refined to form M refined point clouds. These point clouds are then filtered according to the M edge activation intensities output by the encoder and finally merged in the decoder to generate the reconstructed point cloud. Then, the loss function is calculated based on the data generated during the operation of the encoder and decoder, and the encoder and decoder are self-supervised trained with the goal of minimizing the loss function. Finally, the trained encoder is used for semantic key point extraction.
[0024] Reference Figure 2 , in this embodiment, the semantic consistent key point extraction method based on the autoencoder specifically includes the following steps:
[0025] Step 1, in the encoder, extract semantic key points based on the original point cloud of the object, connect all pairs of semantic key points to construct a skeleton containing several connecting edges, and at the same time, the encoder outputs the edge activation intensity of each connecting edge;
[0026] Step 2, in the decoder, generate sub-point clouds based on each connecting edge and merge all sub-point clouds according to the edge activation intensity to generate the reconstructed point cloud;
[0027] Step 3, construct the loss function, which is always composed of weighted fidelity loss, coverage loss, separation loss, shape loss, volume loss, and overlap loss;
[0028] Step 4, with the goal of minimizing the loss function, self-supervised train the encoder and decoder, and perform semantic key point extraction based on the trained encoder.
[0029] Reference Figure 3 , the encoder in this embodiment includes a PointNet++ network, a softmax activation function, and a three-layer fully connected network. The specific implementation process of the encoder for key point extraction, skeleton construction, and edge activation intensity prediction is as follows:
[0030] First, obtain k per-point score matrices and the global feature vector of the original point cloud based on the PointNet++ network, where N is the number of points in the original point cloud;
[0031] Then, after performing softmax normalization on the per-point score matrix using the softmax activation function, calculate the weighted average value to generate the coordinates of k semantic key points, where the weighted average is achieved through matrix multiplication;
[0032] Meanwhile, the three-layer fully connected network receives the global feature vectors extracted by the PointNet++ network and predicts M edge activation intensities activated by the softmax function. The edge activation intensities are mainly used to mask certain connecting edges before the merging stage of the decoder.
[0033] Reference Figure 4 , the decoder receives the semantic key points generated by the encoder as input and uniformly samples on the connecting edges between the semantic key points to generate M initial sub-point clouds P1, P2, …, P M . The number of sampling points n i for each connecting edge is proportional to the length of the connecting edge. For each initial sub-point cloud it is refined by adding local offsets that offset the point-wise positions . Among them, the local offset B i is a trainable parameter of the network, and L2 regularization is used to maintain the locality of the refinement process. Finally, using the activation intensities generated by the encoder, the refined M sub-point clouds are merged into a unified reconstructed point cloud That is:
[0034]
[0035] where a i is the edge activation intensity of the i-th connecting edge.
[0036] In the process of implementing point cloud reconstruction, a core challenge is how to quantify the similarity between the input original point cloud X and the reconstructed point cloud . The reconstructed point cloud is composed of multiple sub-point clouds which are determined whether to be masked by the edge activation intensity a i . This embodiment solves this problem by adopting a composite Chamfer distance function (CCD). This composite Chamfer distance function is an extension based on the traditional Chamfer distance, enabling it to more accurately measure the difference between the reconstructed point cloud composed of multiple sub-point clouds and the original point cloud. Similar to the conventional Chamfer distance, the composite Chamfer distance function consists of a forward fidelity loss and a backward coverage loss. The difference is that the reconstruction result is composed of multiple sub-point clouds, while the input is a single point cloud. This structural asymmetry leads to the corresponding asymmetry in the design of the fidelity loss and the coverage loss to adapt to the actual situation of the problem.
[0037] In this embodiment, the fidelity loss is obtained by a direct extension of the Chamfer distance, considering the influence of the activation intensity on each sub-point cloud. Its calculation process is as follows:
[0038]
[0039] where L f is the fidelity loss, is a point in the sub-point cloud , and p0 is a point in the original point cloud X.
[0040] In this embodiment, the calculation process of the coverage loss is as follows:
[0041] First, calculate the coverage loss of any point p0 in the original point cloud X, including the following steps:
[0042] Step 301, initialize the remaining sub-point cloud set cumulative activation intensity current point coverage loss
[0043] Step 302, search for the nearest neighbor of point p0 in the remaining sub-point cloud set and determine the sub-point cloud to which the point belongs and its edge activation intensity a i ;
[0044] Step 303, weight the distance from the point to point p0 into the loss, and update the current point coverage loss and the cumulative activation intensity, as:
[0045]
[0046] Step 304, judge whether holds:
[0047] If so, perform Step 305;
[0048] Otherwise, directly output the current point coverage loss
[0049] Step 305, after removing the sub-point cloud from the remaining sub-point cloud set , judge whether the remaining sub-point cloud set R is an empty set:
[0050] If so, let the current point coverage loss and then output it, where γ is a penalty term, for example, γ can be set to 20;
[0051] Otherwise, return to Step 302;
[0052] After obtaining the coverage loss of all points in the original point cloud X, the coverage loss is obtained by aggregating the coverage losses of all points. For example, the average value of the coverage losses of all points is directly used as the coverage loss.
[0053] In the specific application process, the sub-point cloud with a greater contribution to the coverage will have a smaller backward coverage loss. And due to the penalty mechanism of the edge activation intensity, the activation intensity of these parts will increase. Relatively, the fidelity loss will reduce the activation intensity of the edges that do not exist in the skeleton. By taking the composite Chamfer distance function as part of the loss function, the network is forced to generate a set of sub-point clouds with reasonable activation values, thus promoting the training process. The composite Chamfer distance function L in this embodiment CDD is composed of the fidelity loss L f and the coverage loss L c weighted as follows:
[0054] L CDD = λ f ·L f + λ c ·L c
[0055] where λ f and λ c are weight coefficients.
[0056] To make up for the deficiencies of the composite Chamfer distance function in terms of the semantic consistency of key points and the robustness to point cloud interference, this embodiment also introduces an additional loss function, that is, the individual loss is mainly used to measure the matching degree between the key points output by the encoder and the point cloud shape.
[0057] The individual loss is calculated for the original point cloud data of a single shape. Assume that k semantic key points are output where k << N. The semantic key points should have the following characteristics: they should be relatively dispersed, covering the volume of the entire object as much as possible, while still being close to the original point cloud and not overlapping with each other. Therefore, the individual loss in this embodiment consists of four parts: separation loss, shape loss, volume loss, and overlap loss.
[0058] The separation loss is used to maximize the distance between each semantic key point and its neighboring key points in the semantic key point set K, thereby promoting a more dispersed distribution of semantic key points. The calculation process of the separation loss in this embodiment is:
[0059]
[0060] where L sep is the separation loss, and e jis the j-th semantic key point, and KNN(e j , K) is the nearest neighbor key point of the semantic key point e j in the set of semantic key points K. δ is a factor to prevent gradient explosion. For example, it can be set to δ = 0.01, so as to effectively prevent the separation loss value from tending to infinity when all semantic key points are wrongly estimated to be at the same position.
[0061] Since there is no maximum distance limit when the separation loss moves the semantic key points away from their neighbors, it may cause the semantic key points to be far from the target object itself or even farther. To avoid this situation, in this embodiment, a shape loss is introduced to ensure that the semantic key points fit more closely to the shape of the target object. Specifically, the shape loss calculates the distance between each semantic key point in the set of key points K and its nearest neighbor point in the input original point cloud. The calculation process of the shape loss in this embodiment is as follows:
[0062]
[0063] where L shape is the shape loss, e j is the j-th semantic key point, and KNN(e j , K) is the nearest neighbor point of the semantic key point e j in the original point cloud X.
[0064] Although the separation loss and the shape loss help to optimize the distribution of the semantic key points, they do not directly consider the distribution of the semantic key points on the entire shape of the target object. To ensure that the semantic key points can fully cover the entire target object, in this embodiment, a volume loss is introduced, which is realized by calculating the difference between the estimated semantic key points and the three-dimensional volume of the input original point cloud. The calculation process of the volume loss in this embodiment is as follows:
[0065] L volume = |vol(K) - vol(X)|
[0066] where L volume is the volume loss, vol(K) is the volume of the minimum bounding box of the set of semantic key points K, and vol(X) is the volume of the minimum bounding box of the original point cloud X.
[0067] In addition, to avoid multiple key points being estimated to the same three-dimensional position, in this embodiment, the concept of overlap loss is also introduced to calculate the degree of overlap between the key points. The overlap loss is calculated by considering the distance between each pair of semantic key points in the set of semantic key points. If the distance between two semantic key points is less than a certain threshold, they are considered to overlap. The calculation process of the overlap loss in this embodiment is as follows:
[0068]
[0069] Among them, L overlap is the overlap loss, and e i , e j are the i-th and j-th semantic key points respectively. ||e i -e j || 2 represents the square of the Euclidean distance between the semantic key points e i , e j . τ1 is a preset threshold used to determine whether two semantic key points are close enough to be considered overlapping. The summation symbol iterates over all pairs of semantic key points, and [·] is an indicator function that takes the value 1 when the condition ||e i -e j || 2 < τ1 holds, and 0 otherwise. By minimizing the overlap loss, the model can be encouraged to spread the key points out and prevent them from clustering at the same location. The total individual loss L IND can be summarized as a weighted sum of the above loss components, i.e.:
[0070] L IND = λ sep ·L sep + λ shape ·L shape + λ volume ·L volume + λ overlap ·L overlap
[0071] Among them, λ sep , λ shape , λ volume , λ overlap are weight coefficients;
[0072] Therefore, the final loss function L can be set as L = L CDD + L IND .
[0073] As a preferred implementation, this embodiment also introduces an interdependence loss into the loss function L. Specifically, the original point cloud is randomly rotated twice, and two sets of new semantic key points are generated by the encoder. The interdependence loss is calculated through these two sets of semantic key points, aiming to evaluate the semantic consistency between the two sets of semantic key points. Specifically, the interdependence loss includes a consistency loss and a pose loss, and its calculation process is as follows:
[0074] First, use the rotation matrices R A , R B to apply rotations to the original point cloud respectively to obtain the first point cloud and the second point cloud;
[0075] Then, input the first point cloud and the second point cloud into the encoder respectively to obtain the first set of semantic key points corresponding to the first point cloud and the second set of semantic key points corresponding to the second point cloud;
[0076] Subsequently, calculate the consistency loss based on the consistency of the coordinates and indices of the first set of semantic key points and the second set of semantic key points, which is:
[0077]
[0078] where L consist is the consistency loss, k is the number of semantic key points in the first set of semantic key points and the second set of semantic key points, is the i-th semantic key point in the first set of semantic key points, is the i-th semantic key point in the second set of semantic key points;
[0079] Meanwhile, calculate the relative rotation R est between the first set of semantic key points and the second set of semantic key points, and calculate the pose loss based on the relative rotation R est and the rotation matrices R A 、R B , which is:
[0080]
[0081] where L pose is the pose loss;
[0082] Finally, weight the consistency loss and the pose loss, that is, obtain the interdependence loss, which is:
[0083] L MUT =λ consist ·L consist +λ pose ·L pose
[0084] where L MUT is the interdependence loss, λ consist 、λ pose are weight coefficients;
[0085] Therefore, the final loss function L can be set as L = L CDD +L IND +L MUT .
[0086] The above are only the preferred embodiments of the present invention, and do not limit the protection scope of the present invention accordingly. Any equivalent structural transformation made under the inventive concept of the present invention by using the content of the specification and drawings of the present invention, or any direct / indirect application in other related technical fields is included in the protection scope of the present invention.
Claims
1. A semantic-consistent key point extraction method based on an autoencoder, characterized in that, It includes the following steps: Step 1: In the encoder, semantic key points are extracted based on the original point cloud of the target, and all pairs of semantic key points are connected to construct a skeleton containing several connecting edges. Meanwhile, the encoder outputs the edge activation intensity of each connecting edge; Step 2: In the decoder, sub-point clouds are generated based on each connecting edge, and all the sub-point clouds are merged according to the edge activation intensity to generate a reconstructed point cloud; Step 3: A loss function is constructed, and the loss function is always composed of a weighted combination of fidelity loss, coverage loss, separation loss, shape loss, volume loss, and overlap loss; Step 4: With the goal of minimizing the loss function, the encoder and the decoder are self-supervised trained, and semantic key point extraction is performed based on the trained encoder.
2. The semantic consistency key point extraction method based on an autoencoder according to claim 1, wherein The specific process of Step 1 is as follows: In the encoder, k per-point score matrices and a global feature vector of the original point cloud are obtained based on the PointNet++ network; After performing softmax normalization on the per-point score matrices, the weighted average is calculated to generate the coordinates of k semantic key points; The global feature vector is processed based on a three-layer fully connected network to predict the edge activation intensity of M connecting edges, where M = k(k - 1) / 2.
3. The method for extracting semantically consistent key points based on an autoencoder according to claim 2, wherein The specific process of Step 2 is as follows: Uniformly sample each connecting edge to generate an initial sub-point cloud P i , where i = 1 to M; Add a learnable local offset to each initial sub-point cloud to generate the sub-point cloud corresponding to each edge It is as follows: Among them, B i is the local offset; Merging all the sub-point clouds according to the edge activation intensity to generate a reconstructed point cloud is as follows: where a i is the edge activation intensity of the i-th connecting edge.
4. The method for extracting semantically consistent key points based on an autoencoder according to claim 3, wherein In Step 3, the calculation process of the fidelity loss is as follows: Among them, L f is the fidelity loss, is the point in the sub - point cloud , and p0 is the point in the original point cloud X.
5. The method for extracting semantically consistent key points based on an autoencoder according to claim 3, wherein In Step 3, the calculation process of the coverage loss is as follows: Calculating the coverage loss of any point p0 in the original point cloud X includes the following steps: Step 301, initialize the remaining subset of point clouds Cumulative activation intensity Current point coverage loss Step 302, search for the nearest neighbor of point p0 in the remaining sub-point cloud set and determine the sub-point cloud to which the point belongs and its edge activation intensity a i ; Step 303, weight the distance from point to point p0 into the loss, and update the current point coverage loss and the cumulative activation intensity as follows: Step 304, determine whether holds: If so, go to Step 305; Otherwise, directly output the current point coverage loss Step 305, remove the sub-point cloud from the remaining sub-point cloud set and then determine whether the remaining sub-point cloud set is an empty set: If so, let the current point covering loss and output it, where γ is the penalty term; Otherwise, return to Step 302; After obtaining the coverage losses of all points in the original point cloud X, the coverage losses of all points are aggregated to obtain the coverage loss.
6. The method for extracting semantically consistent key points based on an autoencoder according to claim 3, wherein In Step 3, the calculation process of the separation loss is as follows: Among them, L sep is the separation loss, e j is the j-th semantic key point, KNN(e j , K) is the nearest neighbor key point of the semantic key point e j in the semantic key point set K, and δ is the anti-gradient explosion factor.
7. The method for extracting semantically consistent key points based on an autoencoder according to claim 3, wherein In Step 3, the calculation process of the shape loss is as follows: Among them, L shape is the shape loss, e j is the j-th semantic key point, and KNN(e j , K) is the nearest neighbor point of the semantic key point e j in the original point cloud X.
8. The method for extracting semantically consistent key points based on an autoencoder according to claim 3, characterized in that In Step 3, the calculation process of the volume loss is as follows: L volume = |vol(K) - vol(X)| where L volume is the volume loss, vol(K) is the volume of the minimum bounding box of the semantic key point set K, and vol(X) is the volume of the minimum bounding box of the original point cloud X.
9. The method for extracting semantically consistent key points based on an autoencoder according to claim 3, wherein In Step 3, the calculation process of the overlap loss is as follows: Among them, L overlap is the overlapping loss, e i , e j are the i-th and j-th semantic key points respectively, and τ1 is the threshold value.
10. The method for extracting semantically consistent key points based on an autoencoder according to any one of claims 1 to 9, characterized in that In Step 3, the loss function also includes a consistency loss and a pose loss, and the calculation process of the consistency loss is as follows: Using the rotation matrices R A and R B to apply rotations to the original point cloud respectively, obtaining a first point cloud and a second point cloud; The first point cloud and the second point cloud are respectively input into the encoder to obtain a first set of semantic key points corresponding to the first point cloud and a second set of semantic key points corresponding to the second point cloud; The consistency loss is calculated based on the consistency of the coordinates and indices of the first set of semantic key points and the second set of semantic key points, and is: Among them, L consist is the consistency loss, k is the number of semantic key points in the first semantic key point set and the second semantic key point set, is the i-th semantic key point in the first semantic key point set, is the i-th semantic key point in the second semantic key point set; Calculate the relative rotation R between the first set of semantic key points and the second set of semantic key points est , and based on the relative rotation R est and the rotation matrix R A , R B Calculate the pose loss as follows: Among them, L pose is the pose loss.