Point cloud classification

Abstract

To provide a computer-implemented method for performing point cloud classification.SOLUTION: A method comprises steps of: selecting a point cloud for classification; obtaining a plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud; obtaining a graph structure representation of the plurality of subsets, where the graph structure representation has a node associated with each subset and one or more edges connecting the nodes; inputting the graph structure representation into a graph encoder convolutional neural network; obtaining a graph structural embedding that encapsulates a structural relationship between the plurality of subsets; deriving a point cloud representation from the graph structure embedding; inputting the point cloud representation into a classification neural network to obtain a classification of the selected point cloud; and classifying the point cloud representation.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] The present invention relates to training and using point cloud classifiers, and in particular to computer-implemented methods, computer programs, and information programming apparatus. [Background technology]

[0002] Three-dimensional (3D) scene analysis, which automatically assigns meaningful labels to scene content, is an increasingly important technique with wide applications across photogrammetry, remote sensing, computer vision, and robotics. With the increasing availability of 3D point cloud data, it is becoming important in this field to be able to accurately classify 3D point clouds, especially for applications such as autonomous driving, augmented reality, medical imaging, and geographic information systems.

[0003] A core problem in this field is that point cloud classification models are trained on a specific labeled source domain dataset, but are required to classify features in a large number of unseen, unlabeled target domains during use. In practice, the source domain for training is typically a dataset with a large number of labeled, pristine (clean) examples, while the target domain is an unseen dataset with only a small number of labeled examples or no labeled examples at all, and typically contains noisy data points. For example, the target dataset may involve different or inconsistent data acquisition processes. Therefore, to accurately classify features, a classification model must have a certain level of domain generalization that enables it to accurately classify features in a new, unseen, unlabeled domain despite only being trained on another labeled domain. For example, in the specific example of autonomous driving, 3D point clouds are used to represent the environment surrounding a vehicle. Here, a classification model trained on one domain, such as a specific city dataset, may face difficulties in accurately classifying features in a different domain, such as a different city, due to the diversity of the environment.

[0004] An important aspect of improving the accuracy of point cloud classification models relates to improving the level of 3D domain generalization. Essentially, this approach is a machine learning technique that improves the performance of a classification model through identifying and learning generalizable features of data in a source domain, so that those generalizable features can be better identified in a target domain. Specifically, this can involve identifying and learning various representations and relationships in the labeled feature data of the source domain so that those generalizable features can be more accurately and easily identified in an unseen, unlabeled target domain that shares a common label space with the source domain. Improved identification of these generalized features directly translates to improved classification accuracy of features in the target domain.

[0005] It is desirable to improve the accuracy of point cloud classification. Summary of the Invention

[0006] According to a first aspect, a computer-implemented method for performing point cloud classification is disclosed herein, the method comprising: Select the point cloud for classification, obtaining a plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud; obtaining a graph-structured representation of the plurality of subsets, the graph-structured representation having a node associated with each subset and one or more edges connecting the nodes; inputting the graph structure representation into a graph encoder convolutional neural network to obtain a graph structure embedding that encapsulates structural relationships between the plurality of subsets; Deriving a point cloud representation from the graph structure embedding; classifying the point cloud representation by inputting the point cloud representation into a classification encoder neural network to obtain a classification of the selected point cloud; It has the following.

[0007] The present invention has been described with reference to specific embodiments. Other embodiments are within the scope of the following claims. For example, the steps of the invention can be performed in a different order and still achieve desirable results.

[0008] Those skilled in the art will appreciate that, except where mutually exclusive, features described with respect to any of the above aspects may also be applied mutatis mutandis to any other aspect, and, except where mutually exclusive, features described herein may be applied to any aspect and / or combined with any other feature described herein. [Brief explanation of the drawings]

[0009] Reference will now be made, by way of example, to the accompanying drawings, in which: [Figure 1] FIG. 10 is a diagram showing a comparative method. [Figure 2] FIG. 10 is a diagram showing a comparative method. [Figure 3] FIG. 10 is a diagram showing a comparative method. [Figure 4] FIG. 10 is a diagram showing a comparative method. [Figure 5] FIG. [Figure 6] FIG. [Figure 7] FIG. [Figure 8] FIG. [Figure 9] FIG. [Figure 10] FIG. 1 illustrates the training process. [Figure 11] FIG. [Figure 12] 1 is a diagram useful for understanding an embodiment. [Figure 13] The results are shown in a table. [Figure 14] The table shows the results. [Figure 15] FIG. 1 shows an apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0010] Figure 1 illustrates the first comparative method (Comparative Method 1). Comparative Method 1 can be referred to as MetaSets (Huang) (Huang et al., 2022, 'MetaSets: Meta-Learning on Point Sets for Generalizable Representations'). Comparative Method 1 is a method for point cloud classification that aims to learn generalizable representations that can transfer well across different point sets. In particular, it relates to 3D domain generalization (3DDG), a machine learning technique that aims to improve the performance of models trained on a labeled 3D point cloud source domain when performing classification on one or more different, unlabeled, unseen 3D point cloud target domains by learning generalized features. 3DDG aids in the transfer of models trained using synthetic data to various real-world scenarios in applications such as autonomous driving and advanced driver assistance systems (ADAS). To attempt to achieve this form of unsupervised domain generalization on point clouds, Comparison Method 1 subjects a point set in a specific source domain to several different data transformations / augments, each with a different geometry prior, and then attempts to meta-learn a point cloud representation from a classification task performed on this group of carefully designed, differently transformed point sets containing specific geometry priors. Figure 1 illustrates the framework of the meta-learning approach, including a specific meta-task in which examples of different data transformations / augments used as input are random dropping, non-uniform density, perspective, projection, etc. By performing a large number of different data transformations / augments, each with a different geometry prior, Comparison Method 1 aims to provide a MetaSet capable of inducing a larger set of geometry priors that are more likely to contain geometry priors similar to those from the unseen target domain.Thus, this aims to provide a classification method capable of improved domain generalization. However, a drawback of this approach is that it takes the complete point cloud for processing as a single monolithic entity for each case, and in doing so, ignores the intricate complexities and nuances inherent within the object and the inherent geometric structure of the object that persists regardless of domain. Therefore, it is desirable to improve the classification accuracy of Comparison Method 1.

[0011] Figure 2 shows the second comparative method (Comparative Method 2). Comparative Method 2 can be referred to as part-based feature representation (Wei et al., 2022, 'Learning Generalizable Part-based Feature Representation for 3D Point Clouds'). Comparative Method 2 aims to provide unsupervised domain generalization for point cloud object classification. It does so by providing a generalizable part-based feature representation and using a part-based domain generalization network (PDG). However, to do so, this method requires constructing a part-template feature space shared by the source and target domains. As shown in Figure 2, shapes from separate domains are first organized into part-level features and then represented by the part-template features. These transformed part-level features are called aligned part-based representations and are then aggregated by a part-based feature aggregation module. As shown in Figure 2, given a point cloud from the source or target domain, it is first processed by a feature encoder and organized into part-level features. The part-level features are then converted into aligned part-based features by aligning them with the part template features. The aligned part-based features are then aggregated into a global representation by a part-based feature aggregation module. A problem with this method is that, although it discusses using part-level features during training, these part-level features are obtained by performing farthest point sampling (FPS) to specifically sample M center points to construct M parts. However, FPS is inherently random and therefore does not guarantee that the identified parts are meaningful. Furthermore, in contrast to the implementation of the present application, comparative method 2 does not consider that relationships between parts may be useful and therefore does not consider learning relationships between parts.On the other hand, we propose to identify parts of point clouds, especially meaningful parts, and their relationships to learn domain-invariant features.

[0012] Figure 3 shows the third comparative method (Comparative Method 3). Comparative Method 3 can be referred to as SUG (Huang et al., 2023, 'SUG: Single-dataset Unified Generalization for 3D Point Cloud Classification'). Comparative Method 3 aims to provide unsupervised domain generalization for object point cloud classification. To do so, it proposes a Single-Dataset Unified Generalization (SUG) framework that utilizes only a single source dataset to mitigate unexpected domain differences faced by the trained source model. In particular, it proposes to design a Multi-Grained Sub-Domain Alignment (MSA) method that can constrain learned representations to be domain-independent and discriminative by performing a multi-grained feature alignment process across divided subdomains from a single source dataset. A sample-level domain-aware attention (SDA) strategy is then presented, which can selectively enhance adaptive samples from different subdomains according to sample-level inter-domain distances in an attempt to avoid negative transfer. Figure 3 shows a SUG framework consisting of multi-granularity subdomain alignment (MSA) and sample-level domain-aware attention (SDA) to address the one-to-many domain generalization problem. However, the problem with comparison method 3 is that it shares the combined shortcomings of comparison methods 1 and 2. In particular, it takes the complete point cloud as a single monolithic entity for processing, and in so doing, ignores the intricate complexities and nuances inherent within the object and the inherent geometric structure of the object that persists regardless of domain.For the same reason, it does not consider decomposing an object into parts, let alone meaningful parts, much less considering that relationships between parts may be useful, and therefore does not consider learning relationships between parts. It is therefore desirable to improve the accuracy of comparison method 3.

[0013] Figure 4 shows the fourth comparative method (Comparative Method 4). Comparative Method 4 can be referred to as SRG-Net (Hu et al., 2022, 'Unsupervised Segmentation for Terracotta Warrior Point Cloud (SRG-Net)'). Comparative Method 4 aims to automatically segment 3D point cloud data, particularly of the Chinese Terracotta Warriors, store the fragment data in a database, and assist archaeologists in matching actual fragments with those in the database. To achieve this, it performs several high-level operational steps that can be decomposed into three broad stages: coarse segmentation of the point cloud into clusters, unsupervised refinement of the clusters, and finally, once this labeling is performed, supervised training on soft labels. In particular, it first coarsely labels the input point cloud with cluster labels using a Seed Region Growing (SRG) algorithm. It then uses a three-stage encoder-decoder architecture with two branches: an 'Edge-Conv' branch for generating a dynamic graph from the KNN and a 'Graph-Conv' branch for creating a bottleneck. A segmenter network is then used on the concatenated three dynamic graphs and the bottleneck to predict a class label for each point on the point cloud. A subsequent refinement strategy is then used to attempt to achieve a better cluster label assignment for each point in the point cloud. Finally, a supervised training stage is used, whereby predicted labels are assigned in an unsupervised manner, and the entire network is trained on the predicted labels in a supervised manner to obtain the final segmentation results. Therefore, comparative method 4 first encodes the input point cloud using an STN (transformer-based model) and then passes it to a dynamic graph convolutional neural network (DGCNN) encoder.The encoder creates dynamic graphs by using a K-nearest neighbor (KNN) algorithm at different stages of the process. Therefore, a problem with this approach is that all graphs are created for embeddings, not meaningful parts of the input point cloud. In fact, instead, it directly feeds the entire monolithic point cloud into a multi-layer GCN-based network to learn a partial graph representation without enforcing a geometry-preserving structure. Therefore, comparative method 4 ignores the intricate complexity and nuance inherent in objects and the inherent geometric structure of objects that persists regardless of domain. For the same reason, it does not consider decomposing objects into parts, let alone meaningful parts, much less considering that relationships between parts may be useful, and therefore does not consider learning relationships between parts. Furthermore, comparative method 4 uses a supervised approach, which disadvantageously requires annotated and pseudo-annotated data for classification. In other words, it requires ground truth, since predicted labels are used as soft labels for supervised training of the segmenter, and ground truth labels are used for loss calculation. Therefore, it is desirable to improve the accuracy of Comparison Method 4.

[0014] Aspects of the present application aim to address these shortcomings in the current state-of-the-art of point cloud classification.

[0015] One aim of aspects of the present application is to determine a way to segment an input point cloud that provides a subset collection of parts of the input point cloud that can be considered meaningful, rather than random. In other words, aspects of the present application aim to identify and decompose the input point cloud into parts that can be considered to represent, for example, some fundamental or consistent aspect of the object that the point cloud data describes, such that those parts are present, on average or substantially, in all objects of that classification and are therefore useful, for example, for classifying the objects in the point cloud.

[0016] One aspect of what is considered meaningful is, for example, a part that is recognizable by a human as semantically significant; i.e., in the chair example, one way a human might identify and understand this concept is as a composite structural concept formed from entailed concepts such as legs, back, and seat, which are configured in a particular relationship. Here, concepts such as legs, back, and seat are semantically significant parts. Another aspect of what is considered meaningful is a geometrically significant part, in the sense that it represents or relates to properties that are considered to define the unique geometric structure of a particular object. Of course, semantically significant and geometrically significant parts are not mutually exclusive; they may be the same, share overlap, or be completely separate in certain circumstances.

[0017] In other words, it has been realized that the inaccuracies and failures of current state-of-the-art point cloud classification methods are in part due to the fact that they either do not decompose objects into parts at all, but instead treat point clouds monolithically during processing, or they decompose point clouds into random parts, such as by using farthest point sampling. However, randomly decomposing point clouds is only useful for reducing the processing load of point cloud classification and does little to improve the accuracy of the classification itself. Therefore, to improve classification accuracy, aspects of the present application have determined that it would be desirable to be able to identify and determine parts of objects that are useful for identifying those objects, e.g., by being parts that are present on average or substantially present and in a particular relationship in all objects of a particular classification, for example, by determining geometrically meaningful parts in that they are thought to represent, relate to, and / or define parts of the unique geometric structure of a particular object that persists regardless of domain.

[0018] In particular, as mentioned above, real-world objects are often composed of two or more identifiable parts, each of which may relate to concepts inherent in all or most objects of that same class. For example, in the chair example above, one way humans understand it is as a composite structural concept formed from entailed concepts, such as legs, back, and seat, which are arranged in a specific relationship. Given that the class concept of chair is a human construct, all chairs share at least most of these features, often in the same relationship. Thus, in a high-level sense, one way humans identify a chair is by determining the existence of the concepts that make up a chair and determining that they exist in a specific relationship to one another. This same principle applies to most objects in the 3D world; they can be said to be composed of specific identifiable parts that are in specific identifiable relationships.

[0019] Thus, one aim of aspects of the present application is to seek a way to approximate and / or incorporate a similar procedure in a point cloud classification model by identifying so-called 'meaningful' parts of objects of a particular class, so that those features are useful and can be used to pinpoint the class of the object by determining their presence and their structural relationships, and / or their presence and structural relationships are on average or substantially invariant across all objects in that class regardless of, for example, domain. Those meaningful parts may be what humans understand, such as the legs and back of a chair, or they may or may not correlate to what humans understand, but the same principles apply, and aspects of the present application relate to a system that breaks down objects into smaller component parts, determines the relationships between them, and from this determines the classification of the object.

[0020] As further described below, a training process is first required in which a set of parts that are considered and determined to be meaningful for a particular class of objects, and their structural relationships, can be learned from the labeled source domain across different classes, thereby enabling the system, once trained, to determine objects from a target, different domain that share the classification of the source domain. Accordingly, aspects of the present application seek to determine parts within objects that share a classification, which are considered meaningful in the sense that they are present on average or substantially in all point clouds of objects of a particular class, and are therefore useful and can be used to accurately identify the classification of an object by determining the presence of those meaningful parts and their structural relationships. Aspects of the present application, in particular, identify parts that are geometrically meaningful for a particular object class, which are determined to represent or be associated with properties that are considered to define the unique geometric structure of a particular object that persists regardless of domain.

[0021] Certain implementations of the present application can provide a point cloud classification model that can exploit meaningful part-level information (with fine granularity) without incurring additional annotation, rendering costs, and / or requiring prior knowledge of the point cloud data.

[0022] According to a particular implementation of this application, it is possible to learn a high-quality representation of an input point cloud by utilizing both unsupervised partial decomposition and graph structure induction on the source point cloud.

[0023] Certain implementations of the present application provide point cloud classification with improved classification accuracy in the context of domain generalization over multiple unseen domains that share a common label space with the source domain.

[0024] According to certain implementations of the present application, a point cloud classification model is provided that is capable of learning local geometry-preserving graph structure representations in or of point clouds that are persistent across a particular class domain, thereby enabling the learning of better generalized representations.

[0025] According to certain implementations of the present application, an improved point cloud classification model is provided that allows for improved classification accuracy.

[0026] Figure 5 illustrates a classification process according to one particular implementation. Figure 5 illustrates components of a point cloud classification model 1 configured to take an unlabeled point cloud as input and perform classification of the point cloud, e.g., to output point cloud classification labels and / or to identify real-world object(s) within the point cloud, e.g., to identify one or more real-world objects within the point cloud. The received unlabeled point cloud data is therefore in the target domain to be classified. The point cloud data may be received, for example, from a sensor on, for example, an autonomous vehicle or a robotic arm or an augmented reality device.

[0027] The point cloud is augmented by the point cloud classification model in two ways. First, it is decomposed or segmented into subsets, each of which is a part of the original point cloud, or in other words, a smaller point cloud formed from a part of the original point cloud. Second, a graph structure is induced on the subsets associated with the parts to form a graph-structured representation of the parts of the point cloud. The graph structure is formed with nodes to the parts and edges connecting the nodes. In other words, the graph structure represents the structural relationships between the parts.

[0028] Following expansion, the graph structure representation is passed through a machine learning encoder, which obtains a graph structure embedding having extracted and encapsulated information defining structural relationships between parts based on the graph structure representation. The graph structure embedding forms a representation of the original point cloud. This point cloud representation is then classified by a machine learning classification model, e.g., to output classification labels identifying one or more real-world objects in the point cloud. The machine learning encoder, e.g., a graph encoder, has network weights and can be trained as described below.

[0029] 5, the decomposition of the input point cloud 2 into parts is performed by an Unsupervised Part Decomposition (UPD) module 3, which performs an unsupervised segmentation of the point cloud into parts, in particular parts that are deemed to be meaningful, e.g., as broadly outlined above. The specific functionality of the UPD module 3 and how it determines meaningful parts are described in more detail below.

[0030] As can be seen in Figure 5, an example input point cloud 2 is provided that a human could identify as a chair, but this depiction is merely for ease of visual understanding. The input point cloud 2 is presented as conventional input data and may form or represent any number of different real-world objects belonging to any number of different domains. Implementations of the present disclosure apply equally to any domain and any point cloud input thereto, providing an unseen target domain that shares labels and classifications with the source domain on which the system was trained. Throughout this application, a point cloud representing a chair will be used as an example.

[0031] The output 5 of the UPD module 3 is meaningful portions of the point cloud into which the input point cloud 2 has been segmented. For ease of visual understanding, the different portions into which the UPD module 3 has determined to segment the input point cloud 2 are shown as portions of chairs with different hatching, as shown in the UPD output 4 of Figures 5 to 9. However, this is merely a visual representation to aid understanding and is not intended to be limiting.

[0032] 5, graph structure induction to meaningful parts is performed by a Graph Structure Induction (GSI) module 5, which performs fine-grained graph structuring, for example, by inducing nodes to meaningful parts and connecting them with edges. The specific functionality of the GSI module 5 and the manner in which it determines nodes and edges are described in more detail below.

[0033] The output of the GSI module 5 is a graph-structured representation of meaningful parts formed by nodes and edges. In particular, to facilitate visual understanding, the output of the GSI module 5 is visually depicted as meaningful parts of a segmented chair with embedded nodes and connected edges, as shown in the GSI output 6 of Figures 5 to 9. However, this is merely a visual representation to aid understanding and is not limiting.

[0034] The graph-structured representation of the meaningful parts is then passed through a machine-learning graph encoder 7, which in the particular implementation of FIG. 5 may be a graph encoder convolutional neural network. The graph encoder 7 may be a graph encoder convolutional neural network (GCN). For example, it may be any number of basic GCN layers stacked together back-to-back. In the state of the art, multiple variations of the basic GCN layer exist. For example, any one or more of these GCN layers may be used in combination with each other. A specific example is provided in Kipf & Welling, 2017, 'Semi-Supervised Classification with Graph Convolutional Networks'.

[0035] The graph encoder 7 is configured to extract and encapsulate information defining the relationships between parts represented by the graph structure representation and output a graph structure embedding accordingly. The graph structure embedding forms a representation of the original point cloud. The graph encoder 7 has network weights and can be trained as described below.

[0036] The graph structure embedding that forms a representation of the original point cloud is then input to a classifier 9 that includes a machine learning classification model that outputs classification labels for classification. The machine learning classification model may be a classification encoder neural network, such as a multi-layer perceptron (MLP) structure. The classifier 9 outputs classification labels that, for example, identify the real-world object(s) in the point cloud. The classifier 9 encoder has network weights and can be trained as described below.

[0037] FIG. 6 illustrates the classification process according to one implementation.

[0038] Step S11 comprises selecting the point cloud for classification, where selecting may for example simply mean choosing.

[0039] Step S13 includes obtaining a plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud, the plurality of portions being capable of being considered meaningful and may be considered geometrically meaningful.

[0040] Step S15 obtains a graph structure representation of the plurality of subsets, the graph structure representation having a node associated with each subset and one or more edges connecting the nodes.

[0041] Step S17 comprises obtaining a graph structure embedding that encapsulates the structural relationships between the plurality of subsets, which comprises inputting the graph structure representation into a graph encoder convolutional neural network.

[0042] Step S19 comprises deriving a point cloud representation from the graph structure embedding.

[0043] Step S21 comprises classifying the point cloud representation, which comprises inputting the point cloud representation into a classification encoder neural network to obtain a classification of the selected point cloud.

[0044] Any of these steps may include the processing described with reference to Figure 5. For example, the UPD module 3 may perform the processing of step S13, and / or the GSI module 5 may perform any of the processing of step S15, and / or the graph encoder 7 may perform any of the processing of step S17, and / or the classifier 9 may perform any of the processing of step S21.

[0045] Therefore, a classification approach such as that described above with reference to FIG. 5 may be used.

[0046] FIG. 7 shows a process according to a more specific and possible implementation of the process of UPD module 3 described in relation to FIG.

[0047] The UPD module 3 is configured to decompose or segment the input point cloud 2 into a plurality of parts, in particular parts that are deemed meaningful as mentioned above, e.g., geometrically meaningful. In particular, one aspect of the present application to which the inventor has contributed is the provision of a method for identifying and determining parts within a point cloud object that can be deemed meaningful, in the sense that they are useful and can be used to accurately identify object classifications by determining the presence of meaningful parts within the point cloud object and their structural relationships. A particular example of parts that are deemed meaningful is the identification of parts that are deemed geometrically meaningful in that they represent or relate to properties that are considered to define the unique geometric structure of a particular object classification, regardless of domain and that are invariant across domains.

[0048] In the particular implementation of Figure 7, the UPD module 3 performs unsupervised partial decomposition, which is a machine learning technique that segments point clouds into meaningful parts in an unsupervised manner without any prior knowledge of the point cloud data. This has emerged as a powerful technique that allows for the analysis of 3D data at a finer granularity by dividing objects into semantically significant parts.

[0049] Conventional unsupervised partial decomposition may determine parts or segments using a clustering algorithm, but it requires the user to predefine the number of parts or segments in advance. In the implementation of the present application, the inventors have advantageously identified and contributed that the most effective hyperparameter value for the range of the number of parts (k) is between 2 and 6. In other words, it has been determined that constraining the range of the number of parts (k), which is the number of clusters in the clustering algorithm, provides the most useful partial decomposition that can later be used to accurately classify the point cloud object. In other words, it has been identified that the most useful range for the number of meaningful parts to look for and identify within the point cloud object is between 2 and 6.

[0050] Thus, in operation, the UPD module 3 uses a clustering algorithm 12 to output a subset of 2 to 6 smaller constituent parts of the original point cloud.

[0051] In a particular implementation, it was determined that spectral clustering was the most effective, efficient, and useful clustering algorithm to use. An example of spectral clustering can be found in von Luxburg, 2007, 'A Tutorial on Spectral Clustering', Statistics and Computing, 17 (4), 2007. However, it should be noted that other clustering algorithms, such as BIRCH clustering, can also be applied.

[0052] However, it is advantageous to determine an optimal fraction value within this range. To achieve this, the inventors of the present application have advantageously implemented an improved weighted version of Shannon entropy. A comparable method can be found in Weinmann et al., 2015, 'Semantic Point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers', ISPRS Journal of Photogrammetry and Remote Sensing.

[0053] In particular, as shown in Figure 7, the UPD module 3 uses a clustering algorithm 12 followed by a geometry-based entropy calculation 14 that feeds back into the clustering algorithm 12 to segment the input point cloud 2 in an unsupervised manner. Note that there is no training for this module.

[0054] In a particular implementation, the UPD module 3 uses spectral clustering followed by entropy minimization to segment portions of the point cloud in an unsupervised manner. In particular, the problem of optimal meaningful clustering of the input point cloud 2 is modeled as a discrete optimization problem using the metric of entropy. Again, it is noted that any clustering algorithm may be used, but in a particular advantageous implementation, spectral clustering is used as it has been found to give the best results.

[0055] In particular, the use of improved weighted Shannon entropy determines, based in part on geometry, the optimal number of clusters into which a point cloud is decomposed or segmented into associated subsets. This geometry-based approach assesses the informativeness of the segmented clusters and their potential for further subdivision.

[0056] First, the point cloud is decomposed or segmented into k separate parts, each part representing a candidate subset that may be determined to be a subset. Then, for each individual part, its geometric center is calculated, where the number of points in each part is n. Then, a 3D covariance matrix S for each part is calculated. Next, the eigenvalues ​​of the 3D covariance matrix are calculated:

number

[0057] These eigenvalues ​​are used to define the local 3D structure. More precisely, by using these eigenvalues, we can obtain, for example, the linearity L λ , planarity P λ , and scattering S λ The degree of can be interpreted as:

[0058] In other words, L λ ,P λ ,S λ These degree features, ∈[0,1], represent 1D, 2D, and 3D properties:

number

[0059] Finally, determining an appropriate neighborhood size can be configured as a way to prioritize one of these three geometric features. This prioritization is measured using a weighted Shannon entropy measure (E i ), where the number of points in the candidate subset associated with the part is w i Let:

number

[0060] Therefore, the weighted Shannon entropy (E i) is determined to be smallest, the optimal number of portions k is determined to have been reached, and the input point cloud 2 is thereby decomposed into k subsets, each associated with a determined portion of the point cloud. Each portion is thereby identified, for example, as meaningful, for example as being particularly geometrically meaningful.

[0061] Thus, in one aspect, the above method uses weighted Shannon entropy along with geometry-preserving features of linearity, planarity, and scattering defined by the dominant eigenvalues, to identify and determine which parts and how many parts we may consider to be meaningful, e.g., geometrically meaningful, within a particular input point cloud 2.

[0062] In light of the above, in other words, an alternative algorithmic representation of the functionality of UPD module 3 can be presented as follows:

number

[0063] For completeness and ease of reference, a list of the variables outlined in all the foregoing formulas is provided here:

number

[0064] Thus, in a particular implementation, UPD module 3 uses improved spectral clustering for point cloud segmentation, characteristically using an improved weighted variant of Shannon entropy to advantageously optimize this segmentation, thereby addressing the central problem of determining the optimal number of parts in a point cloud without any prior knowledge.

[0065] Thus, the UPD module 3 advantageously automatically and unsupervisedly obtains multiple subsets of the input point cloud 2, each subset associated with an identified portion of the input point cloud 2. These subsets and associated portions may be considered meaningful, as discussed above. For example, a particular example of a portion considered meaningful is the identification of portions considered to be geometrically meaningful in that they represent or relate to properties that are considered to define the unique geometric structure of a particular object classification, regardless of domain, and that are invariant across domains. Nevertheless, it has been determined that the subsets and associated portions created by implementations of the present application are advantageously useful for improving the accuracy of object identification and classification.

[0066] Thus, according to certain implementations of the present application, a single-step unsupervised clustering algorithm is provided that automatically divides the entire input point cloud 2 into multiple clusters without the need for additional processes, which clusters may be considered meaningful, as discussed above. Moreover, advantageously, the process is completely unsupervised and does not require ground truth for comparison, as in current state-of-the-art classification processes.

[0067] Thus, the UPD module 3 provides subsets of the point cloud from which structural relationships between them are learned and used for classification of the point cloud in other aspects of the present application.

[0068] Training for UPD Module 3 will be discussed later.

[0069] FIG. 8 illustrates a process according to a more specific and possible implementation of the process of GSI module 5 described in relation to FIG.

[0070] In one aspect, the GSI module 5 of a particular implementation is configured to receive multiple subsets of the input point cloud 2 associated with multiple portions of the input point cloud 2 and obtain a graph structure representation of the subsets, the graph structure representation having a node associated with each subset and one or more edges connecting the nodes, thus in other words inducing a graph structure for the subsets associated with the portions.

[0071] In a particular implementation, the GSI module 5 comprises a partial feature encoder 11 and a graph creation module 13, both of which may be configured to process either serially or in parallel, as they both take as input at least multiple subsets of the input point cloud 2, each subset being associated with a portion of the input point cloud 2. In one particular preferred implementation, as outlined in the representation of Figure 8, the processing is configured serially, with the partial feature encoder 11 performing its function first and then outputting to the graph creation module 13 to perform its function.

[0072] The partial feature encoder 11 may be, for example, a partial feature encoder neural network, which generates node-representative embeddings of the subsets associated with the portions. The graph creation module 13 generates edge-representative embeddings of the subsets associated with the portions. The partial feature encoder 11 has network weights and can be trained as described below.

[0073] In certain implementations, the partial feature encoder 11 further decomposes or segments the subsets associated with the portions received from the UPD module 3. In particular, it may perform a finer-granularity decomposition into smaller portions, which may be achieved, for example, using a farthest point sampling (FPS) method. Thus, the partial feature encoder 11 may perform fine-granularity portion creation. In certain implementations, the partial feature encoder 11 may take the form of a point cloud encoder, such as PointNet, PointNet++, or DGCNN.

[0074] Regardless of the specific mode of operation, the partial feature encoder 11 thereby creates node representation embeddings for the subsets associated with the portion. The node embeddings may be as many as the number of subsets associated with the portion, e.g., one per subset, and may further be, for example, centered for each subset. Alternatively, there may be more or fewer node embeddings than subsets. Other modes of deriving nodes are envisioned and may relate to current node creation methods. Training of the partial feature encoder 11 is described below.

[0075] In a specific implementation, the graph creation module 13 may execute a Euclidean-based graph creation method to determine edge representation embeddings of the subsets associated with the parts. In particular, the graph creation module 13 may create edge connections between the extracted subsets associated with the parts based on proximity. For example, an edge may be determined if the Euclidean distance between the subsets associated with the parts is below a specified threshold t. In other words, a distance metric, specifically the Euclidean distance, between the clusters forming the subsets is used, and thresholding is performed to form edges between the subsets associated with the parts, which advantageously helps create a geometrically meaningful graph structure representation. In the implementation of the present application, t is a hyperparameter determined as a result of extensive experiments by the inventors.

[0076] Regardless of the specific mode of operation, the graph creation module 13 thereby creates a graph representation embedding of the subset associated with the portion. Other modes of deriving and connecting edges are also envisioned and may relate to state-of-the-art edge creation methods.

[0077] The functionality of the GSI module 5 for performing graph structure induction as outlined in this application is a novel application in the field of point clouds and offers clear advantages in the accuracy and efficiency of the graphical representation of point clouds, in particular in determining the structure of the point cloud, such as the intrinsic geometric structure of a subset associated with a portion of the input point cloud 2.

[0078] Thus, the GSI module 5 outputs a graph-structured representation of the subsets, comprising a node representation embedding and a graph representation embedding. This graph-structured representation can be said to encapsulate and represent the structural relationships between the subsets associated with the part. This graph-structured representation aims to capture the underlying geometric and structural relationships of the previously determined subsets, facilitating the subsequent determination and identification of graph-structured embeddings that can be used to accurately classify objects in the point cloud.

[0079] Therefore, according to the implementation of this application, by forming edges between meaningfully segmented point cloud subsets associated with parts, it is possible to build geometrically meaningful graph-structured representations, which can be thought of as geometry-preserving graphs as templates for classifying objects of that class in point clouds from unseen domains.

[0080] 5, the output of the GSI module 5, a graph structure representation having nodes associated with each subset and one or more edges connecting those nodes, is then passed to a graph encoder 7. As mentioned above, in the particular implementation of FIG. 5, the graph encoder 7 may be a graph encoder neural network, such as a graph encoder convolutional neural network (GCN), such as a two-layer GCN.

[0081] The graph encoder 7 is configured to extract and encapsulate information defining the relationships between the parts represented by the graph structure representation and output a graph structure embedding accordingly, which forms a representation of the original point cloud.

[0082] In other words, the graph encoder is used to understand, encapsulate and encode / embed structural relationships in a graph-structured representation of the point cloud. Thus, in other words, the output of the graph encoder 7 can be thought of as a graph-structured embedding that contains information about the subset of parts that have been determined to be meaningful, e.g., geometrically meaningful parts, and that contains information about the structural relationships of those parts.

[0083] Thus, referring to some of the aims of this application discussed above, in layman's terms, the graph structure embedding output of the graph encoder 7 can be thought of as containing information that identifies portions of the point cloud (represented by the subset) and specifies structural relationships between those portions, e.g., portions that have been determined to be meaningful, e.g., geometrically meaningful. This information can then be used to identify objects in the point cloud by classification, since those portions that are in a particular structural relationship can be considered to be shared by the majority or most or all objects in that particular classification, or simply in a sufficient proportion to be useful for accurate classification, and these determinations can then be used to identify the classification of the objects in the point cloud.

[0084] In other words, at this stage, the system can be said to have extracted and encapsulated from the point cloud the information needed to classify the object it depicts, which in the practice of this application is considered to be the identification of parts of the object within the point cloud, particularly meaningful parts, and the identification of relationships between those parts.

[0085] Note that the graph encoder 7 is shown in this particular implementation as being separate from the GSI module 5, and it may not be part of the GSI module 5, although in some particular implementations it may be considered part of the GSI module 5.

[0086] The graph structure embeddings output from the graph encoder 7 are input to a classifier 9 for classification of the objects in the point cloud. The graph structure embeddings can be considered to be point cloud representations in that the node and edge embeddings and their structural relationships, e.g., relating to subsets associated with (meaningful) parts, can be considered to represent the original point cloud input to the system, e.g., representing information about the inherent geometric structure of the objects in the point cloud.

[0087] As mentioned above, the classifier 9 may be a machine learning classification model configured to output a classification label for classification. The machine learning classification model may be a classification encoder neural network, such as a multi-layer perceptron (MLP) structure. The classifier 9 outputs a classification label that identifies, for example, one or more real-world objects in a point cloud. The classifier 9 encoder has network weights and can be trained as described below.

[0088] The combined functionality of the UPD module 3, GSI module 5, graph encoder 7, and classifier 9 described in the implementation aspects of the present application has been determined to be particularly advantageous for providing accurate point cloud classification.

[0089] Figure 9 shows a classification process according to a more specific and possible implementation of the classification process of Figure 5. Any of the features of Figures 5 to 8 may be included here.

[0090] In particular, compared to the process described with reference to FIG. 5, the particular implementation of FIG. 9 additionally includes a feature encoder 15 and a feature concatenator 17 .

[0091] In particular, the functionality of the classification process described above with reference to Figures 5, 6, 7, and 8 may be included here, and therefore redundant description will be avoided. Thus, the functionality of Figure 9 may be the same as that previously described with reference to Figures 5, 6, 7, and 8, except for the following additional optional features:

[0092] In particular, the input point cloud 2 input to the UPD module 3 and then processed by the GSI module 5 and graph encoder 7 as described above may be passed in parallel and / or separately through different feature encoders 15. The feature encoders 15 are encoders that perform a different form of feature enhancement, extraction or encoding on the input point cloud 2 compared to the functions of the UPD module 3 and GSI module 5 described above. In a first example, the exact manner of feature encoding performed is not important, since the purpose of the parallel processing through the feature encoders 15 is to provide additional decomposition of the input point cloud 2, e.g., in a different way or from a different process perspective, compared to the functions of the UPD module 3 and GSI module 5.

[0093] This is particularly advantageous because all processing of the input point cloud 2 to perform classification inevitably loses or reduces the amount of information in the original point cloud. Thus, while the advantageous features of the UPD module 3, GSI module 5, and graph encoder 7 in implementation aspects of the present application combine to facilitate the creation of a point cloud representation that can enable highly accurate classification, the fact remains that the point cloud representation created for classification still has reduced information compared to the input point cloud 2. In fact, this is the entire purpose of the representation; all point cloud classification models process the input point cloud in a manner that reduces the amount of information. Therefore, for further improved classification accuracy, it is advantageous to additionally include a further, complementary manner of decomposing or segmenting the input point cloud 2 in addition to that described with reference to the UPD module 3, GSI module 5, and graph encoder 7 above. Thus, the particular implementation of FIG. 9 complements the above method described with reference to FIG. 5 with an additional encoding of the features of the input point cloud 2, which is combined with the above-mentioned graph structure embedding before inputting this combined information into the classifier 9.

[0094] Thus, feature encoder 15 may perform encoding according to any prior art method of point cloud encoding. For example, feature encoder 15 may perform point cloud shape encoding. Thus, a particular implementation may use a trained point cloud shape encoder taken from a state-of-the-art system point cloud shape encoder system. However, any suitable point cloud encoder selection may be used.

[0095] The output from the feature encoder 15 is therefore a different feature embedding, which is input to the feature concatenator 17 where it is mathematically concatenated with the graph structure embedding output from the graph encoder 7 as described in any previous implementation.

[0096] The output of the feature connector 17 combines the feature embeddings from the feature encoder 15 with the graph structure embeddings from the graph encoder 7 to form the point cloud representation that is input to the classifier 9 for classification of the objects in the input point cloud 2. Apart from this difference, the function of the classifier 9 is as described above.

[0097] This particular implementation therefore takes the advantageous improved classification provided by the systems described with reference to Figures 5, 6, 7 and 8 and supplements it with an additional form of feature encoding of the input point cloud 2. This combined encoding provides a further improvement in the accuracy of the point cloud classification.

[0098] Figure 10 illustrates a training process according to one implementation. In particular, Figure 10 depicts a logic flow diagram for training a point cloud classification model 1 as described in any previous implementation. Accordingly, many aspects of the training process of Figure 10 are the same as those previously shown and described, e.g., with reference to Figures 5 through 9, because training a point cloud model for point cloud classification can, in certain instances, be considered a specific version of using a point cloud classification model for classification; in the training process, the classification of the point cloud is known and can be used for comparison with a predicted classification for further training; in typical use, the classification of the point cloud is unknown and the predicted classification is the final output classification.

[0099] The input point cloud is from a training source domain with associated known labels, and therefore the classification output by the aforementioned point cloud classification model can be considered a predicted classification, which is compared to the known labels and this comparison can be used to train the point cloud classification model. In the aforementioned use of point cloud classification, the input point cloud can be from an unlabeled target domain, and the classification output is considered to be the final classification.

[0100] In particular, for example, the UPD module 3, GSI module 5, graph encoder 7 (graph neural network), classifier 9, partial feature encoder 11, graph creation module 13, feature encoder 15 (i.e., the shape encoder described in relation to Figure 9 in certain implementations), and feature concatenator 17 shown and described in relation to Figure 10 may be the same mechanisms as described in any previous implementation, particularly as described with reference to Figures 5 to 9. Therefore, duplication of the above description of these mechanisms will be avoided.

[0101] In particular, the functionality of each of the following mechanisms, namely, UPD module 3, GSI module 5, graph encoder 7 (graph neural network), classifier 9, partial feature encoder 11, graph creation module 13, point cloud feature encoder 15 (i.e., the shape encoder described in connection with FIG. 9 in a specific implementation), and feature connecter 17, was described above with reference to FIGS. 5 through 9 in the use phase of point cloud classification model 1, in which point cloud classification model 1 was used to classify point clouds in an unseen target domain. In the training process of FIG. 10, the functionality of each of these mechanisms is the same as that described above and will not be further described here, except for the following details, which are described in more detail below: the input point cloud is from a source domain of labeled point cloud data, and point cloud classification model 1 receives this input point cloud and processes it as described above, with the output from classifier 9 being considered the predicted classification labels. The predicted classification labels are then compared to the known classification labels of the input point cloud, and a comparison is made between the known labels and the predicted labels, and the network weights of at least one of the classifier 9 and the graph encoder convolutional network 7 are adjusted based on this comparison. This process is then performed iteratively until the classification loss between the known labels and the predicted labels is determined to be at a minimum.

[0102] At this point, it may be determined that the point cloud classification model 1 has been trained to accurately classify point cloud objects of that particular classification across any domains that share the same label space, regardless of the domain. In other words, it may be determined that the point cloud classification model 1 has learned the domain-invariant features of the point cloud classification, as described above, and in particular, has determined the portions of the point cloud for that classification that are deemed meaningful, as described above. For example, a particular example of a portion that is deemed meaningful is the identification of portions that are deemed geometrically meaningful in that they represent or relate to properties that are deemed to define the unique geometric structure of the particular object classification, regardless of the domain, that are invariant across domains, and therefore are useful and can be used to accurately identify the classification of the object, as provided in other aspects of the present application.

[0103] The training process of Figure 10 includes the mechanisms of the classification process of Figure 9 described above, including the optional feature of using a point cloud feature encoder 15, particularly a feature encoder 15 in certain implementations of a point cloud shape encoder. This mechanism is optional, but is depicted here for completeness.

[0104] The UPD module 3 and the GSI module 5 are as previously described in Figures 7 and 8, respectively, and any mechanism described as any particular implementation is also any particular implementation here. Similarly, the graph encoder 7 is shown as being within the GSI module 5, but as previously described, it may be within the GSI module 5 or may be separate from the GSI module 5.

[0105] In the first step S101, an input point cloud (P) from the training data with corresponding labels (L) is selected.

[0106] In optional implementations that include a point cloud feature encoder 15 in the form of a point cloud shape encoder, such as that shown in FIG. 10, this must first be trained. Thus, in optional step S103, the input point cloud is passed through any suitable existing architecture of point cloud feature encoder 15, such as a point cloud shape encoder. Because this is the state of the art, it will not be described in further detail. Briefly, however, FIG. 10 illustrates an example of training a shape encoder, in which the input point cloud trains a shape encoder using a loss-minimization evaluation to adjust the encoder's network weights until it is determined that the classification loss between the predicted classification labels and the known classification labels is minimal. At this point, it is determined that the shape encoder has been trained to determine the shape encoding of the point cloud for a particular classification. This process is repeated across the source-domain training data so that the shape encoder is trained across multiple classifications. Once trained, the shape encoder is used as the aforementioned feature encoder 15 in optional implementations that use a feature encoder, such as that described with reference to FIG. 9.

[0107] Step S105, as described with reference to Figure 9, is an optional step in which feature encoder 15 is used in addition to UPD module 3, GSI module 5, and graph encoder 7 of implementations of the present application, such as those described with reference to Figures 5-8. Here, as described with reference to Figure 9, feature encoder 15 complements the functionality of the systems described with reference to Figures 5-8 to provide additional forms of feature encoding (e.g., shape encoding) that are combined with the graph structure embedding output from the combined functionality of UPD module 3, GSI module 5, and graph encoder 7. This advantageously further improves the accuracy of the classification provided by implementations of the present application.

[0108] In implementations where step S105 is included, it may occur before, after, or in parallel with step S107, as described below.

[0109] In step S105, the input point cloud is input to the trained shape encoder 15, which outputs a shape encoding of the input point cloud ('encoded shape features' shown in Figure 10). As shown, this stage may be frozen in the sense that there is no backward pass through the trained shape encoder 15.

[0110] In step S107, the input point cloud is input to UPD module 3. This is the same UPD module 3 described above with reference to Figures 5 to 9, and functions according to the description provided in connection with any of those embodiments, which will not be repeated here. For example, the functionality of UPD module 3 may be as described above with reference to Figure 7. UPD module 3 thus outputs a subset of the input point cloud associated with multiple parts, which, as described above, are deemed meaningful ('semantically significant parts' shown in Figure 10) or potentially meaningful when considered during the training phase. As described above, UPD module 3 is unsupervised and not trained.

[0111] In step S109, subsets of the input point cloud associated with the plurality of portions are input to the GSI module 5. This is the same GSI module 5 described above with reference to Figures 5 to 9, and functions according to the description provided in connection with any of those embodiments, which will not be repeated here. For example, the functionality of the GSI module 5 may be as described above with reference to Figure 8. The graph encoder 7 (graph neural network) then outputs a graph structure embedding ('encoded graph feature' shown in Figure 10) that encapsulates the structural relationships between the previously determined subsets. Note that while Figure 10 shows the partial feature encoder 11 operating in a parallel logical flow to the graph creation module 11, in a preferred particular implementation, as described above with reference to Figure 8, the partial feature encoder 11 and the graph creation module operate serially, with the graph creation module 13 processing after the partial feature encoder 11.

[0112] In implementations that include step S105, such as the system described above with reference to Figure 9 and shown in Figure 10, the shape embeddings ('encoded shape features' as shown in Figure 10) then need to be combined with the graph structure embeddings ('encoded graph features' as shown in Figure 10) in step S110 to derive a point cloud representation for classification. This combination is performed in the same feature concatenation module 17 as described above with reference to Figure 9. Alternatively, in implementations without step S105, the graph structure embeddings output from the graph encoder 7 may be determined to be a point cloud representation and output directly to the classifier 9, and therefore step S110 is not required.

[0113] In step S111, the point cloud representation is input to classifier 9, which is the classifier described above with reference to any of Figures 5 to 9. Classifier 9 then outputs a predicted classification for the input point cloud, which is compared to the known classification (L). In particular, in embodiments of the present application, this comparison can be performed in the form of computing a loss, in particular a classification loss defined by a categorical cross-entropy loss function. Categorical cross-entropy loss functions are known functions and will not be described further here (see, for example, Anqi Mao, Mehryar Mohri, and Yutao Zhong, 2023, 'Cross-Entropy Loss Functions: Theoretical Analysis and Applications').

[0114] If, in step S113, it is determined that the classification loss, for example as determined using a categorical cross-entropy loss function, is minimal ('Yes'), the training process for that point cloud is terminated and the point cloud classification model, including the partial feature encoder 11, graph encoder 7, and classifier 9, is determined to have been trained to accurately identify the input point cloud.

[0115] If, in step S113, it is determined that the classification loss, e.g., as determined using a categorical cross-entropy loss function, is not minimized ('No'), then based on this computational comparison, at least one network weight of at least one of the following encoders is adjusted: graph encoder 7 and classifier 9. The adjustment of the at least one network weight is, for example, to reduce or bring the computed difference and / or the computed classification loss closer to zero.

[0116] After adjusting the network weights accordingly, the classification process is run again, e.g., starting with step S109, optionally including step S105 as described above. This process is run iteratively until the classification loss between the known labels and the predicted labels is determined to be minimal. At this point, it may be determined that the point cloud classification model 1 has been trained to accurately classify point cloud objects of that particular classification across any domains that share the same label space, regardless of the domain. In other words, it may be determined that the point cloud classification model 1 has learned domain-invariant features of the point cloud classification, as described above, and in particular, for example, has determined a subset associated with portions of the point cloud for that classification that are deemed meaningful, as described above. For example, a particular example of portions that are deemed meaningful is the identification of portions that are deemed geometrically meaningful in that they represent or relate to properties that are deemed to define the unique geometric structure of a particular object classification that are invariant across domains, regardless of the domain, and therefore are useful and can be used to accurately identify the classification of the object, as provided in other aspects of the present application.

[0117] This training process is then repeated until the loss is determined to be minimal ('Yes'), after which the training process for that input point cloud is completed and it is determined that the point cloud classification model, including the partial feature encoder 11, graph encoder 7, and classifier 9, has been trained to accurately identify the input point cloud.

[0118] This training process can then be repeated across the source domain training data point cloud so that the point cloud classification model 1 is trained across multiple classifications.

[0119] In an alternative example not shown, training of the shape encoder 15 may be performed in parallel or simultaneously with the training performed in steps S105 and S107. In this case, step S103 shown in FIG. 10 is eliminated, and step S105 is modified so that the shape encoder 15 is not initially trained and is not frozen to allow a backward pass. Steps S107 through S111 are performed as described above. Step S113 is performed as described above, with the modification that if the classification loss is determined to be not minimal ('No'), then in addition to adjusting at least one network weight of at least one of the graph encoder 7 and the classifier 9, at least one network weight of the feature encoder 15 is also adjusted. Thus, implementations of the present application contemplate both training aspects of the optional feature encoder 15 before other aspects of the system, and alternatively, training the feature encoder 15 simultaneously with training other aspects of the system.

[0120] FIG. 11 is a diagram illustrating the training process according to one implementation.

[0121] Step S21 comprises selecting a point cloud with a known classification from training data comprising a plurality of point clouds from a source domain.

[0122] Step S23 comprises obtaining a plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud, the plurality of portions being capable of being considered geometrically meaningful.

[0123] Step S25 obtains a graph structure representation of the plurality of subsets, the graph structure representation having a node associated with each subset and one or more edges connecting the nodes.

[0124] Step S27 comprises obtaining a graph structure embedding that encapsulates the structural relationships between the plurality of subsets, which comprises inputting the graph structure representation into a graph encoder convolutional neural network.

[0125] Step S29 comprises deriving a point cloud representation from the graph structure embedding.

[0126] Step S31 comprises classifying the point cloud representation, which comprises inputting the point cloud representation into a classification encoder neural network to obtain a predicted classification of the selected point cloud.

[0127] Step S33 comprises comparing the predicted classification with the known classification of the selected point cloud.

[0128] Step S35 includes adjusting at least one network weight of one of the graph encoder convolutional neural network and the classification encoder neural network based on the comparison.

[0129] Any of these steps may include the processing described with reference to Figure 5 or Figure 10. For example, the UPD module 3 may perform the processing of step S23, and / or the GSI module 5 may perform any of the processing of step S25, and / or the graph encoder 7 may perform any of the processing of step S27, and / or the classifier 9 may perform any of the processing of step S31.

[0130] Therefore, a training approach such as that described above with reference to FIG. 10 may be used.

[0131] Figure 12 is a diagram depicting an example task that may be performed by an implementation of the point cloud classification model 1 as described above. The example task may be in the field of augmented reality. The example task is identifying objects depicted by a point cloud acquired from a sensor in a room. The source domain training data is simulated synthetic data of objects in the room; therefore, this example task is an example of a 'sim-to-real' scenario in point cloud classification.

[0132] The image labeled 'Training' shows a source domain for training, containing simulated data of objects in a room, where these objects have known classifications. Because the objects in the source domain all have known classifications, they are stored as a point cloud and input into the system during the training phase, as described above in the implementation of this application with reference to FIG. 10 . For example, the objects present in the illustrated source domain are a chair, a sofa, and a shelving system. Objects from this source domain are used to train the system to create graph-structured representations, graph-structured embeddings, and point cloud representations that are accurately classified by the system as described above, by adjusting the network weights of the associated encoders of point cloud classification model 1 as described above. Thus, point cloud classification model 1 trained using any of the processes described herein can be used to accurately identify objects in point clouds sensed in a real room, as shown in the image labeled 'Test', which represents an unseen target domain for point cloud classification model 1. It will therefore be appreciated that improved processes in the implementations of the present application described above, such as improved decomposition, feature identification, and feature extraction of point cloud objects, can lead to more accurate point cloud classification performance, and in particular, improved domain generalization of point cloud classification.

[0133] While Figure 12 illustrates a potential application in the field of augmented reality, implementations of the present application have broad applicability across all fields involving the collection and processing of point cloud data for classification. Further example fields include autonomous driving, robotics, photogrammetry, remote sensing, and computer vision. For example, point cloud data can be received from, for example, a sensor on, for example, an autonomous vehicle or a robotic arm or an augmented reality device.

[0134] For example, in the field of autonomous driving, point clouds are used to represent the environment surrounding a vehicle. For example, implementations of the present application in providing improved domain generalization from a source domain to an unseen target domain may advantageously enable a vehicle to perform well in many different locations, regardless of the diversity of the environment. In current state-of-the-art systems, a point cloud classification model trained on one city dataset (the source domain) may face difficulties in adapting to a new city (the unseen target domain). However, by applying aspects of the implementations of the present application, the point cloud classification model may enable, for example, a vehicle trained in a particular city to then operate in a new city without difficulty, reliably and accurately performing point cloud classification regardless.

[0135] In the field of robotics, application of aspects of the implementations of the present application may allow a robot trained to lift objects in a simulated factory environment, for example, to be advantageously used to lift objects in new environments by benefiting from the domain generalization capabilities of the implementations of the present application.

[0136] Figure 13 shows a table of results for models trained by several different state-of-the-art processes and the model trained by the training process described here (labeled 'Ours') on the PointDA-10 dataset (Qin et al., 2019, 'PointDAN: A Multi-Scale 3D Domain Adaption Network for Point Cloud Representation'). These results are in the form of class-level accuracy, i.e., (total number of correctly predicted samples) / total number of samples.

[0137] In this table you will find the following references: - Setting - the setting in which the experiment is performed. Supervised (Source & Target) refers to training on the source and target domains and testing directly on the target domain. w / o Adapt refers to training directly on the source domain and testing on the target domain without any adaptation techniques. DA refers to a source with data and labels, unlabeled target data is seen during training. DG refers to training only on the source domain. - X → Y, X → Z - refers to X as the source domain, and the model is trained only on X. After that, it is tested on Y and Z. - M - ModelNet domain in the PointDA-10 dataset - S - ShapeNet domain in the PointDA-10 dataset -S * - ScanNet domain in the PointDA-10 dataset - Avg - average accuracy of the rows.

[0138] In particular, the bottom two rows of the table show results from two specific implementations of the present application, referred to in the table as 'Ours' and 'Ours: Joint Training'. These implementations are in line with the specific implementations described with reference to Figures 9 and 10 of the present application, in which optional feature encoder 15 is used. In particular, in the table, 'Ours' and 'Ours: Joint Training' refer to the combined operation of UPD module 3, GSI module 5, graph encoder 7, and classifier 9 (e.g., as described with reference to Figures 5 through 8), and complementing this functionality with the use of optional feature encoder 15 (e.g., as described with reference to Figures 9 and 10).

[0139] In the results shown, 'Ours' is trained in the manner described with reference to Figure 10, where the feature encoder 15 is trained first before training other aspects of the system. 'Ours:Joint Training' is trained as described above in the alternative implementation of Figure 10, where the feature encoder 15 is trained at the same time as training other aspects of the system.

[0140] As can be seen, the models trained by the training process described here (labeled 'Ours' and 'Ours: Joint Training') consistently outperform all previous state-of-the-art domain generalization methods by a significant margin in terms of delivering classification accuracy. In particular, our proposed method demonstrates its suitability for sim-to-real scenarios (MS) where the target dataset is a real scan of an object with various challenges (occlusions, missing parts, deformations, etc.). * and SS * ) shows significant performance improvements.

[0141] Figure 14 shows a table of results for models trained by several different state-of-the-art processes and the model trained by the training process described here (labeled 'Ours') on the GraspNetPC-10 dataset (H.S. Fang, C. Wang, M. Gou and C. Lu, 2020, 'GraspNet-1 Billion: A Large-Scale Benchmark for General Object Grasping'). These results are in the form of class-level accuracy, i.e., (total number of correctly predicted samples) / total number of samples.

[0142] In this table you will find the following references: - DA - Domain Adaptation - DG - Domain Generalization - M - ModelNet dataset - S - ShapeNet dataset -S * - ScanNet dataset - Syn - Synthetic domain - RS - Realsense domain - Kin - Kinect domain.

[0143] In particular, the bottom two rows of the table show results from two specific implementations of the present application, referred to in the table as 'Ours' and 'Ours: Joint Training.' These implementations are in line with the specific implementations described with reference to Figures 9 and 10 of the present application, in which optional feature encoder 15 is used. In particular, in the table, '(Ours)' and '(Ours: Joint Training)' refer to the combined operation of UPD module 3, GSI module 5, graph encoder 7, and classifier 9 (e.g., as described with reference to Figures 5 through 8), and complementing this functionality with the use of optional feature encoder 15 (e.g., as described with reference to Figures 9 and 10).

[0144] In the results shown, 'Ours' is trained in the manner described with reference to Figure 10, where the feature encoder 15 is trained first before training other aspects of the system. 'Ours:Joint Training' is trained as described above in the alternative implementation of Figure 10, where the feature encoder 15 is trained at the same time as training other aspects of the system.

[0145] As can be seen, models trained by the training process described herein (labeled 'Ours' and 'Ours: Joint Training') consistently outperform all previous state-of-the-art domain generalization methods by a significant margin in terms of providing classification accuracy. In particular, our method demonstrates significant performance improvements in Sim-to-Real scenarios (Syn-Kin and Syn-RS) where the target datasets are real scans of objects cropped from real indoor scenes scanned using two different LiDAR scanners (Kinect and Intel RealSense). In particular, the method according to the implementation described in this application demonstrates significant performance improvements in Sim-to-Real scenarios (Syn-to-Kin and Syn-to-RS). Thus, the implementation of this application is particularly advantageous in providing accurate classification of point cloud data with unseen real-world data, even when trained on synthetic data.

[0146] Thus, according to a specific implementation of the present application, given a source domain consisting of a point cloud and labels, it is possible to train a highly generalizable point cloud classification model that learns improved class separation, thereby achieving better classification accuracy for multiple unseen domains that share a common label space with the source domain.

[0147] According to certain implementations of the present application, it is possible to identify and learn relationships and graph structures between parts of point clouds that persist across a particular class of domains.

[0148] Certain implementations of the present application provide a novel and advantageous technique for classifying point cloud objects in unseen datasets (domain generalization) by decomposing point clouds into local sub-features and learning their relationships with each other by incorporating graph structure.

[0149] Particular implementations of this application provide meaningful partial decomposition of point cloud objects in an unsupervised manner for better domain generalization and incorporate a geometry-preserving graph structure into point cloud objects that is consistent across domains and achieves state-of-the-art performance in classification.

[0150] According to certain implementations of the present application, advantageously, there is no need for annotated or pseudo-annotated data, and the classification process is completely unsupervised.

[0151] Any of the training processes described herein may have multiple iterations. The number of iterations may depend on determining after each (or some) iteration whether the difference, embedding difference, or loss between the first and second representations is below an error threshold, or on determining whether the difference, embedding difference, or loss between the first and second representations is below an error threshold for a certain (predetermined, predetermined) number of iterations. This determination may instead or in addition to the above determination examples include determining whether a set number of iterations have been performed. For example, the training process may terminate based on whichever occurs first: error convergence or a maximum number of iterations. In other words, the considerations discussed with respect to step S113 may apply to any of the training processes.

[0152] Extracting a representation of a graph is sometimes referred to as embedding the graph or generating a graph embedding (or node embedding). Extracting a representation of a graph is sometimes referred to as representation learning. Extracting a representation of a graph may be thought of as extracting a representation for each node in the graph and combining these node representations (or node-level representations) to obtain a representation of the graph. A node embedding or node representation may include information about a node about some or all of the other nodes in the graph and the connections to (and between) it.

[0153] A GNN has layers that may be referred to as message passing layers that are responsible for combining node and edge information into node embeddings. This process is sometimes referred to as graph convolution. For example, a node embedding may contain information about that node and other nodes (and edges) in its neighborhood. It is understood that there are several specific ways in which a representation may be extracted from a graph, and the specific method used is not important to the methods disclosed herein.

[0154] 15 is a block diagram of an information processing apparatus 10 or computing device 10, such as a data storage server, embodying the present invention and which may be used to implement some or all of the method operations and perform some or all of the tasks of an embodiment of the apparatus embodying the present invention. The computing device 10 may be used to perform any of the method steps described above, such as steps S11-S21, steps S21-S35, steps S101-S113, and / or any of the processes described above.

[0155] The computing device 10 has a processor 993 and memory 994. Optionally, the computing device also includes a network interface 997 for communicating with other such computing devices, such as other computing devices of invention embodiments. Optionally, the computing device also includes one or more input mechanisms 996, such as a keyboard and mouse, and a display unit 995, such as one or more monitors. These elements may facilitate user interaction. These components may be connectable to one another via a bus 992.

[0156] The memory 994 may include a computer-readable medium, which term may refer to a single medium or multiple media (e.g., centralized or distributed databases and / or associated caches and servers) configured to carry computer-executable instructions. Computer-executable instructions may include, for example, instructions and data that are accessible by a computer (e.g., one or more processors) and cause the computer to perform one or more functions or operations. For example, computer-executable instructions may include instructions for implementing a method disclosed herein or any method steps disclosed herein, e.g., any of steps S11-S21, steps S21-S35, steps S101-S113, and / or any process described above. Accordingly, the term “computer-readable storage medium” may also include any medium capable of storing, encoding, or carrying a set of instructions for execution by a machine, causing the machine to perform any one or more of the method steps of the present disclosure. Accordingly, the term “computer-readable storage medium” may be interpreted to include, but is not limited to, solid-state memory, optical media, and magnetic media. By way of example, and not limitation, such computer-readable media may include non-transitory computer-readable storage media, including random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), compact disc read-only memory (CD-ROM) or other optical disk storage, magnetic disk storage or other magnetic storage devices, flash memory devices (e.g., solid-state memory devices).

[0157] The processor 993 is configured to control the computing device and perform processing operations, for example, executing computer program code stored in memory 994 to perform any of the method steps described herein. The memory 994 stores data read and written by the processor 993, and may store the training data, network weights, patches, updated patches, embeddings, vectors, graphs, representations, difference quantities, formulas, and / or other data and / or programs for performing any of the method steps and / or processes described above. As referred to herein, a processor may include one or more general-purpose processing devices, such as a microprocessor, a central processing unit, or the like. The processor may include a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, or a processor implementing another instruction set or a combination of multiple instruction sets. The processor may also include one or more special-purpose processing devices, such as, for example, an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. In one or more embodiments, the processor is configured to execute instructions to perform the processes and operations described herein. The processor 993 may be considered to include any of the modules described above. Any process described as being implemented by a module may also be implemented as a computational method, such as by the processor 993.

[0158] The display unit 995 may display representations of data stored by the computing device, such as images, and / or difference quantities, and / or graphs, and / or detected objects, and / or GUI windows, and / or interactive representations that allow a user to interact with apparatus 10, for example, by drag-and-drop or selection interactions, and / or any other output described above, as well as cursors and dialog boxes and screens that allow a user to interact with programs and data stored on the computing device. The input mechanism 996 may allow a user to input data and instructions into the computing device, such as by allowing a user to enter any of the user inputs described above.

[0159] The network interface (network I / F) 997 can be connected to a network, such as the Internet, and can be connected to other such computing devices via the network. The network I / F 997 can control data input / output from / to other devices via the network. Other peripheral devices, such as a microphone, speaker, printer, power supply unit, fan, case, scanner, tracker ball, etc., may also be included in the computing device.

[0160] A method embodying the present invention may be performed on a computing device / apparatus 10, such as that shown in FIG. 15. Such a computing device need not have all of the components shown in FIG. 15, but may be configured with a subset of these components. For example, apparatus 10 may have a processor 993 and a memory 994 connected to processor 993. Alternatively, apparatus 10 may have a processor 993, a memory 994 connected to processor 993, and a display 995. A method embodying the present invention may be performed by a single computing device that communicates with one or more data storage servers over a network. The computing device may also be the data storage itself, storing at least a portion of the data.

[0161] Methods embodying the present invention may be performed by a plurality of computing devices operating in cooperation with one another, one or more of which may be data storage servers that store at least a portion of the data.

[0162] The invention may be implemented in digital electronic circuitry, or in computer hardware, firmware, software, or in combinations of them. The invention may also be implemented as a computer program or computer program product, that is, a computer program tangibly embodied in a non-transitory information carrier, such as a machine-readable storage device, or in a propagated signal, for execution by, or to control the operation of, one or more hardware modules.

[0163] A computer program may be in the form of a stand-alone program, a computer program portion, or two or more computer programs, may be written in any type of programming language, including compiled or interpreted languages, and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a data processing environment. A computer program may be deployed to be executed on one module or on multiple modules at one location or distributed across multiple locations and interconnected by a communication network.

[0164] The method steps of the present invention can be performed by one or more programmable processors that execute computer programs to perform the functions of the present invention by processing input data and generating output. Apparatus of the present invention can be implemented as programmed hardware or as special purpose logic circuitry including, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit).

[0165] Processors suitable for the execution of a computer program include, by way of example, both general and special purpose microprocessors, and any one or more processors of any kind of digital computer. Generally, a processor receives instructions and data from a read-only memory or a random-access memory or both. The essential elements of a computer are a processor for executing instructions coupled to one or more memory devices for storing instructions and data.

[0166] The above-described embodiments of the invention may be used to advantage independently of any other of these embodiments or in any feasible combination with one or more other of these embodiments.

[0167] Alternative descriptions of the present invention are set forth below as numbered appendices.

[0168] Clause 1. A computer-implemented method for performing point cloud classification, comprising: Select the point cloud for classification, obtaining a plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud; obtaining a graph-structured representation of the plurality of subsets, the graph-structured representation having a node associated with each subset and one or more edges connecting the nodes; inputting the graph structure representation into a graph encoder convolutional neural network to obtain a graph structure embedding that encapsulates structural relationships between the plurality of subsets; Deriving a point cloud representation from the graph structure embedding; classifying the point cloud representation by inputting the point cloud representation into a classification encoder neural network to obtain a classification of the selected point cloud; A computer-implemented method comprising:

[0169] Clause 2. The computer-implemented method of clause 1, further comprising inputting the selected point cloud into a pre-trained feature encoder neural network to obtain feature embeddings of the selected point cloud.

[0170] Appendix 3. The computer-implemented method of Appendix 2, wherein deriving a point cloud representation from the graph structure embedding further comprises concatenating the graph structure embedding and the feature embedding to create the point cloud representation.

[0171] Clause 4. The computer-implemented method of any one of clauses 2 to 3, wherein the pre-trained feature encoder neural network is a shape encoder and the feature embeddings are shape embeddings.

[0172] Appendix 5. The computer-implemented method of any one of Appendixes 1 to 4, wherein obtaining the plurality of subsets of the selected point cloud associated with the plurality of portions of the selected point cloud further comprises inputting the selected point cloud to an unsupervised partial decomposition module, the unsupervised partial decomposition module performing unsupervised segmentation of the selected point cloud into the plurality of subsets.

[0173] Clause 6. Obtaining a graph structure representation of the plurality of subsets further comprises inputting the plurality of geometrically meaningful parts to a graph structure induction module having a partial feature encoder and a graph creation module; the partial feature encoder is a partial feature encoder neural network and generates the nodes as node representation embeddings of the plurality of subsets associated with the plurality of portions; the graph creation module generates the edges as edge representation embeddings of the subsets associated with the portions. 6. A computer-implemented method according to any one of claims 1 to 5.

[0174] Appendix 7. Performing the point cloud classification is part of a training process for a point cloud classification model, the training process comprising: selecting a point cloud for classification comprises selecting a point cloud with a known classification from training data comprising a plurality of point clouds from a source domain; in the step of classifying the point cloud representation, the classification being a predicted classification of the selected point cloud; Having that, comparing the predicted classification to the known classification of the selected point cloud; adjusting at least one network weight of one of the graph encoder convolutional neural network and the classification encoder neural network based on the comparison; 7. The computer-implemented method of any of claims 1 to 6, further comprising:

[0175] Clause 8. The computer-implemented method of clause 7, wherein comparing the predicted classification to the known classification comprises determining whether classification loss is minimal.

[0176] Appendix 9. A computer program product which, when executed on a computer, causes the computer to perform a method comprising a process of point cloud classification, the process comprising: Select the point cloud for classification, obtaining a plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud; obtaining a graph-structured representation of the plurality of subsets, the graph-structured representation having a node associated with each subset and one or more edges connecting the nodes; inputting the graph structure representation into a graph encoder convolutional neural network to obtain a graph structure embedding that encapsulates structural relationships between the plurality of subsets; Deriving a point cloud representation from the graph structure embedding; classifying the point cloud representation by inputting the point cloud representation into a classification encoder neural network to obtain a classification of the selected point cloud; A computer program comprising:

[0177] Appendix 10. An information processing device having a memory and a processor coupled to the memory, the processor configured to execute a process comprising point cloud classification, the process comprising: Select the point cloud for classification, obtaining a plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud; obtaining a graph-structured representation of the plurality of subsets, the graph-structured representation having a node associated with each subset and one or more edges connecting the nodes; inputting the graph structure representation into a graph encoder convolutional neural network to obtain a graph structure embedding that encapsulates structural relationships between the plurality of subsets; Deriving a point cloud representation from the graph structure embedding; classifying the point cloud representation by inputting the point cloud representation into a classification encoder neural network to obtain a classification of the selected point cloud; An information processing device having the above.

[0178] Appendix 11. The computer-implemented method of Appendix 1, wherein the point cloud representation is the graph structure embedding.

[0179] Appendix 12. The computer-implemented method of Appendix 2, wherein the step of obtaining feature embeddings of the selected point cloud is performed in parallel with the steps of obtaining the plurality of subsets, and / or obtaining graph-structured representations of the plurality of subsets, and / or obtaining graph-structured embeddings, and / or deriving a point cloud representation.

[0180] Clause 13. The computer-implemented method of Clause 5, wherein the unsupervised partial decomposition module uses a clustering algorithm to create candidate subsets.

[0181] Clause 14. The computer-implemented method of Clause 13, wherein the clustering algorithm is spectral clustering.

[0182] Appendix 15. The computer-implemented method of any one of Appendixes 13 to 14, wherein the unsupervised partial decomposition module determines the plurality of subsets from the candidate subsets, the determining including identifying when the Shannon entropy of the candidate subset is determined to be minimum.

[0183] Clause 16. The computer-implemented method of clause 5, wherein the number of subsets is in the range of 2 to 6.

[0184] Clause 17. The computer-implemented method of Clause 6, wherein the partial feature encoder neural network performs farthest point sampling to decompose the plurality of subsets into finer-grained segmentations to generate the node representation embeddings.

[0185] Appendix 18. The computer-implemented method of Appendix 6, wherein the graph creation module performs a Euclidean-based graph creation method to determine the edge representation embedding of the plurality of subsets associated with the plurality of parts, and in particular, the graph creation module creates edge connections between the plurality of subsets associated with the plurality of parts based on proximity, and optionally, edge connections are determined if the Euclidean distance between the plurality of subsets associated with the plurality of parts is below a specified threshold t.

[0186] Appendix 19. The computer-implemented method of Appendix 6, wherein the graph encoder convolutional neural network encapsulates the structural relationships between the subsets associated with the portions by extracting the information from the nodes and one or more edges connecting the nodes.

[0187] Clause 20. The computer-implemented method of Clause 6, wherein the graph encoder convolutional neural network is two-layered.

[0188] Clause 21. The computer-implemented method of Clause 6, wherein the classification encoder neural network is a multi-layer perceptron.

[0189] Clause 22. The computer-implemented method of Clause 8, wherein the classification loss is determined using a categorical cross-entropy loss function.

[0190] Clause 23. The computer-implemented method of Clause 8, wherein the training process is terminated if the classification loss is determined to be minimal.

[0191] Clause 24. The computer-implemented method of Clause 8, wherein if the classification loss is determined to be not minimal, the method further comprises iteratively performing the training process until the classification loss is determined to be minimal.

[0192] Appendix 25. The computer-implemented method of Appendix 8, wherein when the method includes the partial feature encoder, the adjusting step based on the comparison further comprises adjusting at least one network weight of the partial encoder.

[0193] Appendix 26. The computer-implemented method of Appendix 8, wherein the point cloud for classification relates to point cloud data from a sensor, and the sensor may be used in an autonomous vehicle, a robot, or an augmented reality device.

[0194] Clause 27. An information processing device having a memory and a processor connected to the memory, the processor configured to execute a process comprising point cloud classification, the process comprising: Select the point cloud for classification, obtaining a plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud; obtaining a graph-structured representation of the plurality of subsets, the graph-structured representation having a node associated with each subset and one or more edges connecting the nodes; inputting the graph structure representation into a graph encoder convolutional neural network to obtain a graph structure embedding that encapsulates structural relationships between the plurality of subsets; Deriving a point cloud representation from the graph structure embedding; classifying the point cloud representation by inputting the point cloud representation into a classification encoder neural network to obtain a classification of the selected point cloud; An information processing device having the above.

[0195] Clause 28. A computer-implemented method for training a point cloud classification model, comprising: selecting point clouds for classification, including selecting point clouds with known classifications from training data having a plurality of point clouds from a source domain; obtaining a plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud; obtaining a graph-structured representation of the plurality of subsets, the graph-structured representation having a node associated with each subset and one or more edges connecting the nodes; inputting the graph structure representation into a graph encoder convolutional neural network to obtain a graph structure embedding that encapsulates structural relationships between the plurality of subsets; Deriving a point cloud representation from the graph structure embedding; classifying the point cloud representation by inputting the point cloud representation into a classification encoder neural network to obtain a predicted classification of the selected point cloud; comparing the predicted classification to the known classification of the selected point cloud; adjusting at least one network weight of one of the graph encoder convolutional neural network and the classification encoder neural network based on the comparison; A computer-implemented method comprising:

[0196] Clause 29. A computer program that, when executed on a computer, causes the computer to perform a method comprising training a point cloud classification model, the method comprising: selecting point clouds for classification, including selecting point clouds with known classifications from training data having a plurality of point clouds from a source domain; obtaining a plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud; obtaining a graph-structured representation of the plurality of subsets, the graph-structured representation having a node associated with each subset and one or more edges connecting the nodes; inputting the graph structure representation into a graph encoder convolutional neural network to obtain a graph structure embedding that encapsulates structural relationships between the plurality of subsets; Deriving a point cloud representation from the graph structure embedding; classifying the point cloud representation by inputting the point cloud representation into a classification encoder neural network to obtain a predicted classification of the selected point cloud; comparing the predicted classification to the known classification of the selected point cloud; adjusting at least one network weight of one of the graph encoder convolutional neural network and the classification encoder neural network based on the comparison; A computer program comprising:

[0197] Clause 30. An information processing device having a memory and a processor coupled to the memory, the processor configured to execute a method comprising training a point cloud classification model, the method comprising: selecting point clouds for classification, including selecting point clouds with known classifications from training data having a plurality of point clouds from a source domain; obtaining a plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud; obtaining a graph-structured representation of the plurality of subsets, the graph-structured representation having a node associated with each subset and one or more edges connecting the nodes; inputting the graph structure representation into a graph encoder convolutional neural network to obtain a graph structure embedding that encapsulates structural relationships between the plurality of subsets; Deriving a point cloud representation from the graph structure embedding; classifying the point cloud representation by inputting the point cloud representation into a classification encoder neural network to obtain a predicted classification of the selected point cloud; comparing the predicted classification to the known classification of the selected point cloud; adjusting at least one network weight of one of the graph encoder convolutional neural network and the classification encoder neural network based on the comparison; An information processing device having the above.

Claims

1. 1. A computer-implemented method for performing point cloud classification, comprising: Select the point cloud for classification, obtaining a plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud; obtaining a graph-structured representation of the plurality of subsets, the graph-structured representation having a node associated with each subset and one or more edges connecting the nodes; inputting the graph structure representation into a graph encoder convolutional neural network to obtain a graph structure embedding that encapsulates structural relationships between the plurality of subsets; Deriving a point cloud representation from the graph structure embedding; classifying the point cloud representation by inputting the point cloud representation into a classification encoder neural network to obtain a classification of the selected point cloud; A computer-implemented method comprising:

2. 10. The computer-implemented method of claim 1, further comprising inputting the selected point cloud into a pre-trained feature encoder neural network to obtain feature embeddings for the selected point cloud.

3. 3. The computer-implemented method of claim 2, wherein deriving a point cloud representation from the graph structure embedding further comprises concatenating the graph structure embedding and the feature embedding to create the point cloud representation.

4. 4. The computer-implemented method of claim 2 or 3, wherein the pre-trained feature encoder neural network is a shape encoder and the feature embeddings are shape embeddings.

5. 2. The computer-implemented method of claim 1, wherein obtaining the plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud further comprises inputting the selected point cloud to an unsupervised partial decomposition module, the supervised partial decomposition module performing unsupervised segmentation of the selected point cloud into the plurality of subsets.

6. Obtaining a graph structure representation of the plurality of subsets further comprises inputting the plurality of geometrically meaningful parts to a graph structure induction module having a partial feature encoder and a graph creation module; the partial feature encoder is a partial feature encoder neural network and generates the nodes as node representation embeddings of the plurality of subsets associated with the plurality of portions; the graph creation module generates the edges as edge representation embeddings of the subsets associated with the portions.

10. The computer-implemented method of claim 1.

7. Performing the point cloud classification is part of a training process for a point cloud classification model, the training process comprising: selecting a point cloud for classification comprises selecting a point cloud with a known classification from training data comprising a plurality of point clouds from a source domain; in the step of classifying the point cloud representation, the classification being a predicted classification of the selected point cloud; Having that, comparing the predicted classification to the known classification of the selected point cloud; adjusting at least one network weight of one of the graph encoder convolutional neural network and the classification encoder neural network based on the comparison; The computer-implemented method of claim 1 , further comprising:

8. The computer-implemented method of claim 7 , wherein comparing the predicted classification to the known classification comprises determining whether a classification loss is minimal.

9. 1. A computer program which, when executed on a computer, causes the computer to perform a method comprising a process of point cloud classification, the process comprising: Select the point cloud for classification, obtaining a plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud; obtaining a graph-structured representation of the plurality of subsets, the graph-structured representation having a node associated with each subset and one or more edges connecting the nodes; inputting the graph structure representation into a graph encoder convolutional neural network to obtain a graph structure embedding that encapsulates structural relationships between the plurality of subsets; Deriving a point cloud representation from the graph structure embedding; classifying the point cloud representation by inputting the point cloud representation into a classification encoder neural network to obtain a classification of the selected point cloud; A computer program comprising:

10. 1. An information processing device having a memory and a processor coupled to the memory, the processor configured to execute a process comprising point cloud classification, the process comprising: Select the point cloud for classification, obtaining a plurality of subsets of the selected point cloud associated with a plurality of portions of the selected point cloud; obtaining a graph-structured representation of the plurality of subsets, the graph-structured representation having a node associated with each subset and one or more edges connecting the nodes; inputting the graph structure representation into a graph encoder convolutional neural network to obtain a graph structure embedding that encapsulates structural relationships between the plurality of subsets; Deriving a point cloud representation from the graph structure embedding; classifying the point cloud representation by inputting the point cloud representation into a classification encoder neural network to obtain a classification of the selected point cloud; An information processing device having the above.

Images