Method and system for graph-based panorama segmentation

Abstract

Methods and systems for graph-based point cloud panoptic segmentation are described herein. The method receives points of a point cloud having semantic labels from a first class. In addition, a plurality of unified cluster feature vectors are also received from a second class. Each unified cluster feature vector is extracted from a point cluster in the point cloud. A graph comprising nodes and edges is constructed from the plurality of unified cluster feature vectors. Each node of the graph is a unified feature vector, and each edge of the graph indicates a relationship between each two nodes of the graph. The edges of the graph are represented as an adjacency matrix, where the adjacency matrix indicates whether an edge exists or does not exist between each two nodes. The graph is fed to a graph convolutional neural network for predicting an instance label for each node or an attribute for each edge, where the attribute for each edge is used to assign the instance label to each node. The method combines the points of the first class having semantic labels with the points of the second class having instance labels to generate points having both semantic labels and instance labels.

Description

[0001] This patent application claims priority to U.S. Patent Application Serial No. 17 / 204,930, filed March 17, 2021, entitled “Method and System for Graph-Based Panoramic Segmentation,” the entire contents of which are incorporated herein by reference. Technical Field

[0002] This invention relates to point cloud processing, and more particularly to a method and system for graph-based point cloud panoramic segmentation. Background Technology

[0003] Perception is an indispensable part of various intelligent / autonomous systems in fields such as autonomous driving, autonomous manufacturing, inspection, and medical diagnosis. In recent years, artificial intelligence has been widely applied to perception, enabling computers to learn to interpret and understand the visual world using digital images or point clouds. Using images generated by cameras or three-dimensional (3D) point clouds generated by LiDAR (light detection and ranging) sensors, computers can use deep learning to learn models to accurately "perceive" the environment (i.e., identify and classify objects in images or point clouds), and intelligent / autonomous systems can react to what they "perceive" in the environment.

[0004] Computers can also use segmentation methods to process 2D images or 3D point clouds to perceive their surroundings, which are the environments within the 2D images or 3D point clouds. Segmentation methods aim to generate labels for each pixel in an image, or for each point in a point cloud, where each point in the point cloud corresponds to a category (e.g., object class) identified in the image or point cloud (e.g., a car or pedestrian). Multiple pixels / points with the same label represent a mask (fragment) used to identify the object described by the category (e.g., a tree, person, or bicycle). While some segmentation methods can detect and identify “complete” fragments, such as bicycles, cars, or buildings, others can detect and identify parts of a “complete” fragment (e.g., the seat, wheels, and pedals of a bicycle), rather than identifying the entire bicycle. What is identified in an image or point cloud depends on which categories the segmentation method is used to identify.

[0005] Traditionally, there are two segmentation methods: semantic segmentation and instance segmentation. Semantic segmentation methods label pixels of an image or 3D points of a point cloud with corresponding semantic labels, where the semantic label indicates the category detected in the image or point cloud. For example, all cars detected in an image or point cloud are labeled with one semantic label, the sky with another, and so on for pedestrians and buildings. Thus, each category has a semantic label. Instance segmentation methods identify each instance of an object represented by a category in an image or point cloud, and then label each instance of the object with an instance label (e.g., each car, each bicycle, or each building in a scene has a corresponding instance label). Therefore, semantic segmentation methods label each pixel of an image or each point of a point cloud with a semantic label representing its category, while instance segmentation methods primarily use instance labels to label the points of instances of objects represented by each category. The methods and techniques used to implement semantic segmentation and instance segmentation are significantly different.

[0006] Another recently emerging segmentation type is panoptic segmentation. Panoptic segmentation is particularly useful for autonomous vehicles. It is an organic combination of semantic segmentation methods and instance segmentation methods. It can define either a "stuff" category or a "things" category. The "stuff" category is labeled as described above for semantic segmentation, and the "things" category is labeled as described above for instance segmentation. The user of the panoptic segmentation method decides which categories are "things" and "stuff". Typically, instances of the "things" category are the necessary and important instances for successfully implementing panoptic segmentation. For example, in the case of autonomous vehicles, pedestrians, animals, and bicycles are important; therefore, these are "things". Instances of buildings, sky, and curbs are less important; therefore, these are "stuff". For autonomous vehicles, knowing that buildings exist may be important, but knowing the boundaries of each building (instances of buildings) is not very important.

[0007] Known panoptic segmentation methods typically employ both semantic segmentation and instance segmentation methods simultaneously. Det2Seg describes such a panoptic segmentation method, proposed in "Zuo, C. et al., Det2Seg: A Two-Stage Approach for Road Object Segmentation from 3D Point Clouds," 2019 IEEE International Conference on Visual Communications and Image Processing (VCIP), Sydney, Australia, 2019. In Det2Seg, in addition to semantic segmentation, two stages of instance segmentation are applied: a coarse-grained stage and a fine-grained stage. The coarse-grained stage uses a sophisticated deep learning model.

[0008] Det2Seg requires complex deep learning models for instance segmentation. The computational cost of implementing and running complex deep learning models on computing systems is typically high, which limits the real-time performance of panoptic segmentation using the two-stage method described above.

[0009] Therefore, it is desirable to provide improved methods and systems for point cloud panoramic segmentation. Summary of the Invention

[0010] This paper describes methods and systems for graph-based panoramic segmentation of point clouds. Exemplary embodiments describe methods and systems applied to points with semantic labels (things) to generate a graph, the graph comprising nodes and edges, which are provided to a graph convolutional neural network (GCNN) for node classification or edge classification. Node classification predicts instance labels for the points, and edge classification predicts edge attributes for assigning instance labels to the points, outputting points with both instance and semantic labels. Points with both instance and semantic labels are combined with points with only semantic labels (i.e., points with stuff semantic labels) to generate a point cloud, the point cloud including points with semantic labels as well as points with both instance and semantic labels. To identify instance labels, the semantic labels of the semantic segmentation method are utilized without relying on complex deep learning models; therefore, the required computation is less than that of complex deep learning models.

[0011] According to one aspect, the present invention provides a method for graph-based panoramic segmentation. The method includes: receiving points of a point cloud with semantic labels from a first category, and receiving a plurality of unified clustering feature vectors from a second category; each unified clustering feature vector is extracted from a cluster of points in the point cloud. The method constructs a graph including nodes and edges from the plurality of unified clustering feature vectors. Each node of the graph is a unified feature vector, and each edge of the graph indicates the relationship between every two nodes and is represented as an adjacency matrix. The adjacency matrix indicates whether an edge exists between every two nodes. The method feeds the nodes and the adjacency matrix to a graph convolutional neural network, which predicts an instance label for each node or an attribute for each edge. The attribute of each edge is used to assign the instance label to each node. The method combines points with semantic labels from the first category with points with instance labels from the second category to generate a labeled point cloud, the labeled point cloud including points with both semantic and instance labels.

[0012] Each unified clustering feature vector represents a set of multiple clusters. Furthermore, each unified clustering feature vector is extracted from multiple points in the point cloud. Additionally, the method constructs a graph from the multiple unified clustering feature vectors, such that nodes in the graph are associated with the unified feature vectors, and the edges of the graph are represented as adjacency matrices. The adjacency matrix indicates whether an edge exists between any two nodes. The method feeds the nodes and the adjacency matrix into the graph convolutional neural network for prediction to determine instance segments. Finally, the method combines the instance segments with semantic segments to determine panoramic segments for the graph-based panoramic segmentation.

[0013] In some examples of the method, the elements of the adjacency matrix are determined using at least one similarity distance between every two nodes. In some examples, the at least one similarity distance is a combination of cosine similarity and Euclidean distance. In some examples of the method, the elements of the adjacency matrix are determined using a criterion that states an edge exists if the cosine similarity between two nodes is greater than a predetermined threshold and the Euclidean distance between the two nodes is less than another predetermined threshold.

[0014] In some examples of the method, the graph convolutional neural network is used for node classification to predict the instance label for each node of the graph, and each point in the point cloud is labeled with the instance label of its respective node to form one of the instance segments.

[0015] In some examples of the method, the graph convolutional neural network is used for edge classification to predict the properties of the edge between every two nodes. The graph assigns instance labels to the nodes connected together by at least one edge. Furthermore, each point in the point cloud is labeled with the instance label of its respective node to form one of the instance segments.

[0016] In some examples of the method, each unified clustering feature vector is extracted from multiple points in a point cloud using at least one of a learnable sparse convolution operation and a PointNet model, wherein the learnable sparse convolution operation and at least one of the PointNet model map the multiple points of the cluster to a 1×k vector, where k is a hyperparameter. In some examples of the method, the unified clustering feature vector includes the centroid value of each cluster, generating a unified clustering feature vector of size 1×(k+3).

[0017] In some examples of the method, each point in the point cloud includes at least the spatial coordinates and semantic label of the point, wherein the semantic label of the point is determined by a semantic segmentation method.

[0018] In some examples of the method, the plurality of clusters are determined using at least one of k-means clustering, clustering around a centroid, and density-based clustering (DBSCAN).

[0019] In some examples, the present invention describes a system for graph-based panoptic segmentation using a graph convolutional neural network, the system comprising: a memory for storing instructions; and one or more processors coupled to the memory and configured to execute the instructions, such that the system: receives points of a point cloud with semantic labels from a first category; receives a plurality of unified clustering feature vectors from a second category, each unified clustering feature vector being extracted from a cluster of points in the point cloud; and constructs a graph including nodes and edges from the plurality of unified clustering feature vectors. Each node of the graph is a unified feature vector, and each edge of the graph indicates a relationship between every two nodes and is represented as an adjacency matrix. The adjacency matrix indicates whether an edge exists between every two nodes. The nodes and the adjacency matrix are fed into the graph convolutional neural network, which is used to predict an instance label for each node or an attribute for each edge. The attribute of each edge is used to assign the instance label to each node. Points with semantic labels from the first category are combined with points with instance labels from the second category to generate points with both semantic and instance labels.

[0020] In some examples of the system, the elements of the adjacency matrix are determined using at least one similarity distance between every two nodes. In some examples of the system, the at least one similarity distance is a combination of cosine similarity and Euclidean distance. In some examples of the system, the elements of the adjacency matrix are determined using a criterion that states an edge exists if the cosine similarity between two nodes is greater than a predetermined threshold and the Euclidean distance between the two nodes is less than another predetermined threshold.

[0021] In some examples of the system, the graph convolutional neural network is used for node classification to predict the instance label for each node of the graph, and each point in the point cloud is labeled with the instance label of its respective node to form one of the instance segments.

[0022] In some examples of the system, the graph convolutional neural network is used for edge classification to predict the properties of the edge between every two nodes. The graph assigns instance labels to the nodes connected together by at least one edge. Furthermore, each point in the point cloud is labeled with the instance label of its respective node to form one of the instance segments.

[0023] In some examples of the system, each unified clustering feature vector is extracted from multiple points in a point cloud using at least one of a learnable sparse convolution operation and a PointNet model, wherein the learnable sparse convolution operation and at least one of the PointNet model map the multiple points of the cluster to a 1×k vector, where k is a hyperparameter. In some examples of the system, the unified clustering feature vector includes the centroid value of each cluster, generating a unified clustering feature vector of size 1×(k+3).

[0024] In some examples of the system, each point in the point cloud includes at least the spatial coordinates and semantic label of the point, wherein the semantic label of the point is determined by a semantic segmentation method.

[0025] In some examples of the system, the plurality of clusters are determined using at least one of k-means clustering, clustering around a centroid, and density-based clustering (DBSCAN). Attached Figure Description

[0026] Figure 1 A block diagram of an exemplary processing system that can be used to implement the methods and systems disclosed herein is shown;

[0027] Figure 2 This is a block diagram of an exemplary instance segmentation subsystem provided as an example of the present invention;

[0028] Figure 3 This is an illustrative example of the output of some components of the instance segmentation subsystem provided as an example of the present invention;

[0029] Figure 4 This is a hypothetical example of a graph representation of a unified clustering feature vector provided as an example of the present invention;

[0030] Figure 5 This is a flowchart illustrating an exemplary method for training instance segmentation, provided as an example of the present invention.

[0031] Figure 6 This is a flowchart of an exemplary inference method for determining panoptic segmentation using graph representation and GCNN, provided as an example of the present invention.

[0032] The same reference numerals may be used to denote the same components in different figures. Detailed Implementation

[0033] This invention has been carried out with reference to the accompanying drawings, in which embodiments are illustrated. However, many different embodiments may be used, and therefore the description should not be construed as limiting oneself to the embodiments set forth herein. Rather, these embodiments are provided to make the invention thorough and complete.

[0034] Exemplary embodiments describe methods and systems for graph-based point cloud panorama segmentation. The methods and systems include identifying semantic labels for each point in a point cloud, and then considering only some of the identified semantic labels belonging to the category "things" for instance segmentation. Points in each fragment of "things" are clustered, and a uniform clustering feature vector is determined for each cluster of points with the same semantic label. The uniform clustering feature vector is used to generate a graph including nodes and edges, where each uniform clustering feature vector is a node of the graph. The edges of the graph describe the relationships between every two nodes. The edges between every two nodes are represented as an adjacency matrix. Exemplary embodiments describe methods and systems for predicting instance labels for each node using a GCNN for node classification. For node classification, the adjacency matrix is ​​determined using a criterion based on a similarity metric between every two nodes. During inference, the GCNN predicts instance labels for each node. The methods and systems assign the instance labels to corresponding points of each node. Other exemplary embodiments describe methods and systems for predicting attributes for each node using a GCNN for edge classification. For edge classification, the adjacency matrix is ​​determined using values ​​of a similarity metric. During inference, the GCNN predicts whether an edge exists between every two nodes in the graph. The method and system assign instance labels to nodes connected by edges, and further assign the instance labels to the corresponding points of each node. Points with only semantic labels (i.e., points of the stuff category) are combined with points with both semantic and instance labels (i.e., points of the things category) to generate a labeled point cloud, which includes points with semantic labels and points with both semantic and instance labels, and is the output of the graph-based panoramic segmentation.

[0035] Figure 1 This is a block diagram of an exemplary simplified processing system 100, which can be used to implement the embodiments disclosed herein. The exemplary processing system 100 or variations thereof described below can be used to implement the panoramic segmentation system 102. Other processing systems may be adapted to implement the embodiments described in this invention and may include components different from those discussed below. Although Figure 1 A single instance of each component is shown, but there may be multiple instances of each component in the processing system 100 (e.g., when the processing system 100 is used to implement a graph-based panoramic segmentation system 102).

[0036] The processing system 100 may include one or more processing devices 104, such as processors, microprocessors, graphics processing units (GPUs), tensor processing units (TPUs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), special-purpose logic circuits, or combinations thereof. The processing system 100 may optionally include one or more input / output (I / O) interfaces 106 to enable connectivity with one or more optional input devices 108 and / or output devices 110. The processing system 100 may include one or more network interfaces 112 for wired or wireless communication with other processing systems. The one or more network interfaces 112 may include wired links (e.g., Ethernet cables) and / or wireless links (e.g., one or more antennas) for intra-network and / or inter-network communication.

[0037] The processing system 100 may also include one or more storage units 114, which may include mass storage units such as solid-state drives, hard disk drives, disk drives, and / or optical disk drives. In some exemplary embodiments, the storage unit 114 may include a database 116 for storing training datasets that can be used to train portions of the graph-based panoptic segmentation system 102, as described in further detail below. Although Figure 1 The illustration shows storage unit 114 including database 116, but in an alternative embodiment, database 116 may be included in one or more remote storage units that can be remotely accessed via network interface 112. Database 116 may need to be loaded into memory 118 before being used by processing device 104.

[0038] Processing system 100 may include one or more non-transitory memories 118, which may include volatile or non-volatile memories (e.g., flash memory, random access memory (RAM), and / or read-only memory (ROM)). The non-transitory memories 118 may store instructions to be executed by processing device 104, such as to perform the exemplary methods described in this invention. Memory 118 may store other software (e.g., instructions to be executed by processing device 104), such as operating systems and other applications / functions. In some examples, one or more databases 116 may be provided by external memory (e.g., an external drive that communicates with processing system 100 via wired or wireless communication) or may be provided by transient or non-transitory computer-readable media. Examples of non-transitory computer-readable media include RAM, ROM, erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, CD-ROM, or other portable storage. In some examples, the graph-based panoramic segmentation system 102 may be software, comprising machine-readable instructions stored in memory 118 and executable by processing device 100. The graph-based panoramic segmentation system 102 includes a semantic segmentation subsystem 120, an instance segmentation subsystem 122, and a fusion subsystem 124. Each of subsystems 120, 122, and 124 may be software, comprising machine-readable instructions stored in memory 118 and executable by processing device 100. In some examples, one or more of the semantic segmentation subsystem 120, instance segmentation subsystem 122, and fusion subsystem 124 may implement a method for performing inference tasks relying on machine learning. In some examples, the graph-based panoramic segmentation system 102 may be implemented as a single hardware component (e.g., a system-on-a-chip including components that perform the functions of the semantic segmentation subsystem 120, instance segmentation subsystem 122, and fusion subsystem 124). In other examples, multiple chips (each performing a task of a component of the graph-based panoramic segmentation system 102) are executed by their respective hardware components (e.g., separate chips within the processing system 100).

[0039] A bus 126 may exist, providing communication between the components of the processing system 100, including processing device 104, optional I / O interface 106, network interface 112, storage unit 114, and / or memory 118. Bus 126 may be any suitable bus architecture, including memory bus, peripheral bus, or video bus.

[0040] exist Figure 1 In this system, optional input devices 108 (e.g., keyboard, mouse, microphone, touchscreen integrated into a display device, which may include a user interface (UI) and / or keypad) and optional output devices 110 (e.g., display devices that may include a UI, speakers, and / or printers) are displayed external to the processing system 100. In other examples, one or more of the input devices 108 and / or output devices 110 may be internal components of the processing system 100. Input device 108 may include a display device with a display screen and UI navigation device (e.g., touchscreen input device, mouse, or handheld controller). Output device 110 may also include a display device with a display screen and UI navigation device for displaying the generated results of the graph-based panoramic segmentation system 102.

[0041] The panoramic segmentation system 102 assigns instance labels to pixels or point cloud points of images belonging to the "things" category, identifying each instance of an object represented by the "things" category, while assigning semantic labels to pixels or point cloud points of images belonging to the "stuff" category, ignoring instances of objects belonging to the "stuff" category. For example, if cars belong to the "things" category and buildings belong to the "stuff" category, the graph-based panoramic segmentation system 102 can assign instance labels to pixels or point cloud points of each car, generating a mask represented by each instance label. The instance label for each instance of a car is unique; therefore, the panoramic segmentation system 102 generates a unique mask for each instance of a car. Furthermore, the graph-based panoramic segmentation system 102 can also assign semantic labels to pixels or point cloud points of images belonging to the "stuff" category, without assigning instance labels to the pixels of the image or the point cloud points. The semantic labels for all instances of buildings are the same; therefore, the panoramic segmentation system 102 can generate a single mask for all buildings. Typically, the graph-based panoramic segmentation system 102 uses user-defined "stuff" and "things" categories. In the following exemplary embodiments, the 3D points (points) of a point cloud are considered; however, this should not be regarded as a limitation, but is only for illustration.

[0042] The graph-based panorama segmentation system 102 includes a semantic segmentation subsystem 120 and an instance segmentation subsystem 122. The semantic segmentation subsystem 120 labels each point with semantic tags, generating masks for all categories (e.g., roads, buildings, trees, cars, pedestrians) in things and stuff. The semantic segmentation subsystem 120 can implement numerous methods with varying accuracies, including the method proposed in "Cheng, Ran et al., '2-S3Net: Attentive Feature Fusion with Adaptive Feature Selection for Sparse Semantic Segmentation Network', arXiv preprint, arXiv:2102.04530 (2021)". It should be understood that the method implemented by the semantic segmentation subsystem 120 is an example, not a limitation, and is for illustrative purposes only. Various semantic segmentation methods that accept point clouds as input and label the points of the point cloud with semantic tags can be implemented by the semantic segmentation subsystem 120.

[0043] Instance segmentation subsystem 122 is used to mark points belonging to the `things` category using instance labels of `things`; instance labels are used for... things Each instance of an object in a category is unique. The outputs of semantic segmentation 120 and instance segmentation 122 are merged to generate a panoramic segmentation, where points in the stuff category and points in the things category are labeled with tags; points in the stuff category have semantic tags, and points in the things category have both instance tags and semantic tags.

[0044] Figure 2 This is a block diagram of an exemplary instance segmentation provided by an exemplary embodiment. The instance segmentation subsystem 122 executes multiple modules, including a filtering module 202, a clustering module, an embedding module 206, and a graph representation module 208. The instance segmentation subsystem 122 also includes a graph convolutional neural network (GCNN) 210. The instance segmentation subsystem 122 receives the output from the semantic segmentation subsystem 120, i.e., points in a point cloud with semantic labels. It may also receive a labeled dataset for training the GCNN 210. Each labeled dataset includes multiple labeled point clouds. Each labeled point cloud includes multiple points, wherein each of the multiple points is labeled with a ground truth semantic label and an instance label to train the GCNN 210.

[0045] Not all points in the point cloud are processed by the instance segmentation subsystem 122; only points with semantic labels belonging to the "things" category are processed by the instance segmentation subsystem 122. The filtering operation 202 has a list of "things" categories and their corresponding semantic labels (e.g., the tree category has semantic label 1, the car category has semantic label 2). The filtering module 202 selects only points with semantic labels belonging to the "things" category for processing by the clustering module 204, which groups points with the same semantic label into point clusters. These point clusters are fed to the embedding module 206, which extracts a unified clustering feature vector from each point cluster. The unified clustering feature vector is fed to the graph representation 208 to create a graph of nodes and edges. The graph, including nodes and edges, is fed to a graph convolutional network (GCNN) 210 to predict the instance labels of nodes or the attributes of edges to determine the instance labels of the nodes connected by the edges.

[0046] Figure 3 This is an illustrative example of the output of some components of the instance segmentation subsystem 122 provided in the exemplary embodiment. For example, points 302 with semantic labels include labels 1 and 2, where 1 refers to a tree and 2 refers to a car. These fragments have no instance labels, such that the two fragments 302 of the car only have semantic label 2. Filter 202 retains only the things category 304 for further processing in embedding 206.

[0047] refer to Figure 2 The clustering module 204 is used to divide the points received from the filtering module 202 into clusters based on a similarity metric. The clustering module 204 applies a clustering operation to multiple points for each semantic label. The clustering module 204 groups points with internal similarity. Exemplary embodiments apply different types of clustering methods. For example, k-means clustering using Mahalanobis distance, partition around medoids (PAM) clustering, or density-based clustering (DBSCAN), where DBSCAN is a non-parametric method. The clustering module 204 groups multiple points into a cluster. Each cluster can have a different number of points than other clusters. Figure 3In the illustration, the output of clustering 204 is cluster points 306; each pattern corresponds to a point cluster 308 (only two of them are labeled). The clustering pattern is a cluster label representing each point cluster. Points at this stage have both semantic and cluster labels. It can be observed that point clusters 308 have different sizes. In other words, different numbers of points form each point cluster 308. The method implemented by clustering 204 is applied to each semantic label in the point cloud. After feeding the cluster points 306 to the graph representation 208 and GCNN 210, the output includes points with semantic and instance labels, displayed as a mask 310 (described in detail below).

[0048] exist Figure 2In this process, the embedding module 206 encodes multiple points of each cluster into a unified 1xk-dimensional feature vector, where k is a hyperparameter stored in memory. An exemplary embodiment may use learnable sparse convolution operations, such as the Minkowski convolutional neural network in “Choy, Christopher, JunYoung Gwak and Silvio Savarese, 4d Spatio-Temporal Convnets: Minkowski Convolutional Neural Networks,” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019. Other exemplary embodiments may use the PointNet model, such as the PointNet model in "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation" by Qi, Charles R. et al., Proceedings of the IEEE conference on computer vision and pattern recognition, 2017. It should be understood that these two methods are examples, not limitations, and are for illustrative purposes only. Other embedding methods 206 that encode an n×m matrix and extract a unified clustering feature vector from that n×m matrix may be applicable. This n×m matrix represents a point cluster, comprising n points that constitute the cluster, where each point has m elements. At this processing stage, each point in the cluster can have five parameters (m): normal x, y, z coordinates, intensity value, and semantic label. Therefore, regardless of the number of points in the cluster, the output of the embedding module 206 is a feature vector of size 1×k, i.e., the unified clustering feature vector.

[0049] An exemplary embodiment can concatenate a unified clustering feature vector with the three centroid values ​​(x-dimensional centroid value, y-dimensional centroid value, and z-dimensional centroid value) of each point cluster, such that the size of the unified clustering feature vector is 1×(k+3). It should be understood that any method described below for a unified clustering feature vector of size 1×k also applies to a unified clustering feature vector with the added centroid data, i.e., a unified clustering feature vector of size 1×(k+3). It should be understood that other data can also be concatenated with the unified clustering feature vector. The centroid value concatenation is merely an example and not a limitation, and is for illustrative purposes only.

[0050] The graph representation module 208 receives the output of the embedding module 206 (i.e., the unified feature vector of each node cluster) and outputs a graph including nodes and edges; wherein each node is the unified feature vector of the node cluster, and each edge is represented by an adjacency matrix. The adjacency matrix is ​​a square matrix describing the edge connections between two nodes in the graph.

[0051] Figure 4 This is a hypothetical example illustrating the operations performed by the graph representation module 208. The graph representation module 208 receives a set of uniform clustering feature vectors 402, which includes n uniform clustering feature vectors (402-1, 402-2, 402-3, ..., 402-n). The set of uniform clustering feature vectors 402 represents nodes 404 (404-1, 404-2, 404-3, 404-n) of 410, where each uniform clustering feature vector (402-1, 402-2, 402-3, ..., 402-n) is a node (404-1, 404-2, 404-3, ..., 404-n) in 410. Edges 406 (one of which is labeled) can connect nodes (404-1, 404-2, ...). Edges connecting nodes are represented in adjacency matrix 408, which indicates the relationships between nodes 404. The graph representation module 410 is fed to GCNN 210 for training. Exemplary embodiments describe methods and systems for training GCNN 210 in a supervised learning algorithm for node classification to predict instance labels for nodes 404. In such exemplary embodiments, each node 404 is labeled with an instance label. Nodes are labeled with the instance labels of their respective nodes.

[0052] Exemplary embodiments describe methods and systems for training GCNN 210 in a supervised learning algorithm for edge classification to predict the properties of edge 406. In such exemplary embodiments, GCNN 210 predicts the properties of edges 406 connecting nodes 404. Nodes 404 interconnected by at least one edge 406 are grouped and labeled with instance labels using the group. Points with a uniform feature vector associated with each node 404 in the group are labeled with the instance labels of the group.

[0053] GCNN 210 receives input X, which is an n×k matrix, where n is the number of point clusters and also the number of unified cluster feature vectors 402, and k is the size of each unified cluster feature vector. It also receives an adjacency matrix 406A (edge ​​406 information).

[0054] Node classification

[0055] An exemplary embodiment describes a method and system for training a GCNN 210 to perform node classification and predict the instance label for each node 404 using a supervised learning algorithm. The training dataset stored in database 116 contains a labeled point cloud with both semantic and instance labels. Therefore, the target instance label for each node 404 is known during the training of the GCNN 210.

[0056] GCNN 210 is a multi-layer convolutional neural network that processes graphs (e.g., graph data) and can be understood as a neural network consisting of a first layer (usually called the input layer), multiple hidden layers, and a final layer (usually called the output layer). The term "multiple" here has no specific metric.

[0057] The processing of each layer in GCNN 210 is relatively simple. Simply put, the operation H of each layer... l+1 Equation (1) represents:

[0058] H l+1 =f(H l ,A)=σ(AH l W l +b l (1)

[0059] Where f is a function, H 0 =X, l is the layer number, σ is the activation function (linear or nonlinear), W l It is the weight matrix of layer l, b l This is the bias of layer l. The activation function ensures that the values ​​passed to subsequent layers are within an adjustable, expected range. Parameter W l and b l This was learned during the training of GCNN 210. A is the adjacency matrix 406.

[0060] The adjacency matrix is ​​symmetric. When training GCNN 210 for node classification, adjacency matrix 406 can be filled with 1s and 0s, where a value of 1 indicates a connection and a value of 0 indicates no connection. Since there are no self-loops, the diagonal values ​​of adjacency matrix 408 are 0.

[0061] A similarity metric is used to determine whether an edge 406 connecting two nodes 404 exists. An exemplary embodiment uses Euclidean distance and cosine similarity as the similarity metrics. Each node 404 is a uniform clustering feature vector 402. The cosine similarity metric between every two nodes in the graph is calculated according to the following equation:

[0062]

[0063] Among them, A i and B i These are elements of vectors A and B, which are unified clustering feature vectors 402 associated with two nodes 404.

[0064] The Euclidean distance between the two nodes 404 can be calculated using the following equation:

[0065]

[0066] Among them, A i and B i These are elements of vectors A and B, which are unified clustering feature vectors 402 associated with two nodes 404.

[0067] If the cosine similarity cos(θ) between the uniform clustering feature vectors of two nodes 404 is greater than a predetermined threshold Th cos Furthermore, the Euclidean distance d(A,B) between these two nodes is less than a predetermined threshold Th. d If the distance is such that an edge exists between the two nodes, then it is determined that an edge exists between them. Although the exemplary embodiment uses the distance described above, other distances may be applied, including a generalized form of the Euclidean distance and the Minkowski distance.

[0068] An exemplary embodiment may apply a self-loop edge, that is, add 1 to edge 406 of node 404, which itself is included in the computation of equation (1). An exemplary embodiment may perform a normalization method that changes the input data to a general scale for faster convergence during training. The normalization method may be batch normalization, layer normalization, group normalization, or other normalization methods. In some examples, the activation function is a linear function, such as a binary step, rectified linear unit (ReLU), leaky ReLU, identity, or stochastic ReLU; while in other examples, the activation function is a nonlinear function, such as a sigmoid, step, tanh, swish, inverse square root unit (ISRU), softplus, nonlinear square, linear square root reciprocal, exponential linear unit, and other types of nonlinear functions.

[0069] Training GCNN 210 involves adjusting the parameters (weights W) of each layer l. l and bias b l An exemplary method utilizes backpropagation to train a GCNN 210. Backpropagation is used to adjust (also called update) the values ​​of parameters (e.g., weights) in the GCNN 210 so that the error (or loss) in the output becomes smaller. For example, a defined loss function (e.g., cross-entropy loss) is computed by forward propagating the input to the output of the GCNN 210 and updating the parameters using a gradient algorithm (e.g., gradient descent) to reduce the loss function. This process is completed iteratively, called "adjustment". With each iteration (called a cycle), the cross-entropy loss is reduced until the parameters of the GCNN 210 are optimized. After the GCNN 210 is trained, the weights and biases are fixed and can be used in real-time to predict output values; in other words, to perform inference. It should be understood that other GCNN210 training methods can use alternative methods instead of backpropagation. Backpropagation is an example and not a limitation, and is for illustrative purposes only. Exemplary embodiments may describe methods and systems that can use other loss functions, including the mean squared error loss function.

[0070] The output is a matrix Y, indicating the probability that each node 404 is one of the instance labels of the target instance labels in the training dataset. During training, target (real) instance labels are obtained from the training dataset, where each point in the point cloud of the training dataset has an instance label. During training, each node 404 has an instance label for its points. During inference, GCNN 210 predicts the instance label for each node, and the points are labeled using the instance label of their respective predicted nodes.

[0071] Side classification

[0072] An exemplary embodiment uses a supervised learning algorithm to train GCNN 210 to perform edge classification to predict whether the attribute of each edge 406 is 1 (connection exists) or 0 (no connection exists). In other words, GCNN 210 determines whether an edge 406 exists between two nodes 404. During training, the representation of nodes 404 in 410 is the same as that of nodes 404 in the node classification described above (i.e., each node is a uniform clustering feature vector 402). Edges 406, represented as adjacency matrix 408, have a cosine similarity value (as calculated in equation (2)) and a Euclidean similarity value (as calculated in equation (3)) for every two nodes 404. GCNN 210 is trained as described above, where nodes 404 and adjacency matrix 408 are used as inputs. The output of GCNN 210 is a matrix Y indicating nodes 404 and their respective predicted edges 406; Y has the same format as adjacency matrix 408. The output Y is compared with the true adjacency matrix 408, indicating the relationships between nodes 404 obtained from the training dataset. The target output is the true adjacency matrix 408. It is compared with the output Y. Furthermore, the cross-entropy loss is determined and backpropagated to update the weights.

[0073] During inference, GCNN 210 predicts the properties of each edge 406 between every two nodes 404. Nodes 404 connected by the edges 406 predicted by the GCNN are assigned instance labels. Nodes are labeled using the instance labels of their respective nodes.

[0074] Figure 5 This is a flowchart of an exemplary method for training instance segmentation provided by an exemplary embodiment. Instance segmentation method 500 receives points from a point cloud labeled with semantic tags, considers points of the "things" category for instance segmentation 122, and filters out points of the "stuff" category 502. Method 500 applies a clustering method to divide the point cloud into clusters. Each point cluster includes multiple points, and each point has at least its spatial coordinates and a semantic tag 504. Point clusters can have different numbers of points; therefore, method 500 encodes the multiple points of each of the n clusters in the point cloud into a unified clustering feature vector of size 1×k, generating a total of n unified clustering feature vectors 506.

[0075] Then, method 500 represents the unified clustering features as a graph including nodes and edges, such that each unified clustering feature vector 402 is a node 404 of the graph, and the edges 406 of the graph (representing the relationships between the nodes) are represented as an adjacency matrix 508. For node classification, the adjacency matrix is ​​determined using a criterion based on a similarity metric between every two nodes. For edge classification, the adjacency matrix is ​​determined using the values ​​of the similarity metric.

[0076] Node 404 (carrying the unified clustering feature vector) and adjacency matrix 408 are fed to train GCNN 210 to predict the instance label of a node or the attribute of an edge, depending on whether GCNN 210 is trained to classify node 404 or edge 406. Through training, GCNN 210 can update its weights and biases to learn to predict the target instance label of each node 404 or the target attribute of each edge, depending on whether GCNN 210 is trained to classify node 404 or edge 406. GCNN 210 predicts the instance label of each node 404 or the edge attribute of each edge 406. The instance label of each node or the edge attribute of each edge is compared with the true instance label of node 404 or the attribute of edge 406 in the training dataset, depending on whether GCNN 210 is trained to classify node 404 or edge 406. Before GCNN 210 is optimally trained, the loss is calculated and backpropagated to update the weights and biases of GCNN 210.

[0077] Figure 6 This is a flowchart of an exemplary inference method for graph representation and GCNN-based panoptic segmentation provided by an exemplary embodiment. During training, GCNN 210 learns appropriate weights and biases. When optimally trained, GCNN 210 can be used to perform a given inference task. An exemplary embodiment describes a method 600 for performing inference to classify nodes 404 and predict their instance labels. Other exemplary embodiments describe a method for performing inference to classify edges 406 and predict the properties of edges 406 between nodes 404. The prediction type depends on whether GCNN 210 is trained to classify nodes 404 or edges 406. Inference method 600 receives a unified clustering feature vector 402 for each cluster of points of the things category and points 602 with semantic labels of the stuff category, and represents each unified clustering feature vector 402 as a node 404 of 410 at box 604. Method 600 determines an adjacency matrix 408 describing the edges of 410 at box 606. As described above, an adjacency matrix 408 is determined for either node 404 classification or edge 406 classification. For node classification, the adjacency matrix is ​​determined using a criterion based on a similarity metric between every two nodes. For edge classification, the adjacency matrix is ​​determined using the value of the similarity metric. The method propagates the input (nodes and adjacency matrix) of GCNN 210 to the output using trained weights and biases.

[0078] If GCNN 210 is used to classify node 404, then GCNN 210 predicts an instance label for each node 404. Then, method 600 groups all nodes 404 with the same instance label and assigns the predicted instance label to their respective point clouds.

[0079] If GCNN 210 is used to classify edge 406, then GCNN 210 predicts the properties of the edge between every two nodes. During inference, GCNN 210 classifies each edge 406 to determine its presence 608. Nodes 404 connected by the edges 406 predicted by GCNN 210 are assigned instance labels. Points associated with each node 404 are labeled using the instance label of their respective nodes.

[0080] The method disclosed in this invention can be executed by a subsystem module, routine, or software subroutine of the processing system 100. The coding of the software used to perform the steps of the method is also within the scope of understanding of those skilled in the art regarding the method. The method for graph-based panoramic segmentation may include more or fewer steps than those shown and described, and these steps may be performed in different orders. Computer-readable instructions executable by the processor of the processing system 100 may be stored in the memory 118 of the processing system or in a computer-readable medium. It should be emphasized that, unless otherwise stated, the steps of the method do not need to be performed in the exact order shown; similarly, the various steps of the method may be performed in parallel rather than sequentially.

[0081] It is understood that once the method for graph-based panoramic segmentation described in this invention is implemented, it can be executed automatically by the processing system 100, which is convenient for users as it does not require manual interaction.

[0082] Those skilled in the art should understand that, for the sake of convenience and brevity, the specific working processes of the above-described systems, devices, and units can be referred to the corresponding processes in the above-described method embodiments, and will not be repeated here.

[0083] In the described embodiments, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the described system embodiments are merely examples. Furthermore, units or components can be combined or integrated into another system, or some features can be omitted or not performed. Additionally, the mutual coupling or direct coupling or communication connections shown or discussed can be implemented through some interface. Indirect coupling or communication connections between systems or units can be implemented electronically, mechanically, or otherwise.

[0084] This invention may be embodied in other specific forms without departing from the subject matter of the claims. The exemplary embodiments described are to be regarded in all respects as illustrative rather than restrictive. Selected features from one or more of the foregoing embodiments may be combined to create alternative embodiments not explicitly described, and features suitable for such combinations are to be understood to fall within the scope of this invention.

[0085] All values ​​and sub-ranges within the scope of the disclosure are also disclosed. Furthermore, while the systems, devices, and processes disclosed and illustrated herein may include a specific number of elements / components, these systems, devices, and components may be modified to include more or fewer such elements / components. For example, while any element / component disclosed may be a single quantity, embodiments disclosed herein may be modified to include multiple such elements / components. The subject matter described herein is intended to cover and include all suitable technical modifications.

[0086] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of the embodiment solution according to actual needs.

[0087] In addition, the functional units in the exemplary embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0088] When these functions are implemented as software functional units and sold or used as independent products, they can be stored in a storage medium, including several instructions to instruct a computer device (which may be a personal computer, server, or network device) to perform all or part of the steps of the methods described in the embodiments of this application. The aforementioned storage medium includes any medium capable of storing program code, such as a universal serial bus (USB) flash drive, a removable hard disk, read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk, etc.

[0089] The above description is merely a specific implementation method and is not intended to limit the scope of protection. Any variations or substitutions that can be easily conceived by those skilled in the art are within the scope of protection. Therefore, the scope of protection should be determined by the scope of the claims.

Claims

1. A method for graph-based panoramic segmentation, characterized in that, The method includes: Receive points from the point cloud with semantic labels from the first category; Multiple unified clustering feature vectors are received from the second category, each unified clustering feature vector being extracted from a point cluster in the point cloud, wherein the second category belongs to points in the point cloud with semantic labels received from the first category; A graph comprising nodes and edges is constructed from the plurality of unified clustering feature vectors, wherein each node of the graph is a unified feature vector, and each edge of the graph indicates the relationship between every two nodes of the graph and is represented as an adjacency matrix, wherein the adjacency matrix indicates whether there is an edge between every two nodes; The nodes and the adjacency matrix are fed into a graph convolutional neural network, which is used to predict the instance label of each node or the attribute of each edge, wherein the attribute of each edge is used to assign the instance label to each node. The points with semantic labels in the first category are combined with the points with instance labels in the second category to generate points that have both semantic labels and instance labels. Each uniform cluster feature vector is extracted from multiple points in a point cloud using at least one of a learnable sparse convolution operation and a PointNet model, wherein the learnable sparse convolution operation and at least one of the PointNet model map the multiple points of the cluster to... vector, where, It's a hyperparameter; The unified clustering feature vector includes the centroid value of each cluster, and is generated to a size of The unified clustering feature vector.

2. The method according to claim 1, characterized in that, The elements of the adjacency matrix are determined using at least one similarity distance between every two nodes.

3. The method according to claim 2, characterized in that, The at least one similarity distance is the cosine similarity and Euclidean distance between them.

4. The method according to claim 3, characterized in that, The elements of the adjacency matrix are determined using a criterion that states that an edge exists if the cosine similarity between two nodes is greater than a predetermined threshold and the Euclidean distance between the two nodes is less than another predetermined threshold.

5. The method according to claim 4, characterized in that, The graph convolutional neural network is used for node classification to predict the instance label of each node in the graph, and each point in the point cloud is labeled with the instance label of its corresponding node.

6. The method according to claim 3, characterized in that, The graph convolutional neural network is used for edge classification to predict the properties of the edge between every two nodes, the nodes of the graph being connected together by at least one edge, the at least one edge being assigned an instance label, and each point in the point cloud being labeled with the instance label of its corresponding node.

7. The method according to any one of claims 1 to 6, characterized in that, Each point in the point cloud includes at least its spatial coordinates and semantic label.

8. The method according to any one of claims 1 to 6, characterized in that, The multiple clusters are determined using at least one of k-means clustering, clustering around the centroid, and density-based clustering (DBSCAN).

9. A system for graph-based panoramic segmentation using graph convolutional neural networks, characterized in that, The system includes: Memory, used to store instructions; One or more processors, coupled to the memory and used to execute the instructions, to: Receive points from the point cloud with semantic labels from the first category; Multiple unified clustering feature vectors are received from the second category, each unified clustering feature vector being extracted from a point cluster in the point cloud, wherein the second category belongs to points in the point cloud with semantic labels received from the first category; A graph comprising nodes and edges is constructed from the plurality of unified clustering feature vectors, wherein each node of the graph is a unified feature vector, and each edge of the graph indicates the relationship between every two nodes of the graph and is represented as an adjacency matrix, wherein the adjacency matrix indicates whether there is an edge between every two nodes; The nodes and the adjacency matrix are fed into a graph convolutional neural network, which is used to predict the instance label of each node or the attribute of each edge, wherein the attribute of each edge is used to assign the instance label to each node. The points with semantic labels in the first category are combined with the points with instance labels in the second category to generate points that have both semantic labels and instance labels. Each uniform cluster feature vector is extracted from multiple points in a point cloud using at least one of a learnable sparse convolution operation and a PointNet model, wherein the learnable sparse convolution operation and at least one of the PointNet model map the multiple points of the cluster to... vector, where, It's a hyperparameter; The unified clustering feature vector includes the centroid value of each cluster, and is generated to a size of The unified clustering feature vector.

10. The system according to claim 9, characterized in that, The elements of the adjacency matrix are determined using at least one similarity distance between every two nodes.

11. The system according to claim 10, characterized in that, The at least one similarity distance is the cosine similarity and Euclidean distance between them.

12. The system according to claim 11, characterized in that, The elements of the adjacency matrix are determined using a criterion that states that an edge exists if the cosine similarity between two nodes is greater than a predetermined threshold and the Euclidean distance between the two nodes is less than another predetermined threshold.

13. The system according to claim 12, characterized in that, The graph convolutional neural network is used for node classification to predict the instance label of each node in the graph, and each point in the point cloud is labeled with the instance label of its corresponding node.

14. The system according to claim 11, characterized in that, The graph convolutional neural network is used for edge classification to predict the properties of the edge between every two nodes, the nodes of the graph being connected together by at least one edge, the at least one edge being assigned an instance label, and each point in the point cloud being labeled with the instance label of its corresponding node.

15. The system according to any one of claims 9 to 14, characterized in that, Each point in the point cloud includes at least its spatial coordinates and semantic label.

16. The system according to any one of claims 9 to 14, characterized in that, The multiple clusters are determined using at least one of k-means clustering, clustering around the centroid, and density-based clustering (DBSCAN).

17. A computer-readable medium, characterized in that, The computer-readable medium includes instructions that, when executed by a processing device of a processing system, cause the processing system to perform the method according to any one of claims 1 to 8.

18. A computer program, characterized in that, The computer program includes instructions that, when executed by a processing device of a processing system, cause the processing system to perform the method according to any one of claims 1 to 8.

Images