Semantic segmentation method for new category discovery in combination with voxels and space coordinates
By combining the semantic segmentation method of voxel and spatial coordinates, the combination of voxelization processing and multi-head attention mechanisms, the problems of inefficient and poor adaptability of point cloud data processing in the existing technology are solved, and efficient three-dimensional semantic segmentation and new category discovery are achieved.
Patent Information
- Application Number
- CN202411884394.1
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-20
- Publication Date
- 2025-04-15
AI Technical Summary
The existing three-dimensional semantic segmentation methods are inefficient in processing large-scale point cloud data, and are difficult to adapt to dynamically changing environments and new categories, especially in large-scale and complex environments, making it difficult to maintain high-precision segmentation effect.
The semantic segmentation method combining voxel and spatial coordinates is adopted to convert point cloud data into structured voxel format through voxelization processing, while retaining the original spatial coordinates, high-dimensional features are extracted using the MinkowskiNet model, and clustered through the MiniBatchKMeans algorithm to dynamically identify the neighborhood radius, integrate local and global features through the multi-head attention mechanism to improve the segmentation ability of the model.
Effectively process large-scale point cloud data, maintain spatial integrity, improve the model's performance in three-dimensional semantic segmentation and new category discovery tasks, and improve segmentation accuracy and adaptability in complex environments.
Smart Images
Figure CN119832235A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of computer vision technology, and in particular to a semantic segmentation method for discovering new categories by combining voxels and space coordinates. Background Art
[0002] In the field of new category discovery, significant progress has been made in recent years, with the earliest solutions introducing pairwise similarity predictions and learnable clustering objectives to enhance transfer learning across domains and tasks. These foundational works have laid the foundation for more sophisticated methods, such as modifications to deep embedding clustering, self-supervised learning frameworks, and multimodal data methods. Recent advances have introduced various innovative strategies, including dual-branch learning frameworks for visual category discovery, neighborhood contrastive learning, and unified objectives to simplify the discovery process and improve model robustness. In addition, meta-learning-based methods and class-incremental frameworks that handle dynamic properties have been developed. Further contributions in this area include the development of frameworks for specific challenges, such as leveraging mutual knowledge distillation, and implementing new loss functions to improve latent space separability. The integration of these multiple methods has significantly improved the ability to effectively discover and learn new categories in complex and dynamic environments.
[0003] In 2018, Georgia Institute of Technology published a paper titled "Learning to cluster in order to transfer across domains and tasks" at the International Conference on Representation Learning ICLR, which is considered the first solution to the problem of new category discovery. This innovative method leverages learned pairwise similarity predictions to promote unsupervised learning across a variety of domains and tasks. By integrating learnable clustering objectives into neural networks, it significantly enhances cross-domain and cross-task transfer learning, marking an important advance in the field and setting a basic standard for future research. In 2019, the Visual Geometry Group of the University of Oxford published a paper titled "Learning to discover novel visual categories via deep transfer clustering" at the International Conference on Computer Vision ICCV, studying the use of prior knowledge of related labeled categories to discover new object categories in unlabeled image collections. The article modified Deep Embedding Clustering (DEC) for transfer learning, introducing enhancements such as representation bottlenecks, temporal integration, and consistency to improve clustering results. Crucially, the article designed a method to estimate the number of categories in unlabeled data, using labeled categories as a diagnostic tool. This dual approach of combining modified DEC and category number estimation enables efficient discovery and learning of new object categories in unlabeled data.
[0004] In 2020, the Visual Geometry Group at the University of Oxford published a paper titled "Automatically discovering and learning new visual categories with ranking statistics" at the International Conference on Representation Learning ICLR, which defined the term "Novel Category Discovery" and laid the foundation for research on what was later called the new category discovery problem. This study introduced a framework for identifying new categories in image datasets without pre-labeled data, using three key strategies: self-supervised learning to train image representations on labeled and unlabeled data to reduce bias; rank statistics for knowledge transfer when clustering unlabeled images; and joint optimization of objective functions on labeled and unlabeled datasets to enhance classification and clustering performance.
[0005] In 2021, the University of Trento published a paper titled "Neighborhood contrastive learning for novel class discovery" at the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), which expanded previous research, defined "Novel Class Discovery" as an independent research field, and established a more focused research framework. The article developed neighborhood contrastive learning (NCL), which focuses on learning different representations for clustering by leveraging local neighborhoods in the embedding space. This method enhances the model's ability to distinguish new categories in unlabeled samples by mixing features of labeled and unlabeled data, collecting false positive pairs, and generating hard negative samples.
[0006] In 2022, Google Research published a paper titled "Novel class discovery without forgetting" at the European Conference on Computer Vision ECCV, focusing on identifying new categories in unlabeled data while retaining the accuracy of previously learned categories. The article uses pseudo-latent representation to reduce forgetting, mutual information-based regularization to enhance new category discovery, and uses known category identifiers for better generalization reasoning. This method addresses the need to incrementally identify new categories in dynamic, real-world applications while maintaining performance on known categories.
[0007] In 2023, ShanghaiTech University published a paper titled "Class-relation knowledge distillation for novel class discovery" at the International Conference on Computer Vision ICCV, emphasizing the transfer of knowledge from known categories to new categories. The method revolves around class relationship representation, based on the predicted class distribution of known categories. This is supplemented by a unique knowledge distillation strategy that maintains informative class relations when training new categories. In addition, the framework has a learnable weighting function that allows adaptive knowledge transfer based on semantic similarity between known and new categories.
[0008] Currently, 3D point cloud semantic segmentation is of great significance in applications such as autonomous driving, robotics, and urban planning. However, existing 3D semantic segmentation methods have some significant defects. Traditional methods are mainly divided into voxel-based and point-based methods. Voxel-based methods provide structured data representation, but they are prone to losing fine-grained details due to spatial discretization. On the other hand, although point-based methods retain detailed spatial information, they face complexity problems when processing large-scale point cloud data, resulting in low computational efficiency. In addition, traditional methods usually only focus on predefined semantic categories and lack the ability to generalize to categories that have not been seen in the training phase. This makes it difficult for them to adapt to dynamically changing environments and new categories in practical applications, especially in large-scale and complex environments, where existing methods find it difficult to maintain high-precision segmentation effects, and this problem becomes more prominent as objects and conditions in the environment continue to change. Summary of the invention
[0009] The purpose of the present invention is to solve the problems existing in the prior art and to propose a semantic segmentation method that combines voxels and spatial coordinates to discover new categories.
[0010] In order to achieve the above object, the present invention adopts the following technical solutions:
[0011] A semantic segmentation method combining voxels and spatial coordinates for new category discovery, comprising the following steps:
[0012] Input 3D point cloud data, voxelize the point cloud data, generate an integrated data set, and perform feature extraction on the integrated data set;
[0013] Cluster the feature vector set, calculate the centroid of each cluster, and select representative points;
[0014] Map the feature vector to a low-dimensional hash space and dynamically identify the neighborhood radius of the representative point based on its spatial distribution;
[0015] Perform linear transformation on the feature vectors of the representative point and its neighborhood points to generate query, key and value vectors, calculate the attention score by the dot product of the query vector and the key vector, generate the attention vector, calculate and assign the clustering attention vector, and fuse the local attention features with the global spatial features extracted from the dataset;
[0016] The fused features are input into a multi-layer perceptron to obtain semantic segmentation results of known and unknown categories.
[0017] Preferably, in order to ensure information conversion between the voxelized representation and the original spatial framework of the point cloud, it also includes establishing a bijective mapping to link each voxel to its corresponding point set.
[0018] Preferably, when extracting features from the integrated data set, a MinkowskiNet model is used to extract features from the integrated data set, and a high-dimensional feature representation of each voxel is output.
[0019] Preferably, the process of clustering the feature vector set is specifically: inputting the feature vector set into the MiniBatchKMeans algorithm, and organizing the point cloud data into multiple clusters according to feature similarity.
[0020] Preferably, the centroid is the average value of all feature vectors in the cluster, and the representative point is the point in the cluster that is closest to the centroid by minimizing the Euclidean distance.
[0021] Preferably, in the process of dynamically identifying the neighborhood radius of the representative point according to its spatial distribution, the radius is adjusted according to the spatial differences within the cluster to ensure that the neighborhood size is related to the distribution context of the point.
[0022] Preferably, the neighborhood points are identified within the radius of the corresponding representative point and used to evaluate the points in the hash bucket that fall within the radius of the representative point. The distance is calculated using the original spatial coordinates of the point cloud. The selection of neighborhood points is determined based on their spatial proximity to the representative point to ensure spatial relevance and accuracy.
[0023] Preferably, when the attention score is calculated by the dot product of the query vector and the key vector, normalization is performed through a function to obtain an attention weight.
[0024] Preferably, the attention vector is generated by calculating a weighted sum of attention weight value vectors.
[0025] Compared with the prior art, the present invention has the following beneficial effects:
[0026] The present invention effectively combines voxel geometry data with original spatial coordinates, converts the original point cloud data into a structured voxel format, while retaining its original spatial coordinates, ensuring spatial integrity, and can efficiently process large-scale point cloud data. Starting from the voxelized point cloud, each voxel is mapped to the point set from which it originates, retaining detailed spatial information. Subsequently, high-dimensional features are extracted from the data and clustered to identify representative points in each cluster and capture the key features of each cluster. High-dimensional data is managed and neighborhoods in the point cloud are segmented through neighboring hash mapping. By mapping dense feature vectors to low-dimensional hash spaces, local geometric relationships are efficiently organized and analyzed, enhancing the model's ability to parse complex spatial structures. Finally, the spatial feature attention mechanism optimizes spatial relationship understanding through a multi-head attention mechanism. This process integrates local feature vectors from neighborhood points with global feature backgrounds, dynamically adjusts the focus based on spatial proximity and context, and enhances the model's ability to perform detailed segmentation and new category discovery. BRIEF DESCRIPTION OF THE DRAWINGS
[0027] Figure 1 A flowchart of a semantic segmentation method combining voxels and spatial coordinates for new category discovery proposed in the present invention. DETAILED DESCRIPTION
[0028] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0029] Reference Figure 1 , a semantic segmentation method combining voxels and spatial coordinates for new category discovery, comprising the following steps:
[0030] S101: Input 3D point cloud data, expressed as Each point is represented by
[0031] S102: Point cloud data Perform voxel processing to generate a structured voxel grid Voxelization function f vox Point Cloud Aggregation into a structured voxel grid Expressed as:
[0032] S103: To ensure the information conversion between the voxel representation and the original spatial framework of the point cloud, a bijective mapping is established Each voxel Link to its corresponding point set The mapping is represented as:
[0033] S104: Integrate the voxelized data with the original spatial coordinates to generate a structured dataset with rich spatial details Integration function f int Synthesized from and The information is expressed as:
[0034] S105: Use MinkowskiNet model to integrate data sets Perform feature extraction and output high-dimensional feature representation of each voxel The model processing process is expressed as: where f i Representation and is the eigenvector associated with the i-th element in , where N is the number of points.
[0035] S101~S105 proposed a voxel geometry data integration method to convert the original point cloud data into a structured voxel format while retaining the original spatial coordinates to ensure spatial integrity, and used the MinkowskiNet model for high-dimensional feature extraction.
[0036] S201: Input the feature vector set F into the MiniBatchKMeans algorithm, and organize the point cloud data into k clusters according to feature similarity, expressed as: C1, C2, …, C k = MiniBatchKMeans(F,k), where C k Represents the set of indices of points belonging to the kth cluster.
[0037] MiniBatchKMeans is used to cluster and identify representative points to capture key features;
[0038] S202: For each cluster C k , calculate its centroid μ in the feature space k , the centroid is represented as the average of all feature vectors in the cluster, expressed as: Where |C k | is the number of points in the kth cluster.
[0039] S203: In each cluster C k In the example, the centroid μ is selected by minimizing the Euclidean distance k The nearest point is taken as the representative point P of the cluster rep , expressed as
[0040] S301: Apply neighbor hash mapping to the feature space extracted from the point cloud, and map these feature vectors to a low-dimensional hash space, represented as bucket(h(f i )mod m). Where, f i represents the feature vector extracted from each point, m represents the number of hash buckets, and h is the hash function that assigns these feature vectors to specific buckets in the hash table. The index variable i traverses each point in the sequence from 1 to N, where N represents the total number of points in the point cloud. The union operation ∪ aggregates all hash buckets to ensure full coverage of the feature space.
[0041] The neighborhood space segmentation module uses neighbor hash mapping to manage high-dimensional data and segment neighborhoods.
[0042] S302: Based on the representative point P rep The spatial distribution of is dynamically calculated around the neighborhood radius R. This radius is adjusted according to the spatial diversity within the cluster to ensure that the neighborhood size is relevant to the distribution context of the point, expressed as:
[0043] R=γ·ρ rep , where γ is the adaptive scaling factor, ρ rep is the range value obtained from the feature vector, determined by network training.
[0044] S303: Identify neighboring points within the radius of each representative point. This step evaluates the points in the hash bucket that fall within the radius of the representative point, and calculates the distance using the original spatial coordinates of the point cloud. The selection of neighboring points is based on their distance to the representative point P. rep The spatial proximity of is determined to ensure spatial correlation and accuracy, expressed as
[0045] in, Represents the representative point P rep The set of points within radius R, P rep For the selected cluster representative point, R defines the scope of the neighborhood, including the point cloud For points that fall within the boundaries of this space, the Euclidean distance is used to ensure that only points within the range of R are considered for neighborhood analysis.
[0046] S401: For representative point P rep and its neighboring points The feature vector of is linearly transformed to generate query (Q), key (K) and value (V) vectors, expressed as: Among them, W Q ,W K and W V Denote the weight matrices for query, key, and value vectors, respectively, are the feature vectors of the representative point and its neighborhood points respectively.
[0047] S402: Calculate the attention score by taking the dot product of the query vector and the key vector, and normalize it by the Softmax function to obtain the attention weight α, which is expressed as: Among them, d k is the dimension of the key vector, ensuring the stability of the attention scores at different scales.
[0048] S403: Use the attention weight α to calculate the weighted sum of the value vector to generate an attention vector A, expressed as: A=αV.
[0049] S404: For each cluster, calculate the attention vector A cluster , by averaging the feature dimensions of representative points and neighborhood points in the cluster, it is expressed as: Among them, d is the feature dimension, A i Represents the attention result of each feature in the cluster.
[0050] S405: The calculated cluster attention vector A cluster Assigned to all points in the cluster to ensure that the local attention effect permeates the entire cluster, expressed as: Among them, F updated Represents the updated feature vector for each point in the cluster, including the original features and the attention vector A cluster .
[0051] S406: Combine local attention features with global spatial features extracted from the dataset Fusion to form a rich feature set F final , expressed as: in, Represents a fusion operation.
[0052] S407: F final Input into the multi-layer perceptron to obtain the semantic segmentation results of known categories and unknown categories.
[0053] Compared with the prior art, the present invention:
[0054] 1. A voxel geometry data integration method was designed to convert the original point cloud data into a structured voxel format while retaining the original spatial coordinates to ensure spatial integrity, and to efficiently process large-scale point cloud data through an index mapping mechanism.
[0055] 2. A neighborhood space segmentation module is designed to manage high-dimensional data through neighboring hash mapping, segment the neighborhood in the point cloud, map dense feature vectors into low-dimensional hash space, and efficiently organize and analyze local geometric relationships.
[0056] 3. A spatial feature attention mechanism is designed to integrate local feature vectors from neighborhood points with global features through a multi-head attention mechanism, dynamically adjust the focus according to spatial proximity and context, and improve the performance of the model in three-dimensional semantic segmentation and new category discovery tasks.
[0057] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can make equivalent replacements or changes according to the technical scheme and inventive concept of the present invention within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.
Claims
1. A semantic segmentation method combining voxels and spatial coordinates for new category discovery, characterized by: The following steps are involved: Input 3D point cloud data, voxelize the point cloud data, generate an integrated data set, and perform feature extraction on the integrated data set; Cluster the feature vector set, calculate the centroid of each cluster, and select representative points; Map the feature vector to a low-dimensional hash space and dynamically identify the neighborhood radius of the representative point based on its spatial distribution; Perform linear transformation on the feature vectors of the representative point and its neighborhood points to generate query, key and value vectors, calculate the attention score by the dot product of the query vector and the key vector, generate the attention vector, calculate and assign the clustering attention vector, and fuse the local attention features with the global spatial features extracted from the dataset; The fused features are input into a multi-layer perceptron to obtain semantic segmentation results of known and unknown categories.
2. The semantic segmentation method for discovering new categories by combining voxels and spatial coordinates according to claim 1, characterized in that: To ensure the information conversion between the voxelized representation and the original spatial framework of the point cloud, a bijective mapping is also established to link each voxel to its corresponding point set.
3. The semantic segmentation method for discovering new categories by combining voxels and spatial coordinates according to claim 1, characterized in that: When extracting features from the integrated data set, a MinkowskiNet model is used to extract features from the integrated data set, and a high-dimensional feature representation of each voxel is output.
4. The semantic segmentation method for discovering new categories by combining voxels and spatial coordinates according to claim 1, characterized in that: The process of clustering the feature vector set is specifically as follows: inputting the feature vector set into the MiniBatchKMeans algorithm, and organizing the point cloud data into multiple clusters according to feature similarity.
5. The semantic segmentation method for discovering new categories by combining voxels and spatial coordinates according to claim 1, characterized in that: The centroid is the average value of all feature vectors in the cluster, and the representative point is the point in the cluster that is closest to the centroid by minimizing the Euclidean distance.
6. The semantic segmentation method for discovering new categories by combining voxels and spatial coordinates according to claim 1, characterized in that: In the process of dynamically identifying the neighborhood radius of the representative point according to its spatial distribution, the radius is adjusted according to the spatial differences within the cluster to ensure that the neighborhood size is related to the distribution context of the point.
7. The semantic segmentation method for discovering new categories by combining voxels and spatial coordinates according to claim 1, characterized in that: The neighborhood points are identified within the radius of the corresponding representative points and are used to evaluate the points in the hash bucket that fall within the radius of the representative points. The distance is calculated using the original spatial coordinates of the point cloud. The selection of neighborhood points is determined based on their spatial proximity to the representative points to ensure spatial relevance and accuracy.
8. The semantic segmentation method for discovering new categories by combining voxels and spatial coordinates according to claim 1, characterized in that: When the attention score is calculated by the dot product of the query vector and the key vector, normalization is performed through a function to obtain an attention weight.
9. The semantic segmentation method for discovering new categories by combining voxels and spatial coordinates according to claim 8, characterized in that: The attention vector is generated by calculating the weighted sum of the attention weight value vector.