Feature extraction method, device, storage medium and program product for point cloud data

Abstract

A feature extraction method, device, storage medium and program product of point cloud data are provided. The technical field relates to artificial intelligence. The method comprises the following steps: obtaining voxel features of point cloud data, wherein the voxel features comprise semantic features and spatial features of the point cloud; clustering the voxel features based on the semantic features to obtain at least one super class, and determining a mask rate of each super class, wherein the mask rate is used to represent the inter-class difference and intra-class difference between the super classes; and performing self-supervised learning on the point cloud feature extraction model according to the voxel features and the mask rate. The technical solution provided by the present application can improve the overall performance of the automatic driving system in a complex traffic environment and reduce the risk of traffic accidents.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a point cloud data feature extraction method, device, storage medium and program product. BACKGROUND

[0002] With the rapid development of autonomous driving technology, the processing and analysis of three-dimensional point cloud data become increasingly important. Point cloud feature extraction is a key link in these applications, which directly affects the performance of downstream tasks such as target detection, semantic segmentation and scene understanding. However, obtaining a large amount of labeled point cloud data often consumes time and effort, so developing efficient self-supervised learning methods to extract point cloud features has become a research hotspot.

[0003] Currently, many researchers have proposed point cloud self-supervised learning methods based on mask autoencoders. These methods usually use a random mask strategy to randomly mask a part of the input point cloud, and then train the model to reconstruct the masked part. Through the above way, the point cloud feature extraction model can learn the spatial structure and local geometric features of the point cloud without manual annotation.

[0004] However, in the autonomous driving scenario, the random mask strategy commonly used by existing methods has serious defects. Point cloud data in the autonomous driving environment usually contains complex and diverse elements such as vehicles, pedestrians, buildings and roads, etc. These elements differ significantly in importance and complexity. The above problems can lead to a decrease in the perception ability of autonomous driving systems in critical areas in complex traffic environments, increasing the risk of traffic accidents. SUMMARY

[0005] The present application provides a point cloud data feature extraction method, electronic device, storage medium and program product, which can improve the overall performance of the autonomous driving system in complex traffic environments and reduce the risk of traffic accidents.

[0006] In a first aspect of the present application, a point cloud data feature extraction method is provided, comprising:

[0007] Obtaining voxel features of point cloud data, the voxel features containing semantic features and spatial features of the point cloud;

[0008] Clustering the voxel features based on the semantic features to obtain at least one super-class, and determining the mask rate of each super-class, the mask rate representing the inter-class difference and intra-class difference between the super-classes;

[0009] Self-supervised learning of the point cloud feature extraction model according to the voxel features and the mask rate.

[0010] Optionally, the voxel features of the point cloud data are obtained, comprising:

[0011] acquire spatial features and semantic features of the point cloud data to obtain point cloud features;

[0012] voxelize and average pool the point cloud features to obtain voxel features.

[0013] Optionally, the clustering the voxel features based on the semantic features to obtain at least one super-class and determining a mask rate of each super-class comprises:

[0014] clustering the voxel features based on the semantic features to obtain at least one super-class and determining a cluster center of each super-class;

[0015] dividing each super-class into a plurality of super-class groups based on a first average distance between the cluster centers corresponding to each super-class, the super-class groups being used to represent inter-class differences between each super-class;

[0016] acquiring a second average distance between the voxel features inside each super-class and the cluster center, the second average distance being used to represent intra-class differences inside the super-class;

[0017] determining a mask rate of each super-class based on the first average distance and the second average distance of each super-class.

[0018] Optionally, the determining a mask rate of each super-class based on the first average distance and the second average distance of each super-class comprises:

[0019] determining a basic mask rate based on the first average distance corresponding to each super-class group;

[0020] determining an intra-class consistency coefficient based on the second average distance of each super-class, the intra-class consistency coefficient being inversely proportional to the second average coefficient;

[0021] taking a product of the basic mask rate and the intra-class consistency coefficient corresponding to each super-class as the mask rate.

[0022] Optionally, the self-supervised learning of the point cloud feature extraction model according to the voxel features and the mask rate comprises

[0023] masking the voxel features based on the mask rate;

[0024] reconstructing semantic features and spatial features through a coding-decoding device based on the masked voxel features to obtain reconstructed semantic features and reconstructed spatial features;

[0025] acquiring a loss value of the reconstructed semantic features and the reconstructed spatial features;

[0026] adjusting parameters in the point cloud feature extraction model based on the loss value.

[0027] Optionally, the semantic feature reconstruction and the spatial feature reconstruction of the voxel feature processed by the codec comprise:

[0028] determining a first weight factor corresponding to the semantic feature reconstruction and a second weight factor corresponding to the spatial feature reconstruction, wherein the first weight factor is greater than the second weight factor;

[0029] based on the first weight factor and the second weight factor, the semantic feature reconstruction and the spatial feature reconstruction of the voxel feature processed by the encoder are performed.

[0030] Optionally, the semantic feature reconstruction and the spatial feature reconstruction of the voxel feature processed by the codec comprise:

[0031] the spatial feature reconstruction of the voxel feature processed by the activated codec is performed;

[0032] based on a start time factor, the semantic feature reconstruction of the voxel feature processed by the activated codec is performed, wherein the start time factor is determined according to the degree of completion of the spatial feature reconstruction when the codec is activated to perform the semantic feature reconstruction in the process of training the point cloud feature extraction model.

[0033] In a second aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the point cloud data feature extraction method according to any one of the above aspects when executing the computer program.

[0034] In a third aspect of the present application, a non-transitory computer readable storage medium is provided, which stores a computer program executable by a processor to implement the point cloud data feature extraction method according to any one of the above aspects.

[0035] In a fourth aspect of the present application, a computer program product is provided, comprising a computer program executable by a processor to implement the point cloud data feature extraction method according to any one of the above aspects.

[0036] In summary, the one or more technical solutions provided in the present application have at least the following technical effects or advantages:

[0037] By obtaining the voxel features of the point cloud data, including semantic features and spatial features, and clustering the voxel features based on the semantic features to obtain superclasses, and then determining the mask rate of each superclass, the importance of different regions in the point cloud data is identified and distinguished. Compared with the random masking strategy commonly used in the prior art, the above-mentioned adaptive masking strategy based on semantic and spatial information can more specifically guide the model to focus on the key regions in the point cloud data. By combining voxel features and mask rates for model training, the point cloud feature extraction model trained by the method can more accurately capture the features of various complex elements in the autonomous driving scene, such as vehicles, pedestrians, buildings, and roads. This method effectively solves the defects of the existing random masking strategy in processing complex and diverse point cloud data in autonomous driving environments, and improves the model's perception ability of key regions. Finally, through the feature extraction of the point cloud data in the target region by the trained point cloud feature extraction model, more accurate and effective feature representation can be obtained, thereby providing more reliable input for downstream tasks such as target detection, semantic segmentation, and scene understanding, which helps to improve the overall performance of the autonomous driving system in complex traffic environments and reduce the risk of traffic accidents. BRIEF DESCRIPTION OF DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present application or prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings described below are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0039] Figure 1 is a flowchart of a point cloud data feature extraction method provided by an embodiment of the present application.

[0040] Figure 2 is a flowchart of a point cloud data feature extraction method provided by an embodiment of the present application.

[0041] Figure 3 is a comparison diagram of true value semantic features and encoder output heat maps provided by an embodiment of the present application.

[0042] Figure 4 is a distribution diagram of semantic categories and mask rates in a scene provided by an embodiment of the present application.

[0043] Figure 5 is a structural diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0044] In order to make the objects, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below in combination with the drawings in the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the protection scope of the present application.

[0045] Before the technical solutions of the present application are described, it is necessary to analyze the deficiencies of the prior art point cloud self-supervised learning method in detail. The existing point cloud self-supervised learning method mainly consists of two parts: proxy task and mask sampling strategy. The proxy task is used to guide the neural network to reconstruct the point cloud information and improve the feature extraction ability of the point cloud encoder, while the mask sampling strategy is used to sparsify the input data and determine the area for point cloud reconstruction to improve the learning efficiency and robustness of the model. These two parts work together to provide feature representation for downstream tasks. However, the prior art still has the following deficiencies:

[0046] Deficiency of proxy task:

[0047] 1.1 Learning limitation dominated by spatial features:

[0048] The existing proxy task method mainly focuses on the reconstruction of spatial features. For example, some methods predict whether each masked voxel is occupied to promote the learning of the model, and the core idea is to enhance the understanding of the scene by reasoning the existence of objects in the three-dimensional scene. Some other methods focus on the reconstruction of the three-dimensional coordinates of the point cloud in each masked voxel, and achieve higher precision scene reconstruction by guiding the model to learn the fine geometric structure of the point cloud data. Some other methods introduce geometric features such as normal information to enhance the spatial understanding ability of the model.

[0049] Although these methods can effectively capture the geometric structure of the point cloud, they often ignore the importance of semantic information. The specific manifestations are:

[0050] (a) Limitation of learning features: the point cloud self-supervised learning process can only learn the features required by the shallow layer of the downstream task network, and has limited promotion effect on advanced tasks such as three-dimensional object detection, three-dimensional semantic segmentation, and occupancy grid prediction. For example, taking the reconstruction of the three-dimensional coordinates or geometric features of the point cloud as the proxy task of the self-supervised learning method, it can only bring 0.85% and 1.06% improvement in L2 mAP index for three-dimensional object detection.

[0051] (b) Insufficient feature discrimination: The lower information quantity in the proxy task leads to poor feature discrimination. This makes the features of different point clouds not specific enough after passing through the point cloud encoder obtained by the current point cloud self-supervised learning, making it difficult for downstream tasks such as recognition and classification. Experiments show that the feature similarity of deep networks is nearly 50% larger than that of shallow networks.

[0052] 1.2 Lack of learning of advanced semantic relationships:

[0053] Although some methods attempt to introduce innovative proxy task designs, such as using jigsaw puzzles and mask autoencoders together to understand the spatial relationships between voxels, these methods still mainly focus on the reconstruction of spatial structures rather than the extraction of semantic information. Although this design can learn the arrangement and connection rules of voxels in three-dimensional space and thus capture the overall structural information of the scene at a higher level, it still lacks understanding of the semantic relationships and high-level concepts of objects in the scene.

[0054] Insufficient mask sampling strategies:

[0055] 2.1 Uniformity problem of mask strategy:

[0056] Existing mask sampling strategies, such as random masking or position-based masking, have the following problems:

[0057] (a) Ignoring regional differences: These strategies treat all voxels equally and ignore the inconsistency of voxel features in different regions, making it challenging for the model to reconstruct complex regions. Specifically, by randomly selecting voxels in three-dimensional space for masking, the model can infer the content of the masked voxels through neighborhood information. However, this method does not consider the differences in different positions in point cloud data, especially when dealing with regions with large distance differences in three-dimensional scenes. This may lead to uneven masking, affecting the model's generalization ability.

[0058] (b) Scattered focus: Unable to prioritize the classes or regions of interest according to the needs of downstream tasks, but evenly distribute the focus of the model's learning in all regions. This even distribution may lead to insufficient feature extraction of key regions, affecting the final task performance. For example, when using random masking as the mask sampling strategy for self-supervised learning methods, it can only bring a 1.84% improvement in L2mAP for three-dimensional object detection.

[0059] 2.2 Insufficient handling of sparse regions:

[0060] Some improved methods attempt to solve part of the problem by considering distance factors or special processing for sparse areas. For example, some methods propose that the mask rate should decrease with distance from oneself, using a higher mask rate for points closer to oneself and a lower mask rate for points farther away. Some other methods apply a lower mask rate in sparse areas to ensure that the model can stably reconstruct in these areas. However, these methods still fail to fully solve the above problems, especially in terms of combining semantic information and adapting to complex scenes.

[0061] 2.3 Lack of adaptability to different scales:

[0062] Although some methods propose mask strategies for different granularities to maintain consistent mask ranges, separate mask strategies are implemented according to different granularities of point cloud data. The advantage of this strategy is that it can flexibly adjust the mask rate according to the characteristics of data at different scales, thereby ensuring that the model can effectively learn at each scale. However, this method still fails to fully consider the distribution characteristics of semantic information at different scales, which may lead to insufficient learning in some key semantic areas.

[0063] Based on the above detailed analysis, the present application proposes a new point cloud self-supervised learning method to overcome the shortcomings of the prior art. Please refer to Figure 1 , Figure 1 is a flowchart of a feature extraction method for point cloud data provided by an embodiment of the present application. The method can be implemented by a computer program, can be implemented by a single-chip microcomputer, and can also run on a self-supervised learning system of a point cloud feature extraction model based on the von Neumann architecture. The computer program can be integrated in an application or run as a standalone tool application. Specifically, the method can include the following steps:

[0064] Step 101, obtaining voxel features of point cloud data, the voxel features containing semantic features and spatial features of the point cloud.

[0065] Wherein, the voxel feature refers to the feature information contained by each non-empty grid cell after discretizing the three-dimensional point cloud data into a regular three-dimensional grid structure. The voxel feature in the embodiment of the present application can be understood as a comprehensive feature representation, mainly composed of spatial features and semantic features.

[0066] Further, the semantic feature refers to a high-level feature representation reflecting the semantic information of objects or scenes in the point cloud data. In the embodiments of the present application, the semantic feature can be understood as a high-dimensional feature vector extracted by a pre-trained model. The feature vector encodes high-level semantic information such as the possible object category to which the voxel belongs and the scene structure. The semantic feature is used to enhance the model's understanding of the scene content, provide semantic-related guidance for the adaptive masking strategy, and serve as an important reconstruction target in the feature reconstruction process. By introducing the semantic feature, the present application can better capture the semantic structure in the scene and improve the model's performance in various downstream tasks such as three-dimensional object detection and semantic segmentation.

[0067] Correspondingly, the spatial feature refers to a low-level feature describing the geometric structure and spatial distribution of the point cloud data. In the embodiments of the present application, the spatial feature can be understood as the three-dimensional coordinates of the voxel, which directly reflects the position information of the voxel in the three-dimensional space. In addition, the spatial feature can also include the density of the points in the voxel, the normal vector and other geometric information. The spatial feature is used to retain the basic geometric structure of the point cloud and provide accurate spatial information for the model. In the self-supervised learning process, the reconstruction of the spatial feature helps the model learn the local geometric features and spatial distribution of the point cloud.

[0068] In the subsequent adaptive masking process, the semantic feature is used to guide the development of the masking strategy, while the spatial feature ensures the spatial consistency of the mask. In the feature reconstruction task, the model simultaneously learns to reconstruct both types of features, thereby obtaining a more comprehensive understanding of the three-dimensional scene. Through the cascade reconstruction guided by the center kernel alignment, the present application can also emphasize the reconstruction of the spatial feature and the semantic feature at different levels, achieving more fine-grained feature learning.

[0069] On the basis of the above-mentioned embodiments, as an optional embodiment, in step 101: the step of obtaining the voxel features of the point cloud data can further include the following steps:

[0070] Step 201: obtaining the spatial features and semantic features of the point cloud data to obtain the point cloud features.

[0071] Specifically, the system first obtains the spatial features of the point cloud data, including the three-dimensional coordinate information of each point, which directly reflects the geometric structure and distribution of the point cloud in the three-dimensional space. At the same time, in order to enhance the model's understanding of the scene content, the present application also introduces the extraction of semantic features. The semantic features can be extracted by a pre-trained MinkUNet model. This model can effectively aggregate the context information of the point cloud to generate a high-dimensional semantic feature vector.

[0072] By combining spatial features and semantic features, a comprehensive point cloud feature representation can be obtained. The above feature representation not only contains the basic geometric information of the point cloud, but also encodes the high-level semantic information such as possible object categories and scene structures. In this way, the system pays attention to the spatial structure and semantic content of the point cloud during the self-supervised learning process, thereby improving the performance of the model in various downstream tasks such as three-dimensional object detection and semantic segmentation.

[0073] Notably, the extraction process of semantic features can be completed offline, thereby helping to speed up the entire self-supervised learning process.

[0074] Step 202: Voxelization and average pooling of point cloud features to obtain voxel features.

[0075] Specifically, the system first performs voxelization processing on the obtained point cloud features. The voxelization process discretizes the continuous three-dimensional space into a regular three-dimensional grid structure, with each grid cell being called a voxel. This makes the originally irregularly distributed point cloud data mapped to a fixed-size three-dimensional grid, greatly simplifying the complexity of subsequent processing.

[0076] After voxelization, the present application uses the method of average pooling to aggregate the point cloud features located in the same voxel. The average pooling operation calculates the average value of the features of all points in each voxel, thereby obtaining the representative features of the voxel.

[0077] For example, the final voxel features can be defined as:

[0078] S all ={s i ∣j=1,K,N v}。

[0079] Step 102, clustering the voxel features based on the semantic features to obtain at least one super class, and determining the mask rate of each super class.

[0080] After obtaining the voxel features of the point cloud data, the present application proposes an adaptive masking strategy based on semantic features. This strategy aims to optimize the masking process by considering the semantic properties of voxel features, thereby improving the effect of self-supervised learning. Specifically, the present application first clusters the voxel based on the semantic features of the voxel, dividing the voxel into multiple super classes. Since the true semantic categories cannot be directly obtained during the self-supervised learning process, using a clustering algorithm to group semantic voxel features is an effective alternative.

[0081] Among them, the super class refers to a set of semantically similar voxel collections obtained by clustering the semantic features of point cloud data. In the embodiments of the present application, the super class can be understood as a group of voxels with similar semantic attributes, which may be distributed at different positions in three-dimensional space but close to each other in the semantic feature space. The super class is used to group the original point cloud data according to its semantic features, so that different strategies can be taken for different semantic regions in the subsequent adaptive mask process.

[0082] Further, after clustering is completed, the present application determines a corresponding mask rate for each super class. The mask rate refers to the proportion of a specific region or feature that is masked in the self-supervised learning process.

[0083] Among them, the mask rate refers to the proportion of a specific region or feature that is masked in the self-supervised learning process. In the embodiments of the present application, it can be understood as the degree of masking the voxel features in each super class, which is a value between 0 and 1, indicating the proportion of the masked voxels in the total voxels in the super class. The mask rate is used to regulate the degree of information shielding in the self-supervised learning process, thereby affecting the learning difficulty of the model and the complexity of the reconstruction task. By setting different mask rates for different super classes, the present application can guide the model to pay more attention to key regions and complex structures in the point cloud data, while avoiding over-learning in simple regions. This adaptive masking strategy helps to improve the understanding ability of the model for complex three-dimensional scenes, enhances the effectiveness and robustness of feature extraction, and provides more high-quality feature representation for subsequent downstream tasks such as object detection and semantic segmentation.

[0084] In order to quantify the inter-class difference, the present application adopts the farthest class sampling algorithm. This algorithm first calculates the distance between the cluster centers of each super class, and then selects the combination of super classes with the largest average inter-class distance. This method can effectively identify the super classes with the most significant semantic difference and assign them a higher basic mask rate.

[0085] Among them, the inter-class difference and the intra-class difference are two important concepts describing the distribution characteristics of data. The inter-class difference refers to the degree of feature difference between different classes or groups, while the intra-class difference reflects the degree of feature variation within the same class or group. In the embodiments of the present application, the inter-class difference can be understood as the degree of difference in semantic features between different super classes, which reflects the degree of separation of each super class in the semantic space; the intra-class difference can be understood as the degree of variation of voxel features within the same super class, which reflects the consistency of the semantics within the super class.

[0086] The inter-class difference and the intra-class difference are closely related to the mask rate, and they jointly determine the final mask rate of each super-class. Specifically, the greater the inter-class difference, the more significant the semantic difference between different super-classes, and in this case, the application tends to assign a higher basic mask rate to the super-class with a greater inter-class difference. This is because the region with a significant semantic difference usually contains more abundant information, and by increasing the mask rate, the model can be prompted to learn more discriminative features. As for the intra-class difference, the application adopts an opposite strategy: the smaller the intra-class difference, i.e., the higher the intra-class consistency, the more similar the semantic features within the super-class, and in this case, the application will retain more original information by reducing the mask rate, because the reconstruction difficulty of these regions is relatively low.

[0087] On the basis of the above-mentioned embodiments, as an optional embodiment, in step 102: based on the semantic feature, the voxel feature is clustered to obtain at least one super-class, and the mask rate of each super-class is determined. This step can further include the following steps:

[0088] S301, based on the semantic feature, the voxel feature is clustered to obtain at least one super-class, and the cluster center of each super-class is determined.

[0089] Specifically, in order to group the voxels with similar semantics, it is necessary to first cluster the voxel features to form super-classes with similar semantics. The application adopts a clustering method based on semantic features, such as the K-means algorithm, to group the voxel features. The reason for choosing the K-means algorithm is that it is simple and efficient, and can effectively divide the data into a predetermined number of clusters without supervision. In this embodiment, the number N of clusters can be set according to the specific application scenario and the complexity of the point cloud data, and a value that can reasonably reflect the semantic diversity of the scene can usually be selected.

[0090] At the beginning of the clustering process, the system first randomly initializes N cluster centers. Then, for each voxel feature, the distance between it and all cluster centers is calculated, and the voxel is assigned to the super-class represented by the cluster center with the closest distance. After completing a round of assignment, the system recalculates the cluster center of each super-class, i.e., the average value of all voxel features in the super-class. This process is iterated until the position of the cluster center no longer changes significantly or the maximum number of iterations is reached.

[0091] For example, for each super-class k i , the cluster center of the super-class is calculated, i.e., the average value of all voxel semantic features in the super-class, and the formula is as follows:

[0092]

[0093] In the formula, represents the super-class ki semantic feature of the jth voxel, representing a super-class k i number of voxels.

[0094] S302, based on the first average distance between the clustering centers corresponding to each super-class, dividing each super-class into a plurality of super-class groups, the super-class groups being used to represent the inter-class differences between each super-class.

[0095] Specifically, after completing the clustering of super-classes and the determination of clustering centers, in order to quantify and utilize the inter-class differences between super-classes, thereby identifying super-class groups with significant semantic differences, a more refined guidance is provided for the subsequent adaptive masking strategy. The present application proposes using the farthest class sampling algorithm to achieve this goal, which can effectively divide the super-classes into groups with different degrees of semantic difference.

[0096] Further, the present application uses the farthest class sampling algorithm to achieve this goal. First, select any n1 super-classes from all super-classes, and calculate the average distance between the clustering centers of these n1 super-classes, which is the first average distance. By calculating the first average distance of all possible n1 super-class combinations, find the combination with the largest average inter-class distance, which is denoted as K1. This combination represents the n1 super-classes with the most significant semantic difference. Next, select n2 super-classes from the remaining super-classes, and repeat the above process to find the combination with the second largest average inter-class distance, denoted as K2. Finally, the remaining super-classes automatically form the third group K3, which has the smallest average inter-class distance.

[0097] The super-class grouping method based on the above first average distance can effectively identify the super-class group with the most significant semantic difference, providing clear guidance for the subsequent masking strategy. The strategy of dividing super-classes into three groups allows for a more flexible and detailed approach in subsequent masking rate allocation, prompting the model to pay more attention to areas with significant semantic differences during self-supervised learning, while also not neglecting areas with high semantic similarity. This method provides an implicit semantic structure guidance for the self-supervised learning process, which helps to improve the model's understanding of complex scenes and the effectiveness of feature extraction.

[0098] For example, first define the set K = {k i ∣i = 1, 2, K, N c}, where N c is the total number of super-classes, and then select any n1 super-classes from the set K and calculate the distance between the clustering centers. The average inter-class distance set D inter is calculated as follows:

[0099]

[0100] where, The distance between two super-class cluster centers, which can be measured using cosine similarity or other distance metrics. Select a combination of n1 super-classes from the set K.

[0101] Next, find the n1 super-class partition with the largest average inter-class distance, denoted as K1. The difference between these super-classes is the largest, so a higher base mask rate can be set for them to increase the difficulty of reconstruction and the attention of the model.

[0102] Subsequently, we take the difference set between K and K1 as the input for the next loop. From the remaining super-classes, select n2 super-classes again, repeat the above steps, and obtain the second farthest average inter-class distance partition K2. The remaining super-classes form K3, which has the smallest inter-class difference. In this way, the super-classes can be divided into three groups: K1, K2, and K3, which have the largest, second largest, and smallest inter-class differences, respectively. The larger the inter-class difference, the more difficult the reconstruction, so a higher base mask rate can be set.

[0103] S303, obtaining a second average distance between the voxel features inside each super-class and the cluster center, the second average distance being used to represent the intra-class difference inside the super-class.

[0104] Specifically, after completing the division of super-classes and the quantification of inter-class differences, in order to further understand the internal structure of each super-class and provide more comprehensive information for the subsequent adaptive mask strategy, the present application further proposes a method to evaluate the intra-class difference inside each super-class. By calculating the average distance between the voxel features inside each super-class and its cluster center, i.e., the second average distance, the intra-class difference is quantified. The second average distance refers to a quantitative indicator inside a single super-class for measuring the tightness of the distribution of voxel features within the super-class. In the embodiments of the present application, the second average distance can be understood as the average value of the distances between all voxel features within the super-class and its cluster center.

[0105] For example, define the distance between a voxel and a cluster center, calculate the second average distance D intra of the voxel features that are too far away from the super-class center.

[0106]

[0107] wherein, represents the distance between the voxel feature and the cluster center , λ is the distance threshold, only the voxel features whose distance from the cluster center exceeds λ are taken into account, and 1{·} represents the indicator function, indicating that only the voxel features with a distance greater than the threshold are considered.

[0108] By calculating the second average distance, the application can evaluate the internal consistency of each super class. A smaller second average distance indicates that the voxel feature distribution within the super class is relatively tight, and the semantic consistency is high; while a larger second average distance indicates that there is a larger semantic change or heterogeneity within the super class. The quantification of such intra-class differences provides an important basis for subsequent adaptive masking strategies. For example, for a super class with high internal consistency, a lower masking rate can be used because the reconstruction difficulty of these regions is relatively low; while for a super class with large internal differences, a higher masking rate may be needed to enable the model to learn more complex feature structures.

[0109] S304, determine the masking rate of each super class based on the first average distance and the second average distance of each super class.

[0110] Specifically, after completing the quantification of the inter-class difference and the intra-class difference of the super class, in order to formulate an adaptive masking strategy for each super class that takes into account both its position in the overall semantic space and its internal structure. The application proposes a method that considers both differences to determine the masking rate of each super class. Specifically, the process can also include the following steps:

[0111] S401, determine the basic masking rate based on the first average distance corresponding to each super class group.

[0112] Specifically, for the three groups K1, K2, K3, corresponding to the super class set with semantic difference degree from large to small. Set different levels of basic masking rate for the three groups The super class group with greater semantic difference should have a higher basic masking rate.

[0113] S402, determine the intra-class consistency coefficient based on the second average distance of each super class, and the intra-class consistency coefficient is inversely proportional to the second average coefficient.

[0114] Specifically, after determining the basic masking rate, considering the structural features within the super class, in order to enable the masking strategy to better adapt to the complexity and diversity of point cloud data. The application further introduces an intra-class consistency coefficient to more finely adjust the masking rate of each super class.

[0115] The intra-class consistency coefficient is a crucial numerical indicator used in this invention to quantify the degree of semantic consistency within a superclass. In this embodiment, it can be understood as a coefficient between 0 and 1, calculated based on the second average distance of the superclass. The closer the value is to 1, the higher the semantic consistency within the superclass. By multiplying it by the base masking rate, the intra-class consistency coefficient can finely adjust the masking strength according to the semantic structure within the superclass, thereby allowing the masking strategy to better adapt to local semantic changes in the point cloud data. For superclasses with high internal consistency, a higher coefficient reduces the masking rate, avoiding over-masking of these relatively simple semantic regions; while for superclasses with low consistency, a lower coefficient increases the masking rate, prompting the model to focus more on these complex regions, increasing the learning difficulty and the model's robustness.

[0116] For example, the formula for calculating the intra-class consistency coefficient is as follows:

[0117]

[0118] In the formula, This represents the maximum intra-class distance among all superclasses, used for normalization.

[0119] S403. The product of the base mask rate and the intra-class consistency coefficient corresponding to each superclass is used as the mask rate.

[0120] Specifically, after determining the base mask rate and intra-class consistency coefficient, this invention multiplies the base mask rate and intra-class consistency coefficient corresponding to each superclass to obtain the final mask rate of each superclass.

[0121] For example, the calculation formula is expressed as follows:

[0122]

[0123] S103. The point cloud feature extraction model is trained based on voxel features and masking rate to obtain a trained point cloud feature extraction model, wherein the masking rate is used to assist the point cloud feature extraction model in focusing on key regions in the point cloud data.

[0124] Specifically, such as Figure 2 As shown, Figure 2 This is a flowchart illustrating the feature extraction method for point cloud data provided in an embodiment of the present invention. The point cloud feature extraction model refers to a deep learning model used to extract meaningful feature representations from raw point cloud data. In this embodiment, it can be understood as an encoder-decoder structure based on a sparse voxel transform, used to learn the spatial and semantic features of point clouds. The model includes a voxel feature extraction module, an adaptive masking module, a cascaded reconstruction module, and a progressive learning module, which together constitute a complete self-supervised learning system.

[0125] The voxel feature extraction module is the foundation of the entire model, responsible for extracting initial features from raw point cloud data. This module first obtains the spatial features of the point cloud, including three-dimensional coordinate information, which directly reflects the geometric structure of the point cloud. At the same time, in order to enhance the model's understanding of the scene content, the module also introduces the extraction of semantic features, which can be done through a pre-trained MinkUNet model. MinkUNet can effectively aggregate the context information of the point cloud, generating high-dimensional semantic feature vectors. Subsequently, the module voxelizes these features, discretizing the continuous three-dimensional space into a regular three-dimensional grid structure, with each grid cell referred to as a voxel. Through average pooling operations, the module aggregates the point cloud features located within the same voxel, ultimately obtaining voxel features that integrate spatial and semantic information.

[0126] The adaptive masking module, based on voxel features, implements an intelligent masking strategy. This module first clusters voxels based on semantic features, forming several superclasses. A superclass can be understood as a group of voxels with similar semantic attributes. Subsequently, the module determines an appropriate masking rate for each superclass by considering the inter-class differences between superclasses and the intra-class differences within superclasses. Specifically, the module uses the farthest class sampling algorithm to quantify inter-class differences, calculating the distances between the cluster centers of each superclass and selecting the combination of superclasses with the largest average inter-class distance. For intra-class differences, the module introduces an intra-class consistency coefficient, reflecting the degree of change in voxel features within the same superclass. By combining inter-class differences and intra-class differences, the module generates an adaptive masking rate for each superclass, a strategy that can better guide the model to focus on important areas and improve learning efficiency.

[0127] The cascaded reconstruction module is responsible for performing the reconstruction task of spatial features and semantic features. This module uses a center kernel alignment guided strategy to weight the reconstruction of spatial features and semantic features at different network layers. Specifically, the cascaded reconstruction module calculates a center kernel alignment factor for each layer, which reflects the relevance of the features at that layer to downstream tasks. Based on this factor, the module dynamically adjusts the weights of spatial feature and semantic feature reconstruction. In shallow network layers, the module focuses more on the reconstruction of spatial features, as these layers mainly capture local geometric information; while in deep network layers, it focuses more on the reconstruction of semantic features to learn more advanced semantic representations. This cascaded reconstruction strategy enables the model to better learn spatial and semantic information at different levels, thereby improving the quality of feature representation.

[0128] The progressive learning module implements a differential gating progressive learning strategy to control the learning process of spatial feature reconstruction and semantic feature reconstruction. The module dynamically adjusts the starting time of semantic feature reconstruction based on the completion degree of spatial feature reconstruction. Specifically, the progressive learning module evaluates the completion degree of spatial feature reconstruction by calculating the mutual information metric center kernel alignment factor between two adjacent training rounds. When the completion degree reaches a preset threshold, the progressive learning module activates the semantic feature reconstruction task. This progressive learning method enables the model to first master the basic spatial structure and then gradually learn complex semantic information, thereby improving the stability and effectiveness of training.

[0129] It should be noted that each module in the point cloud feature extraction model can be executed separately or as a whole to implement the point cloud data feature extraction method. When executed as a whole, these modules closely cooperate to form an end-to-end self-supervised learning process. The voxel feature extraction module first processes the input raw point cloud data to generate voxel features containing spatial and semantic information. Then, the adaptive mask module generates a mask based on these voxel features to create conditions for subsequent reconstruction tasks. Next, the cascaded reconstruction module uses the generated mask to reconstruct the masked voxel features while considering the characteristics of different network layers. Finally, the progressive learning module controls the entire learning process to ensure that the model learns spatial and semantic features in the appropriate order and rhythm.

[0130] Through the above overall cooperation, the point cloud feature extraction model can effectively learn the comprehensive feature representation of point cloud data and provide rich and effective features for various downstream tasks. At the same time, the model also improves the overall understanding ability of 3D scenes and enhances the processing ability of partially occluded or noisy data. This self-supervised learning method not only reduces the dependence on a large amount of labeled data, but also provides a high-quality pre-trained model for point cloud related tasks.

[0131] Specifically, after obtaining the voxel features and determining the mask rate of each supercategory, the present application further proposes a self-supervised learning method based on voxel features and mask rates to optimize the performance of the point cloud feature extraction model. This method aims to improve the model's understanding of three-dimensional scenes and the quality of feature representation by combining spatial and semantic information and an adaptive mask strategy.

[0132] Specifically, the system first masks the voxel features according to the determined mask rate to create a hybrid dataset containing masked and unmasked voxels. This masking operation simulates the partial information loss that may exist in real-world scenarios, helping to improve the model's robustness. Subsequently, these processed data are input into the point cloud feature extraction model for training.

[0133] During training, the model is required to simultaneously reconstruct spatial and semantic features, which reflect the geometric structure and high-level semantic information of the point cloud, respectively. To better utilize feature information at different levels, this invention employs a cascaded reconstruction strategy guided by center kernel alignment. This strategy dynamically adjusts the weights of spatial and semantic feature reconstruction based on the relevance of different network layers to the downstream task. Specifically, in shallow networks, the focus is on spatial feature reconstruction, as these layers primarily capture local geometric information; while in deep networks, more attention is paid to semantic feature reconstruction to learn higher-level semantic representations.

[0134] Furthermore, to ensure the stability and effectiveness of the training process, this invention introduces a differentially gated progressive learning strategy. This strategy monitors the completion rate of spatial feature reconstruction and dynamically adjusts the timing of initiating the semantic feature reconstruction task, avoiding training instability issues that may arise from introducing complex tasks too early. In this way, the model can progressively learn from basic spatial structures to complex semantic concepts, achieving a smoother and more effective feature learning process. Finally, the model parameters are optimized by minimizing the reconstruction loss, which comprises two parts: spatial feature reconstruction loss and semantic feature reconstruction loss.

[0135] The comprehensive self-supervised learning method described above not only improves the model's understanding of point cloud data but also provides richer and more effective feature representations for various downstream tasks in the absence of a large amount of labeled data. Through this approach, the self-supervised learning method of this invention can significantly enhance the performance and generalization ability of point cloud feature extraction models, laying a solid foundation for complex 3D scene understanding tasks.

[0136] S104. Use the trained point cloud feature extraction model to extract features from the point cloud data of the target area.

[0137] Specifically, the system first acquires raw point cloud data of the target area, which may come from devices such as LiDAR scanning or depth cameras. Subsequently, the system preprocesses this raw point cloud data, including noise reduction and downsampling, to improve the efficiency of subsequent processing.

[0138] Next, the system inputs the preprocessed point cloud data into the trained point cloud feature extraction model. This model processes the input point cloud data based on the previously learned adaptive masking strategy and cascaded reconstruction method. In this process, the model first converts the point cloud data into voxel features, which include spatial and semantic features.

[0139] Then, the encoder part of the model performs multi-level feature extraction and transformation on these voxel features to generate a latent representation. This latent representation contains high-level features of the point cloud data, including both geometric and semantic information.

[0140] Finally, the system extracts the final feature representation from this latent representation as the feature extraction result of the target region point cloud data.

[0141] On the basis of the above-mentioned embodiments, the training process of the point cloud feature extraction model can further include the following steps:

[0142] S501, mask processing the voxel features based on the mask rate.

[0143] Specifically, for each super-class, the system first calculates the number of voxels that need to be masked according to its corresponding mask rate. Then, a corresponding number of voxels are randomly selected within the super-class for the masking operation. The masking operation includes replacing the features of the selected voxels with a specific mask token, usually a predefined vector or a zero vector. This approach ensures that the model cannot directly access the masked information during training, forcing the model to learn to infer and reconstruct the missing information from the surrounding unmasked voxels.

[0144] Notably, the masking strategy of the present application not only considers the semantic differences between super-classes, but also reflects the feature distribution within the super-class through the intra-class consistency coefficient, making the masking process more fine-grained and targeted.

[0145] S502, reconstruct the semantic features and the spatial features by the encoder-decoder on the voxel features after the mask processing.

[0146] After completing the mask processing of the voxel features, the present application uses an encoder-decoder structure to reconstruct the semantic features and spatial features of the voxel features after the mask processing. The model is trained through the reconstruction task to be able to infer and recover the complete scene representation from the partially visible information.

[0147] Further, the present application uses an encoder-decoder structure based on sparse voxel transformer (DSVT) to perform the above-mentioned tasks. First, the voxel features after the mask processing are input into the encoder. The encoder is composed of multiple DSVT layers, each of which can capture spatial and semantic information at different scales. The voxel features that are not masked are directly input into the encoder, while the voxel features that are masked are replaced with a specific mask token. The encoder generates a latent representation through multiple levels of feature extraction and transformation. Then, this latent representation is input into the decoder. The task of the decoder is to reconstruct the complete voxel features, including the masked part, based on this latent representation.

[0148] In an implementable embodiment, the embodiment of the present application adds deep supervision in the point cloud self-supervised method, since the shallower layers of the encoder receive weaker information feedback from the supervision signal. Only half of the decoder layers are used for the reconstruction of non-final layers to avoid the expensive computational burden. In addition, since the point cloud encoder follows hierarchical feature learning, the shallow layers learn local geometric features and spatial distribution information, while the deep layers capture high-level semantic information. Not all layers are suitable for performing the same reconstruction. To further improve the effect and efficiency of the reconstruction, the present application adopts a center kernel alignment guided cascade reconstruction strategy, which weights the spatial feature and semantic feature reconstruction loss of each layer.

[0149] Specifically, the process can further include the following steps:

[0150] S601, determining a first weight factor corresponding to semantic feature reconstruction and a second weight factor corresponding to spatial feature reconstruction, wherein the first weight factor is greater than the second weight factor.

[0151] Specifically, the system calculates the center kernel alignment factor of each layer, which reflects the relevance of the features of the layer to the downstream tasks. Then, the average value of the center kernel alignment factors of the three downstream tasks is defined as the proportion of spatial information α i (i = 1, 2, …, L), where L is the total number of layers of the network. Based on this proportion, the second weight factor of the spatial feature reconstruction is set to The first weight factor of the semantic feature reconstruction is set to

[0152] It is worth noting that since the deep network is more inclined to capture high-level semantic information, in most cases, the first weight factor will be greater than the second weight factor. The reasons for this setting are multifaceted: first, it takes into account the hierarchical feature learning characteristics of the point cloud encoder, that is, the shallow layers mainly learn local geometric features and spatial distribution information, while the deep layers capture more high-level semantic information. By setting different weights for different layers, the feature learning ability of each layer can be better utilized.

[0153] Secondly, this strategy can alleviate the gradient decay problem in deep supervision. Since the shallower layers of the encoder receive weaker information feedback from the supervision signal, by increasing the weight of the deep semantic feature reconstruction, the learning of the deep features can be enhanced. Furthermore, this method can balance the learning of spatial information and semantic information, avoiding the model from paying too much attention to one aspect and ignoring the other.

[0154] Finally, by dynamically adjusting the weights, the model can more flexibly adapt to the requirements of different types of downstream tasks. The effect of this center core alignment guided cascade reconstruction strategy is significant. It can improve the feature learning ability of the model at different levels, so that the learned features contain rich geometric information and have high-level semantic understanding. On the basis of improving the effect of self-supervised learning, it can also provide more comprehensive and effective feature representation for various downstream tasks.

[0155] S602, based on the first weight factor and the second weight factor, the voxel feature processed by the mask is reconstructed by the encoder.

[0156] In another possible implementation, the reconstruction of semantic features fundamentally depends on the accurate reconstruction of spatial features, such as local geometric features. Error amplification caused by parallel learning will lead to instability in semantic feature reconstruction. The present application proposes a multi-task switching strategy, namely differential gating progressive learning, which adjusts the start time of semantic feature reconstruction according to the completion of spatial feature reconstruction. The activation difference between two adjacent rounds can reflect the degree of training completion

[0157] Specifically, the process can further include the following steps:

[0158] S701, activate the codec to reconstruct the spatial features of the voxel feature processed by the mask.

[0159] Specifically, spatial features are the basis of point cloud data and reflect the geometric structure of objects and scenes. Accurate spatial feature reconstruction lays a solid foundation for subsequent semantic feature reconstruction. The above method conforms to the human cognitive process, that is, first understand the shape and position of the object, and then give it semantic meaning. Secondly, spatial features are usually more stable and consistent than semantic features, and are not easily affected by annotation bias or semantic ambiguity. Starting with a relatively simple and stable task helps the model establish a basic understanding of point cloud data and avoids the training instability that may be caused by directly processing complex semantic information.

[0160] Furthermore, by first focusing on spatial feature reconstruction, the model can establish an understanding of point cloud data at a lower level, which provides a good starting point for subsequent high-level semantic feature learning. Finally, this progressive learning strategy can effectively prevent the error amplification problem that may be caused by parallel learning, especially in the semantic feature reconstruction task. By first ensuring the accurate reconstruction of spatial features, the uncertainty and instability in the semantic feature reconstruction process can be greatly reduced.

[0161] S702, activate the codec to perform semantic feature reconstruction on the voxel features processed by the mask based on a start time factor, wherein the start time factor is determined according to the degree of completion of spatial feature reconstruction when the codec is activated to perform semantic feature reconstruction in the point cloud feature extraction model training process.

[0162] Specifically, based on the completion of spatial feature reconstruction, the application proposes a semantic feature reconstruction activation mechanism based on a start time factor, which is a core component of the differential gating progressive learning strategy. The design of this mechanism aims to ensure that the semantic feature reconstruction task starts at the most appropriate time, thereby improving the stability and effectiveness of training.

[0163] Among them, the start time factor refers to a dynamic adjustment parameter used to determine when to start the semantic feature reconstruction task in the point cloud self-supervised learning process. In the embodiment of the application, the start time factor can be understood as a value between 0 and 1, reflecting the degree of completion of spatial feature reconstruction, and accordingly controlling the starting time of the semantic feature reconstruction task.

[0164] When the start time factor is 0, only spatial feature reconstruction is performed; when it approaches 1, the spatial feature has been well reconstructed and the semantic feature reconstruction task can be started. The start time factor is used to implement the differential gating progressive learning strategy, which dynamically adjusts the start time of semantic feature reconstruction to ensure that the model learns more complex semantic information step by step on the basis of mastering the basic spatial structure, thereby improving the stability and effectiveness of training.

[0165] Specifically, the start time factor is dynamically determined according to the degree of completion of spatial feature reconstruction in the point cloud feature extraction model training process. When calculating, first calculate the activation difference through the mutual information degree center kernel alignment factor between two adjacent training rounds. Considering that the shallow network mainly learns local geometric features, the application calculates the average center kernel alignment factor of the first four layers as the degree of completion of spatial feature reconstruction at the current round t. Exemplarily, the calculation formula of the start time factor is:

[0166]

[0167] In the formula, is the degree of completion of spatial feature reconstruction at the tth round, δ is the gating factor, indicating the start threshold of semantic feature reconstruction, and γ is the slope factor, controlling the growth rate of β t .

[0168] Based on the start time factor, the application activates the codec to perform semantic feature reconstruction on the voxel features processed by the mask at the appropriate time. Specifically, when β tWhen the preset threshold is reached, the model begins to perform the semantic feature reconstruction task. In this process, the encoder continues to process the input voxel features, but now focuses more on extracting high-level semantic information. The decoder is trained to reconstruct these semantic features, including the masked parts.

[0169] The semantic feature reconstruction activation mechanism based on the start time factor ensures that the semantic feature reconstruction task begins on the basis of a good understanding of spatial features, which greatly reduces the instability of training. Spatial features provide the necessary context and structural information for semantic features, making the learning of semantic features more accurate and effective. Secondly, this progressive learning method simulates the learning order from basic shapes to complex concepts in the human cognitive process, which helps the model to establish a more reasonable and hierarchical feature representation. Thirdly, by dynamically adjusting the start time of semantic feature reconstruction, the model can better adapt to point cloud data of different complexity. For simple scenes, it can enter the semantic learning stage faster, while for complex scenes, it gives more time to master the basic spatial structure. Finally, this method effectively prevents the error amplification problem that may be caused by parallel learning, especially when dealing with complex semantic information.

[0170] S503, obtaining a loss value of the reconstructed semantic feature and the reconstructed spatial feature.

[0171] Specifically, the spatial feature reconstruction loss can be represented as:

[0172]

[0173] wherein, is the spatial feature of the masked voxel predicted by the model, is the corresponding real spatial feature. For semantic features, a smooth L1 loss function is also used to calculate the reconstruction error:

[0174]

[0175] wherein, is the semantic feature of the masked voxel predicted by the model, is the corresponding real semantic feature.

[0176] S504, adjusting the parameters in the point cloud feature extraction model based on the loss value.

[0177] Specifically, first, the spatial feature reconstruction loss and the semantic feature reconstruction loss are combined to form a comprehensive loss function. Then, the gradient of the loss function with respect to the parameters of each layer of the model is calculated using the backpropagation algorithm. The Adam optimizer is used to update the model parameters, which can adaptively adjust the learning rate and help handle the sparsity and non-uniformity of point cloud data. By minimizing the reconstruction error, the model can gradually learn more accurate and meaningful feature representations.

[0178] The above embodiment introduces the self-supervised learning process of the point cloud feature extraction model. Based on the above embodiment, the above process will be illustrated in the context of point cloud self-supervised learning in an autonomous driving scenario.

[0179] As shown in Table 1, Table 1 is a self-supervised method parameter setting table provided by an embodiment of the present application. The present application uses a laser radar as a sensor to obtain input point cloud data. In the construction of the point cloud feature extraction model, the present application selects a fine-tuned encoder based on the sparse voxel transformer (DSVT). Specifically, the encoder consists of four DSVTs, each containing eight attention layers. The decoder part consists of two DSVTs in the last layer, and another DSVT is used for decoding in the non-last layer. The feature dimension of the model is set to 192, and the voxel size is set to [0.32, 0.32, 0.1875].

[0180] In the inter-class difference guided adaptive mask module, the present application sets the expected number of super-class partitions (n1, n2, n3) to (3, 3, 2), and the corresponding base mask rate is set to (0.9, 0.45, 0). It is worth noting that this setting means that some super-classes will not be masked. For intra-class difference guided adaptive masking, the present application sets the distance threshold λ to 0.6.

[0181] In the center kernel alignment guided cascade reconstruction module, the proportion of spatial information is set to [0.97, 0.88, 0.78, 0.64, 0.55, 0.44, 0.36, 0.31]. This setting reflects the different emphasis of different network layers on spatial information and semantic information.

[0182] For the differential gating progressive learning strategy, the present application sets the slope factor γ to 10 and the gating factor δ to 0.2. The selection of these parameters aims to achieve a smooth transition between spatial feature reconstruction and semantic feature reconstruction.

[0183] The present application selects a point cloud self-supervised learning on the Waymo autonomous driving scene dataset, and evaluates the performance of three-dimensional target detection on the same dataset. The evaluation indicators include standard three-dimensional mean average precision (mAP) and mAP weighted by orientation precision (mAPH). Specifically, for the vehicle category, an intersection over union (IoU) threshold of 0.7 is used; for other categories, an IoU threshold of 0.5 is used. The performance evaluation is divided into two difficulty levels: level 1 is applicable to target boxes containing more than five point clouds, and level 2 is applicable to target boxes containing at least one point cloud. The present application mainly uses L2mAP and L2mAPH as key evaluation indicators.

[0184] Table 1 Self-supervised method parameter setting table

[0185]

[0186] In the inter-class difference guided adaptive mask, in order to determine the number of superclasses of semantic features, the present application embodiment first determines the semantic categories that can represent the autonomous driving scene. Table 2 lists the correspondence between the eight semantic categories and the categories on the downstream SemanticKITTI scene, nuScenes Lidarseg scene and Waymo scene. Based on this, the superclass number N is set to 8.

[0187] Table 1 Correspondence table between semantic categories and scene categories

[0188]

[0189] Please refer to Figure 3 , Figure 3 A comparison diagram of true value semantic features and encoder output heat maps provided by the present application embodiment is shown in FIG. X(a) as the true value semantic features, FIG. X(b) as the heat map output by the encoder when reconstructing using semantic features, and FIG. X(c) as the heat map output by the encoder when not reconstructing using semantic features. After introducing semantic voxel features as a proxy task, the latent features extracted by the encoder are more aligned with the semantic features and have higher mutual information, which is beneficial for downstream tasks that rely on semantic information.

[0190] Please refer to Table 3, which is a comparison experiment table of reconstruction targets. When only using semantic features, the performance is not as good as only using spatial features, which indicates that it is necessary to reconstruct spatial features. Second, supplementing semantic voxel features as a proxy task, compared with various spatial features, L2mAP is increased by 0.69% and 0.48%, respectively.

[0191] Table 2 Comparison experiment table of reconstruction targets

[0192]

[0193] Please refer toFigure 4 Figure 4 A semantic category and mask rate distribution diagram in a scenario provided by an embodiment of the present application. As shown in Figure 4 (a) and Figure 4 (b), compared with random masking, the inter-class and intra-class difference guided adaptive masking assigns different regions with opposite trends of mask rates according to the difficulty of reconstruction. In addition, it can be seen that regions such as vehicles and pedestrians that are highly focused in downstream tasks have higher mask rates, which force these semantic categories to undergo complex reconstruction. In addition, regions such as roads and buildings have lower mask rates, which reduce the attention in the reconstruction process. The inter-class and intra-class difference guided adaptive masking of the embodiment of the present application promotes the mutual information between latent features and semantic objects.

[0194] Please refer to Table 4, which is a comparison experiment table of different mask sampling strategies. The method of the embodiment of the present application achieves an improvement of 0.51% in L2mAP. This shows that smoother information flow between unmasked and masked features will enhance the mutual information between latent features and semantic features.

[0195] Table 3 Comparison experiment table of different mask sampling strategies

[0196]

[0197] Please refer to Table 5, which is a comparison experiment table of different multi-task weighting strategies. On the one hand, due to higher mutual information between each layer of latent features and input point clouds, deep supervision can improve model performance. On the other hand, compared with fixed weighting methods, the center kernel alignment guided cascaded reconstruction proposed by the present application follows hierarchical feature learning, which can improve L2mAP by 0.34%.

[0198] Table 4 Comparison experiment table of different multi-task weighting strategies

[0199]

[0200] Please refer to Table 6, which is a comparison experiment table of different multi-task switching strategies. First of all, all switching strategies bring improvements compared with continuously enabling semantic feature reconstruction. This shows that following progressive learning is crucial in multi-agent task learning. In addition, the derivative-based method uses the gradient of the pre-task loss to determine whether to start the semantic task. The achievement-based method defines the pre-task accuracy rate and the ratio of a single task as achievement, and triggers advanced tasks based on it. The differential gated progressive learning proposed by the present application deeply studies the mutual information of features, rather than relying on objective indicators such as gradients and accuracy. The more accurate switching strategy of the present application improves L2mAP by 0.21%.

[0201] ​Table 6 Comparative Experiment Table of Different Multitasking Switching Strategies

[0202]

[0203] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5 As shown, the electronic device may include a processor 510, a communications interface 520, a memory 530, and a communication bus 540. The processor 510, communications interface 520, and memory 530 communicate with each other via the communication bus 540. The processor 510 can call logical instructions from the memory 530 to execute a feature extraction method for point cloud data.

[0204] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0205] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the feature extraction method for point cloud data provided by the above methods.

[0206] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a feature extraction method for point cloud data provided by the methods described above.

[0207] The device embodiments described above are merely illustrative, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place, or can be distributed to multiple network units. Part or all of the modules can be selected to achieve the purposes of the embodiments according to actual needs. Those skilled in the art can understand and implement without creative labor.

[0208] Through the description of the above embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software and the necessary general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in each embodiment or some parts of the embodiments.

[0209] Finally, it should be noted that: the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A feature extraction method of point cloud data, characterized by, The method comprises the following steps: acquiring voxel features of point cloud data, wherein the voxel features include semantic features and spatial features of the point cloud; clustering the voxel features based on the semantic features to obtain at least one super-class, and determining a mask rate of each super-class; training a point cloud feature extraction model according to the voxel features and the mask rate, to obtain a trained point cloud feature extraction model, wherein the mask rate is used to represent the inter-class difference and the intra-class difference between the super-classes; extracting features of point cloud data of a target region through the trained point cloud feature extraction model; the self-supervised learning of the point cloud feature extraction model according to the voxel features and the mask rate comprises masking the voxel features based on the mask rate; reconstructing semantic features and spatial features of the masked voxel features through an encoder-decoder, to obtain reconstructed semantic features and reconstructed spatial features; acquiring a loss value of the reconstructed semantic features and the reconstructed spatial features; adjusting parameters in the point cloud feature extraction model based on the loss value; the reconstructing semantic features and spatial features of the masked voxel features through the encoder-decoder comprises: activating the encoder-decoder to reconstruct spatial features of the masked voxel features; activating the encoder-decoder to reconstruct semantic features of the masked voxel features based on a start time factor, wherein the start time factor is determined according to the degree of completion of the spatial feature reconstruction when the encoder-decoder is activated to reconstruct semantic features during the training process of the point cloud feature extraction model; the calculation formula of the start time factor is: In the formula, is the spatial feature reconstruction completion degree of the tth round, δ is a gating factor, indicating the starting threshold of semantic feature reconstruction, and γ is a slope factor, controlling the growth rate of β t .

2. The feature extraction method of point cloud data according to claim 1, characterized in that, the acquiring voxel features of point cloud data comprises: acquiring spatial features and semantic features of point cloud data, to obtain point cloud features; performing voxelization and average pooling on the point cloud features, to obtain voxel features.

3. The feature extraction method of point cloud data according to claim 1, characterized in that, the clustering the voxel features based on the semantic features to obtain at least one super-class, and determining a mask rate of each super-class comprises: clustering the voxel features based on the semantic features to obtain at least one super-class, and determining a cluster center of each super-class; dividing each super-class into a plurality of super-class groups based on the first average distance between the cluster centers corresponding to each super-class, wherein the super-class groups are used to represent the inter-class difference between the super-classes; acquiring a second average distance between the voxel features inside each super-class and the cluster center, wherein the second average distance is used to represent the intra-class difference inside the super-class; determining a mask rate of each super-class based on the first average distance and the second average distance of each super-class.

4. The feature extraction method of point cloud data according to claim 3, characterized in that, the determining a mask rate of each super-class based on the first average distance and the second average distance of each super-class comprises: determining a basic mask rate based on the first average distance corresponding to each super-class group; determining an intra-class consistency coefficient based on the second average distance of each super-class, wherein the intra-class consistency coefficient is inversely proportional to the second average distance; multiplying the basic mask rate and the intra-class consistency coefficient corresponding to each super-class to obtain the mask rate.

5. The feature extraction method of point cloud data according to claim 1, characterized in that, the reconstructing semantic features and spatial features of the masked voxel features through the encoder-decoder comprises: determine a first weight factor corresponding to the semantic feature reconstruction, and a second weight factor corresponding to the spatial feature reconstruction, wherein the first weight factor is greater than the second weight factor; perform semantic feature reconstruction and spatial feature reconstruction on the voxel feature processed by the mask based on the first weight factor and the second weight factor through an encoder.

6. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the feature extraction method of the point cloud data according to any one of claims 1-5 when executing the computer program.

7. A non-transitory computer-readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the feature extraction method of the point cloud data according to any one of claims 1-5.

8. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the feature extraction method of the point cloud data according to any one of claims 1-5. The computer program is executed by the processor to implement the feature extraction method of the point cloud data according to any one of claims 1-5.

Images