Point cloud segmentation network lightweight method, device and equipment based on knowledge distillation

Abstract

The application discloses a kind of point cloud segmentation network lightening method based on knowledge distillation, by sending point cloud and label information into teacher model, student model, to respectively obtain first feature map and the point feature importance score of first feature map, second feature map, again, first feature map, second feature map are classified and sampled based on importance score, label information, further processing obtains global structured information, local structured information, then the cross-entropy loss of label information and student model prediction result is combined, and the total loss of semantic segmentation is obtained;The application proposes the class balance sampling strategy of classification sampling, effectively alleviate the problem caused by the sparsity of point cloud and class imbalance, and global structured information extraction and local structured information extraction are designed for point cloud semantic segmentation, and the knowledge learned by teacher model is effectively transmitted.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of point cloud segmentation, in particular to a point cloud segmentation network lightweight method, device and equipment based on knowledge distillation. BACKGROUND

[0002] With the improvement of the performance of three-dimensional laser sensors, RGB-D cameras (such as Kinect, RealSense and Apple depth camera) and other three-dimensional acquisition devices, three-dimensional data with rich geometric shape texture information can be obtained. Such sensors can be applied to scenes with different measurement accuracies, and with the improvement of data accuracy, it is possible to detect the defects of small parts of consumer electronic products, and the development of deep learning provides an effective technical means for the field of industrial defect detection, so real-time processing of large-scale three-dimensional point clouds based on deep learning has become a research hotspot. Since PointNet solved the rotation invariance and permutation invariance of point clouds, various new point cloud deep learning architectures have been proposed, and the performance of deep learning models in point cloud segmentation tasks has been continuously improved. However, in order to obtain better performance, the current point cloud semantic segmentation model often uses a deeper and wider model, which means greater computational load and storage capacity. Complex models have better performance, but the high storage space and computational resource consumption make it difficult to meet the requirements of high real-time tasks (autonomous driving) or lightweight carriers (mobile devices and edge devices).

[0003] How to compress and accelerate the point cloud deep model, and how to use the deep learning model to directly and efficiently process large-scale point clouds is the current research difficulty and hotspot. Knowledge distillation, as a special model compression method, adopts the training method of transferring knowledge from the teacher model to the student model, which can improve the accuracy of the lightweight student model with lower inference delay, weaker computing power and less storage capacity to close to the level of the teacher model, meeting the deployment and operation requirements of resource-constrained platforms.

[0004] Existing knowledge distillation strategies are mostly applied to image-based deep learning models, and the methods they take mainly include logits distillation, feature distillation, attention distillation and structured information distillation. Among them, structured information distillation is proposed for image semantic segmentation models, which is used to extract the structured relationship between pixels. However, due to the sparsity and class imbalance of three-dimensional point clouds, the above methods have little effect when directly applied to point cloud processing tasks, so some scholars have studied the knowledge distillation strategy for point cloud deep learning models.

[0005] Solution one: the patent with the title of "a method for knowledge distillation of point cloud segmentation network" proposes a method for knowledge distillation of point cloud segmentation network, which distills knowledge from points and voxels, which can effectively alleviate the sparsity problem of point cloud. At the same time, in order to alleviate the problem caused by the unordered density and unevenness of point cloud, a sampling strategy based on learning difficulty and distance proximity is designed. In addition, the method effectively transmits the spatial information of point cloud by constructing the affinity relationship between points in voxel and voxels, and improves the distillation performance.

[0006] Solution two: the patent with the title of "a point cloud scene segmentation method based on knowledge distillation and semantic fusion" designs a double flow graph neural network, the teacher model learns the local dense point cloud information, and the student model learns the sampled sparse global point cloud information. Then, through the construction of distillation module by adversarial learning, the detailed information is transmitted to the feature of sparse point cloud. Finally, through the dynamic graph context semantic information fusion module, the high-level semantic relationship and spatial position relationship of dense local point cloud feature and sparse global point cloud feature are learned at the same time, and the global context semantic information is fused into the local area, so that more accurate segmentation result is obtained. Due to the need of double flow execution, this method will introduce additional parameter quantity and calculation amount.

[0007] Solution three: the patent with the title of "target recognition method and target recognition model training method and device" designs three distillation modules, which are respectively a class conduction distillation module based on the 3D feature map output by the student model and the teacher model, a feature conduction distillation module based on the 2D feature map output by the student model and the teacher model, and a prediction conduction distillation module based on the recognition result output by the student model and the teacher model. The method makes the student model trained to achieve the target detection and recognition effect close to the teacher model.

[0008] Among the above three technical solutions, solution one considers the sparsity and uneven density of point cloud, and effectively improves the distillation effect by point-by-point and voxel-by-voxel distillation method, but this method is not suitable for point-based point cloud deep neural network. Solution two only considers the sparsity problem of point cloud, and does not consider the structured information, and needs double flow execution, which will introduce additional parameter quantity and calculation amount. Solution three transmits the learned knowledge of the teacher model from multiple angles, so that the student model achieves the performance close to the teacher model, but this method is not suitable for point cloud semantic segmentation model. Therefore, the above solutions need to be further optimized. SUMMARY

[0009] The purpose of the present application is to provide a point cloud segmentation network lightweight method based on knowledge distillation to solve the above problems.

[0010] The first aspect of the present application provides a point cloud segmentation network lightweight method based on knowledge distillation, comprising the following steps:

[0011] Step A: the collected point cloud data and the point-by-point labeled label information are respectively sent into the student model and the teacher model, and then the first feature map output by the backbone network of the teacher model is extracted and the first feature map is calculated , and the second feature map output by the backbone network of the student model is extracted

[0012] Step B: based on the label information and the importance score , the first feature map and the second feature map are respectively classified and sampled to obtain the first saliency feature map based on the teacher model and the second saliency feature map based on the student model , wherein each point feature in the first saliency feature map is a first saliency point, and each point feature in the second saliency feature map is a second saliency point.

[0013] Step C: a first point-by-point similarity matrix between each first saliency point is constructed , a second point-by-point similarity matrix between each second saliency point is constructed , and the first point-by-point similarity matrix and the second point-by-point similarity matrix are subjected to knowledge distillation to obtain global structured information ;

[0014] Step D: first local information of similarity between each first saliency point and its neighbor node is constructed , second local information of similarity between each second saliency point and its neighbor node is constructed , the first local information and the second local information are subjected to knowledge distillation to obtain local structured information ;

[0015] Step E: on the basis of knowledge distillation, the cross-entropy loss of the label information and the prediction result of the student model is calculated , and the global structured information and the local structured information are combined to obtain the total loss of point cloud semantic segmentation .

[0016] Furthermore, step A specifically includes the following steps:

[0017] Step A1: Collect the point cloud data and point-by-point label information Input the student model and the teacher model, among which This indicates the number of points in the point cloud. The feature count of a point is represented; then the first feature map output by the backbone network of the teacher model is extracted. ,in, This indicates the number of points in the point cloud. Represents the first feature map The number of channels; extracting the second feature map from the student model backbone network output. ,in This indicates the number of points in the point cloud. Representing the second feature map The number of channels;

[0018] Step A2: Calculate the first feature map Importance score of all features The importance score The Middle Importance score of each feature For the first feature map The permutation-invariant maximum value of the corresponding point feature along the channel dimension is expressed as follows:

[0019]

[0020] Furthermore, step B specifically includes the following steps:

[0021] Step B1: Utilize tag information For the first feature map Importance score of all features Perform classification and aggregation to obtain the total set of importance score categories. ,in Indicates the number of categories. Indicates the first A set of importance scores for point features in each category;

[0022] Step B2: For the total set of importance score categories From the importance score set of point features for each category, the first index of the n key point clouds is extracted. , will the first index The aggregation yields the second index. , among which dimension In the formula This indicates the number of key point clouds for each category. denotes the number of categories;

[0023] Step B3: extracting a first saliency feature map corresponding to the first feature map Step B4: extracting a second saliency feature map wherein denotes the number of channels of the first feature map, and the second saliency feature map each point feature in the second saliency feature map is a second saliency point; extracting a second saliency feature map corresponding to the second feature map wherein denotes the number of channels of the first feature map, and the second saliency feature map each point feature in the second saliency feature map is a second saliency point.

[0024] Further, the step C specifically comprises the following steps:

[0025] Step C1: constructing a first point-wise similarity matrix between the first saliency points constructing a second point-wise similarity matrix between the second saliency points , which is expressed as follows: denotes the transpose operation:

[0026]

[0027] wherein and denote the index of the first saliency point in the first saliency feature map, and denote the index of the second saliency point in the second saliency feature map, denotes the first saliency point in the first saliency feature map, denotes the first saliency point in the first saliency feature map, denotes the second saliency point in the second saliency feature map, denotes the second saliency point in the second saliency feature map; Step C2: performing knowledge distillation on the first point-wise similarity matrix and the second point-wise similarity matrix , which is expressed as follows:

[0028]

[0029] ​​​​​​

[0030] wherein, and denotes the index of the first salient point / second salient point in the first saliency feature map / second saliency feature map, denotes the number of classes, denotes the number of keypoint clouds per class, denotes the similarity between the th first salient point and the th first salient point, denotes the similarity between the th second salient point and the th second salient point.

[0031] Further, the step D specifically comprises the following steps:

[0032] Step D1: taking each first salient point as a center point, then finding its th neighbor node; taking each second salient point as a center point, then finding its th neighbor node;

[0033] Step D2: constructing the first local information of the similarity between each first salient point and its neighbor node, specifically, constructing a graph for each first salient point and its neighbor node respectively, wherein the vertex , the edge , denotes the similarity between the first salient point and its neighbor node, which can be expressed as follows, wherein denotes the transpose operation:

[0034]

[0035] wherein, denotes the th first salient point in the first feature map , denotes the th neighbor node adjacent to the first salient point;

[0036] constructing the second local information of the similarity between each second salient point and its neighbor node, specifically, constructing a graph for each second salient point and its th neighbor node wherein the vertex , the edge , denotes the similarity between the second salient point and its neighbor node:

[0037]

[0038] In the formula: represents the second feature map the first second salient point, represents the first neighbor node adjacent to the second salient point;

[0039] Step D3: Knowledge distillation is performed on the first local information and the second local information , expressed as the following formula:

[0040]

[0041] In the formula, represents the number of categories, represents the number of key point clouds per category, represents the number of neighbor nodes for composition of the first salient point / second salient point, represents the similarity of the first salient point and its neighbor nodes, represents the similarity of the second salient point and its neighbor nodes.

[0042] Further, the step E specifically includes the following steps:

[0043] Step E1: The label information is passed to the student model for guidance, that is, the cross-entropy loss of the label information and the prediction result of the student model is calculated:

[0044]

[0045] In the formula, represents the number of points in the point cloud, represents the number of categories, is the network prediction point is the probability of belonging to category, is its one-hot encoded corresponding label, represents the index of the point in the point cloud, represents the category

[0046] Step E2: The total loss of the point cloud semantic segmentation is calculated, expressed as the following formula:

[0047]

[0048] In the formula, represents the calculated label information ​Compared with student model prediction results Cross-entropy loss, Indicates the first balancing weight. This indicates the second balancing weight. Represents global structured information. It represents local structured information.

[0049] A second aspect of the present invention provides a lightweight point cloud segmentation network based on knowledge distillation, applied to the lightweight point cloud segmentation network based on knowledge distillation method described in the first aspect, comprising:

[0050] The first feature information extraction module, which has a built-in teacher model, is used to receive point cloud data. Preset label information And based on point cloud data Extracting the first feature map from the backbone network output of the teacher model And calculate the first feature map. Importance score of each feature point ;

[0051] The second feature information extraction module, which has a built-in student model, is used to receive point cloud data. Preset label information And based on point cloud data Extracting the second feature map from the backbone network output of the student model and output student model prediction results. ;

[0052] The category balance sampling module receives the first feature map output by the first feature information extraction module. and importance score Receive the second feature map output by the second feature information extraction module Based on tag information And importance score For the first feature map respectively Second feature map Classification sampling was performed to obtain the first salient feature map based on the teacher model. Second saliency feature map based on student model The first saliency feature map Each point feature in the graph is called a first salient point and a second salient feature graph. Each point feature in the matrix is ​​called a second salient point;

[0053] The global structured information extraction module receives the first salient feature map output by the category balance sampling module. Second saliency feature map Calculate the first point-by-point similarity matrix for each first salient point. For the second significance point Find the second pointwise similarity matrix. Align the first point-by-point similarity matrix Similarity matrix with the second point Obtain global structured information ;

[0054] The local structured information extraction module receives the first salient feature map output by the category balance sampling module. Second saliency feature map Construct neighborhood relationships for the first salient point and calculate the first local information. Construct neighborhood relationships for the second saliency point and calculate the second local information. Align the first local information With the second part of the information Obtain local structured information ;

[0055] The model training module is used to receive the global structured information output by the global structured information extraction module. Local structured information output by the local structured information extraction module And the student model prediction results output by the second feature information extraction module. The cross-entropy loss is calculated by comparing the label information with the student model's prediction results. Correcting erroneous information passed from the teacher model to the student model, combined with global structured information. Local structured information To obtain the total loss for point cloud semantic segmentation. .

[0056] A third aspect of the present invention provides an electronic device, comprising:

[0057] One or more processors; a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the method of the first aspect.

[0058] By adopting the above solution, compared with the prior art, the present invention has the following beneficial effects:

[0059] I. In the process of extracting the first and second saliency feature maps, label information is used... And importance score For the first feature map respectively Second feature map The classification sampling is performed, that is, a category balance sampling strategy is used, and problems caused by the sparsity of the point cloud and the category imbalance are effectively alleviated,

[0060] II. A global structured information part and a local structured information part are designed for point cloud semantic segmentation, and the knowledge learned by the teacher model is effectively transferred to the student model through the integration of the global structured information and the local structured information.

[0061] III. The application is suitable for a point / voxel-based point cloud deep neural network, and belongs to a pluggable structure, does not introduce additional parameter quantity and calculation quantity, is not only suitable for large-scale point cloud application scenarios, but also can be easily applied to any point cloud semantic segmentation network. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can obtain other forms of variations based on these drawings without creative labor.

[0063] Figure 1 is a step flowchart of the embodiment of the present application;

[0064] Figure 2 is a category balance sampling schematic diagram of the embodiment of the present application;

[0065] Figure 3 is a local structured information extraction schematic diagram of the embodiment of the present application;

[0066] Figure 4 is a device block diagram of the embodiment of the present application;

[0067] Figure 5 is a device block diagram of the embodiment of the present application. DETAILED DESCRIPTION

[0068] The technical solutions in the embodiments of the present application will be described clearly below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0069] The existing knowledge distillation strategy suitable for point cloud semantic segmentation usually needs to further process the point cloud (such as voxelization), which leads to poor portability and applicability of the method. Figures 1-4As shown, the present application provides a point cloud segmentation network lightweight method based on knowledge distillation. By inputting point cloud data X and label information Y into a teacher model and a student model, the first feature map based on the teacher model is obtained and the importance score of all point features of the first feature map , the second feature map based on the student model , the first feature map , the second feature map based on the importance score , and the label information Y are classified and sampled, and the first saliency feature map and the second saliency feature map are obtained through further processing. The global structured information and the local structured information of each point feature in the first saliency feature map and the second saliency feature map are extracted, and finally the cross-entropy loss of the label information Y and the prediction result of the student model is combined to obtain the total loss of semantic segmentation . Thus, not only the knowledge learned by the teacher model is effectively transferred to the student model, but also the method can be easily migrated to a point cloud semantic segmentation model of any architecture, and the application range is wide, i.e. it can be applied not only to the field of industrial defect detection, but also to other large-scale point cloud scenarios. The method includes the following steps:

[0070] Step A: input the collected point cloud data and the point-by-point labeled label information into the student model and the teacher model, respectively, then extract the first feature map output by the backbone network of the teacher model and calculate the first feature map , and extract the second feature map output by the backbone network of the student model;

[0071] Specifically, step A1: input the collected point cloud data and the point-by-point labeled label information into the student model and the teacher model, wherein represents the number of points in the point cloud, such as , represents the number of features of the point, which is set according to the actual situation, and in this embodiment is 6, i.e. including the three-dimensional coordinates of the point and the color information For example, the teacher model is a pre-trained RandLA-Net network (performing 5 feature aggregations), while the student model is an untrained RandLA-Net network (performing 4 feature aggregations). Then, the first feature map output from the teacher model's backbone network is extracted. ,in The number of points in a point cloud is represented as follows: , Represents the first feature map The number of channels; extracting the second feature map from the student model backbone network output. ,in The number of points in a point cloud is represented as follows: , Representing the second feature map The number of channels, in a network where both the teacher and student models are RandLA-Net, the aforementioned backbone network is interpreted as a U-Net structure consisting of feature extraction and feature aggregation; Step A2: Calculate the first feature map. Importance score of all features It should be noted that the importance score This can be viewed as an attention mechanism, where the teacher model focuses more on the points where the importance score is considered. The Middle Importance score of each feature For the first feature map The permutation-invariant maximum value of the corresponding point feature along the channel dimension is expressed as follows:

[0072]

[0073] Step B: As Figure 2 As shown, based on tag information And importance score For the first feature map respectively Second feature map Classification sampling was performed to obtain the first salient feature map based on the teacher model. Second saliency feature map based on student model The first saliency feature map Each point in the graph is characterized by its first salient point and its second salient feature graph. Each point in the matrix is ​​a second salient point;

[0074] Specifically, step B1: Utilizing tag information For the first feature map Importance score of all features Perform classification and aggregation to obtain the total set of importance score categories. ,in Indicates the number of categories. Indicates the first The set of importance scores for point features of each category, the number of categories is set according to the actual situation, such as setting the number of point cloud segmentation categories to 13 in this embodiment; Step B2: Set the total set of importance score categories From the importance score set of point features for each category, the first index of the n key point clouds is extracted. , will the first index The aggregation yields the second index. , among which dimension In the formula This indicates the number of key point clouds for each category. This represents the number of categories, where the keypoint cloud is the point cloud with the highest importance score in each category. A high importance score can be selected from the top 20% of points, but the specific number can be determined based on the actual situation. In this embodiment, This means the number of indexes is 1 / 5 of the original number of points, making 20% ​​of the point features more relevant. However, it should be noted that the categories... The values ​​are preferably the same, but some deviation is allowed between categories, such as the ratio of the minimum value to the maximum value being 80%-100%; Step B3: Through the second index Extraction and first feature map The corresponding first saliency feature map ,in The first saliency feature map represents the number of channels in the first feature map. Each point in the matrix is ​​characterized as a first salient point; through the second index... Extraction and second feature map The corresponding second saliency feature map ,in The second saliency feature map represents the number of channels in the second feature map. Each point feature in the sample is a second salient point; the above sampling is class-balanced sampling, that is, the number of samples collected for each class is the same or basically the same.

[0075] Step C: Construct the first pointwise similarity matrix between each first salient point. Construct a second point-by-point similarity matrix between each second saliency point. It should be noted that the point-by-point similarity matrix is ​​defined as follows: for example, the value at position (i,j) in the matrix represents the similarity between the i-th point and the j-th point. This definition can be found in the patent "A Method for Knowledge Distillation of Point Cloud Segmentation Network". The first point-by-point similarity matrix... and the second point-by-point similarity matrix Perform knowledge distillation to obtain global structured information. The distillation principle can be found in the paper "Structured Knowledge Distillation for Semantic Segmentation";

[0076] Specifically, step C1: Construct the first point-by-point similarity matrix between each first salient point. Construct a second point-by-point similarity matrix between each second saliency point. It is expressed as the following formula, where Indicates the transpose operation:

[0077]

[0078] in In and This represents the index of the first salient point in the first salient feature map. middle and This indicates the index of the second saliency point in the second saliency feature map. The first saliency feature graph represents the first saliency feature. The first significant point, The first saliency feature graph represents the first saliency feature. The first significant point, The second saliency feature graph represents the first The second significant point, The second saliency feature graph represents the first The second significant point;

[0079] Step C2: For the first pointwise similarity matrix and the second point-by-point similarity matrix Knowledge distillation can be expressed as follows:

[0080]

[0081] In the formula, and This indicates the index of the first salient point / second salient point in the first salient feature map / second salient feature map. Indicates the number of categories. This represents the number of keypoint clouds in each category. Keypoint clouds are those with high importance scores in each category. A high importance score can be selected from the top 20% of points, but the specific number can be determined based on the actual situation. Indicates the first The first salient point and the first Similarity between the first salient points Indicates the first a second salient point. a second salient point.

[0082] Step D: As shown, the first local information of the similarity between each first salient point and its neighbor nodes is constructed Figure 4 , the second local information of the similarity between each second salient point and its neighbor nodes is constructed , the first local information of the similarity between each first salient point and its neighbor nodes is constructed , the second local information of the similarity between each second salient point and its neighbor nodes is constructed , the first local information of the similarity between each first salient point and its neighbor nodes is constructed , the second local information of the similarity between each second salient point and its neighbor nodes is constructed ;

[0083] Specifically,

[0084] Step D1: Taking each first salient point as a center point, then finding its surrounding neighbor nodes; taking each second salient point as a center point, then finding its surrounding neighbor nodes;

[0085] Step D2: The first local information of the similarity between each first salient point and its neighbor nodes is constructed , specifically, respectively constructing a graph for each first salient point and its neighbor nodes, wherein the vertex , the edge , represents the similarity of the first salient point and its neighbor nodes, which can be expressed as follows, wherein represents the transpose operation:

[0086]

[0087] In the formula: represents the first feature map , the first salient point, represents the neighbor node adjacent to the first salient point;

[0088] The second local information of the similarity between each second salient point and its neighbor nodes is constructed , specifically, respectively constructing a graph for each second salient point and its surrounding neighbor nodes , wherein the vertex , the edge , represents the similarity of the second salient point and its neighbor nodes:

[0089]

[0090] wherein: denotes the second feature map denotes the first second salient point, denotes the first neighbor node adjacent to the second salient point;

[0091] Step D3: Knowledge distillation is performed on the first local information and the second local information , which is expressed as follows:

[0092]

[0093] wherein, denotes the number of categories, denotes the number of key point clouds per category, denotes the number of neighbor nodes for composition of the first / second salient point, denotes the similarity of the first salient point and its neighbor nodes, denotes the similarity of the second salient point and its neighbor nodes.

[0094] Step E: Since the teacher model inevitably transmits some incorrect information, in order to slow down the impact of these information, the label information is transmitted to the student model for guidance, that is, on the basis of knowledge distillation, the cross-entropy loss between the label information and the prediction result of the student model is calculated, since the label information is correct, the cross-entropy loss between the label information and the prediction result of the student model is calculated, so as to prevent the student model from being misled by the incorrect information of the teacher model, and the cross-entropy loss is combined with the global structured information and the local structured information to obtain the total loss of point cloud semantic segmentation.

[0095] Specifically, step E1: the label information is transmitted to the student model for guidance, that is, the cross-entropy loss between the label information and the prediction result of the student model is calculated:

[0096]

[0097] wherein, denotes the number of points in the point cloud, denotes the number of categories, Network prediction point To belong to The probability of the category, It is the corresponding tag for its one-hot encoding. This represents the index of a point in the point cloud. Indicates category;

[0098] Step E2: Calculate the total loss for point cloud semantic segmentation, expressed as follows:

[0099]

[0100] In the formula, Indicates the calculation of label information Compared with student model prediction results Cross-entropy loss, Indicates the first balancing weight. This indicates the second balancing weight. Represents global structured information. This represents local structured information. In this example, we can let... In practical applications, this method can be easily applied to any point cloud semantic segmentation network simply by changing the teacher and student models.

[0101] In summary, the present invention has the following beneficial effects:

[0102] I. In the process of extracting the first and second saliency feature maps, label information is used... And importance score For the first feature map respectively Second feature map Classification sampling, specifically the use of a class-balanced sampling strategy, effectively mitigates the problems caused by the sparsity and class imbalance of point clouds.

[0103] Second, a global structured information component and a local structured information component were designed for point cloud semantic segmentation. By integrating the global structured information and the local structured information, the knowledge learned by the teacher model can be effectively transferred to the student model.

[0104] Third, this invention is applicable to point cloud deep neural networks based on points / voxels, and is a pluggable structure that does not introduce additional parameters or computational load. It is not only suitable for large-scale point cloud application scenarios, but can also be easily applied to any point cloud semantic segmentation network.

[0105] This invention also provides a lightweight point cloud segmentation network based on knowledge distillation, applied to the aforementioned lightweight point cloud segmentation network based on knowledge distillation, such as... Figure 5 As shown, it includes:

[0106] The first feature information extraction module, which has a built-in teacher model, is used to receive point cloud data. Preset label information And based on point cloud data Extracting the first feature map from the backbone network output of the teacher model And calculate the first feature map. Importance score of each feature point ;

[0107] The second feature information extraction module, which has a built-in student model, is used to receive point cloud data. Preset label information And based on point cloud data Extracting the second feature map from the backbone network output of the student model and output student model prediction results. ;

[0108] The category balance sampling module receives the first feature map output by the first feature information extraction module. and importance score Receive the second feature map output by the second feature information extraction module Based on tag information And importance score For the first feature map respectively Second feature map Classification sampling was performed to obtain the first salient feature map based on the teacher model. Second saliency feature map based on student model The first saliency feature map Each point feature in the graph is called a first salient point and a second salient feature graph. Each point feature in the matrix is ​​called a second salient point;

[0109] The global structured information extraction module receives the first salient feature map output by the category balance sampling module. Second saliency feature map Calculate the first point-by-point similarity matrix for each first salient point. Find the second point-by-point similarity matrix for the second salient point. Align the first point-by-point similarity matrix Similarity matrix with the second point Obtain global structured information ;

[0110] The local structured information extraction module receives the first salient feature map output by the category balance sampling module. Second saliency feature map Construct neighborhood relationships for the first salient point and calculate the first local information. Construct neighborhood relationships for the second saliency point and calculate the second local information. Align the first local information With the second part of the information Obtain local structured information ;

[0111] The model training module is used to receive the global structured information output by the global structured information extraction module. Local structured information output by the local structured information extraction module And the student model prediction results output by the second feature information extraction module. The cross-entropy loss is calculated by comparing the label information with the student model's prediction results. Correcting erroneous information passed from the teacher model to the student model, combined with global structured information. Local structured information To obtain the total loss for point cloud semantic segmentation. .

[0112] The present invention also provides an electronic device, such as... Figure 5 As shown, it includes: one or more processors; and a memory for storing one or more programs; wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the above-described method. Since the electronic device described in this embodiment is a device used to implement the method, those skilled in the art can understand the specific implementation methods and various variations of the electronic device in this embodiment based on the methods described in this application. Therefore, how the electronic device implements the methods in the embodiments of this application will not be described in detail here. Any device used by those skilled in the art to implement the methods in the embodiments of this application falls within the scope of protection of this application.

[0113] In the description of this specification, references to terms such as "an embodiment," "some embodiments," "example," "specific example," or "optional embodiment," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0114] The above-described embodiments do not constitute a limitation on the protection scope of the technical solutions. Any modification, equivalent replacement and improvement made within the spirit and principle of the above-described embodiments should be included in the protection scope of the technical solutions.

Claims

1. A lightweight method for point cloud segmentation networks based on knowledge distillation, characterized in that, Includes the following steps: Step A: Collect the point cloud data and point-by-point label information The data are fed into the student model and the teacher model respectively, and then the first feature map output by the backbone network of the teacher model is extracted. And calculate the first feature map Importance score of all features Extract the second feature map output by the backbone network of the student model. ; Step B: Based on tag information And importance score For the first feature map respectively Second feature map Classification sampling was performed to obtain the first salient feature map based on the teacher model. Second saliency feature map based on student model The first saliency feature map Each point in the graph is characterized by its first salient point and its second salient feature graph. Each point in the matrix is ​​a second salient point; Step C: Construct the first pointwise similarity matrix between each first salient point. Construct a second point-by-point similarity matrix between each second saliency point. And for the first point-by-point similarity matrix and the second point-by-point similarity matrix Perform knowledge distillation to obtain global structured information. ; Step D: Construct the first local information on the similarity between each first salient point and its neighboring nodes. Construct second local information on the similarity between each second saliency point and its neighboring nodes. For the first part of the information and second part information Perform knowledge distillation to obtain locally structured information. ; Step E: Calculate tag information based on knowledge distillation. Compared with student model prediction results Cross-entropy loss Combined with global structured information Local structured information To obtain the total loss for point cloud semantic segmentation. ; Step C specifically includes the following steps: Step C1: Construct the first pointwise similarity matrix between each first salient point. Construct a second point-by-point similarity matrix between each second saliency point. It is expressed as the following formula, where Indicates the transpose operation: in In and This represents the index of the first salient point in the first salient feature map. middle and This indicates the index of the second saliency point in the second saliency feature map. The first saliency feature graph represents the first saliency feature. The first significant point, The first saliency feature graph represents the first saliency feature. The first significant point, The second saliency feature graph represents the first The second significant point, The second saliency feature graph represents the first The second significant point; Step C2: For the first pointwise similarity matrix and the second point-by-point similarity matrix Knowledge distillation can be expressed as follows: In the formula, and This indicates the index of the first salient point / second salient point in the first salient feature map / second salient feature map. Indicates the number of categories. This indicates the number of key point clouds for each category. Indicates the first The first salient point and the first Similarity between the first salient points Indicates the first The second significant point and the first Similarity between the second salient points.

2. The lightweight point cloud segmentation network method based on knowledge distillation according to claim 1, characterized in that, Step A specifically includes the following steps: Step A1: Collect the point cloud data and point-by-point label information Input the student model and the teacher model, among which This indicates the number of points in the point cloud. The feature count of a point is represented; then the first feature map output by the backbone network of the teacher model is extracted. ,in, This indicates the number of points in the point cloud. Represents the first feature map The number of channels; extracting the second feature map from the student model backbone network output. ,in This indicates the number of points in the point cloud. Representing the second feature map The number of channels; Step A2: Calculate the first feature map Importance score of all features The importance score The Middle Importance score of each feature For the first feature map The permutation-invariant maximum value of the corresponding point feature along the channel dimension is expressed as follows:

3. The lightweight point cloud segmentation network method based on knowledge distillation according to claim 1, characterized in that, Step B specifically includes the following steps: Step B1: Utilize tag information For the first feature map Importance score of all features Perform classification and aggregation to obtain the total set of importance score categories. ,in Indicates the number of categories. Indicates the first A set of importance scores for point features in each category; Step B2: For the total set of importance score categories From the importance score set of point features for each category, the first index of the n key point clouds is extracted. , will the first index The aggregation yields the second index. , among which dimension In the formula This indicates the number of key point clouds for each category. Indicates the number of categories; Step B3: Through the second index Extraction and the first feature map The corresponding first saliency feature map ,in The first saliency feature map represents the number of channels in the first feature map. Each point in the matrix is ​​characterized as a first salient point; through the second index... Extraction and second feature map The corresponding second saliency feature map ,in The second saliency feature map represents the number of channels in the second feature map. Each point in the matrix is ​​a second salient point.

4. A lightweight point cloud segmentation network method based on knowledge distillation according to any one of claims 1 to 3, characterized in that, Step D specifically includes the following steps: Step D1: Using each first salient point as the center point, then search for its surrounding points.

1. Neighboring nodes; using each second salient point as the center point, then search its surrounding nodes. One neighboring node; Step D2: Construct the first local information on the similarity between each first salient point and its neighboring nodes. Specifically, this involves constructing a graph for each first salient point and its neighboring nodes. , where the vertex ,side , The similarity between the first salient point and its neighboring nodes can be expressed as the following formula, where Indicates the transpose operation: In the formula: Represents the first feature map The Middle The first significant point, Indicates the number of adjacent points to the first saliency point. One neighboring node; Constructing second local information on the similarity between each second saliency point and its neighboring nodes. Specifically, this involves examining each second saliency point and its surrounding areas. Construct a graph using neighboring nodes. , where the vertex ,side , Indicate the similarity between the second salient point and its neighboring nodes: In the formula: Representing the second feature map The Middle The second significant point, Indicates the number of adjacent points to the second saliency point. One neighboring node; Step D3: For the first local information and second part information Knowledge distillation can be expressed as follows: In the formula, Indicates the number of categories. This indicates the number of key point clouds for each category. This represents the number of neighboring nodes used to construct a graph from the first salient point / the second salient point. This represents the similarity between the first salient point and its neighboring nodes. This indicates the similarity between the second salient point and its neighboring nodes.

5. A lightweight point cloud segmentation network method based on knowledge distillation according to any one of claims 1 to 3, characterized in that, Step E specifically includes the following steps: Step E1: Transfer label information The information is passed to the student model for guidance, namely, to calculate the label information. Compared with student model prediction results Cross-entropy loss: In the formula, This indicates the number of points in the point cloud. Indicates the number of categories. Network prediction point To belong to The probability of the category, It is the corresponding tag for its one-hot encoding. This represents the index of a point in the point cloud. Indicates category; Step E2: Calculate the total loss for point cloud semantic segmentation, expressed as follows: In the formula, Indicates the calculation of label information Compared with student model prediction results Cross-entropy loss, Indicates the first balancing weight. This indicates the second balancing weight. Represents global structured information. It represents local structured information.

6. A lightweight point cloud segmentation network based on knowledge distillation, applied to the lightweight point cloud segmentation network based on knowledge distillation method according to any one of claims 1 to 5, comprising: The first feature information extraction module, which has a built-in teacher model, is used to receive point cloud data. Preset label information And based on point cloud data Extracting the first feature map from the backbone network output of the teacher model And calculate the first feature map. Importance score of each feature point ; The second feature information extraction module, which has a built-in student model, is used to receive point cloud data. Preset label information And based on point cloud data Extracting the second feature map from the backbone network output of the student model and output student model prediction results. ; The category balance sampling module receives the first feature map output by the first feature information extraction module. and importance score Receive the second feature map output by the second feature information extraction module Based on tag information And importance score For the first feature map respectively Second feature map Classification sampling was performed to obtain the first salient feature map based on the teacher model. Second saliency feature map based on student model The first saliency feature map Each point feature in the graph is called a first salient point and a second salient feature graph. Each point feature in the matrix is ​​called a second salient point; The global structured information extraction module receives the first salient feature map output by the category balance sampling module. Second saliency feature map Calculate the first point-by-point similarity matrix for each first salient point. For the second significance point Find the second pointwise similarity matrix. Align the first point-by-point similarity matrix Similarity matrix with the second point Obtain global structured information ; The local structured information extraction module receives the first salient feature map output by the category balance sampling module. Second saliency feature map Construct neighborhood relationships for the first salient point and calculate the first local information. Construct neighborhood relationships for the second saliency point and calculate the second local information. Align the first local information With the second part of the information Obtain local structured information ; The model training module is used to receive the global structured information output by the global structured information extraction module. Local structured information output by the local structured information extraction module And the student model prediction results output by the second feature information extraction module. The cross-entropy loss is calculated by comparing the label information with the student model's prediction results. Correcting erroneous information passed from the teacher model to the student model, combined with global structured information. Local structured information To obtain the total loss for point cloud semantic segmentation. .

7. An electronic device, comprising: One or more processors; Memory, used to store one or more programs; Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 5.

Images