A point cloud segmentation method based on multi-feature input and matrix supervision

Abstract

The application relates to a point cloud segmentation method based on multi-feature input and matrix supervision, which comprises the following steps: obtaining point cloud training and test data by acquiring a data set, selecting a benchmark network to decouple and classify point cloud data on the overall data set, evaluating the best output result of the feature input benchmark network, connecting by calculating the KL divergence of the result obtained in the training process of each benchmark network, performing iterative training until a preset training number is reached, combining the outer product matrix KL divergence of the multi-feature mutual learning network to online train the mutual learning network, and finally obtaining a point cloud semantic segmentation result. By means of mutual learning, different networks learn different features at the same time, knowledge is transmitted between the networks, the understanding and utilization of multi-feature information of each network are enhanced, the generalization performance of the network is greatly improved, the point cloud segmentation network has better robustness and more accurate segmentation precision when facing actual segmentation scenes.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and in particular to a point cloud segmentation method based on multi-feature input and matrix supervision. BACKGROUND

[0002] Point cloud segmentation is a popular field of point cloud research, which has important value in applications such as autonomous driving, remote sensing, indoor and outdoor scene planning, and VR / AR. Point cloud data has the advantages of rich information, high precision, and easy data acquisition. However, due to its large and complex information, effective extraction of point cloud features becomes a key and difficult problem. Most current methods only stay at the surface of abstracting point cloud features, or use traditional detection operators, or directly use deep learning encoders to map features and then input the features into a deep learning network for training. However, this approach has some limitations, such as different emphasis on feature extraction for each method, resulting in different utilization of each feature, which ultimately affects network performance.

[0003] To overcome this limitation, one possible solution is to decouple representation learning, which involves separating each feature and separating features that may interfere with each other, allowing the network to learn each feature separately. However, in the field of point cloud segmentation, research on feature decoupling and deep mutual learning is not sufficient. At the same time, existing deep mutual learning methods rarely consider using decoupled multiple features or different feature groups for the same mutual learning task, and rarely explore more efficient mutual learning mechanisms. Therefore, how to segment point clouds is a problem that needs to be considered.

[0004] It should be noted that the information disclosed in the above background section is only used to strengthen the understanding of the background of the present disclosure, and therefore can include information that does not constitute prior art known to those of ordinary skill in the art. SUMMARY

[0005] The present application aims to overcome the shortcomings of the prior art and provides a point cloud segmentation method based on multi-feature input and matrix supervision, which solves the problem of existing methods in handling data with complex features, difficulty in effectively extracting and utilizing information of different features, and the problem of deep mutual learning methods mainly based on the input of the same feature, which cannot be targeted at different feature inputs, and the problem of insufficient network improvement effect of existing deep mutual learning.

[0006] The purpose of the present application is achieved by the following technical solution: a point cloud segmentation method based on multi-feature input and matrix supervision, the point cloud segmentation method comprising:

[0007] Step one, obtain the point cloud training and test data set, and use multiple reference networks to form a multi-feature mutual learning network to decouple and classify the point cloud data on the overall data set;

[0008] Step two, input the split point cloud data into the selected reference network to test the accuracy, calculate the average accuracy and average intersection over union value of the prediction, and select the good decoupling feature combination;

[0009] Step three, during the training of each reference network, the obtained results are used to calculate the KL divergence to connect the learned knowledge between the reference networks, and each reference network receives different feature data, so that each reference network focuses on processing its corresponding data;

[0010] Step four, iterate according to the training of step three until the preset training times are reached, and finally obtain the point cloud semantic segmentation result by combining the outer product matrix KL divergence online training mutual learning network through the multi-feature mutual learning network.

[0011] The step one specifically includes the following contents:

[0012] Download the S3DIS public data set to obtain point cloud training and test data, extract the position, color and label information of each point, calculate the normal vector of each point itself using the open3d library through the position information, and obtain p={p1, p2,..., pN}∈R3+C, where N represents the number of points, and 3+C represents the number of channels. N}∈R N*(3+C) , 3 is the three-dimensional coordinate information xyz information of the point, and C includes color rgb information and point cloud surface normal vector component (n x , n y , n z ) information;

[0013] Select point transformer as the reference network, all points in the point cloud have three-dimensional coordinate information xyz, color information rgb, and point cloud surface normal vector component (n x , n y , n z ) information, and the overall features are split and decoupled according to these information, and the feature combination is selected and input into the network.

[0014] The reference network includes an encoder, a decoder and a classifier, the encoder is used to extract feature information, which is composed of a down-sampling module and a transformer module; the decoder is used to gradually condense semantic information using feature information, which is composed of an up-sampling module and a transformer module; the classifier is used to classify all point clouds through the semantic information condensed by the decoder.

[0015] The step of inputting the split point cloud data into the selected benchmark network to test accuracy includes:

[0016] A1, n x n y n z The information features of the six RGB channels are input into a point transformer network with input channels of 6, and the information features of the six XYZRGB channels are input into a point transformer network with input channels of 6.

[0017] A2. By sampling the farthest point, a number of points are uniformly selected in the overall point cloud space. Then, the K-nearest neighbor method is used to construct a spherical neighborhood with these points as the center and a preset distance as the radius. After sampling within the neighborhood, K points are obtained. After passing through a fully connected network, global pooling is performed to concentrate the information of the K-nearest neighbors on the center point. This reduces the point cloud data while increasing the features of each point, achieving the effect of extracting features within the machine neighborhood of key points.

[0018] A3. Perform downsampling and upsampling sequentially. After sampling, the data is further processed by the transformer module in the point transformer network to extract point cloud features. The point cloud data is then processed iteratively multiple times until the preset downsampling and upsampling sizes are reached.

[0019] A4. The results are passed through a fully connected network as a classification layer. The last layer of the fully connected network is a softmax layer. The data obtained by the fully connected network is normalized and the final results are used for prediction.

[0020] A5. Perform cross-entropy calculation on the obtained category prediction and the real object category to obtain the loss value as the loss. After backpropagation, train this network, and then start training again from step A1 until the set number of training times is reached.

[0021] The A3 step specifically includes the following:

[0022] A31. The downsampled data is further processed by the transformer module in the point transformer network to extract point cloud feature information. The transformer module then transforms the point cloud data x i pass Pointwise feature transformations are performed on three different fully connected layers, ψ and α, to obtain... ψ(x i ) and α(x i) three features, and the relative distance between all points in K-neighbor and the center point is taken as a feature to obtain position encoding δ through a fully connected layer, and then the formula Transform the obtained features, where γ is a fully connected layer, ρ is a normalization layer, and is a Hadamard product. After transformation, y i is the final feature obtained by the transformer module;

[0023] A32, repeat steps A2 and A31 until a preset downsampling size is reached;

[0024] A33, use the trilinear interpolation method to fill in the points, and combine the features obtained during downsampling as the final upsampling result;

[0025] A34, further extract point cloud feature information from the data after upsampling through the transformer module in the point transformer network, and repeat the processing content of the point cloud data in step A31 through the transformer module;

[0026] A35, repeat steps A33 and A34 until a preset upsampling size is reached.

[0027] The third step specifically includes the following contents:

[0028] B1, divide the S3DIS data Dataset into two groups of features Dataset1 and Dataset2 according to the features selected in step two;

[0029] B2, input Dataset1 and Dataset2 into student networks model1 and model2 respectively for training at each training time to obtain two predicted values logits1 and logits2;

[0030] B3, calculate the cross-entropy loss of logits1 and logits2 predicted by each student network and the label label to obtain loss_ce1 and loss_ce2;

[0031] B4, calculate the KL divergence loss of the outer product matrix between each student network, take the outer product of logits1 and logits2 with itself to obtain two matrices, and take KL_loss1 and KL_loss2 between the two matrices as mutual learning loss;

[0032] B5, use the sum of loss_ce1 and KL_loss1 loss to update model1, and use the sum of loss_ce2 and KL_loss2 loss to update model2.

[0033] The application has the following advantages: a point cloud segmentation method based on multi-feature input and matrix supervision, decoupling features and allowing different networks to simultaneously learn different features through mutual learning, transferring knowledge between each other to enhance the understanding and utilization of multi-feature information by each network, greatly improving the generalization performance of the network, and making the point cloud segmentation network have better robustness and more accurate segmentation accuracy when facing actual segmentation scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0034] Figure 1 A multi-feature mutual learning structure of the application is shown in the figure.

[0035] Figure 2 An outer product matrix calculation schematic of the application is shown in the figure. DETAILED DESCRIPTION

[0036] To make the purposes, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in combination with the drawings herein is not intended to limit the protection scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative work are within the protection scope of the present application. The present application will be further described below in combination with the drawings.

[0037] The present application specifically relates to a multi-feature input mutual learning point cloud semantic segmentation method, which aims to solve the problem that it is difficult to effectively extract and utilize information of different features when processing data with complex features. The complex features are separated and given to different networks for processing, so that each network can focus on the features it is good at or interested in, thereby improving the extraction efficiency and quality of the features. Most deep mutual learning methods are based on the input of the same features, and there is less research on the input of different features. The input of different features between mutual learning input networks is explored, and different feature processing networks are connected through mutual learning, so that each network can obtain supplementary or reference information from other networks, thereby enhancing the understanding and utilization of multi-feature information by each network. To solve the problem that the traditional deep mutual learning has insufficient effect on network improvement, a similarity measurement method based on label outer product matrix is proposed to improve the deep mutual learning process, so that the mutual learning effect between networks is more obvious and effective. Specifically, the following contents are included:

[0038] S1, obtain point cloud training and test data by downloading S3DIS public dataset, the dataset contains 6 scenes in total, each scene is composed of several regions, each region contains several point cloud data, each point cloud data has three features of xyz representing position information, three features of rgb representing color information, and a class label representing the object category it belongs to, the segmentation task is mainly tested on the 5th scene, and trained and verified on other scenes. After downloading, the position, color and label information of each point are extracted by the program, and the normal vector of each point is calculated by the position information using the open3d library. Finally, p = {p1, p2,..., p N} ∈ R N*(3+C) , 3 + C is the number of channels, 3 is the three-dimensional coordinate information xyz information of the point, C includes color rgb information and point cloud surface normal vector component n x n y n z information;

[0039] S2, select point transformer as the reference network, the network specific information is as follows:

[0040] The overall network is composed of an encoder, a decoder and a classifier, the encoder is composed of a down-sampling module and a transformer module corresponding to the output dimension of the down-sampling, which is used to extract feature information; the decoder is composed of an up-sampling module and a transformer module corresponding to the output dimension of the up-sampling, which is used to gradually condense semantic information using feature information; the final classification layer uses the semantic information condensed by the decoder to classify all point clouds.

[0041] S3, decoupling classification of point cloud data on the overall dataset, all points in the point cloud have three-dimensional coordinate xyz information, color rgb information, point cloud surface normal vector component (n x , n y , n z ) information, so the overall features are disassembled and decoupled according to these information, for example, select rgb, xyzrgb and other feature combinations to input into the network, and try various possible feature combinations.

[0042] S4, input the point cloud information into the encoder of the reference network, the disassembled point cloud data is input into the selected reference network to test the accuracy, since the selected reference network is point transformer, the specific way is as follows:

[0043] Substep 1: input the selected feature data into the point transformer network containing the corresponding channel, such as n x n y n zThe information features of the six RGB channels are input into a point transformer network with input channels of 6, and the information features of the six XYZRGB channels are input into a point transformer network with input channels of 6.

[0044] Sub-step 2: Select several points evenly in the overall point cloud space through farthest point sampling, and then use the K-Nearest Neighbor method to construct a spherical neighborhood with these points as the center and a specific radius. Sample K points in the neighborhood, pass them through a fully connected network and then perform global pooling to concentrate the information of the K nearest neighbors on the center point. This reduces the number of points in the point cloud while increasing the features of each point, achieving the effect of extracting features from several key points and their neighborhoods.

[0045] Sub-step 3: The data sampled by the downsampling network is passed through the transformer module in the point transformer network to further extract point cloud feature information. Specifically, the transformer module transforms the point cloud data x i pass Pointwise feature transformations are performed on three different fully connected layers, ψ and α, to obtain... ψ(x i ) and α(x i These three features, along with the relative distances between all points and the center point during the K-nearest neighbor step, are used to obtain the position encoding δ through a fully connected layer. Then, the obtained features are transformed: Where γ is the fully connected layer, ρ is the normalized layer, and ⊙ is the Hadamard product. After the above transformation, y i The features finally obtained by the transformer layer;

[0046] Sub-step 4: Repeat steps 2 and 3 until the preset downsampling size is reached. As a preferred option, a total of 5 downsampling modules and 5 transformer modules are selected to form the encoder. After the data passes through the 5 downsampling modules, the number of feature channels changes as follows: (input_channels, 32), (32, 64), (64, 128), (128, 256), (256, 512). Finally, the data is processed into 512-channel features.

[0047] Sub-step 5: Use trilinear interpolation to fill in the points and upsample the data obtained in the previous step. This can be seen as the inverse process of downsampling. At the same time, it combines the features obtained during the previous downsampling as the final upsampling result.

[0048] Substep 6: The data sampled by the upsampling network is further processed through the transformer module designed by the point transformer author to extract point cloud feature information. The specific steps and substeps are the same as substep 3;

[0049] Substep 7: Steps 5 and 6 are repeatedly executed until the preset upsampling size is reached. As a preferred embodiment, five upsampling modules and five transformer modules are selected to form the decoder, and the five upsampling feature numbers are (512, 512), (512, 256), (256, 128), (128, 64), and (64, 32). Finally, the data is processed into 32-channel features;

[0050] Substep 8: The results are processed through a fully connected network as a classification layer, and the feature processing changes to (32, 13). The final sampling results are predicted, and the last layer of the fully connected network is a softmax layer. The data processed by the network is normalized, and the 13-channel values are limited to 0-1 and added to 1. The final data is called logits, which represents the prediction of each input point category;

[0051] Substep 9: The obtained category prediction and the real object category are subjected to cross-entropy (Cross-Entropy) operation to obtain the loss value as the loss. After back-propagation (Back-propagation), the benchmark network is trained, and then the training starts from substep 1 again until the set training rounds are reached;

[0052] S5. To provide better mutual learning student branches for the subsequent mutual learning network training, evaluate the best output results of these feature inputs into the benchmark network, calculate the predicted mean accuracy (Mean accuracy) and mean intersection over union (Mean Intersection over Union) values respectively, and select the good decoupling feature combination. Due to the excellent miou performance during individual training, n x n y n z rgb and xyzrgb features;

[0053] S6. According to the design of the multi-feature mutual learning network as shown in Figure 1 , the features are grouped and input into different student networks. The specific steps are as follows:

[0054] Substep 1: The S3DIS data Dataset is divided into Dataset1 and Dataset2 according to the features selected in S5. As a preferred embodiment, the features of Dataset1 are xyzrgb, and the features of Dataset2 are nx n y n z rgb;

[0055] Sub-step 2: During each round of training, Dataset1 and Dataset2 are input into student networks model1 and model2 respectively for training, wherein the structures of model1 and model2 are both the pointtransformer reference network mentioned in step S4, and finally two sets of predicted values logits1 and logits2 are obtained, and the channel number of each set of logits is 13, which represents the probability of each point belonging to 13 categories.

[0056] Sub-step 3: Calculate the cross-entropy loss of logits1 and logits2 predicted by each student network and the label label to obtain loss_ce1 and loss_ce2.

[0057] Sub-step 4: Calculate the mutual learning KL divergence loss. Unlike the general mutual learning that calculates the KL loss of two logits1 and logits2, the present application calculates the KL loss of the outer product matrix between each student network. As shown in the following formula, the outer product matrix is obtained by performing outer product on the predicted data of the reference network itself, and logits1 and logits2 are respectively performed on themselves to obtain two matrices, and the KL_loss1 and KL_loss2 between the two matrices are used as mutual learning loss. Figure 2

[0058] Sub-step 5: Use the sum of loss_ce1 and KL_loss1 to update model1, and use the sum of loss_ce2 and KL_loss2 to update model2.

[0059] S7, train for several rounds according to the step S6 until the set training round number is reached.

[0060] Under the condition of multi-feature input and outer product matrix supervision in the present application, n x n y n z rgb, xyzrgb, the miou of this feature combination reaches 72.34%, which is about 2% higher than the existing one.

[0061] ​The present application separates complex features and gives them to different networks for processing, so that each network can focus on the features it is good at or interested in, thereby improving the efficiency and quality of feature extraction; the different feature processing networks are connected in a mutual learning manner, so that each network can obtain supplementary or reference information from other networks, thereby enhancing the understanding and use of each network for multi-feature information; for the case that the KL divergence supervision label effect is relatively weak in mutual learning, a similarity measurement method based on outer product matrix is proposed, that is, each network obtains a probability prediction value vector after network layer, and performs outer product on itself, which is equivalent to that each network has two independent repeated tests under its own predicted probability, and calculating the KL divergence of this matrix can make the mutual learning effect between networks more obvious and effective; different feature networks can be trained simultaneously and knowledge can be transferred between them. Unlike traditional deep mutual learning methods, the method of the present application does not require all networks to share the same data source or input features, but allows each network to be trained according to the features designed or selected by itself. In this way, each network can focus on the features it is good at or interested in, and can obtain supplementary or reference information from other networks.

[0062] The above only describes the preferred embodiments of the present application, and it should be understood that the present application is not limited to the forms disclosed herein, should not be considered as excluding other embodiments, and can be used in various other combinations, modifications and environments, and can be modified within the scope of the concepts described herein by the above teachings or related art or knowledge. Any modification and change made by those skilled in the art without departing from the spirit and scope of the present application shall be within the protection scope of the appended claims of the present application.

Claims

1. A method for point cloud segmentation based on multi-feature input and matrix supervision, characterized in that: The point cloud segmentation method comprises: Step one, obtaining point cloud training and test data from a data set, and using multiple benchmark networks to form a multi-feature mutual learning network to decouple and classify point cloud data on the overall data set; Step two, input the split point cloud data into the selected benchmark network to test the accuracy, calculate the average accuracy and average intersection over union value of the prediction, and select the good decoupling feature combination; Step three, calculate the KL divergence by the results obtained during the training of each benchmark network to connect them, so that the benchmark networks transfer the knowledge learned by each other, and each benchmark network receives different feature data, so that each benchmark network focuses on processing its corresponding data; Step four, iterate according to the training of step three until the preset training times are reached, combine the outer product matrix KL divergence online training mutual learning network through the multi-feature mutual learning network, and finally obtain the point cloud semantic segmentation result; The step one specifically comprises the following contents: Download the S3DIS public dataset to obtain point cloud training and test data, extract the position, color and label information of each point, calculate the normal vector of each point itself through the position information by using the open3d library, and obtain , wherein 3+C represents the number of channels, 3 is the three-dimensional coordinate information xyz information of the point, and C includes color rgb information and point cloud surface normal vector component (n x , n y , n z ) information; Select point transformer as the benchmark network, all points in the point cloud have three-dimensional coordinate xyz information, color rgb information, point cloud surface normal component (n x , n y , n z ) information, the overall feature is disassembled and decoupled for these information, and the feature combination is selected into the network; The step three specifically comprises the following contents: B1, divide the data Dataset of S3DIS into Dataset1 and Dataset2 according to the features selected in step two; B2, input Dataset1 and Dataset2 into student network model1 and model2 respectively for training at each training time, and obtain two prediction values logits1 and logits2; B3, calculate the cross-entropy loss of logits1 and logits2 predicted by each student network and the label label to obtain loss_ce1 and loss_ce2; B4, calculate the KL divergence loss of the outer product matrix between each student network, and perform outer product on logits1 and logits2 respectively to obtain two matrices, and KL_loss1 and KL_loss2 between the two matrices are used as mutual learning loss; B5, use the sum of loss_ce1 and KL_loss1 loss to update model1, and use the sum of loss_ce2 and KL_loss2 loss to update model2. 2.The method of claim 1, wherein: The benchmark network comprises an encoder, a decoder and a classifier, the encoder is used to extract feature information and is composed of a down-sampling module and a transformer module; the decoder is used to gradually condense semantic information using feature information, and is composed of an up-sampling module and a transformer module; the classifier is used to classify all point clouds through the semantic information condensed by the decoder. 3.The method of claim 1, wherein: The step two specifically comprises the following contents: A1, n x n y n z The information features of the six channels of rgb are input into the pointtransformer network with input_channels=6 channels, and the information features of the six channels of xyzrgb are input into the pointtransformer network with input_channels=6 channels. A2, uniformly select a plurality of points in the overall point cloud space by farthest point sampling, then use the K-nearest neighbor method to construct a spherical neighborhood with the plurality of points as the center and a preset distance as the radius, sample K points in the neighborhood, perform global pooling after passing through a fully connected network, and then perform global pooling to concentrate the K-nearest neighbor information on the center point, reduce the point cloud data while increasing the features of each point, and achieve the effect of extracting the features in the neighborhood of the key point machine; A3, sequentially perform downsampling and upsampling, further extract point cloud features of the data after sampling through the transformer module in the point transformer network, and perform multiple iterations on the point cloud data until the preset downsampling and upsampling size is reached; A4, the result is processed through a fully connected network as a classification layer, the last layer of the fully connected network is a softmax layer, the data obtained by processing the fully connected network is normalized, and the result obtained by the final adoption is predicted; A5, cross-entropy operation is performed on the obtained category prediction and the real object category to obtain a loss value as a loss, the network is trained after back propagation, and then the training is started again from step A1 until the set number of training times is reached.

4. The method of claim 3, wherein: The A3 step specifically includes the following contents: A31, the down-sampled data is further extracted through a transformer module in the point transformer network to obtain point cloud feature information, and the point cloud data is transformed through the transformer module by , , three different fully connected layers, to obtain , and three features, and the relative distances between all points in the K-nearest neighbor and the center point are taken as features to obtain position encoding through a fully connected layer, and then the above features obtained by processing are transformed through a formula , wherein is a fully connected layer, is a normalization layer, is a Hadamard product, and after transformation, the final features of the transformer module are obtained . A32, repeat steps A2 and A31 until the preset downsampling size is reached; A33, use the tri-linear interpolation method to supplement points, upsample the obtained data, and combine the features during downsampling as the final upsampling result; A34, further extract point cloud feature information of the data after upsampling through the transformer module in the point transformer network, and repeat the processing content of the point cloud data in step A31 through the transformer module; A35, repeat steps A33 and A34 until the preset upsampling size is reached.

Images