A weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity relationships.

Abstract

This invention discloses a weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity relationships, comprising: acquiring scene-level point cloud data; constructing a semantic segmentation network architecture; the semantic segmentation network having two branches and an affinity enhancement module; the two branches extracting multi-scale spatial features and multi-scale color and texture features from the scene-level point cloud data, respectively; the affinity enhancement module concatenating the spatial and color and texture features at multiple scales and constructing an affinity matrix; and the semantic segmentation module performing semantic segmentation prediction based on the spatial and color and texture features at multiple scales, and optimizing the prediction results using the affinity matrix. This invention, by considering the affinity relationships of objects at different scales and the different modal information of the point cloud, can identify objects that are easily confused in terms of spatial or color and texture.

Description

Technical Field

[0001] This invention relates to the field of point cloud data semantic segmentation technology, and more specifically to a weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity relationships. Background Technology

[0002] Point cloud semantic segmentation refers to identifying the content and location of points in a point cloud by finding all points belonging to a certain category. Traditional supervision signals contain category information for all points, and annotation requires a lot of manpower and resources, limiting its application scenarios. Scene-level weak supervision refers to supervision signals that only contain the category to which the scene belongs, without the category information of any individual point. Its annotation is the most efficient, perfectly solving the problem of high annotation costs that limit full supervision.

[0003] The key to successful weakly supervised semantic segmentation of point clouds lies in generating complete and accurate localization regions for the segmented objects. Existing methods generally utilize class activation maps and multi-instance loss to generate pseudo-labels, which are typically limited to a small portion of the most discriminative regions and have low quality, far from sufficient for completing weakly supervised semantic segmentation tasks. Existing methods often ignore small-scale objects, performing well only for large-scale objects (such as walls and floors), and generally focus more on point cloud spatial information, leading to the problem of segmentation results confusing highly similar categories in spatial information.

[0004] Therefore, how to provide a weakly supervised semantic segmentation method for point clouds that can take into account objects of multiple scales and can distinguish objects that are easily confused in terms of space or color and texture is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] In view of this, the present invention provides a weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity relationships. By considering the affinity relationships of objects at different scales and the different modal information of point clouds, it can identify objects that are easily confused in terms of space or color texture.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity relationships includes:

[0008] Acquire scene-level point cloud data, which includes multiple scenes and category labels for each scene;

[0009] A semantic segmentation network architecture is constructed; the semantic segmentation network has two branches and an affinity enhancement module, and both branches include a feature extraction module and a semantic segmentation module.

[0010] During the training phase, one branch inputs complete point cloud information of scene-level point cloud data, and the feature extraction module on this branch extracts multi-scale color and texture features of the scene-level point cloud data; the other branch inputs spatial information of scene-level point cloud data, and the feature extraction module on this branch extracts multi-scale spatial features of the scene-level point cloud data.

[0011] The affinity enhancement module splices together spatial features and color texture features at multiple scales and constructs an affinity matrix.

[0012] The semantic segmentation modules on the two branches perform semantic segmentation prediction based on spatial features and color texture features at multiple scales, and optimize the prediction results using the affinity relation matrix.

[0013] Furthermore, the scene-level point cloud data is represented in the following form:

[0014]

[0015] Where D represents scene-level point cloud data; P m ∈R N×(3+K) , representing the m-th point cloud scene, with input information including 3D coordinates and other K-dimensional features; R N×(3+K) This represents a point cloud scene, where R represents a real number, N represents the number of points in the scene, and 3 represents the three coordinates (X, Y, Z); y m ∈(0,1) C , represents the category label of the scene, and each scene contains C categories; M represents the number of point cloud scenes.

[0016] Furthermore, both branches of the feature extraction module use Pointnet++, which can aggregate multi-scale features, as the backbone network. Its structure includes four downsampling layers and two upsampling layers. The upsampling layers gradually reduce the number of sampling points, and the upsampling layers gradually restore the number of sampling points.

[0017] Furthermore, the construction process of the affinity relationship matrix is ​​as follows:

[0018] The spatial features extracted from different upsampling layers in one branch of the feature extraction module are concatenated to obtain the spatial feature concatenation result.

[0019]

[0020] in, These represent the features obtained by different upsampling networks in the feature extraction module. The dimension representing the F1 feature is N1×d1. The dimension representing the F2 feature is N1×d2, where N1 represents the tensor length, d1 represents the tensor width of the F1 feature, and d2 represents the tensor width of the F2 feature. The features were sampled from the teacher network in the semantic segmentation module and aggregated into highly abstract features. Indicates F u The dimension of the feature is N1×d u d u Indicates F u The tensor width of the feature;

[0021] The color and texture features extracted from different upsampling layers in another branch of the feature extraction module are concatenated to obtain the color and texture feature concatenation result.

[0022] Based on the cosine similarity between the spatial feature splicing result and the color texture feature splicing result, an affinity relationship matrix is ​​constructed, the expression of which is:

[0023]

[0024]

[0025] Where i,j=1,2,…,N1; Let F represent the affinity matrix, which has dimensions 2N1×2N1; θ represents the threshold for selecting the affinity matrix; <·> represents the inner product, and ||·|| represents L2 normalization; M [i,·] represents the i-th row vector of the affinity matrix; F M [j,·] represents the vector of the j-th row of the affinity matrix.

[0026] Furthermore, the two branches of the semantic segmentation module respectively include a teacher network and a student network;

[0027] The teacher network uses MIL-loss constraints to find the most identifiable regions in each scene and optimizes pseudo-labels using the affinity relation matrix. High-confidence pseudo-labels are selected and self-supervised loss is used to supervise the predictions of the student network.

[0028] The student network performs semantic segmentation prediction on the multi-scale features extracted by the feature extraction module and outputs the final segmentation result.

[0029] Furthermore, the loss function of the semantic segmentation module consists of three parts, and its expression is as follows:

[0030] L = L mil +L self +L consist

[0031] Among them, Lmil Prediction of loss-constrained teacher networks; L self The loss is used to constrain the student network by taking the high-confidence result of the teacher network prediction output; L consist The loss is L1 loss, which constrains the teacher network of the two branches.

[0032] Furthermore, L mil The expression for loss is:

[0033]

[0034] in, This represents the scene-level category prediction obtained after average pooling of the logits for each point; N represents the number of classification points. Let y[c] represent the logits of the teacher network predicting C categories for N1 points; y[c] represents the known scene-level category labels.

[0035] L self The expression for loss is:

[0036]

[0037] Where φ[i,c]=softmax(S seg [i,c]) represents the probability that point i belongs to category c; Y[i,c]∈{0,1} represents the high-confidence pseudo-label generated by the teacher network;

[0038] L consist The expression for loss is:

[0039]

[0040] in, This represents the teacher network prediction results after the spatial information input branches are optimized using the affinity relation matrix. This indicates the teacher network prediction results after optimization via the affinity relation matrix for the input branch emphasizing color information.

[0041] Furthermore, the expression for optimizing pseudo-labels using the affinity matrix is ​​as follows:

[0042]

[0043] in, This represents the teacher network prediction results after the spatial information input branches are optimized using the affinity relation matrix. This indicates the teacher network prediction results after optimization using the affinity relation matrix for the input branch emphasizing color information. (A) MMA U represents the affinity matrix. segThis represents the teacher network prediction results for spatial information input branches without affinity relation matrix optimization. This represents the teacher network predicting the categories of N1 points based on the unoptimized affinity relation matrix of the input branch emphasizing color information.

[0044] Furthermore, the teacher network consists of an upsampling layer and a classifier;

[0045] The student network consists of an upsampling layer, a Transformer Encoder layer, and a classifier.

[0046] Furthermore, when the teacher network makes predictions, it performs a reweighting operation on the classifier, using the following formula:

[0047]

[0048] in, represents the weight of classifier j in the teacher network after reweighting, and ||·|| represents L2 normalization; The weights of classifier j in the teacher network, w i This represents all categories of the classifier, and ||.|| represents the L2 norm.

[0049] As can be seen from the above technical solution, compared with the prior art, this invention discloses a weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity relationships. Firstly, by considering the affinity relationships of objects at different scales, it solves the problem that existing methods ignore small-scale objects and only perform well in large-scale object categories (such as walls and floors). Secondly, it fully utilizes the information from different modalities of the point cloud, solving the problem that existing methods generally focus more on the spatial information of the point cloud, causing segmentation results to confuse highly similar categories with spatial information. The final generated segmentation network architecture can distinguish objects that are easily confused by spatial or color textures, while the localization region takes into account objects at multiple scales and does not ignore small-scale objects, making the localization region complete and accurate. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0051] Figure 1 The diagram shows the segmentation network architecture provided by this invention. Detailed Implementation

[0052] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] like Figure 1 As shown, this embodiment of the invention discloses a weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity relationships, including:

[0054] Acquire scene-level point cloud data, which includes multiple scenes and category labels for each scene;

[0055] A semantic segmentation network architecture is constructed; the semantic segmentation network has two branches and an affinity enhancement module, and both branches include a feature extraction module and a semantic segmentation module.

[0056] During the training phase, one branch inputs complete point cloud information of scene-level point cloud data, and the feature extraction module on this branch extracts multi-scale color and texture features of the scene-level point cloud data; the other branch inputs spatial information of scene-level point cloud data, and the feature extraction module on this branch extracts multi-scale spatial features of the scene-level point cloud data.

[0057] The affinity enhancement module stitches together spatial features and color texture features at multiple scales and constructs an affinity matrix.

[0058] The semantic segmentation modules on the two branches perform semantic segmentation prediction based on spatial features and color and texture features at multiple scales, and optimize the prediction results using the affinity relation matrix.

[0059] During the training phase of the semantic segmentation network, one of the two branches emphasizes color and texture information (i.e., all point cloud information), while the other emphasizes spatial information. After the semantic segmentation network is trained, during the testing phase, only the complete point cloud information of the scene-level point cloud data needs to be input into the branch emphasizing color and texture information, and the prediction result of that branch is used as the final segmentation result.

[0060] The steps described above will be explained in further detail below.

[0061] The purpose of this invention is to achieve weakly supervised semantic segmentation of 3D vision by locating objects of each category within a point cloud using scene-level category labels. The problem is defined as follows: given M point cloud scenes and category labels for each scene, forming scene-level point cloud data; the goal is to assign a category to each point within the scene.

[0062] The representation of scene-level point cloud data is as follows:

[0063]

[0064] Where D represents scene-level point cloud data; P m ∈R N×(3+K) Let R represent the m-th point cloud scene. The input information includes 3D coordinates and other K-dimensional features, such as a dimension of N×10, where the 10 dimensions include XYZ coordinate information, RGB color information, normal vector, and height information; N×(3+K) This represents a point cloud scene, where R represents a real number, N represents the number of points in the scene, and 3 represents the three coordinates (X, Y, Z); y m ∈(0,1) C , represents the category label of the scene, and each scene contains C categories; M represents the number of point cloud scenes.

[0065] Specifically, the framework of this invention consists of a feature extraction module, a semantic segmentation module, and an affinity enhancement module. The network input includes two parts: spatial information input and color information input, with all RGB channels of the spatial information input set to 0. The input first passes through the feature extraction module, which provides features at different scales. The affinity enhancement module concatenates the multi-scale features and multi-modal features to calculate the affinity matrix. The multi-modal features come from two inputs to the network and have the same network structure and parameters. The semantic segmentation module uses the affinity matrix to help the pseudo-labels correctly classify point by point, and after upsampling, generates accurate segmentation results.

[0066] The above modules are described in detail below.

[0067] 1. Feature Extraction Module

[0068] Both branches of the feature extraction module use Pointnet++, which can aggregate multi-scale features, as the backbone network. Its structure includes four downsampling layers and two upsampling layers. The upsampling layers gradually reduce the number of sampling points, and the upsampling layers gradually restore the number of sampling points.

[0069] In this embodiment of the invention, the downsampling layers progress from shallow to deep, with the number of points varying as follows: N = 40000, N1 = 2048, N2 = 1024, N3 = 512, and N4 = 256. Upsampling restores the number of points in the order from N4 to N2. Shallow layers extract fine-grained features, while deeper layers focus on highly abstract, coarse-grained features. This embodiment selects the classic PointNet++ in point cloud research as the feature extraction network, but other backbone networks with aggregated multi-scale feature extraction capabilities are also applicable.

[0070] 2. Affinity Enhancement Module: This module accepts multi-scale semantic features aggregated by the feature extraction module. The feature extraction module aggregates features from surrounding neighborhood point pairs through downsampling, thus capturing fine-grained features for shallow points and coarse-grained features for deep points.

[0071] The construction process of the affinity relation matrix is ​​as follows:

[0072] ① The spatial features extracted from different upsampling layers in one branch of the feature extraction module are concatenated to obtain the spatial feature concatenation result.

[0073]

[0074] in, These represent the features obtained by different upsampling networks in the feature extraction module, with different network levels aggregating features at different scales. The dimension representing the F1 feature is N1×d1. The dimension representing the F2 feature is N1×d2, where N1 represents the tensor length, d1 represents the tensor width of the F1 feature, and d2 represents the tensor width of the F2 feature. Obtained by sampling from the teacher network in the semantic segmentation module, it aggregates highly abstract features. Indicates F u The dimension of the feature is N1×d u d u Indicates F u The tensor width of the feature. The tensor length is equivalent to the number of sampling points. Since the matrix describes the features of the N1 points sampled by this network layer, that is, each of the N1 points has a vector of length d describing the features, the matrix length is N1, which is the number of sampling points of this network layer.

[0075] In this embodiment of the invention, when constructing the affinity relationship matrix, the F-values ​​of the teacher network are spliced ​​together. u The purpose is F u With deeper network layers and more abstract features, aggregating this dimension can help with classification.

[0076] ② The color and texture features extracted from different upsampling layers in the feature extraction module of another branch are concatenated to obtain the color and texture feature concatenation result. The expression is consistent with the splicing process of F in ①, the difference being that the splicing is performed on the color texture features of the branch that emphasizes color information input (the branch with all point cloud information input).

[0077] ③ In order to consider the semantic information of both space and color texture, we consider the multi-scale semantic features generated by the two branches, i.e. in It emphasizes color information input and combines the two inputs. Then, based on the cosine similarity between the merged spatial feature stitching result and the color and texture feature stitching result, an affinity matrix is ​​constructed, the expression of which is:

[0078]

[0079]

[0080] Where i,j=1,2,…,N1; Let F represent the affinity matrix, which has dimensions 2N1×2N1; θ represents the threshold for selecting the affinity matrix; <·> represents the inner product, and ||·|| represents L2 normalization; M [i,·] represents the i-th row vector of the affinity matrix; F M [j,·] represents the vector of the j-th row of the affinity matrix.

[0081] Since the features used to calculate the affinity relationship matrix emphasize spatial information and color texture information respectively, the segmentation results of this invention take into account the relationships between points across modes.

[0082] 3. Semantic Segmentation Module

[0083] The semantic segmentation module has two branches, namely the teacher network and the student network (i.e., Figure 1 (Teacher and student modules in the text);

[0084] The teacher network uses the MIL-loss constraint to find the most identifiable regions in each scene and optimizes the pseudo-labels using the affinity matrix, as shown in the expression below. This makes the classification results of points of the same category similar, and the classification results of points of different categories different. The pseudo-labels with high confidence are selected and then subjected to self-supervised loss L. self To supervise the predictions of the student network, high-confidence pseudo-labels are treated as real labels, and the prediction results of the student network are constrained by cross-entropy.

[0085] The student network performs semantic segmentation prediction on the multi-scale features extracted by the feature extraction module and outputs the final segmentation result.

[0086] The expression for optimizing pseudo-labels using the affinity matrix is ​​as follows:

[0087]

[0088] In the above formula, This represents the teacher network prediction results after the spatial information input branches are optimized using the affinity relation matrix. This indicates the teacher network prediction results after optimization using the affinity relation matrix for the input branch emphasizing color information. (A)MMA U represents the affinity matrix. seg This represents the teacher network prediction results for spatial information input branches without affinity relation matrix optimization. This represents the teacher network predicting the category of N1 points based on the unoptimized affinity relation matrix of the input branch emphasizing color information.

[0089] Specifically, to ensure the effectiveness of self-supervised loss, the teacher network and student network structures differ to some extent. The teacher network consists of an upsampling layer and a classifier; the upsampling layer upsamples the number of points from N2 = 1024 to N1 = 2048, and the classifier predicts the category of N1 = 2048 points.

[0090] The student network consists of an upsampling layer (upsampling the number of points from N2 = 1024 to N1 = 2048), a Transformer Encoder layer, and a classifier. The upsampling layer and classifier in the student network function the same as those in the teacher network.

[0091] Introducing the Transformer Encoder layer is necessary because the student network outputs the final segmentation result, which requires consideration of long dependencies. Furthermore, experience from joint training shows that maintaining the differences between the self-supervised student network and the teacher network is beneficial for the student network to learn richer semantic features.

[0092] Long-range dependencies refer to the relationships between pixels, especially those pixels that are spatially distant, allowing the learning of contextual relationships within an image. The attention mechanism in the transformer encoder layer relates the value of a node to all nodes, unlike convolution which only considers local relationships. The student network captures global information about long-range dependencies, while the teacher network captures local features, learning semantic features from both directions. Therefore, maintaining the differences between the self-supervised student and teacher networks helps the student network learn richer semantic features.

[0093] In other embodiments, point cloud semantic segmentation conforms to a long-tail data distribution in both scene category and point category aspects. For example, head categories such as "wall" and "floor" appear in every scene and occupy a large proportion. This not only causes head categories to dominate the training optimization direction, but also hinders the inference of the affinity matrix. Therefore, this invention uses classifier reweighting to mitigate the negative impact of long-tail data distribution on semantic segmentation. When the teacher network makes predictions, the classifier is reweighted, setting... The classifier weights for class j in the teacher network are reweighted on W using the following formula:

[0094]

[0095] get The weights of classifier j in the teacher network are reweighted, and ||·|| represents L2 normalization; w i This represents all categories of the classifier, and ||.|| represents the L2 norm.

[0096] In one specific embodiment, the loss function of the semantic segmentation module consists of three parts, and its expression is as follows:

[0097] L = L mil +L self +L consist

[0098] Among them, L mil Prediction of loss-constrained teacher networks; L self The loss is used to constrain the student network by taking the high-confidence result of the teacher network prediction output; L consist The loss is L1 loss, which constrains the teacher network of the two branches.

[0099] Specifically, L mil The expression for loss is:

[0100]

[0101] in, This represents the scene-level category prediction obtained after average pooling of the logits of each point, where N represents the number of classification points; Let y[c] represent the logits (classification scores) of the teacher network predicting C categories for N1 points; y[c] represents the known scene-level category labels.

[0102] L self The expression for loss is:

[0103]

[0104] Where φ[u,c]=softmax(S seg [i,c]) represents the probability that point i belongs to category c; Y[i,c]∈{0,1} represents the high-confidence pseudo-label generated by the teacher network;

[0105] L consist The expression for loss is:

[0106]

[0107] in, This represents the teacher network prediction results after the spatial information input branches are optimized using the affinity matrix. This indicates the teacher network prediction results after emphasizing the color information input and optimizing it using the affinity matrix.

[0108] To further verify the segmentation effect of the method of the present invention, the following experiments were conducted.

[0109] The datasets used are Scannet and S3DIS. For the Scannet dataset, we divided it into 1201 scenes as the training set and 312 scenes as the test set. For the S3DIS dataset, as usual, we used region 5 as the test set and the remaining regions as the training set.

[0110] The verification results are shown in Tables 1 and 2:

[0111] Table 1 shows the performance of different methods on the Scannet dataset.

[0112] method Supervision methods Validation set test set PointNet++ Full supervision —— 33.9 PointCNN Full supervision —— 45.8 KPConv Full supervision —— 68.4 MinkNet Full supervision —— 73.6 MPRM Scenario-level weak supervision 21.9 —— WyPR Scenario-level weak supervision 29.6 24 MIL-Derived Scenario-level weak supervision 26.2 —— This invention Scenario-level weak supervision 37.7 30.6

[0113] Table 2 Performance of different methods on S3DIS

[0114] method Supervision methods test set PointNet++ Full supervision 53.5 PointCNN Full supervision 57.3 KPConv Full supervision 70.6 MinkNet Full supervision 65.4 MPRM Scenario-level weak supervision 10.3 WyPR Scenario-level weak supervision 22.3 MIL-Derived Scenario-level weak supervision 12.9 This invention Scenario-level weak supervision 26.3

[0115] As shown in Tables 1 and 2, under the segmentation evaluation metric of Intersection over Union (IoU), the method of this invention achieves 15.8%, 11.5%, and 8.1% higher IoU on the Scannet dataset compared to existing methods such as MPRM, MIL-Derived, and WyPR, respectively. On the S3DIS dataset, it improves by 6.0%, 13.4%, and 13.4%. Classification segmentation results show that this invention improves the segmentation results for small objects. For example, on the Scannet dataset, this invention improves the IoU for the chair category by 20% and for the table category by 25%. Furthermore, for objects in certain scenes where color information is not readily apparent, visualization results show that this invention can correctly locate them using spatial information.

[0116] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to the method section.

[0117] The above description of the disclosed embodiments enables those skilled in the art to make or use the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity relationships, characterized in that, include: Acquire scene-level point cloud data, which includes multiple scenes and category labels for each scene; Construct a semantic segmentation network architecture; The semantic segmentation network has two branches and an affinity enhancement module. Both branches include a feature extraction module and a semantic segmentation module. During the training phase, one branch inputs complete point cloud information of scene-level point cloud data, and the feature extraction module on this branch extracts multi-scale color and texture features of the scene-level point cloud data; the other branch inputs spatial information of scene-level point cloud data, and the feature extraction module on this branch extracts multi-scale spatial features of the scene-level point cloud data. The affinity enhancement module splices spatial features and color texture features at multiple scales and constructs an affinity matrix. The semantic segmentation modules on the two branches perform semantic segmentation prediction based on spatial features and color texture features at multiple scales, and optimize the prediction results using the affinity relation matrix. The scene-level point cloud data is represented in the following form: Where D represents scene-level point cloud data; P m ∈R N×(3+K) , representing the m-th point cloud scene, with input information including 3D coordinates and other K-dimensional features; R N×(3+K) This represents a point cloud scene, where R represents a real number, N represents the number of points in the scene, and 3 represents the three coordinates (X, Y, Z); y m ∈(0,1) C , represents the category label of the scene, and each scene contains C categories; M represents the number of point cloud scenes; The process of constructing the affinity matrix is ​​as follows: The spatial features extracted from different upsampling layers in one branch of the feature extraction module are concatenated to obtain the spatial feature concatenation result. in, These represent the features obtained by different upsampling networks in the feature extraction module. The dimension representing the F1 feature is N1×d1. The dimension representing the F2 feature is N1×d2, where N1 represents the tensor length, d1 represents the tensor width of the F1 feature, and d2 represents the tensor width of the F2 feature. The features were sampled from the teacher network in the semantic segmentation module and aggregated into highly abstract features. F represents u The dimension of the feature is N1×d u d u F represents u The tensor width of the feature; The color and texture features extracted from different upsampling layers in another branch of the feature extraction module are concatenated to obtain the color and texture feature concatenation result. Based on the cosine similarity between the spatial feature splicing result and the color texture feature splicing result, an affinity relationship matrix is ​​constructed, the expression of which is: Where i,j=1,2,…,N1; Let F represent the affinity matrix, which has dimensions 2N1×2N1; θ represents the threshold for selecting the affinity matrix; <·> represents the inner product, and ||·|| represents L2 normalization; M [i,·] represents the i-th row vector of the affinity matrix; F M [j,·] represents the vector of the j-th row of the affinity relation matrix.

2. The weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity as described in claim 1, characterized in that, Both branches of the feature extraction module use Pointnet++, which can aggregate multi-scale features, as the backbone network. Its structure includes four downsampling layers and two upsampling layers. The upsampling layer gradually reduces the number of sampling points, and the upsampling layer gradually restores the number of sampling points.

3. The weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity as described in claim 1, characterized in that, The semantic segmentation module has two branches, namely a teacher network and a student network. The teacher network uses MIL-loss constraints to find the most identifiable regions in each scene and optimizes pseudo-labels using the affinity relation matrix. High-confidence pseudo-labels are selected and self-supervised loss is used to supervise the predictions of the student network. The student network performs semantic segmentation prediction on the multi-scale features extracted by the feature extraction module and outputs the final segmentation result.

4. The weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity as described in claim 3, characterized in that, The loss function of the semantic segmentation module consists of three parts, and its expression is as follows: L=L mil +L self +L consist Among them, L mil Prediction of loss-constrained teacher networks; L self The loss is used to constrain the student network by taking the high-confidence result of the teacher network prediction output; L consist The loss is L1 loss, which constrains the teacher network of the two branches.

5. A weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity as described in claim 4, characterized in that, L mil The expression for loss is: in, This represents the scene-level category prediction obtained after average pooling of the logits for each point; N represents the number of classification points. Let y[c] represent the logits of the teacher network predicting C categories for N1 points; y[c] represents the known scene-level category labels. L self The expression for loss is: Where φ[i,c]=softmax(S seg [i,c]) represents the probability that point i belongs to category c; Y[i,c]∈{0,1} represents the high-confidence pseudo-label generated by the teacher network; L consist The expression for loss is: in, This represents the teacher network prediction results after the spatial information input branches are optimized using the affinity relation matrix. This indicates the teacher network prediction results after optimization via the affinity relation matrix for the input branch emphasizing color information.

6. The weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity as described in claim 1, characterized in that, The expression for optimizing pseudo-labels using the affinity matrix is ​​as follows: in, This represents the teacher network prediction results after the spatial information input branches are optimized using the affinity relation matrix. This indicates the teacher network prediction results after optimization using the affinity relation matrix for the input branch emphasizing color information. (A) MMA U represents the affinity matrix. seg This represents the teacher network prediction results for spatial information input branches without affinity relation matrix optimization. This represents the teacher network predicting the category of N1 points based on the unoptimized affinity relation matrix of the input branch emphasizing color information.

7. A weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity as described in claim 3, characterized in that, The teacher network consists of an upsampling layer and a classifier; The student network consists of an upsampling layer, a Transformer Encoder layer, and a classifier.

8. A weakly supervised semantic segmentation method for point clouds based on multimodal and multi-scale affinity as described in claim 7, characterized in that, When the teacher network makes predictions, it performs a reweighting operation on the classifier, using the following formula: in, represents the weight of classifier j in the teacher network after reweighting, and ||·|| represents L2 normalization; The weights of classifier j in the teacher network, w i This represents all categories of the classifier, and ||.|| represents the L2 norm.

Images